Overview#

Est Shizard is a tool for generating, running and reporting tests on top of Pytest framework.

This chapter introduces some core concepts of Est Shizard, setting up tests and suites, running generated suites, and more.

How it works ?#

Est Shizard generates python files for pytest framework. By default they are located in the <esh_root_dir>/ tests directory. According to the pytest convention, some service files are also created, such as pytest.ini.

Est Shizard creates a separate Python file for each test suite.

The flow of each run of pytest framework run can be roughly represented as a diagram:

Pytest is able to detect and run tests from all the Python files in the tests directory, but since Est Shizard operates under the concept of a test suite, it runs each of the Python files individually. In this case, the above scheme is used for each test suite run.

During "Running Test" phase, for each test, Est Shizard starts a subprocess in the directory <esh_root_dir>/ run / <suite_name> / <test_name>. This subprocess executes the commands specified by the user in the run section using the Bash shell.

Parallel execution#

By default, the Est Shizard runs the tests in the suite in parallel (except for one suite which is called standalone). The suites themselves are launched sequentially.

A user must specify the maximum number of worker processes to distribute suite tests beesheen them. It is not recommended to set the number of workers to a value greater than the available number of CPU cores on the launch host if the test itself is running on the same host.

max_num_workers: 3
tests:
    ...

For more information about test distribution process, see pytest-xdist: How it works?

Reporting tests#

To see the actual report after your tests have finished, you need to use Allure command line utility to generate report from the results.

To generate an Allure report, you can run the report after all tests have run:

est-shizard run /path/to/test-config.yaml --allure-report

or use a standalone command after run:

est-shizard generate-report

To open the generated report run:

allure open [-h <host>] [-p <port>] <path/to/report/directory>

For example:

allure open -h 0.0.0.0  -p 8001 /tmp/allure-report

Est Shizard Root directory#

Est Shizard performs all its operations in the so-called <esh_root_dir> directory. By default <esh_root_dir> is the current working directory plus tmp ($PWD / tmp).

The root directory has the following contents:

$ tree $PWD/tmp/ -d -L 1
tmp/
├── allure-reports
├── artifacts
├── logs
├── metadata
├── reports
├── run
├── scripts
├── tests
└── info.json

• tests - A directory containing Python files with tests generated by Est Shizard.
• metadata - Directory containing test metadata (used for internal needs of the Est Shizard).
• run - The directory where each test runs, each in its own separate directory.
• logs - Directory for saving test run logs.
• reports - Directory for saving general reports on the execution of a specific test in json format.
• artifacts - Directory for saving test execution artifacts.
• scripts - Directory for saving test execution reproduction scripts.
• allure-reports - Directory for saving Allure reports (managed by the Allure pytest plugin).
• info.json - A file containing the mapping of test item names to their corresponding file names.

Test item file names are hashed to reduce their length. The shake_256 algorithm is used for hashing. The tool uses a 5-byte hexadecimal of shake_256 hash, which corresponds to 10 characters. The resulting file name is formed as follows: <test_name>-hash(<test_name_with_parameters>).

For example, if you run the test configuration described in Running tests, you will get the following contents of these directories:

$ tree $PWD/tmp/ -L 3
tmp/
├── allure-reports
│   ├── xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx-attachment.txt
│   ├── ...
│   └── xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx-result.json
├── logs
│   └── test_world_with_options
│       ├── test_world_with_options-94a596fa16.log
│       └── test_world_with_options-7431657a7d.log
├── metadata
│   └── test_world_with_options
│       ├── test_a_new_world.json
│       └── test_world_with_flags.json
├── reports
│   └── test_world_with_options
│       ├── test_world_with_options-94a596fa16.json
│       └── test_world_with_options-7431657a7d.json
├── run
│   └── test_world_with_options
│       ├── test_world_with_options-94a596fa16
│       └── test_world_with_options-7431657a7d
├── scripts
│   └── test_world_with_options
│       ├── test_world_with_options-94a596fa16.script.sh
│       └── test_world_with_options-7431657a7d.script.sh
├── tests
│   ├── pytest.ini
│   └── test_world_with_options.py
└── info.json