Skip to content

Test parameters#

How test.parameters differs from test.env ?#

Although both of these configuration fields have the same effect - exporting a variable to the environment of the running test subprocess - the parameter has different behavior.

Parameters make the test parameterized. If you specified sets of parameters - the same test command will run several times for all sets of parameters for a given test. If you passed key-value parameter - test will run once.

For example if you run this configuration

max_num_workers: 3
tests:
  - suite: parametrized_test_example
    env:
      FOO: 1
    tests:
    - test: parametrized_test
      env:
        bar: 3
      run: echo "make sim seed=$seed"
      parameters:
        seed:
          brute_list: [1, 2]

this will result in the following output:

est-shizard-demo$ est-shizard run tests.yaml
Run: pytest /tmp/est-shizard-demo/tmp/tests/parametrized_test_example.py -vv -n=3 --est-shizard-core --root-dir=/tmp/est-shizard-demo/tmp --alluredir=/tmp/est-shizard-demo/tmp/allure-reports
======================================================= test session starts =======================================================
platform linux -- Python 3.11.8, pytest-7.4.0, pluggy-1.4.0 -- /tmp/est-shizard-demo/venv/bin/python3.11
cachedir: .pytest_cache
rootdir: /tmp/est-shizard-demo/tmp/tests
configfile: pytest.ini
plugins: rerunfailures-12.0, est-shizard-verif-0.3.0, mock-3.14.0, allure-pytest-2.13.2, xdist-3.3.1, est-shizard-3.0.0
3 workers [2 items]
scheduling tests via LoadScheduling

tmp/tests/parametrized_test_example.py::parametrized_test[seed=2]
tmp/tests/parametrized_test_example.py::parametrized_test[seed=1]
[gw1] [ 50%] PASSED tmp/tests/parametrized_test_example.py::parametrized_test[seed=2]
[gw0] [100%] PASSED tmp/tests/parametrized_test_example.py::parametrized_test[seed=1]

======================================================== 2 passed in 2.97s ========================================================
Running /tmp/est-shizard-demo/tmp/tests/parametrized_test_example.py ended with result OK

As you can see, env does not affect the number of test runs.

Each parameterized test run gets a unique test item name, such as parametrized_test[seed=1] and parametrized_test[seed=2] instead of just parametrized_test.

This name appears in reports and in the names of all service files and directories of Est Shizard, see:

$ tree $PWD/tmp/ -L 3
tmp/
├── allure-reports
│   └── ...
├── logs
│   └── parametrized_test_example
│       ├── 94a596fa16.log
│       └── 7431657a7d.log
├── metadata
│   └── parametrized_test_example
│       └── parametrized_test.json
├── reports
│   └── parametrized_test_example
│       ├── 94a596fa16.json
│       └── 7431657a7d.json
├── run
│   └── parametrized_test_example
│       ├── 94a596fa16
│       └── 7431657a7d
├── scripts
│   └── parametrized_test_example
│       ├── 94a596fa16.script.sh
│       └── 7431657a7d.script.sh
└── tests
    ├── parametrized_test_example.py
    └── pytest.ini

Passing variables through CLI#

You can pass env and parameters with cmd options --env and --parameter. But if you pass env and parameter with the same name, you will get pytest warning - You have 'env' and 'parameter' with common name - <name> and parameter will override env. Please note that by passing env/parameters as CLI options, you apply these values to all tests in the configuration.

--env and --parameter also support external JSON/YAML file paths.

Example JSON file for --env:

{
    "FOO": "foo",
    "BAR": "bar"
}

Example JSON file for --parameter:

{
    "FOO": "foo",
    "RAND":
    {
        "rand_list": ["1", "2"]
    }
}

Example YAML file for --parameter:

FOO: foo
RAND:
    rand_list: ["1", "2"]

In case multiple files are passed to fill the environment variables or parameters, the latter ones take precedence. Thus, in

est-shizard-demo$ est-shizard run tests.yaml --parameter parameters-1.yaml --parameter parameters-2.yaml

entries from both files will be merged into a single collection; conflicting parameters will be resolved with values from parameters-2.yaml.

Test variable precedence#

When more than one environment variable is defined with the same name, Est Shizard uses the most specific variable.

For example, an environment variable defined in a test will override suite environment variables with the same name, while the test executes. An environment variable defined from command line will override suite or test variable with same name.

The order of precedence for variables is (from highest to lowest):

  1. Command line variables specified by --parameter option
  2. Test parameters
  3. Command line variables specified by --env option
  4. Test env
  5. Suite env
  6. Predefined variables

Predefined variables#

Variable Description
ESH_SUITE_NAME Test suite name in the configuration file
ESH_TEST_NAME Test name declared in the configuration file
ESH_TEST_ITEM_NAME Test item name (test name plus parameters)
ESH_TEST_ITEM_NAME_NORMALIZED The value of the ESH_TEST_ITEM_NAME variable, normalized to remove special characters and truncated to 128 characters
ESH_TEST_RUN_DIR Path to test run directory
ESH_ROOT_DIR Path to the root directory where will be placed tmp folder.

Predefined variables are injected into the test environment automatically. They can be overwritten if the user declares a variable with the same name.

Example of predefined variable values:

For the next config:

tests:
  - suite: suite name
    tests:
      - test: test name
        run: echo $seed
        parameters:
          seed:
            brute_list: [1, 2]

the variables will be

  • ESH_SUITE_NAME - suite_name
  • ESH_TEST_NAME - test_name
  • ESH_TEST_ITEM_NAME - test_name[seed=1] for the first test run and test_name[seed=2] for the second.