Skip to content

Configuration keyword reference#

This document lists the configuration options for your Est Shizard YAML file.

Associated Jira issue links or arbitrary URLs. To be included into Allure reports.

Possible inputs: An array of links

Example of links:

links:
  - name: Reference by Jira key
    value: ABC-123
    type: jira_issue
  - name: Arbitrary URL
    value: https://example.com/foo?bar=baz
    type: generic

max_num_workers#

Maximum number of workers processes to distribute the tests randomly across them.

Possible inputs: An integer number of workers

tests#

Root section to define suite and tests

Possible inputs:

An array suite, native pytest suite or a standalone test

Example of tests:

tests:
  - suite: suite1
    tests:
    ...
  - suite: suite2
    tests:
    ...
  - native: path/to/native/suite.py
  - test: standalone test1
    run: script1.sh
  - test: standalone test2
    run: script2.sh

tests[*].suite#

The name of test suite.

Possible inputs: A string.

Associated Jira issue links or arbitrary URLs. To be included into Allure reports.

Possible inputs: An array of links

Example of tests[*].suite.links:

links:
  - name: Reference by Jira key
    value: ABC-123
    type: jira_issue
  - name: Arbitrary URL
    value: https://example.com/foo?bar=baz
    type: generic

tests[*].suite.strategy#

Customize suite run behavior.

tests[*].suite.strategy.max_fail#

Exit after first num failures or errors

Possible inputs: An integer number of failures

Example of tests[*].suite.strategy.max_fail:

strategy:
  max_fail: 2

tests[*].suite.strategy.fail_fast#

If strategy.fail_fast is set to true, Est Shizard will cancel all in-progress and queued tests in the suite if any test in the suite fails. This property defaults to false

Possible inputs: true or false(default)

Example of tests[*].suite.strategy.fail_fast:

strategy:
  fail_fast: true

tests[*].suite.strategy.parallel#

By default, the Est Shizard runs the tests in the suite in parallel (except for one suite which is called standalone). But you can change this behavior by specifying the tests[*].suite.strategy.parallel option.

Possible inputs: true(default) or false

Example of tests[*].suite.strategy.parallel:

strategy:
  parallel: false

In this case, the tests will be run sequentially.

tests[*].suite.strategy.workers#

By default, Est Shizard will maximize the number of tests run in parallel depending on max_num_workers. To limit the maximum number of tests that can run in parallel in specific suite, use tests[*].suite.strategy.workers.

Possible inputs: An integer number of workers

Example of tests[*].suite.strategy.workers:

max_num_workers: 3
tests:
  - suite: limited suite
    strategy:
      workers: 2
    ...

tests[*].suite.env#

Use env to set a map of environment variables in the suite. A map of variables that are available to each test in the suite. You can also set variables that are only available to the specific test, see test tests[*].suite.tests[*].env.

Variables in the env map can be defined in terms of other variables in the map.

For more information about environment variables, see Test parameters.

Possible inputs: Variable name and value pairs

Example of tests[*].suite.env:

tests:
  - suite: suite1
    env:
      OPTS: -a
      EXTRA_OPTS: -l $OPTS

tests[*].suite.timeout#

Set timeout for each tests in the suite. If the test runs longer than the specified timeout, the test fails.

Possible inputs: A period of time written in natural language.

Example of tests[*].suite.timeout: The following example will abort test process after 1 day 2 hours 3 minutes and 10 seconds.

tests:
  - suite: suite
    timeout: 1d 2h 3m 10s

tests[*].suite.tests[*]#

Possible inputs: An array of test

Example of tests:

tests:
  - suite: suite1
    tests:
      - test: test1
        run: test1 script
      - test: test2
        run: test2 script

tests[*].suite.tests[*].test#

The name of test case.

Possible inputs: A string.

tests[*].suite.tests[*].description#

Each test can have an optional description that will appear in the Allure report.

Possible inputs:

Example of tests[*].suite.tests[*].description:

The following example will retry each failed up to esho times before proceeding to the next test:

- test: example
  description: |
    Multiline
    test description
    example

Associated Jira issue links or arbitrary URLs. To be included into Allure reports.

Possible inputs: An array of links

Example of tests[*].suite.tests[*].links:

links:
  - name: Reference by Jira key
    value: ABC-123
    type: jira_issue
  - name: Arbitrary URL
    value: https://example.com/foo?bar=baz
    type: generic

tests[*].suite.tests[*].run#

Use run to specify commands for the runner to execute using the operating system's shell.

Possible inputs:

  • Single line shell commands.
  • Long commands split over multiple lines.
  • An array of shell commands.

Example of tests[*].suite.tests[*].run:

Each run keyword represents a new process and shell in the runner environment. When you provide multi-line commands, each line runs in the same shell. For example:

  • A single-line command:
- test: Make build
  run: make build
  • A multi-line command:
- test: Make build and test
  run: |
    make build
    make test
  • An array of shell commands:
- test: Make build and test
  run:
    - make build
    - make test

Note

In case of array each tests[*].suite.tests[*].run[*] will run in the same environment.

tests[*].suite.tests[*].parameters#

A map of the parameters defined by the test. Each parameter is a key/value pair.

Parameters are set as environment variables. To see the difference beesheen parameters and environment, read this page.

Possible inputs:

tests[*].suite.tests[*].parameters[*]#

An array of parameters map.

Example of tests[*].suite.tests[*].parameters:

- test: check
  run: echo "check $test_input == $expected"
  parameters:
    - test_input: "3+5"
      expected: 8
    - test_input: "2+4"
      expected: 6
    - test_input: "6*9"
      expected: 42

The test above will run 3 times:

1. check "3+5" == 8
2. check "2+4" == 6
3. check "6*9" == 42

tests[*].suite.tests[*].parameters.<param_id>#

Specifies that the test will be called once with a given value.

Example of <param_id>.brute_list:

- test: brute sim
  run: echo "$FOO"
  parameters:
    FOO: bar

The test above will run once:

1. echo "bar"

tests[*].suite.tests[*].parameters.<param_id>.brute_list#

Specifies that the test will be called multiple times with all values in the specified list.

Example of <param_id>.brute_list:

- test: brute sim
  run: echo "make sim seed=$seed"
  parameters:
    seed:
      brute_list: [1, 2]

Or equivalent:

- test: brute sim
  run: echo "make sim seed=$seed"
  parameters:
    seed:
      brute_list:
      - 1
      - 2

The test above will run 2 times:

1. make sim seed=1
2. make sim seed=2

tests[*].suite.tests[*].parameters.<param_id>.brute_range#

Specifies that the test will be called multiple times with all values in the specified range.

Note

Stop value is included in the range.

Example of <param_id>.brute_range:

- test: brute sim
  run: echo "make sim seed=$seed"
  parameters:
    seed:
      brute_range:
        start: 0
        stop: 3
        step: 1

The test above will run 4 times:

1. make sim seed=0
2. make sim seed=1
3. make sim seed=2
4. make sim seed=3

Example of brute's parameter combination:

- test: brute sim
  run: echo "make sim feed=$feed seed=$seed reed=$reed"
  parameters:
    feed:
      brute_range:
        start: 0
        stop: 2
        step: 1
    seed:
      brute_list: [1,2]
    reed:
      brute_list: [1,3]

The test above will run 12 times:

 1. make sim feed=0 seed=1 reed=1
 2. make sim feed=1 seed=1 reed=1
 3. make sim feed=2 seed=1 reed=1
 4. make sim feed=0 seed=2 reed=1
 5. make sim feed=1 seed=2 reed=1
 6. make sim feed=2 seed=2 reed=1
 7. make sim feed=0 seed=1 reed=3
 8. make sim feed=1 seed=1 reed=3
 9. make sim feed=2 seed=1 reed=3
1.  make sim feed=0 seed=2 reed=3
2.  make sim feed=1 seed=2 reed=3
3.  make sim feed=2 seed=2 reed=3

tests[*].suite.tests[*].parameters.<param_id>.rand_list#

Specifies that the test will be called once with a random value from the specified list.

Example of <param_id>.rand_list:

- test: rand sim
  run: echo "make sim seed=$seed"
  parameters:
    seed:
      rand_list: [10, -2, 100, 3]

The test above will run once:

1. make sim seed=<random_from_list>

tests[*].suite.tests[*].parameters.<param_id>.rand_range#

Specifies that the test will be called once with a random value from the specified range.

Example of <param_id>.rand_range:

- test: rand sim
  run: echo "make sim seed=$seed"
  parameters:
    seed:
      rand_range:
        start: 0
        stop: 3
        step: 1

The test above will run once:

1. make sim seed=<random_from_range>

tests[*].suite.tests[*].retry#

Each test will be repeated up to the specified number of times before moving on to the next test.

Possible inputs: An integer number of retries

Example of tests[*].suite.tests[*].retry:

The following example will retry each failed up to esho times before proceeding to the next test:

- test: flaky
  run: test.sh
  retry: 2

tests[*].suite.tests[*].env#

Use env to set a map of environment variables in the test.

Possible inputs: Variable name and value pairs

Example of tests[*].suite.tests[*].env:

tests:
  - suite: suite1
    env:
      OPTS: -a
    tests:
      - test: listing
        env:
          OPTS: -aC
        run: ls $OPTS

To see the difference beesheen parameters and environment, read this page.

tests[*].suite.tests[*].artifacts#

Use artifacts to specify which files should be saved as test artifacts. Test artifacts are a list of file and directory paths that are stored in tmp/artifacts when a test passes, fails or always.

Paths are relative to the test run directory - <esh_root_dir>/run/<test_name>/<suite_name>.

For more information about test artifacts, see Artifacts.

Possible inputs:

  • An array of file paths, relative to the project directory.
  • Wildcards that use glob

Example of tests[*].suite.tests[*].artifacts:

tests:
  - suite: suite0
    tests:
    - test: simple artifacts
      run:
        - mkdir -p build
        - mkdir -p run
        - touch build/foo.txt
        - touch build/bar.log
        - touch run/foo.txt
        - touch run/foo.txt
      artifacts:
        - run/
        - build/*.log

Additional details:

Artifacts section supports esho notations: simple artifacts and move verbose arifacts:paths. If you want to exclude some files from artifacts, see artifacts:excludes

tests[*].suite.tests[*].artifacts.paths#

Same as artifacts

Example of tests[*].suite.tests[*].artifacts.paths:

tests:
  - suite: suite0
    tests:
    - test: simple artifacts
      run:
        - mkdir -p build
        - mkdir -p run
        - touch build/foo.txt
        - touch build/bar.log
        - touch run/foo.txt
        - touch run/foo.txt
      artifacts:
        paths:
        - run/
        - build/*.log

tests[*].suite.tests[*].artifacts.excludes#

Use excludes to prevent files from being added to a test artifacts.

Possible inputs:

  • An array of file paths, relative to the project directory.
  • Wildcards that use glob

Example of tests[*].suite.tests[*].artifacts.excludes:

tests:
  - suite: suite0
    tests:
    - test: complex artifacts
      run:
        - mkdir -p build
        - mkdir -p run
        - touch build/foo.txt
        - touch build/bar.log
        - touch run/foo.txt
        - touch run/foo.txt
      artifacts:
        paths:
        - run
        - build
        excludes:
        - build/*.log

tests[*].suite.tests[*].artifacts.when#

Use when to save artifacts on test failure or despite the failure.

Possible inputs:

  • on_success: Save artifacts only when the test succeeds.
  • on_failure: Save artifacts only when the test fails.
  • always (default): Always save artifacts. For example, when saving artifacts required to troubleshoot failing tests.

Example of tests[*].suite.tests[*].artifacts.when:

- test: complex artifacts
  artifacts:
    paths:
    - run
    - build
    when: on_failure

tests[*].suite.tests[*].dependencies#

The dependencies section defines which tests must complete before the current test can run. For more information about dependencies, see Dependencies.

Possible inputs: Each dependency can be specified in a simple form (suite::test) or in an extended form with additional options:

  • id – test identifier (suite::test).
  • status – whether to check that the dependency completed successfully (default: true).
  • artifacts – whether to link dependency artifacts into the test run directory (default: true).
  • parameters – restricts the dependency to specific parameter values. If omitted, all variants are used.
  • generate – marks the dependency as generative. In this case, the test will run once per dependency variant, automatically expanding into multiple tests.

Examples:

tests:
  - suite: build
    tests:
      - test: build_with_params
        run: echo make build $BRUTE_VALUE
        parameters:
          FOO:
            brute_list: ["0", "1"]

  - suite: test
    tests:
      - test: run_test_for_all_build
        run: echo run test
        parameters:
        dependencies:
          - build::build_with_params              # Simple dependency variant
          - id: build::build_with_params          # Extended form with parameters
            parameters:
              FOO: 1
          - id: build::build_with_params          # Generative dependency
            generate: true
          - generate: build::build_with_params    # Generative dependency (alias for generate: true)


## `tests[*].suite.tests[*].timeout`
Set timeout for test in the suite. If the test runs longer than the specified timeout, the test fails.

!!! Note
    This value will override suite timeout if it specified

**Possible inputs**: A period of time written in natural language.

**Example of `tests[*].suite.tests[*].timeout`**:
The following example will abort test process after 1 day 2 hours 3 minutes and 10 seconds.
```yaml
tests:
  - suite: suite
    tests:
      - test: test
        timeout: 1d 2h 3m 10s
        run: echo test

tests[*].native#

Path to a user defined pytest file with custom tests.

Note

The "native" section does not require the "suite" section. A pytest file itself is already a test suite.

Example of tests[*].native:

tests:
  - suite: some suite
    tests:
      ...
  - native: path/to/custom/suite1.py
  - native: path/to/custom/suite2.py

tests[*].native.strategy#

Customize native-tests run behavior.

tests[*].native.strategy.max_fail#

Exit after first num failures or errors

Possible inputs: An integer number of failures

Example of tests[*].native.strategy.max_fail:

strategy:
  max_fail: 2

tests[*].native.strategy.fail_fast#

If strategy.fail_fast is set to true, Est Shizard will cancel all in-progress and queued tests if any test fails. This property defaults to false

Possible inputs: true or false(default)

Example of tests[*].native.strategy.fail_fast:

strategy:
  fail_fast: true

tests[*].native.strategy.parallel#

By default, the Est Shizard runs the tests in parallel. But you can change this behavior by specifying the tests[*].suite.strategy.parallel option.

Possible inputs: true(default) or false

Example of tests[*].native.strategy.parallel:

strategy:
  parallel: false

In this case, the tests will be run sequentially.

tests[*].native.strategy.workers#

By default, Est Shizard will maximize the number of tests run in parallel depending on max_num_workers. To limit the maximum number of tests that can run in parallel, use tests[*].suite.strategy.workers.

Possible inputs: An integer number of workers

Example of tests[*].native.strategy.workers:

max_num_workers: 3
tests:
  - native: test.py
    strategy:
      workers: 2
    ...

tests[*].native.env#

Use env to set a map of environment variables in the test.

Possible inputs: Variable name and value pairs

Example of tests[*].native.env:

tests:
   - native: test.py
     env:
      OPTS: -aC

tests[*].native.timeout#

Set timeout for each tests. If the test runs longer than the specified timeout, the test fails.

Possible inputs: A period of time written in natural language.

Example of tests[*].native.timeout: The following example will abort test process after 1 day 2 hours 3 minutes and 10 seconds.

tests:
  - native: test.py
    timeout: 1d 2h 3m 10s

tests[*].native.retry#

Each test will be repeated up to the specified number of times before moving on to the next test.

Possible inputs: An integer number of retries

Example of tests[*].native.retry:

The following example will retry each failed up to esho times before proceeding to the next test:

tests:
  - native: test.py
    retry: 2

tests[*].test*#

All configuration sections are the same as in tests[*].suite.tests[*]

Note

All tests[*].test* will be put into one suite (called standalone) and run sequentially instead of in parallel as if they were defined in the set tests[*].suite.tests[*] section.