Skip to content

Executing Tests at a Prompt

The execution-spec-tests test framework uses the pytest framework for test case collection and execution. The fill command is essentially an alias for pytest, which uses several custom pytest plugins to run transition tools against test cases and generate JSON fixtures.

Options specific to execution-spec-tests

The command-line options specific to filling tests can be listed via:

fill --help

See Custom fill Command-Line Options for all options.

Collection - Test Exploration

The test cases implemented in the ./tests sub-directory can be listed in the console using:

fill --collect-only

and can be filtered (by test path, function and parameter substring):

fill --collect-only -k warm_coinbase

Docstrings are additionally displayed when ran verbosely:

fill --collect-only -k warm_coinbase -vv

Execution

By default, test cases are executed for all forks already deployed to mainnet, but not for forks still under active development, i.e., as of time of writing, Q2 2023:

fill

will generate fixtures for test cases from Frontier to Shanghai.

To generate all the test fixtures defined in the ./tests/shanghai sub-directory and write them to the ./fixtures-shanghai directory, run fill in the top-level directory as:

fill ./tests/shanghai --output="fixtures-shanghai"

Test case verification

Note, that the (limited set of) test post conditions are tested against the output of the evm t8n command during test generation.

To generate all the test fixtures in the tests/shanghai/eip3651_warm_coinbase/test_warm_coinbase.py module, for example, run:

fill tests/shanghai/eip3651_warm_coinbase/test_warm_coinbase.py

To generate specific test fixtures from a specific test function or even test function and parameter set, obtain the corresponding test ID using:

fill --collect-only -q -k test_warm_coinbase

This filters the tests by test_warm_coinbase. Then find the relevant test ID in the console output and provide it to fill, for example, for a test function:

fill tests/shanghai/eip3651_warm_coinbase/test_warm_coinbase.py::test_warm_coinbase_gas_usage

or, for a test function and specific parameter combination:

fill tests/shanghai/eip3651_warm_coinbase/test_warm_coinbase.py::test_warm_coinbase_gas_usage[fork_Paris-DELEGATECALL]

Execution for Development Forks

By default, test cases are not executed with upcoming Ethereum forks so that they can be readily executed against the evm tool from the latest geth release.

In order to execute test cases for an upcoming fork, ensure that the evm tool used supports that fork and features under test and use the --until or --fork flag.

For example, as of Q2 2023, the current fork under active development is Cancun:

fill --until Cancun

See: Executing Tests for Features under Development.

Debugging the t8n Command

The --evm-dump-dir flag can be used to dump the inputs and outputs of every call made to the t8n command for debugging purposes, see Debugging Transition Tools.

Other Useful Pytest Command-Line Options

fill -vv            # More verbose output
fill -x             # Exit instantly on first error or failed test case
fill --pdb -nauto   # Drop into the debugger upon error in a test case
fill -s             # Print stdout from tests to the console during execution

Custom fill Command-Line Options

To see all the options available to fill, including pytest and pytest plugin options, use --pytest-help.

To list the options that only specific to fill, use:

fill --help

Output:

usage: fill [-h] [--evm-bin EVM_BIN] [--traces] [--verify-fixtures]
            [--verify-fixtures-bin VERIFY_FIXTURES_BIN] [--solc-bin SOLC_BIN]
            [--filler-path FILLER_PATH] [--output OUTPUT] [--flat-output]
            [--single-fixture-per-file] [--no-html] [--strict-alloc]
            [--ca-start CA_START] [--ca-incr CA_INCR] [--build-name BUILD_NAME]
            [--evm-dump-dir EVM_DUMP_DIR] [--forks] [--fork FORK] [--from FROM]
            [--until UNTIL] [--test-help]

options:
  -h, --help            show this help message and exit

Arguments defining evm executable behavior:
  --evm-bin EVM_BIN     Path to an evm executable that provides `t8n`. Default:
                        First 'evm' entry in PATH.
  --traces              Collect traces of the execution information from the
                        transition tool.
  --verify-fixtures     Verify generated fixture JSON files using geth's evm
                        blocktest command. By default, the same evm binary as for
                        the t8n tool is used. A different (geth) evm binary may
                        be specified via --verify-fixtures-bin, this must be
                        specified if filling with a non-geth t8n tool that does
                        not support blocktest.
  --verify-fixtures-bin VERIFY_FIXTURES_BIN
                        Path to an evm executable that provides the `blocktest`
                        command. Default: The first (geth) 'evm' entry in PATH.

Arguments defining the solc executable:
  --solc-bin SOLC_BIN   Path to a solc executable (for Yul source compilation).
                        Default: First 'solc' entry in PATH.

Arguments defining filler location and output:
  --filler-path FILLER_PATH
                        Path to filler directives
  --output OUTPUT       Directory path to store the generated test fixtures. Can
                        be deleted. If the specified path ends in '.tar.gz', then
                        the specified tarball is additionally created (the
                        fixtures are still written to the specified path without
                        '.tar.gz' suffix). Default: './fixtures'.
  --flat-output         Output each test case in the directory without the folder
                        structure.
  --single-fixture-per-file
                        Don't group fixtures in JSON files by test function;
                        write each fixture to its own file. This can be used to
                        increase the granularity of --verify-fixtures.
  --no-html             Don't generate an HTML test report (in the output
                        directory). The --html flag can be used to specify a
                        different path.
  --strict-alloc        [DEBUG ONLY] Disallows deploying a contract in a
                        predefined address.
  --ca-start CA_START, --contract-address-start CA_START
                        The starting address from which tests will deploy
                        contracts.
  --ca-incr CA_INCR, --contract-address-increment CA_INCR
                        The address increment value to each deployed contract by
                        a test.
  --build-name BUILD_NAME
                        Specify a build name for the fixtures.ini file, e.g.,
                        'stable'.

Arguments defining debug behavior:
  --evm-dump-dir EVM_DUMP_DIR, --t8n-dump-dir EVM_DUMP_DIR
                        Path to dump the transition tool debug output.

Specify the fork range to generate fixtures for:
  --forks               Display forks supported by the test framework and exit.
  --fork FORK           Only fill tests for the specified fork.
  --from FROM           Fill tests from and including the specified fork.
  --until UNTIL         Fill tests until and including the specified fork.

Arguments related to running execution-spec-tests:
  --test-help           Only show help options specific to a specific execution-
                        spec-tests command and exit.

Exit: After displaying help.