Skip to content

Quick Start

Prerequisites

The tools provided by execution-spec-tests use uv (docs.astral.sh/uv) to manage their dependencies and virtual environment.

uv can be installed via curl (recommended; can self-update) or pip (requires Python, can't self-update):

curl -LsSf https://astral.sh/uv/install.sh | sh

pip install uv

If installed via curl, uv will download Python for your target platform if one of the required versions (Python 3.10, 3.11 or 3.12) is not available natively.

Installation

Clone execution-spec-tests and install its dependencies:

git clone https://github.com/ethereum/execution-spec-tests
cd execution-spec-tests
uv sync --all-extras
uv run solc-select use 0.8.24 --always-install

See Installation Troubleshooting if you encounter issues.

Exploring and Filling Test Cases

By default, JSON test fixtures are generated from this repository's Python test cases using the Ethereum Execution Layer Specification (EELS) reference implementation. The resulting JSON fixtures can be executed against execution clients to verify consensus. The process of generating fixtures is often referred to as "filling".

  1. Explore test cases via --collect-only and search for test cases that combine PUSH0 and DELEGATECALL in the EVM functionality introduced in the Shanghai hard fork:

    uv run fill --collect-only -k "push0 and delegatecall" tests/shanghai/

    The fill command is based on pytest. The above command uses the optional pytest arguments:

    • --collect-only only collect test cases; don't execute them.
    • -k <expression> filter test cases by their test case ID based on the given expression.
    • tests/shanghai the directory containing the test cases (tells fill to only discover test cases in this directory; default: tests/).

    Expected console output:

    Screenshot of pytest test collection console output

  2. Fill state_test fixtures for these test cases:

    uv run fill -k "push0 and delegatecall" tests/shanghai/ -m state_test -v

    where:

    • -m state_test only fills test cases marked as a state_test (see all available markers via uv run fill --markers).
    • -v enables verbose output.

    Expected console output:

    Screenshot of fill test collection console output

  3. Verify the generated fixtures:

    a. Check the corresponding fixture file has been generated:

    head fixtures/state_tests/shanghai/eip3855_push0/push0/push0_contract_during_call_contexts.json

    b. Open the generated HTML test using the link provided at the bottom of the console output. This is written to the output directory at:

    ./fixtures/.meta/report_fill.html

Installation Troubleshooting

If you encounter issues during installation, see the Installation Troubleshooting guide.

Next Steps

  1. Learn useful command-line flags.
  2. Execute tests for features under development via the --fork flag.
  3. Optional: Configure VS Code to auto-format Python code and execute tests within VS Code.
  4. Implement a new test case, see Writing Tests.