Execution Spec Tests¶

ethereum/execution-spec-tests is both a collection of test cases and a framework in Python to generate tests for Ethereum execution clients implemented.

The framework collects and executes the test cases in order to generate test fixtures (JSON) which can be consumed by any execution client to verify their implementation of ethereum/execution-specs. The fixtures, which define state transition and block tests, are generated by the framework using one of the t8n command-line tools that are provided by most execution clients, see below for an overview of the supported t8n tools.

---
title: Test Fixture Generation with execution-spec-tests
---
flowchart LR
  style C stroke:#333,stroke-width:2px
  style D stroke:#333,stroke-width:2px
  style G stroke:#F9A825,stroke-width:2px
  style H stroke:#F9A825,stroke-width:2px

  subgraph ethereum/go-ethereum, ...
    C[<code>evm t8n</code>\nexternal executable]
  end

  subgraph ethereum/solidity
    D[<code>solc</code>\nexternal executable]
  end

  subgraph ethereum/EIPs
    E(<code>EIPS/EIP-*.md</code>\nSHA digest via Github API)
  end

  subgraph "ethereum/execution-spec-tests"
    A(<code>./tests/**/*.py</code>\nPython Test Cases)
    B([<code>$ fill ./tests/</code>\nPython Framework])
  end

  subgraph Test Fixture Consumers
    subgraph ethereum/hive
      G([<code>$ hive ...</code>\nGo Test Framework])
    end
    H([Client executables])
  end

  C <-.-> B  
  D <-.-> B
  A --> B
  E <-.-> |retrieve latest spec version\ncheck tested spec version| B
  B -->|output| F(<code>./fixtures/**/*.json</code>\nJSON Test Fixtures)
  F -->|input| G
  F -->|input| H

The generated test fixtures can be used:

Directly by client teams' test frameworks, and,
In the integration tests executed in the ethereum/hive framework.

More information on how to use and download the released test fixtures can be found here.

Transition Tool Support¶

The following transition tools are supported by the framework:

Client	`t8n` Tool	Tracing Support
ethereum/evmone	`evmone-t8n`	Yes
ethereum/execution-specs	`ethereum-spec-evm`	Yes
ethereum/go-ethereum	`evm t8n`	Yes
hyperledger/besu	`evm t8n-server`	No
status-im/nimbus-eth1	`t8n`	Yes

Relationship to ethereum/tests¶

This collection of tests is relatively new (test case development started Q4, 2022) and mainly targets recent and upcoming Ethereum specification changes. It does not replace, but rather complements the existing tests in ethereum/tests.

Motivation¶

The motivation to implement test cases in ethereum/execution-spec-tests is:

To implement test cases as code and ensure that changes, due to spec changes, for example, can be easily made. Moreover, changes are easily understandable and available in version control.
To avoid the 2-step approach often used in ethereum/tests:
1. Code (often unavailable) -> Test case (YAML).
2. Test case (YAML) -> Fixtures (JSON).

Contributing

Contributions via PR are welcome!