Execution Spec Tests¶
ethereum/execution-spec-tests is both a collection of test cases and a framework in Python to generate tests for Ethereum execution clients implemented.
The framework collects and executes the test cases in order to generate test fixtures (JSON) which can be consumed by any execution client to verify their implementation of ethereum/execution-specs. The fixtures, which define state transition and block tests, are generated by the framework using one of the t8n
command-line tools that are provided by most execution clients, see below for an overview of the supported t8n
tools.
---
title: Test Fixture Generation with execution-spec-tests
---
flowchart LR
style C stroke:#333,stroke-width:2px
style D stroke:#333,stroke-width:2px
style G stroke:#F9A825,stroke-width:2px
style H stroke:#F9A825,stroke-width:2px
subgraph ethereum/go-ethereum
C[<code>evm t8n</code>\nexternal executable]
end
subgraph ethereum/solidity
D[<code>solc</code>\nexternal executable]
end
subgraph ethereum/EIPs
E(<code>EIPS/EIP-*.md</code>\nSHA digest via Github API)
end
subgraph "ethereum/execution-spec-tests"
A(<code>./tests/**/*.py</code>\nPython Test Cases)
B([<code>$ fill ./tests/</code>\nPython Framework])
end
subgraph Test Fixture Consumers
subgraph ethereum/hive
G([<code>$ hive ...</code>\nGo Test Framework])
end
H([Client executables])
end
C <-.-> B
D <-.-> B
A --> B
E <-.-> |retrieve latest spec version\ncheck tested spec version| B
B -->|output| F(<code>./fixtures/**/*.json</code>\nJSON Test Fixtures)
F -->|input| G
F -->|input| H
The generated test fixtures can be used:
- Directly by client teams' test frameworks, and,
- In the integration tests executed in the ethereum/hive framework.
More information on how to use and download the released test fixtures can be found here.
Transition Tool Support¶
The following transition tools are supported by the framework:
Client | t8n Tool |
Tracing Support |
---|---|---|
ethereum/evmone | evmone-t8n |
Yes |
ethereum/execution-specs | ethereum-spec-evm |
Yes |
ethereum/go-ethereum | evm t8n |
Yes |
hyperledger/besu | evmtool t8n-server |
Yes |
status-im/nimbus-eth1 | t8n |
Yes |
Relationship to ethereum/tests¶
This collection of tests is relatively new (test case development started Q4, 2022) and mainly targets recent and upcoming Ethereum specification changes. It does not replace, but rather complements the existing tests in ethereum/tests.
Motivation¶
The motivation to implement test cases in ethereum/execution-spec-tests is:
- To implement test cases as code and ensure that changes, due to spec changes, for example, can be easily made. Moreover, changes are easily understandable and available in version control.
- To avoid the 2-step approach often used in ethereum/tests:
- Code (often unavailable) -> Test case (YAML).
- Test case (YAML) -> Fixtures (JSON).
Contributing
Contributions via PR are welcome!