Using Test Fixtures Generated by execution-spec-tests

ethereum/execution-spec-tests generates JSON test fixtures that can be consumed by execution clients directly or via the Hive pyspec simulator:

1. Clients can execute the tests directly against their EVM by implementing a command analogous to the go-ethereum's evm blocktest command, see cmd/evm/blockrunner.go.
2. Hive executes the tests against a fully instantiated client instance using the Engine API.

Here's a top-level comparison of these two approaches:

Consumed via	Scope	Pros	Cons
blocktest-like command	Module test	- Fast feedback loop - Less complex	- Smaller coverage scope
hive --sim ethereum/pyspec	System test / Integration test	- Wider Coverage Scope - Tests more of the client stack	- Slower feedback loop - Harder to debug

Executing a t8n tool via fill is not considered to be the actual test

The fill command uses t8n tools to generate fixtures. Whilst this will provide basic sanity checking of EVM behavior and a sub-set of post conditions are typically checked within test cases, it is not considered the actual test. The actual test is the execution of the fixture against the EVM which will check the entire post allocation and typically use different code paths than t8n commands.

Running blocktest directly within the execution-spec-tests framework

It's possible to execute evm blocktest directly within the execution-spec-tests framework. This is intended to verify fixture generation, see Debugging t8n Tools.

Release Formats

The ethereum/execution-spec-tests repository provides releases of fixtures in various formats (as of 2023-10-16):

Release Artifact	Consumer	Fork/feature scope
fixtures.tar.gz	Clients	All tests until the last stable fork
fixtures_develop.tar.gz	Clients	All tests until the last development fork
fixtures_hive.tar.gaz	Hive	All tests until the last stable fork in hive format
fixtures_develop_hive.tar.gz	Hive	All tests until the last development fork in hive format

The Hive format uses Engine API directives instead of the usual BlockchainTest format.

Obtaining the Most Recent Release Artifacts

Artifacts can be downloaded directly from the release page. The following script demonstrates how the most recent release version of a specific artifact can be downloaded using the Github API:

#!/bin/bash

# requires jq
# sudo apt install jq

# The following two artifacts are intended for consumption by clients:
# - fixtures.tar.gz: Generated up to the last deployed fork.
# - fixtures_develop.tar.gz: Generated up to and including the latest dev fork.
# As of Oct 2023, dev is Cancun, deployed is Shanghai.

ARTIFACT="fixtures_develop.tar.gz"  

OWNER="ethereum"
REPO="execution-spec-tests"

DOWNLOAD_URL=$(curl -s https://api.github.com/repos/$OWNER/$REPO/releases/latest \
                   | jq -r '.assets[] | select(.name=="'$ARTIFACT'").browser_download_url')

# Sanity check for the download URL: contains a version tag prefixed with "v"
if [[ "$DOWNLOAD_URL" =~ v[0-9]+\.[0-9]+\.[0-9]+ ]]; then
    curl -LO $DOWNLOAD_URL
else
    echo "Error: URL does not contain a valid version tag (URL: ${DOWNLOAD_URL})."
    exit 1
fi