Skip to content

Debugging Transition Tools

There are two flags that can help debugging t8n tools or the execution-spec-tests framework:

  1. --evm-dump-dir: Write debug information from t8n tool calls to the specified directory.
  2. --verify-fixtures: Run go-ethereum's evm blocktest command to verify the generated test fixtures.

EVM Dump Directory

The --evm-dump-dir flag tells the framework to write the inputs and outputs of every call made to the t8n command to help debugging or simply understand how a test is interacting with the EVM.

In particular, a script t8n.sh is generated for each call to the t8n command which can be used to reproduce the call to trigger errors or attach a debugger without the need to execute Python.

For example, running:

fill tests/berlin/eip2930_access_list/ --fork Berlin \
    --evm-dump-dir=/tmp/evm-dump

will produce the directory structure:

📁 /tmp/evm-dump/
└── 📁 berlin__eip2930_access_list__test_acl__test_access_list
    ├── 📁 fork_Berlin
    │   ├── 📁 0
    │   │   ├── 📄 args.py
    │   │   ├── 📁 input
    │   │   │   ├── 📄 alloc.json
    │   │   │   ├── 📄 env.json
    │   │   │   └── 📄 txs.json
    │   │   ├── 📁 output
    │   │   │   ├── 📄 alloc.json
    │   │   │   ├── 📄 result.json
    │   │   │   └── 📄 txs.rlp
    │   │   ├── 📄 returncode.txt
    │   │   ├── 📄 stderr.txt
    │   │   ├── 📄 stdin.txt
    │   │   ├── 📄 stdout.txt
    │   │   └── 📄 t8n.sh
    │   └── 📁 1
    │       ├── 📄 args.py
    │       ├── 📁 input
    │       │   ├── 📄 alloc.json
    │       │   ├── 📄 env.json
    │       │   └── 📄 txs.json
    │       ├── 📁 output
    │       │   ├── 📄 alloc.json
    │       │   ├── 📄 result.json
    │       │   └── 📄 txs.rlp
    │       ├── 📄 returncode.txt
    │       ├── 📄 stderr.txt
    │       ├── 📄 stdin.txt
    │       ├── 📄 stdout.txt
    │       └── 📄 t8n.sh
    └── 📄 access_list.json

where the directories 0 and 1 correspond to the different calls made to the t8n tool executed during the test:

  • 0 corresponds to the call used to calculate the state root of the test's initial alloc (which is why it has an empty transaction list).
  • 1 corresponds to the call used to execute the first transaction or block from the test.

Note, there may be more directories present 2, 3, 4,... if the test executes more transactions/blocks.

Each directory contains files containing information corresponding to the call, for example, the args.py file contains the arguments passed to the t8n command and the output/alloc.json file contains the output of the t8n command's --output-alloc flag.

The t8n.sh Script

The t8n.sh script written to the debug directory can be used to reproduce a specific call made to the t8n command during the test session. For example, if a Besu t8n-server has been started on port 3001, the request made by the test for first transaction can be reproduced as:

/tmp/besu/test_access_list_fork_Berlin/1/t8n.sh 3001

which writes the response the from the t8n-server to the console output:

{
  "alloc" : {
    "0x000000000000000000000000000000000000aaaa" : {
      "code" : "0x5854505854",
      "balance" : "0x4",
      "nonce" : "0x1"
    },
    "0x2adc25665018aa1fe0e6bc666dac8fc2697ff9ba" : {
      "balance" : "0x1bc16d674ecb26ce"
    },
    "0xa94f5374fce5edbc8e2a8697c15331677e6ebf0b" : {
      "balance" : "0x2cd931",
      "nonce" : "0x1"
    }
  },
  "body" : "0xf8a0b89e01f89b0180078304ef0094000000000000000000000000000000000000aaaa0180f838f7940000000000000000000000000000000000000000e1a0000000000000000000000000000000000000000000000000000000000000000001a02e16eb72206c93c471b5894800495ee9c64ae2d9823bcc4d6adeb5d9d9af0dd4a03be6691e933a0816c59d059a556c27c6753e6ce76d1e357b9201865c80b28df3",
  "result" : {
    "stateRoot" : "0x51799508f764047aee6606bc6a00863856f83ee5b91555f00c8a3cbdfbec5acb",
    ...
    ...
  }
}

The t8n.sh is written to the debug directory for all supported t8n tools.

Verifying Test Fixtures via evm blocktest

The --verify-fixtures flag can be used to run go-ethereum's evm blocktest command in order to verify the generated JSON test fixtures.

For example, running:

fill tests/berlin/eip2930_access_list/ --fork Berlin \
    --evm-dump-dir==/tmp/evm-dump \
    --evm-bin=../evmone/build/bin/evmone-t8n \
    --verify-fixtures-bin=../go-ethereum/build/bin/evm \
    --verify-fixtures

will additionally run the evm blocktest command on every JSON fixture file and write its output to the EVM dump directory:

📂 /tmp/evm-dump
└── 📂 berlin__eip2930_access_list__test_acl__test_access_list
    ├── 📄 access_list.json
    ├── 📂 fork_Berlin
    │   ├── 📂 0
    │   │   ├── 📄 args.py
    │   │   ├── 📂 input
    │   │   │   ├── 📄 alloc.json
    │   │   │   ├── 📄 env.json
    │   │   │   └── 📄 txs.json
    │   │   ├── 📂 output
    │   │   │   ├── 📄 alloc.json
    │   ... ... ...
    │
    ├── 📄 verify_fixtures_args.py
    ├── 📄 verify_fixtures_returncode.txt
    ├── 📄 verify_fixtures.sh
    ├── 📄 verify_fixtures_stderr.txt
    └── 📄 verify_fixtures_stdout.txt

where the verify_fixtures.sh script can be used to reproduce the evm blocktest command.

Further --verify-fixtures Examples

  1. No fixture verification performed:

    fill

  2. Verify fixtures: Use the first evm binary in the PATH to execute both the t8n and blocktest commands (i.e., same binary used; this must be a geth binary):

    fill --verify-fixtures

  3. Explicitly specify the evm binary to execute the blocktest command (the first evm binary in the PATH is used for t8n commands; --verify-fixtures is not necessary):

    fill --verify-fixtures-bin=../go-ethereum/build/bin/evm

  4. Explicitly set two different evm binaries to execute the t8n and blocktest commands; write debug data to the specified --evm-dump-dir:

    fill --evm-bin=../evmone/build/bin/evmone-t8n \
  --verify-fixtures-bin=../go-ethereum/build/bin/evm \
  --evm-dump-dir=/tmp/evm-dump

Execution scope of evm blocktest

Note, that evm blocktest is not executed per parametrized test case, but rather per test function. This is because each fixture JSON file contains all the parametrized test cases for one test function.

Additionally, it is executed once for all test functions in one module only after all the test cases in the module have been executed and only report the first fixtures from the first failing test function1.

This means that the feedback is not as granular as for test case execution. To improve granularity, and get feedback per fork, for example, the --fork flag can be used to only execute test cases for one particular fork.

  1. This limitation is required to enable support of the pytest-xdist plugin for concurrent test execution across multiple CPUs. To achieve this we use the we apply the --dist loadscope xdist flag in our pytest.ini. ↩