The Shadow Simulator

Ethshadow is a tool to easily configure and run simulated Ethereum networks. Under the hood, it uses Shadow, a discrete-event network simulator that enables us to run simulations with actual Ethereum clients instead of specifically written simulation code.

The advantages of using Ethshadow are as follows.

1. It already includes everything in the simulation (e.g. libp2p, discv5, etc).
2. It uses the same software as the mainnet and the public testnets.
3. If there is any upgades in the supported clients, we can integrate those upgrades easily in the simulation.

If you want to simulate a new Ethereum protocol, what you need to do is just to implement it in supported clients and run it using this simulator.

Installation

Only Linux is supported. For more details, see the Shadow documentation.

Install Go

See the official page

Install Rust

See the official page

Install Docker

See the install page and the non-root user page for the installation.

The Docker daemon must be running while Ethshadow prepares the simulation.

Install Shadow and its dependencies

sudo apt-get install -y cmake findutils libclang-dev libc-dbg libglib2.0-0 libglib2.0-dev make netbase python3 python3-networkx xz-utils util-linux gcc g++
git clone https://github.com/shadow/shadow.git
cd shadow
./setup build --clean
./setup install
echo 'export PATH="${PATH}:/home/${USER}/.local/bin"' >> ~/.bashrc && source ~/.bashrc

Or consult the official page for the installation.

Install CL and EL clients

Ensure that all clients you want to use in the simulation are installed, see the supported client page for notes.

Install Ethshadow

Install Ethshadow by running cargo install --path .

Supported Clients

✅ = Available, works out-of-the-box with latest release

🚧 = Available, works with modifications (see subpage for details)

❌ = Unavailable, does not currently work

❔ = Unavailable, not yet tested

A client is considered to work if it can follow the chain and perform the necessary duties for validating. Other features might not work.

Execution Layer

Consensus Layer

Other

Geth

Installation

Install both geth and `bootnode.

git clone https://github.com/ethereum/go-ethereum.git
cd go-ethereum
git checkout v1.14.11 # The latest tested version
make all
sudo cp build/bin/geth /usr/local/bin/geth # Make it globally accessible
sudo cp build/bin/bootnode /usr/local/bin/bootnode # Make it globally accessible

Or consult the official page for the installation.

Lighthouse

Installation

You need to install both the lighthouse and lcli commands, so it's recommended to install them from source.

sudo apt update && sudo apt install -y git gcc g++ make cmake pkg-config llvm-dev libclang-dev clang
git clone https://github.com/sigp/lighthouse.git
cd lighthouse
git checkout v5.3.0 # The latest tested version
make
make install-lcli

Or consult the official page for the installation.

Getting Started

First, install Ethshadow and its dependencies. Also, make sure lighthouse, lcli, geth, and bootnode are available in your PATH environment variable. (TODO explain how to specify executable paths instead?)

Ethshadow uses, like Shadow, a yaml configuration file. Create a new File, e.g. myfirstsim.yaml.

In this file, you can specify any configuration option Shadow itself supports. There are many options, we will focus on the essentials here. Add the following to your configuration:

general:
  # How much time should we simulate?
  stop_time: 1h
  # Display a progress indicator?
  progress: true

These values will be passed to Shadow. Usually, when using Shadow directly, we would now specify our network topology and hosts to simulate. However, Ethshadow does that for us. Ethshadow introduces several new configuration options, contained in the ethereum section. In its most simple form, it looks like this:

ethereum:
  # Distribute this many validators evenly across all nodes
  validators: 50
  # Create this many nodes with Geth, Lighthouse and a Validator client.
  # Additionally, a host with one boot node for CL and EL each is added.
  nodes: 10

That's it! After adding that, our simulation is ready to run. In a shall, move to the directory your config is in and invoke:

ethshadow myfirstsim.yaml

The first run might take a moment, as Docker will have to pull an image. After some time, Startin Shadow 3.2.0 will be logged, and the simulation will begin. Notice how the Simulation will run at variable speed: it will likely hang for a moment at 00:00:04.999, because all nodes start after giving the boot node five seconds to prepare. As genesis o ccurs at 00:05:00.000, time will pass relatively quickly until then, as nodes only search for peers and wait for genesis. At approximately 00:05:12.000, simulation will take a bit, as the first block is built and all nodes verify it.

While waiting for the simulation to finish, note that a data directory was created next to your configuration file. Feel free to look around in it. For each node, the clients' data directories are included. You can observe the simulation by opening client logs contained within and following as the log gets written. As these logs tend to be a bit noisy, you might also want to check the shadow subdirectory, which contains files where the stdout and stderr of each process is redirected to. Here, you can easily check whether the simulation works by checking for error messages and skipped slots.

Feel free to let the simulation finish or cancel it with Ctrl-C.

Let's take a look at a more sophisticated example (sophisticated.yaml):

general:
  stop_time: 1h
  progress: true

ethereum:
  validators: 60
  nodes:
    - location: europe
      reliability: reliable
      tag: boot
      clients:
        el: geth_bootnode
        cl: lighthouse_bootnode
    - locations:
        - europe
        - na_east
        - na_west
      reliabilites:
        - reliable
        - home
      count:
        per_combination: 5

As you can see, we replaced the simple node count with a list of node specifications. Here, the yaml list has tow items.

In the first one, we define a host located in europe, with a reliable internet connection. We also specify that a Geth bootnode and a Lighthouse bootnode shall be run on that node.

In the second one, we actually specify multimple nodes: notice the count property, which specifies five nodes per combination. Combination here means every possible pair of specified locations and reliabilities: europe with reliable, europe with home, na_east with reliable, and so on. As there are 2 * 3 = 6 combinations, a total of 5 * 6 = 30 nodes will be created. As we specified 60 validators, each node will host 2 validators.

But what is a "location" and a "reliability"? In Ethereum, we have a lot of globally distributed nodes. Therefore, we want to be able to simulate with varying latency between nodes. There are 8 built-in regions: australia, east_asia, europe, na_east, na_west, south_america, south_aftica, and west_asia. Ethshadow has a table with estimated latencies between these regions and will generate a network topology to make Shadow apply these latencies to the traffic between the nodes.

Reliabilities seek to simulate the varying connection qualities available to nodes. As home stakers are important to Ethereum, we want to include them into our simulations. The following reliabilities are available:

You can define your own locations and reliabilities as well as override the default values of the existing ones.

Before we can start a simulation with our more sophisticated simulation, we have to either delete the data directory from the previous run or specify another directory:

ethshadow -d data_sophisticated sophisticated.yaml

Congrats! These are the basics of Ethshadow.