The Shadow Simulator

Ethshadow is a tool to easily configure and run simulated Ethereum networks. Under the hood, it uses Shadow, a discrete-event network simulator that enables us to run simulations with actual Ethereum clients instead of specifically written simulation code.

The advantages of using Ethshadow are as follows.

1. It already includes everything in the simulation (e.g. libp2p, discv5, etc).
2. It uses the same software as the mainnet and the public testnets.
3. If there is any upgades in the supported clients, we can integrate those upgrades easily in the simulation.

If you want to simulate a new Ethereum protocol, what you need to do is just to implement it in supported clients and run it using this simulator.

Installation

Only Linux is supported. For more details, see the Shadow documentation.

Install Go

See the official page

Install Rust

See the official page

Install Docker

See the install page and the non-root user page for the installation.

The Docker daemon must be running while Ethshadow prepares the simulation.

Install Shadow and its dependencies

sudo apt-get install -y cmake findutils libclang-dev libc-dbg libglib2.0-0 libglib2.0-dev make netbase python3 python3-networkx xz-utils util-linux gcc g++
git clone https://github.com/shadow/shadow.git
cd shadow
./setup build --clean
./setup install
echo 'export PATH="${PATH}:/home/${USER}/.local/bin"' >> ~/.bashrc && source ~/.bashrc

Or consult the official page for the installation.

Install CL and EL clients

Ensure that all clients you want to use in the simulation are installed, see the supported client page for notes.

Install Ethshadow

Install Ethshadow by running cargo install --path .

Supported Clients

✅ = Available, works out-of-the-box with latest release

🚧 = Available, works with modifications (see subpage for details)

❌ = Unavailable, does not currently work

❔ = Unavailable, not yet tested

A client is considered to work if it can follow the chain and perform the necessary duties for validating. Other features might not work.

Execution Layer

Consensus Layer

Other

Geth

Installation

Install both geth and `bootnode.

git clone https://github.com/ethereum/go-ethereum.git
cd go-ethereum
git checkout v1.14.11 # The latest tested version
make all
sudo cp build/bin/geth /usr/local/bin/geth # Make it globally accessible
sudo cp build/bin/bootnode /usr/local/bin/bootnode # Make it globally accessible

Or consult the official page for the installation.

Configuration

Bootnode

• executable: Specify path of the bootnode binary to use. Defaults to bootnode, i.e. the executable available in your PATH.

Geth

• executable: Specify path of the geth binary to use. Defaults to geth, i.e. the executable available in your PATH.

Lighthouse

Installation

You need to install both the lighthouse and lcli commands, so it's recommended to install them from source.

sudo apt update && sudo apt install -y git gcc g++ make cmake pkg-config llvm-dev libclang-dev clang
git clone https://github.com/sigp/lighthouse.git
cd lighthouse
git checkout v5.3.0 # The latest tested version
make
make install-lcli

Or consult the official page for the installation.

Prysm

Installation

Download Prysm beacon chain and validator binaries from the Github release page.

curl -L https://github.com/prysmaticlabs/prysm/releases/download/v5.3.0/beacon-chain-v5.3.0-linux-amd64 -o prysm
chmod +x prysm
curl -L https://github.com/prysmaticlabs/prysm/releases/download/v5.3.0/validator-v5.3.0-linux-amd64 -o prysm_vc
chmod +x prysm_vc

Configuration

Prysm beacon chain

• executable: Specify path of the prysm beacon chain binary to use. This field is required and there is no default value for it.

Prysm validator

• executable: Specify path of the prysm validator binary to use. This field is required and there is no default value for it. PATH.

Getting Started

First, install Ethshadow and its dependencies. Also, make sure lighthouse, lcli, geth, and bootnode are available in your PATH environment variable. (TODO explain how to specify executable paths instead?)

Ethshadow uses, like Shadow, a yaml configuration file. Create a new File, e.g. myfirstsim.yaml.

In this file, you can specify any configuration option Shadow itself supports. There are many options, we will focus on the essentials here. Add the following to your configuration:

general:
  # How much time should we simulate?
  stop_time: 1h
  # Display a progress indicator?
  progress: true

These values will be passed to Shadow. Usually, when using Shadow directly, we would now specify our network topology and hosts to simulate. However, Ethshadow does that for us. Ethshadow introduces several new configuration options, contained in the ethereum section. In its most simple form, it looks like this:

ethereum:
  # Distribute this many validators evenly across all nodes
  validators: 50
  # Create this many nodes with Geth, Lighthouse and a Validator client.
  # Additionally, a host with one boot node for CL and EL each is added.
  nodes: 10

That's it! After adding that, our simulation is ready to run. In a shall, move to the directory your config is in and invoke:

ethshadow myfirstsim.yaml

The first run might take a moment, as Docker will have to pull an image. After some time, Starting Shadow 3.2.0 will be logged, and the simulation will begin. Notice how the Simulation will run at variable speed: it will likely hang for a moment at 00:00:04.999, because all nodes start after giving the boot node five seconds to prepare. As genesis o ccurs at 00:05:00.000, time will pass relatively quickly until then, as nodes only search for peers and wait for genesis. At approximately 00:05:12.000, simulation will take a bit, as the first block is built and all nodes verify it.

While waiting for the simulation to finish, note that a data directory was created next to your configuration file. Feel free to look around in it. For each node, the clients' data directories are included. You can observe the simulation by opening client logs contained within and following as the log gets written. As these logs tend to be a bit noisy, you might also want to check the shadow subdirectory, which contains files where the stdout and stderr of each process is redirected to. Here, you can easily check whether the simulation works by checking for error messages and skipped slots.

Feel free to let the simulation finish or cancel it with Ctrl-C.

Let's take a look at a more sophisticated example (sophisticated.yaml):

general:
  stop_time: 1h
  progress: true

ethereum:
  validators: 60
  nodes:
    - location: europe
      reliability: reliable
      tag: boot
      clients:
        el: geth_bootnode
        cl: lighthouse_bootnode
    - locations:
        - europe
        - na_east
        - na_west
      reliabilites:
        - reliable
        - home
      count:
        per_combination: 5

As you can see, we replaced the simple node count with a list of node specifications. Here, the yaml list has two items.

In the first one, we define a host located in europe, with a reliable internet connection. We also specify that a Geth bootnode and a Lighthouse bootnode shall be run on that node.

In the second one, we actually specify multimple nodes: notice the count property, which specifies five nodes per combination. Combination here means every possible pair of specified locations and reliabilities: europe with reliable, europe with home, na_east with reliable, and so on. As there are 2 * 3 = 6 combinations, a total of 5 * 6 = 30 nodes will be created. As we specified 60 validators, each node will host 2 validators.

But what is a "location" and a "reliability"? In Ethereum, we have a lot of globally distributed nodes. Therefore, we want to be able to simulate with varying latency between nodes. There are 8 built-in regions: australia, east_asia, europe, na_east, na_west, south_america, south_aftica, and west_asia. Ethshadow has a table with estimated latencies between these regions and will generate a network topology to make Shadow apply these latencies to the traffic between the nodes.

Reliabilities seek to simulate the varying connection qualities available to nodes. As home stakers are important to Ethereum, we want to include them into our simulations. The following reliabilities are available:

You can define your own locations and reliabilities as well as override the default values of the existing ones.

Before we can start a simulation with our more sophisticated simulation, we have to either delete the data directory from the previous run or specify another directory:

ethshadow -d data_sophisticated sophisticated.yaml

Congrats! These are the basics of Ethshadow.

Advanced Usage

In this section, we will explore some advanced use cases.

• Customize Client Settings: You can run clients with custom CLI parameters, and/or multiple variations of the clients.
• Large Simulations: We have tested simulations with up to 1000 nodes! You need to configure your system to support this.
• Capture Metrics: You can run Prometheus within the simulation to capture the metrics offered by the clients. (Currently CL only)

Customize Client Settings

By default, Ethshadow uses client CLI parameters that ensure that nodes discover each other and use the correct data directory and testnet settings. Furthermore, it automatically set some CLI parameters that are known to be required to make the clients work in Shadow simulations. The default parameters are listed in the client subpages. Also unless specified otherwise, Ethshadow uses the client executable found by the OS via the PATH environment variable.

Often, we want to use specific binaries or override CLI parameters. When trying to test different client configurations, we also may want to run multiple different binaries and configurations. In this page, we will show how to configure such setups.

Client configuration

In the ethereum subsection of your Ethshadow configuration, you can add a clients section to specify client settings. It is a mapping from a client ID to it's settings. You can choose your own client id, or use a default client ID to override the default settings. You can find a client type's default ID and default settings on its client subpage.

Every entry in the clients section has one mandatory parameter: type. Here you instruct Ethshadow what kind of client you want to use. Currently, this is unfortunately also necessary when overriding default client configs. Therefore, a most simple client configuration might look like this:

ethereum:
  clients:
    lighthouse:
      type: lighthouse

This overrides the default lighthouse client config, but doesn't mention any parameters to actually override, so in it's current form does not actually change anything. However, we can now easily add parameters to override. Some client types expose specific parameters to override (see the corresponding client subpage). There are also some parameters available with all client types, which we will explore in the next subsection.

Commonly available client parameters

• executable: Path to client binary to use. Absolute path is recommended. By default, the appropriately named binary in your PATH will be used.
• extra_args: Extra command line arguments to pass to the client after all arguments generated by Ethshadow. Empty by default.
• use_recommended_args: If false, Ethshadow will not use the arguments it deems recommended for this client type. See the client subpages for detailed lists of the recommended arguments. true by default.

Specifying clients in node configuration

After you have created your own client configuration in the clients section, you can now use the client ID in your node configuration section:

ethereum:
  nodes:
    - region: na_east
      reliability: reliable
      count:
        total: 1
    - region: europe
      reliability: reliable
      clients:
        el: reth
        cl: my_lighthouse
        vc: lighthouse_vc
      count:
        per_combination: 1
  clients:
    my_lighthouse:
      type: lighthouse
      extra_args: --awesome-cli-option

The node in na_east uses the default client stack (geth, lighthouse, lighthouse_vc), and the node in europe uses your my_lighthouse, along with reth. Note that you have to specify the full client stack if you want to use a non-default stack, i.e. while lighthouse_vc is the default, omitting it here would create a node without a validator client.

If you want to create multiple variants of a node, you can specify multiple client IDs within a single category. This works similar to specifying multiple regions or reliabilities. For example, changing line 11 above to cl: ["my_lighthouse", "lighthouse"] would create two nodes: one with the default lighthouse and one with your configuration. If you would also specify multiple EL clients (el: ["reth", "geth"]), we would get four nodes: one for every possible combination. Of course, you can modify the number of nodes per combination by adjusting the count property.

Note that the names of the "layers" of the client stack (el, cl, vc) are purely informational and do not influence config generation. You can freely add and remove layers as desired and name them as you like. This is useful for e.g. adding blob spammers, or having multiple or no VCs.

Default client stack

You can override the default client stack with the default_clients setting:

ethereum:
  #... snip ...
  default_clients:
    el: reth
    cl: my_lighthouse
    vc: lighthouse_vc

This will set the client stack for nodes where the stack is not explicitly defined. Note that you can only define a single client per layer here.

Large Simulations

Kernel Configuration

If you want to run the simulation with a lot of nodes, you need to change some limits in the kernel configuration because we will use more resources than the default configuration allows. However, we will only quickly write a bunch of commands here for those who don't want to get into detail. If you want to know why we need to change each of these configurations, please read https://shadow.github.io/docs/guide/system_configuration.html.

echo "fs.nr_open = 10485760" | sudo tee -a /etc/sysctl.conf
echo "fs.file-max = 10485760" | sudo tee -a /etc/sysctl.conf
sudo systemctl set-property user-$UID.slice TasksMax=infinity
echo "vm.max_map_count = 1073741824" | sudo tee -a /etc/sysctl.conf
echo "kernel.pid_max = 4194304" | sudo tee -a /etc/sysctl.conf
echo "kernel.threads-max = 4194304" | sudo tee -a /etc/sysctl.conf

Add the followings lines to /etc/security/limits.conf, but change myname to your username in the machine that you will use to run the simulation.

myname soft nofile 10485760
myname hard nofile 10485760
myname soft nproc unlimited
myname hard nproc unlimited
myname soft stack unlimited
myname hard stack unlimited

If you use the GUI login in your machine, you also need to add the following line in both /etc/systemd/user.conf and /etc/systemd/system.conf.

DefaultLimitNOFILE=10485760

Reboot your machine to make the change effective.

reboot

Swap Space

If you run a lot of nodes, your memory space is probably not enough. You probably need to create some swap space in your storage device. In this example, we will create a 16GB swap file at /swapfile.

sudo fallocate -l 16G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
# Backup the old file
sudo cp /etc/fstab /etc/fstab.bak
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab

Run free -h to check that the swap space is already created.

Capture Metrics

Currently, metrics are captured for Teku and Lighthouse. To capture metrics, simply add a node with the prometheus client to your configuration. IMPORTANT: currently, only nodes mentioned before the Prometheus node are considered for monitoring.

ethereum:
  nodes:
    ...monitored nodes here...
    - location: europe
      reliability: reliable
      tag: monitoring
      clients:
        monitoring: prometheus

To read the metrics after the simulation, simply start Prometheus, for example like this:

prometheus --storage.tsdb.path=<data_dir>/<node_subdir>/prometheus --storage.tsdb.retention.time=30y --config.file=/dev/null

You can use the Prometheus server as usual, for example by connecting a Grafana instance. Note that Shadow is always starting simulations at simulated time 01-01-2000 00:00 UTC.