Introduction to Trin

trin: a fast Rust client for Ethereum access via the Portal Network.

Trin acts as a json-rpc server, like an Ethereum client.

Unlike a typical Ethereum client, trin:

is usable within minutes
limits storage & CPU usage

Continue reading to see how to use trin.

🏗 This is a living document, subject to substantial change without warning.

Quickstart

Trin runs on Linux, MacOS, and Windows. There are two ways to run it: download a binary executable, or install it from source.

Download an executable

The github repository hosts the binaries. Download the latest release for your platform from the releases page.

Extract the compressed file to get the trin executable.

Extraction on Linux / MacOS

The binary is compressed in a tarball, so first we need to extract it.

For example, to extract version 0.1.0:

tar -xzvf trin-v0.1.0-x86_64-unknown-linux-gnu.tar.gz

You now have a trin executable in the current directory.

Run trin

Launch the executable with 2GB local disk space:

trin --mb 2000

Load a block from Ethereum history

Print the block data at height 20,987,654:

BLOCK_NUM=20987654; echo '{"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":["0x'$(printf "%x" $BLOCK_NUM)'", false],"id":1}' | nc -U /tmp/trin-jsonrpc.ipc | jq

For a deeper understanding of how to interact with Ethereum, like invoking a contract function, see the Ethereum data section.

Alternatively, install from source

To get the very latest updates, install from source. This path is intended for power users and developers, who want access to the very latest code.

There are platform-specific build instructions.

Running Trin

Configuration occurs at startup via standard flags. Launch trin with 5GB of storage space like this:

trin --mb 5000

For the full list of flags, run:

trin --help

Data storage limits

When setting a storage usage limit, here's how to think about the tradeoffs:

Storage size	Data access	Network contribution
Smaller	Slower	Less
Larger	Faster	More

Eventually, by default, trin will connect to all Portal Networks. Each network stores different types of data. Some examples are the consensus-layer network for confirming the latest headers, and several execution-layer networks like:

History Network - blocks & receipts
State Network - account info
Transaction Gossip Network - mempool
Canonical Indices Network - tx lookups

For now, only the history network is on by default, because the others are still under active development. At the moment, the state network has only the first one million blocks of state data.

To try out state access, you can turn it on like this:

trin --mb 5000 --portal-subnetworks history,state

Note that to access state, you must also run with history enabled, in order to validate peer responses.

Advanced flags

The following flags all have reasonable defaults and most people won't need to touch them:

Bootnodes

Trin automatically connects to some standard Portal Network bootnodes. Use the --bootnodes cli flag to connect to a specific node or to none.

Private Key management

Trin requires a private key to configure a node's identity. Upon startup, Trin will automatically generate a random private key that will be re-used every time Trin is restarted. The only exceptions are if...

User supplies a private key via the --unsafe-private-key flag, in which case that private key will be used to create the node's identity.
User deletes the TRIN_DATA_DIR or changes the TRIN_DATA_DIR. In which case a new private key will be randomly generated and used.

Networking configuration

Optionally one can specify Trin's network properties:

What sort of connection to query with (HTTP vs IPC)
Port answering Ethereum-related queries
Port for connecting to other nodes

Querying Data

Once Trin is running, you can access Ethereum data by making requests to its endpoint.

The interface for these requests is JSON-RPC, which is a standard way to communicate with Ethereum nodes.

In the following sections, we make queries with:

hand-coded JSON-RPC
using a Web3 library

Serving data for wallets is not covered here. We hope to get there eventually, but the network is not ready quite yet.

Building Queries

If you want to manually query trin, the following patterns can be used, depending on whether Trin was started with --web3-transport as http or ipc. It defaults to ipc.

Whatever the transport, the JSON-RPC format is the same.

Query form

A query for JSON-RPC has the following form for a call to methodname that accepts two parameters: parameter_one and parameter_two.

Query:

{
    "jsonrpc": "2.0",
    "method": "<methodname>",
    "params": ["<parameter_one>", "<parameter_two>"],
    "id":1
}

Succinctly written:

{"jsonrpc":"2.0","method":"<methodname>","params":["<parameter_one>", "<parameter_two>"],"id":1}

In following pages, we'll cover a couple specific examples of queries.

IPC transport

By default, Trin listens on a Unix domain socket file at /tmp/trin-jsonrpc.ipc. This means that you can only access the data from the local machine.

Example command for query (above) to IPC server with socket file located at /tmp/trin-jsonrpc.ipc:

echo '<query>' | nc -U /tmp/trin-jsonrpc.ipc | jq

HTTP transport

If you started Trin with --web3-transport http, you can query it over HTTP.

Command for query (above) to HTTP server on it's default port (8545):

curl -X POST -H "Content-Type: application/json" -d '<query>' localhost:8545 | jq

Response

The "id" will match the request. The result is the data you requested. Alternatively, it may return an error message.

An example successful response:

{
  "jsonrpc": "2.0",
  "id": 1,
  "result": "0x1234"
}

Ethereum data

Trin is designed to serve the JSON-RPC methods that an Ethereum full node would provide. This includes methods the start with the eth_ namespace. Note that not every method is implemented yet.

Here are some methods that are available:

Get Block By Number

Here is an example of making an eth_getBlockByNumber request to a node, to load the block details for block 20,987,654. (In hexadecimal representation, this number is 0x1403f06.)

{"jsonrpc": "2.0", "method": "eth_getBlockByNumber", "params": ["0x1403f06", false], "id": 1}

IPC

By default, trin serves the JSON-RPC methods over an IPC socket. The default location on Linux is /tmp/trin-jsonrpc.ipc.

Make the request in your shell with this one-liner:

BLOCK_NUM=20987654; echo '{"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":["0x'$(printf "%x" $BLOCK_NUM)'", false],"id":1}' | nc -U /tmp/trin-jsonrpc.ipc | jq

Note the following steps in the one-liner above:

Convert the block number from decimal to hexadecimal using printf
Send the request to trin using nc
format the response with jq (optional, for pretty printing)

HTTP

Trin can also serve the JSON-RPC methods over HTTP. It needs to be activated with the flag --web3-transport http. The default port is 8545.

BLOCK_NUM=20987654; curl -X POST -H "Content-Type: application/json" -d '{"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":["0x'$(printf "%x" $BLOCK_NUM)'", false],"id":1}' localhost:8545 | jq

Call a contract function

To read data out of a contract, use the eth_call method.

The data needed to make this call is well populated in the Portal Network for the first 1 million blocks, and is being expanded over time. To activate contract storage access, connect to the Portal state network by running trin with the flag --portal-subnetworks state,history.

Calling a contract fuction usually involves enough formatting that it's helpful to use a tool to build the request, like Web3.py.

Below is an example of one of the earliest contracts posted to Ethereum, which allows you to write "graffiti" on the chain.

Calling Contract with Python

The following python code reads the graffiti from entry 7 in the contract. We read this data (ie~ call this contract fuction) as it was available in block 1,000,000, which has the hash 0x8e38b4dbf6b11fcc3b9dee84fb7986e29ca0a02cecd8977c161ff7333329681e.

from web3 import Web3

# Connect to trin
w3 = Web3(Web3.IPCProvider('/tmp/trin-jsonrpc.ipc'))

# Generate the contract interface
contract = w3.eth.contract(
    address='0x6e38A457C722C6011B2dfa06d49240e797844d66',
    abi='[{"constant":false,"inputs":[],"name":"number_of_claims","outputs":[{"name":"result","type":"uint256"}],"type":"function"},{"constant":true,"inputs":[{"name":"","type":"uint256"}],"name":"claims","outputs":[{"name":"claimant","type":"address"},{"name":"message","type":"string"},{"name":"block_number","type":"uint256"}],"type":"function"},{"constant":false,"inputs":[{"name":"message","type":"string"}],"name":"claim","outputs":[],"type":"function"}]'
)

# Call the contract function
claim = contract.functions.claims(7).call(block_identifier='0x8e38b4dbf6b11fcc3b9dee84fb7986e29ca0a02cecd8977c161ff7333329681e')

print(f"claim graffiti: {claim}")

When running this python script, you should see the output:

claim graffiti: ['0xFb7Bc66a002762e28545eA0a7fc970d381863C42', 'Satisfy Values through Friendship and Ponies!', 50655]

This tells you that the 0xFb7Bc66a002762e28545eA0a7fc970d381863C42 address made the claim, shows you what they wrote, and shows which block number they made the claim in (50,655).

Call contract without Web3

Without a tool like Web3.py, you can build the JSON-RPC request manually. Here is an example of calling the same contract function as above manually:

echo '{"jsonrpc": "2.0", "method": "eth_call", "params": [{"to": "0x6e38A457C722C6011B2dfa06d49240e797844d66", "data": "0xa888c2cd0000000000000000000000000000000000000000000000000000000000000007"}, "0x8e38b4dbf6b11fcc3b9dee84fb7986e29ca0a02cecd8977c161ff7333329681e"], "id": 1}' | nc -U /tmp/trin-jsonrpc.ipc | jq

Which outputs:

{
  "jsonrpc": "2.0",
  "id": 1,
  "result": "0x000000000000000000000000fb7bc66a002762e28545ea0a7fc970d381863c420000000000000000000000000000000000000000000000000000000000000060000000000000000000000000000000000000000000000000000000000000c5df000000000000000000000000000000000000000000000000000000000000002d536174697366792056616c756573207468726f75676820467269656e647368697020616e6420506f6e6965732100000000000000000000000000000000000000"
}

Decoding the result is left as an exercise to the reader.

Portal network data

There are methods for requesting data that are specific to:

Each sub-protocol (history, state, etc.)
- portal_history*
- portal_state*
Discovery protocol
- discv5_*

See the Portal Network JSON-RPC specification here for a comprehensive and interactive view of specific methods available.

Designing a Query

One can identify data by its "content key". The following queries ask Trin to speak with peers, looking for a particular piece of data.

Let us request the block body for block 21,000,000.

Block hash: 0xf5e1d15a3e380006bd271e73c8eeed75fafc3ae6942b16f63c21361079bba709
Selector for a block body: 0x01 (defined in Portal Network spec under the History sub-protocol).
Content key: 0x01f5e1d15a3e380006bd271e73c8eeed75fafc3ae6942b16f63c21361079bba709
Request: portal_historyGetContent, which accepts a content key as a parameter

{"jsonrpc":"2.0","method":"portal_historyGetContent","params":["0x01f5e1d15a3e380006bd271e73c8eeed75fafc3ae6942b16f63c21361079bba709"],"id":1}

IPC

echo '{"jsonrpc":"2.0","method":"portal_historyGetContent","params":["0x01f5e1d15a3e380006bd271e73c8eeed75fafc3ae6942b16f63c21361079bba709"],"id":1}' | nc -U /tmp/trin-jsonrpc.ipc | jq

HTTP

If you have started Trin with --web3-transport http, you can query it over HTTP from any computer that can reach that port.

curl -X POST -H 'Content-Type: application/json' -d '{"jsonrpc":"2.0","method":"portal_historyGetContent","params":["0x01f5e1d15a3e380006bd271e73c8eeed75fafc3ae6942b16f63c21361079bba709"],"id":1}' http://localhost:8545 | jq

Access trin from different computer

If you want to run Trin on one computer and access it from another, launch the trin node with an HTTP transport instead of a default IPC tranport:

trin --web3-transport http

This endpoint is unprotected. Anyone can make requests to your trin node. (This is why the default is IPC)

You probably want to restrict access to your trin node. One way to do that is to firewall off the trin port, and use SSH port forwarding.

SSH port forwarding

Assuming you can SSH into the computer running Trin, you can forward the HTTP port to your local machine, with:

ssh -N -L 8545:127.0.0.1:8545 username@trin-host-computer

Now you can query the trin node from your local machine at http://localhost:8545, as described in Building Queries.

Requirements

Hardware

Suitable:

Processor: x86 or Arm based. Minimum spec TBD.
RAM: Minimum TBD.
Disk: 50 MB

We are eager to hear about a device that is too slow or small to run Trin. The minimum permitted setting for storage usage is technically 1MB. Though the trin binary itself is 41MB at the moment. Tell us if that isn't working for you!

Testing and reports of performance on the following are welcome:

RISC-V based processor.
Resource constrained (CPU/RAM)

Software

Linux, MacOS, or Windows

Network

Testing/reports of low-bandwidth network are welcome.

Trin should be compatible with VPN use, but if you experience difficulty connecting to the network we recommend disabling your VPN.

Monitoring

Once Trin is running, the following may be useful

Logs

If errors are encountered, they will be logged to the console in which Trin was started.

Be aware that The RUST_LOG variable allows for control of what logs are visible.

RUST_LOG=info cargo run -p trin
RUST_LOG=debug cargo run -p trin

If started as a systemd service logs will be visible with:

journalctl -fu <trin-service-name>.service

Disk use

The following locations are where trin stores data by default:

Mac Os: ~/Library/Application Support/trin
Unix-like: $HOME/.local/share/trin

cd /path/to/data
du -sh

CPU and memory use

htop can be used to see the CPU and memory used by trin

Ubuntu: sudo apt install htop
Mac Os: brew install htop

htop

Metrics

Metrics setup no docker

Metrics setup with docker

Metrics setup no docker

Prometheus maintains a database of metrics (trin & system). Grafana converts metrics into graphs. Node exporter provides system information.

graph TD;
Browser-->Grafana-->Prometheus
Prometheus-->Trin & node[Node Exporter]

Install prometheus

Download the latest prometheus https://prometheus.io/download/

curl -LO <link>

Checksum

sha256sum <filename>

Extract

tar xvf <filename>

The directory will contain the binaries for prometheus and promtool. Copy these.

cd <dirname>
sudo cp prometheus /usr/local/bin/
sudo cp promtool /usr/local/bin/

Copy the console files

sudo cp -r consoles /etc/prometheus
sudo cp -r console_libraries /etc/prometheus

Remove the downloaded files

cd ~
rm <tar filename>
rm -r <extracted prometheus directory>

Make a prometheus user

sudo useradd --no-create-home --shell /bin/false prometheus

Create a prometheus data directory

sudo mkdir -p /var/lib/prometheus

Make a config file

sudo nano /etc/prometheus/prometheus.yml

Put this in the config file:

global:
  scrape_interval:     15s
  evaluation_interval: 15s

alerting:
  alertmanagers:
  - static_configs:
    - targets:
rule_files:

scrape_configs:
  - job_name: 'node_exporter'
    static_configs:
      - targets: ['localhost:9100']
  - job_name: 'trin'
    static_configs:
      - targets: ['localhost:9101']

The node_exporter job will gather system data by listening to port 9100. The trin job will gather system data by listening to port 9101.

Update the permissions

sudo chown -R prometheus:prometheus /etc/prometheus
sudo chown -R prometheus:prometheus /var/lib/prometheus

Prometheus will use port 9090 by default. Check it is not used by something else:

sudo lsof -i:9090

Create a service for prometheus

sudo nano /etc/systemd/system/prometheus.service

Include the following, pick another port if 9090 is already in use.

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
User=prometheus
Group=prometheus
Restart=always
RestartSec=5
ExecStart=/usr/local/bin/prometheus \
    --config.file /etc/prometheus/prometheus.yml \
    --storage.tsdb.path /var/lib/prometheus/ \
    --web.console.templates=/etc/prometheus/consoles \
    --web.console.libraries=/etc/prometheus/console_libraries \
    --web.listen-address="localhost:9090"
ExecReload=/bin/kill -HUP $MAINPID

[Install]
WantedBy=multi-user.target

Start the service

sudo systemctl daemon-reload
sudo systemctl start prometheus
sudo systemctl status prometheus
sudo systemctl enable prometheus

Install node exporter

Download the latest node exporter https://prometheus.io/download/#node_exporter

curl -LO <link>

Checksum

sha256sum <filename>

Extract

tar xvf <filename>

The directory will contain the binary for node exporter. Copy this.

cd <dirname>
sudo cp node_exporter /usr/local/bin/

Remove the downloaded files

cd ~
rm <tar filename>
rm -r <extracted node_exporter directory>

Make a node_exporter user and give it permission to the binary.

sudo useradd --no-create-home --shell /bin/false node_exporter
sudo chown -R node_exporter:node_exporter /usr/local/bin/node_exporter

Make a service file:

sudo nano /etc/systemd/system/node_exporter.service

Start the service

sudo systemctl daemon-reload
sudo systemctl start node_exporter
sudo systemctl status node_exporter
sudo systemctl enable node_exporter

Node explorer uses port 9100 by default.

Install grafana

Install

sudo apt-get install -y apt-transport-https software-properties-common wget
sudo wget -q -O /usr/share/keyrings/grafana.key https://apt.grafana.com/gpg.key
echo "deb [signed-by=/usr/share/keyrings/grafana.key] https://apt.grafana.com stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list
sudo apt update
sudo apt install grafana

Open config

sudo nano /etc/grafana/grafana.ini

Modify the http_adr line to use localhost

[server]
;http_addr = # Before
http_addr = localhost # After

Start grafana

sudo systemctl daemon-reload
sudo systemctl start grafana-server
sudo systemctl status grafana-server
sudo systemctl enable grafana-server

This will serve metrics over port 3000.

Generate a grafana dashboard. From trin root directory:

cargo run -p trin -- create-dashboard http://localhost:3000 admin admin http://127.0.0.1:9090

This will create a new monitoring database for trin. This will be visible in the grafana GUI, or directly at a URL similar to: http://localhost:3000/d/trin-app-metrics/trin-app-metrics

If you would like to run the create-dashboard command again, the data source and the dashboard must be deleted, which can be done in the grafana GUI.

Start trin with metrics on

The metrics port must match the trin job set in: /etc/prometheus/prometheus.yml.

cargo run -p trin -- \
    --enable-metrics-with-url 127.0.0.1:<metrics job port> \
    --web3-http-address http://127.0.0.1:<http port> \
    --web3-transport http

For example:

cargo run -p trin -- \
    --enable-metrics-with-url 127.0.0.1:9101 \
    --web3-http-address http://127.0.0.1:8545 \
    --web3-transport http

Updating metrics dashboard

If there are new changes to the metrics dashboard template that you want to view in an already-existing dashboard. The simplest way to update your dashboard is to delete your prometheus datasource and Trin App metrics dashboard, and re-run the create-dashboard command.

View metrics remotely

Trin metrics on a remote machine can be monitored by listening to the grafana address on a local machine.

On local run:

ssh -N -L <port>:127.0.0.1:<port> <user>@<host>

For example

ssh -N -L 3000:127.0.0.1:3000 username@mycomputer

Then navigate to http://127.0.0.1:3000` in a browser and login with username: admin, password: admin. Then navigate to the trin-app-metrics dashboard.

Metrics setup with docker

Install Docker.
Run Prometheus, note that you MUST manually set the absolute path to your copy of Trin's etc/prometheus/:

docker run -p 9090:9090 -v /**absolute/path/to/trin/etc/prometheus**:/etc/prometheus --add-host=host.docker.internal:host-gateway prom/prometheus

Run Grafana:

docker run -p 3000:3000 --add-host=host.docker.internal:host-gateway grafana/grafana:latest

Start your Trin process with:

cargo run -p trin -- --enable-metrics-with-url 0.0.0.0:9100 --web3-http-address http://0.0.0.0:8545 --web3-transport http

The addresses must be bound to 0.0.0.0, because 127.0.0.1 only allows internal requests to complete, and requests from docker instances are considered external.
The --enable-metrics-with-url parameter is the address that Trin exports metrics to, and should be equal to the port to which your Prometheus server is targeting at the bottom of prometheus/prometheus.yml
The --web-transport http will allow Grafana to request routing table information from Trin via JSON-RPC over HTTP

From the root of the Trin repo, run cargo run -p trin -- create-dashboard. If you used different ports than detailed in the above steps, or you are not using docker, then this command's defaults will not work. Run the command with the -h flag to see how to provide non-default addresses or credentials.
Upon successful dashboard creation, navigate to the dashboard URL that the create-dashboard outputs. Use admin/admin to login.

Gotchas

If create-dashboard fails with an error, the most likely reason is that it has already been run. From within the Grafana UI, delete the "json-rpc" and "prometheus" datasources and the "trin" dashboard and re-run the command.
There is a limit on concurrent connections given by the threadpool. At last doc update, that number was 2, but will surely change. If you leave connections open, then new connections will block.

Problems

If you encounter a problem, keep in mind that Trin is under active development. Some issues may be lower on the priority list.

Search for more information

Try searching:

This book
The Trin repository issues

Document the problem

If the problem seems new, raise an issue in the Trin repository. Try to record the problem details and include those in the issue. Include details for how someone else might reproduce the problem you have.

FAQ

The following are frequently asked questions or topics that may be of interest to users.

New submissions are welcome, if you had a question and found the answer elsewhere, submit a pull request or an issue describing the question and the answer. These questions will appear in searches in the book and in the trin repository.

Can I rely on Trin to interact with Ethereum?

Not at present. Trin and the Portal Network more broadly are under active development.

Can Trin be used with a VPN?

Trin should be compatible with VPN use, but if you experience difficulty connecting to the network we recommend disabling your VPN.

Can Trin be used over TOR?

No, the Trin uses uTP, which is not supported in the TOR protocol.

All Command Line Flags

See our guide on how to run trin for the most common flags.

Below is the current list of all command line flags.

Note that these flags may change over time, so run trin --help to get the most up-to-date information for your version.

Usage: trin [OPTIONS] [COMMAND]

Commands:
  create-dashboard  
  help              Print this message or the help of the given subcommand(s)

Options:
      --web3-transport <WEB3_TRANSPORT>
          select transport protocol to serve json-rpc endpoint [default: ipc]
      --web3-http-address <WEB3_HTTP_ADDRESS>
          address to accept json-rpc http connections [default: http://127.0.0.1:8545/]
      --web3-ipc-path <WEB3_IPC_PATH>
          path to json-rpc endpoint over IPC [default: /tmp/trin-jsonrpc.ipc]
      --discovery-port <DISCOVERY_PORT>
          The UDP port to listen on. [default: 9009]
      --bootnodes <BOOTNODES>
          One or more comma-delimited base64-encoded ENR's or multiaddr strings of peers to initially add to the local routing table [default: default]
      --external-address <EXTERNAL_ADDR>
          (Only use this if you are behind a NAT) The address which will be advertised to peers (in an ENR). Changing it does not change which port or address trin binds to. Port number is required, ex: 127.0.0.1:9001
      --no-stun
          Do not use STUN to determine an external IP. Leaves ENR entry for IP blank. Some users report better connections over VPN.
      --no-upnp
          Do not use UPnP to determine an external port.
      --unsafe-private-key <PRIVATE_KEY>
          Hex encoded 32 byte private key (with 0x prefix) (considered unsafe as it's stored in terminal history - keyfile support coming soon)
      --trusted-block-root <TRUSTED_BLOCK_ROOT>
          Hex encoded block root from a trusted checkpoint
      --network <NETWORK>
          Choose mainnet or angelfood [default: mainnet]
      --portal-subnetworks <PORTAL_SUBNETWORKS>
          Comma-separated list of which portal subnetworks to activate [default: history]
      --storage.total <storage.total>
          Maximum storage capacity (in megabytes), shared between enabled subnetworks [default: 1000]
      --storage.beacon <storage.beacon>
          Maximum storage capacity (in megabytes) used by beacon subnetwork
      --storage.history <storage.history>
          Maximum storage capacity (in megabytes) used by history subnetwork
      --storage.state <storage.state>
          Maximum storage capacity (in megabytes) used by state subnetwork
      --enable-metrics-with-url <ENABLE_METRICS_WITH_URL>
          Enable prometheus metrics reporting (provide local IP/Port from which your Prometheus server is configured to fetch metrics)
      --data-dir <DATA_DIR>
          The directory for storing application data. If used together with --ephemeral, new child directory will be created. Can be alternatively set via TRIN_DATA_PATH env variable.
  -e, --ephemeral
          Use new data directory, located in OS temporary directory. If used together with --data-dir, new directory will be created there instead.
      --disable-poke
          Disables the poke mechanism, which propagates content at the end of a successful content query. Disabling is useful for network analysis purposes.
      --ws
          Used to enable WebSocket rpc.
      --ws-port <WS_PORT>
          The WebSocket port to listen on. [default: 8546]
      --utp-transfer-limit <UTP_TRANSFER_LIMIT>
          The limit of max background uTP transfers for any given channel (inbound or outbound) for each subnetwork [default: 50]
  -h, --help
          Print help (see more with '--help')
  -V, --version
          Print version

Concepts

Why does Portal Network exist?

Portal is an important way to support the evolution of the core Ethereum protocol.

To relieve pressure on Ethereum clients, the core protocol will allow full nodes to forget old data in a likely future upgrade.

When that happens, the Portal Network can supply users with that purged data.

How do Portal clients use less space?

Each Portal Network client stores a user-configurable fraction of the data. The client retrieves any missing data from peers, on demand. Just like a full node, the client can cryptographically prove the data it serves.

flowchart TB
    subgraph Full node data: on one computer
    full[Regular full node]
    end
    subgraph Full node data: spread amongst computers
    p1[Portal node]
    p2[Portal node]
    p3[Portal node]
    p1 <--> p2
    p2 <--> p3
    p1 <--> p3

    end

Portal Network

The portal network is a response to two needs. Users should have the ability to:

Access Ethereum using peers (not providers) from 'small' computers.
- An old nice-to-have.
Access historical data once "history expiry" upgrade goes live
- A likely future need.

What is "history expiry"

EIP-4444: Bound Historical Data in Execution Clients is an upgrade that seeks to limit the costs of participating in the network. It does this by allowing the clearing of data older than 1 year.

How the Portal network works

Small users working together

graph TD;
    A & B & C & D & E & F & G & H & I --> id5[complete network data];

The portal network consists of many small nodes that each contribute to the whole. Each node is allocated a specific part of the network to obtain from peers and serve back to the network.

The portal network splits data in to different types (e.g., blocks vs new transactions). Each distinct type is effectively a new network. A portal client such as Trin can be used to operate on each/all of these different sub-protocols.

Dedicated sub-protocols

Users can elect to be part of some sub-networks:

graph TD;
    A & B & C --> id1[(History)]
    D & E & F --> id2[(State)]
    A & G & I --> id4[(Indices)]
    C & E & H --> id3[(Txs)]
    id1[(History)] & id2[(State)] & id3[(Txs)] & id4[(Indices)] --> id5[complete network data];

Peer referrals based on names

Nodes make requests to each other for data. If they don't have the data, they look at their peers and suggest one that is most likely to.

graph LR;
    id1[A requests data] -.-> D & E
    D -.-> F & G
    id1[A requests data] ---> B --->  id2[C has data]
    E -.-> F
    B & G -.-> I

Standard requests

Each node has a name, and only holds data that is similar to that name. Peers can tell who is likely to have what data based on these names.

sequenceDiagram
    Alice-->>Bob: Looking for data 0xabc
    Bob->>Alice: Sorry, but try Charlie (gives address)
    Alice-->>Charlie: Looking for data 0xabc
    Charlie->>Alice: Data 0xabc

Tunable resources

Nodes keep content that is similar to their name. That similarity radius can be made larger to voluntarily hold more data.

graph TD;
    id1[(Alice with big hard drive)]
    id2[(Bob)]
    id4[(Charlie, medium)]

In addition to Trin, several other Portal clients participate in the same network.

Portal vs Standard Clients

This page is not trying to be an unbiased comparison. Obviously we think Portal is cool. Below is why we think so. Of course, we still identify known drawbacks of using the Portal Network.

What are standard clients?

These are also known as Execution Layer clients. The top three by usage are Geth, Nethermind, and Besu. There are more great ones out there.

You would use standard clients if you want to stake ether, or access on-chain contracts or generate transactions without using a third-party service.

Standard client challenges

In general, the following are challenges with all standard clients:

First-time sync: can take days or more
High storage needs: 2 Terabytes or more
Connectivity sensitivity: going offline overnight means spending a while to catch up
Non-trivial CPU usage: you'll notice it running on your laptop

Portal benefits

In contrast, Portal Network was designed to overcome these challenges. For example:

Sync

First-time sync is very fast. You can be up and running in minutes.

All Portal needs to do is download data for the Consensus Layer to identify the tip of the chain signed by stakers. The client can then validate all past Execution Layer data.

Storage

The amount you store is fully configurable. You could run with 10 MB, if you want.

You will help yourself and the network by storing more, like 1-100 Gigabytes. That means you'll get faster local requests and you'll be able to serve more data to others. But it is fully your choice.

Connectivity

Going offline and coming back online is no big deal. You'll be able to sync up again quickly.

CPU

Portal is designed to be very light on CPU. The goal is to make it so you can run it on a Raspberry Pi, or even forget that it's running on your laptop, because it's so light.

Since we're still in early days, users may experience some fluctuations in CPU usage. We will continue to optimize!

Joint Benefits of Portal & Standard Clients

Some things are great about standard clients, so Portal keeps those features, like:

Standard JSON-RPC Endpoint

Portal clients can act as a server for Ethereum data. They do this by hosting the standardized JSON-RPC endpoint. Portal clients are a drop-in replacement for you Web3 script or wallet.

Note that not every endpoint is supported in Portal clients yet, but coverage is expanding over time.

Fully-validated

Whenever Portal clients request data from a peer, they also generate internal cryptographic proofs that the provided content matches the canonical chain.

This happens recursively until the source of consensus. For example, if you request contract storage, Portal clients will:

generate merkle proofs back to the state root
verify the state root against a header
verify that the header was signed/attested by Ethereum stakers

Privacy

There is no single third party that collects every request you make about the Ethereum network.

An individual peer knows if you request data from them, but they don't know what your original RPC query is.

Portal Drawbacks

There are some drawbacks to using Portal clients. Here are some known ones:

Latency

When making a request that requires many different pieces of data under the hood, that requires many network round trips. This can be slow. Reading data out of a contract might take many seconds, instead of milliseconds.

Partial view

The essense of Portal is that you only store a small slice of the total data.

There are some use cases that involve seeing all the data in the network at once. For those, it will often be better to just load a standard client and have all the data locally for analysis.

Caveat: There are still some cases that it might be faster to use Portal, even if you need a wide spread of data. You might be able to enumerate every account on the network faster than it takes a standard client to sync up from scratch, for example.

Offline access

A Portal client is not very useful while offline. Clients depends on requesting missing data from peers. In contrast, standard clients that are offline can still serve all data up until their latest sync point.

Uptime

The primary Ethereum network is maniacal about uptime. If you run a standard client, and have an internet connection, you will be getting network updates.

There are more opportunities for downtime or lag in Portal clients. You might expect something more like 99.5% uptime. (No promises, just a guess. Of course we will aim higher)

Use Cases

The following are examples of people who will be well-suited to using a Portal Network client, like Trin.

All of these examples are speculative about the future. The most plausible users today are the Protocol Researcher and Client Developer.

Laptop wallet user

A user has a laptop that frequently is turned off. When they want to transact, they can turn on Trin and connect their wallet to it.

Benefit: Wallet use without reliance on third party wallet APIs.

Desktop wallet user

A user has a desktop that usually on, but most of the disk is used for other things. When they want to transact, their wallet is already connected to their portal node.

Benefit: Wallet use without reliance on third party wallet APIs. Contributes to network health without using entire disk.

Protocol researcher

A researcher looking to explore the Ethereum protocol, testing out specific aspects and perhaps making experimental changes to the protocol.

Benefit: Spin up a node and play around quickly and with low cost.

Client developer

Ethereum clients are resource-intensive. Developers of those clients can update their client to use Portal Network data and reduce the local burden of their client.

Benefit: Reduce resource usage of an Ethereum client.

Single board computer hobbyist

A raspberry pi 3, or similarly-sized computer with could contribute to network health.

Currently a raspberry pi 4 can run a full node, with consensus and execution clients, however this is a bit tight and requires a ~2TB SSD.

Benefit: Learn about Ethereum, get node access and provide the network with additional robustness.

Mobile user

Trin is not currently configured to run on mobile, however this is plausibly a viable and interesting use case for the future. There are a number of challenges to address first. Mobile does not typically support backrgound use of apps, and the battery life of a mobile device is a concern. So one challenge is how to make the mobile clients contribute back to the network in a way that is not too burdensome on the device.

Benefit: Wallet use without reliance on third party wallet APIs.

Unsuitable users

There are situations where Trin is estimated to not be a good node choice:

Very speedy historical state access. It's possible to retrieve old state, but don't expect sub-second contract reads on state as viewed from a historical block.
Building blocks locally, as a block producer. Random access to the full state and transaction pool is not supported at the speed needed to build competitive blocks.
- We remain hopeful that in the future, you could use an externally-generated block (like one provided by MEV-boost) so that you can act as a validater using a standard Consensus client, with Trin as the Execution client. This probably depends on a future where state witnesses are bundled with the block sent to you by the producer.

Developers

This part of the book is for understanding Trin, and processes around building Trin better.

Where the Trin crates and the Portal Network specification are the source of truth, this section seeks to offer a quicker "key concepts" for getting started.

It seeks to answer questions like:

What do I need to know about the Portal Network?
How do the different components of Trin work together?
What sort of data guarantees are made and how are they achieved?
What things should a new contributor be mindful of?

Quick setup

This is a single page that aims to cover everything required to get Trin running.

Trin is currently in unstable alpha, and should not be used in production. If you run into any bugs while using Trin, please file an Issue!

Building on Debian Based Systems

Prerequisites

Rust installation

Building, Testing, and Running

Note: If you use a VPN, you should disable it before running Trin.

Install dependencies (Ubuntu/Debian):

apt install libclang-dev pkg-config build-essential

Environment variables:

# Optional
export RUST_LOG=<error/warn/info/debug/trace>
export TRIN_DATA_PATH=<path-to-data-directory>

Build, test, and run:

git clone https://github.com/ethereum/trin.git
cd trin

# Build
cargo build --workspace

# Run test suite
cargo test --workspace

# Build and run test suite for an individual crate
cargo build -p trin-history
cargo test -p trin-history

# Run
cargo run

Note: You may also pass environment variable values in the same command as the run command. This is especially useful for setting log levels.

RUST_LOG=debug cargo run

View CLI options:

cargo run -- --help

Building on Arch Based Systems

Before starting, update your system.

sudo pacman -Syu

Then, install rust.

sudo pacman -S rustup
rustup install stable

To check that the rust toolchain was successfully installed, run:

rustc --version

You should see something like:

rustc 1.81.0 (051478957 2024-07-21)

Next, install the required dependencies:

sudo pacman -S openssl clang pkg-config base-devel llvm git

Now you can build, test and run Trin!

git clone https://github.com/ethereum/trin.git
cd trin

# Build
cargo build --workspace

# Run test suite
cargo test --workspace

# Build and run test suite for an individual crate
cargo build -p trin-history
cargo test -p trin-history

# Run help
cargo run -- --help

# Run Trin with defaults
cargo run

Run locally

Serve portal node web3 access over a different port (such as 8547) using the --web3-http-address flag. The --web3-transport for a local node will be over http (rather than ipc).

RUST_LOG=debug cargo run -- \
    --web3-http-address http://127.0.0.1:8547 \
    --web3-transport http \
    --discovery-port 9009 \
    --bootnodes default \
    --mb 200 \
    --no-stun

Connect to the Portal Network mainnet

To immediately connect to the mainnet, you can use the --bootnodes default argument to automatically connect with the default Trin bootnodes.

cargo run -- --bootnodes default

To establish a connection with a specific peer, pass in one or more bootnode ENRs. Pass the ENR as the value for the --bootnodes CLI flag.

cargo run -- --bootnodes <bootnode-enr>

Default data directories

Linux/Unix: $HOME/.local/share/trin
MacOS: ~/Library/Application Support/trin
Windows: C:\Users\Username\AppData\Roaming\trin

Using Trin

In some of the following sections, we make use of the web3.py library.

Connect over IPC

In a python shell:

>>> from web3 import Web3
>>> w3 = Web3(Web3.IPCProvider("/tmp/trin-jsonrpc.ipc"))
>>> w3.client_version
'trin 0.0.1-alpha'

To request a custom jsonrpc endpoint, provide the endpoint and array of params. e.g.:

>>> w3.provider.make_request("portal_historyPing", ["enr:-IS4QBz_40AQVBaqlhPIWFwVEphZqPKS3EPso1PwK01nwDMtMCcgK73FppW1C9V_BQRsvWV5QTbT1IYUR-zv8_cnIakDgmlkgnY0gmlwhKRc9_OJc2VjcDI1NmsxoQM-ccaM0TOFvYqC_RY_KhZNhEmWx8zdf6AQALhKyMVyboN1ZHCCE4w", "18446744073709551615"])
{'jsonrpc': '2.0',
 'id': 0,
 'result': {'enrSeq': '3',
  'dataRadius': '115792089237316195423570985008687907853269984665640564039457584007913129639935'}}

See the JSON-RPC API docs for other standard methods that are implemented. You can use the web3.py API to access these.

Connect over HTTP

First launch trin using HTTP as the json-rpc transport protocol:

cargo run -- --web3-transport http

Then, in a python shell:

>>> from web3 import Web3
>>> w3 = Web3(Web3.HTTPProvider("http://127.0.0.1:8545"))
>>> w3.client_version
'trin 0.0.1-alpha'

The client version responds immediately, from the trin client. The block number is retrieved more slowly, by proxying to Infura.

To interact with trin at the lowest possible level, try netcat:

nc -U /tmp/trin-jsonrpc.ipc
{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":83}
{"jsonrpc":"2.0","id":83,"result":"0xb52258"}{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":84}
{"jsonrpc":"2.0","id":84,"result":"0xb52259"}{"jsonrpc":"2.0","id":85,"params":[],"method":"web3_clientVersion"}
{"jsonrpc":"2.0","id":"85","result":"trin 0.0.1-alpha"}
{"jsonrpc":"2.0","id":86,"params":[],"method":"discv5_nodeInfo"}
{"id":86,"jsonrpc":"2.0","result":"enr:-IS4QHK_CnCsQKT-mFTilJ5msHacIJtU91aYe8FhAd_K7G-ACO-FO2GPFOyM7kiphjXMwrNh8Y4mSbN3ufSdBQFzjikBgmlkgnY0gmlwhMCoAMKJc2VjcDI1NmsxoQNa58x56RRRcUeOegry5S4yQvLa6LKlDcbBPHL4H5Oy4oN1ZHCCIyg"}

For something in between, you may use curl to send requests to the HTTP JSON-RPC endpoint.

Developer stories

Trin is under active development. Perhaps you would like to get involved?

The following are some situations that might resonate.

Issue resolver

Someone who tried out Trin and found an issue, then worked out where it was coming from.

Consider making a pull request to fix the issue.

Ecosystem contributor

Trin, and the Portal Network more broadly are perhaps more quiet than other areas of Ethereum development. Maybe you can see yourself helping out somewhere where you can have a meaningful impact.

Researcher

Someone looking into the Ethereum protocol upgrade path, and thinking through the impact of potential upcoming changes.

There are interesting facets to the Portal network still to be determined.

Perhaps you can be tempted by:

Double batched merkle log accumulators
Topology of content in distributed hash tables
Adversarial scenario planning and mitigation

Hobbyist

Someone looking to poke around and run a node on a single board computer or a mobile device. How small is too small?

Rust developer

Someone looking to build something meaningful in Rust, with interesting architecture and crates:

Cryptography
Peer to peer networking
Async runtimes

Goals

Demonstrate feasibility

Implement the Portal Network and demonstrate its use. Starting with subset of the whole and then expanding from there.

Prioritise Sub-protocols

Primary

Get the History sub-protocol working.

Iron out bugs
Interop with other clients
Monitor network to see if it retains data

Secondary

Start work on the State sub-protocol.

Implementation from the Portal Network specification.

Tertiary

Start work on remaining sub-protocols

Canonical indices
Transaction gossip

Progress status

The Portal Network and Trin are under active development so different components may be in varying stages of progress.

Completed, looking for bugs
Mid-construction
Future, planned features

Some methods to get a sense of the state of development are:

Run trin -- --help and see what flags are available.
Look at recent closed PRs to see what has just been merged.
Look at recent issues that have been opened to see what active foci are.
Look at what examples are shown in the setup and usage guides.
Run trin in the way you think it might work and see what happens.

Architecture

Trin can be understood from different perspectives.

How is code organised?
How does data flow through trin?
How is data stored?
How does testing work?

Workspaces

Trin is a package that can be run:

cargo run -p trin

The trin repository is composed of workspaces that are used by the main Trin package. Their relationship is outlined below.

`trin`

Code for the trin package is located in ./src.

This crate is responsible for the operation of the Trin node functionality.

Startup with different configurations via command line arguments
Starting threads for different important functions such as uTP, Discovery & JSON-RPC.
These threads perform tasks such as listening for peers or requests from a user.

`portalnet`

This crate is responsible for the code that defines the main functions and data structures required for the operation of a Trin node. This includes code for:

Interacting with and managing peers
Determining content to store and share
Database management
Ethereum related data structures

`trin-history`

This crate is responsible for the History sub-protocol. This means interacting with peers to retrieve and distribute the following:

Block headers
Block bodies
Block receipts

Additionally, it is also responsible for the header accumulator, a structure which provides a mechanism to determine whether a given block hash is part of the canonical set of block hashes.

The crate uses the ethportal-api crate to represent the main data type in this crate: the HistoryContentKey. This struct implements the OverlayContentKey trait, which allows it to be treated as a member of the broader family of OverlayContentKeys.

`trin-state`

This crate exists mostly as a stub for future work.

This crate is equivalent in function to the trin-history crate, but instead is responsible for the State sub-protocol.

This means that it is responsible for:

The state of all accounts.
The state of all contracts.
The bytecode of all contracts.

Data in the state network is represented as a tries (tree structures). The network uses proofs against these tries to allow Trin nodes to verify the correctness of data.

`ethportal-api`

This crate seeks to expose the data structures in the Portal Network specification. This includes features such as derived SSZ encoding and convenience functions.

The crate defines traits that may be used across different sub-protocols. For example, the OverlayContentKey may be implemented for content on both the History and State sub-protocols. Thus a function can accept content from both networks via T: OverlayContentKey.

fn handles_content_keys<T: OverlayContentKey>(key: T) {
    // Snip
}

The crate will evolve to provide the types required for the other sub-protocols.

`rpc`

This crate contains implementations of ethportal-api jsonrpsee server API traits in Trin and interface for running the JSON-RPC server.

`utp-testing`

Trin uses Micro Transport Protocol (uTP) a UDP based protocol similar to the BitTorrent protocol. This crate can be used to set up clients and servers to test the protocol on a single machine.

`ethportal-peertest`

Home for integration testing utils used by trin.

Process flow

The following main threads are spawned when Trin is started via ./src/main.rs.

stateDiagram-v2
    trin: trin

    state trin {
        utplistner: UTP listener
        subprotocolhandler: sub-protocol handler
        subprotocolnetworktask: sub-protocol network task
        portaleventshandler: portal events handler
        jsonrpcserver: JSON-RPC server

        main() --> utplistner
        main() --> subprotocolhandler
        main() --> subprotocolnetworktask
        main() --> portaleventshandler
        main() --> jsonrpcserver

    }

Where for each sub-protocol implemented (History, State, Etc.,), a new thread is started.

Here are some of the major components that are called on startup within ./src/lib.rs.

stateDiagram-v2
    collection: configs and services

    state trin {
        main() --> from_cli()
        from_cli() --> run_trin()
        run_trin() --> discovery()
        run_trin() --> utp_listener()
        run_trin() --> header_oracle()
        run_trin() --> portalnet_config
        run_trin() --> storage_config

    }

    portalnet_config --> collection
    storage_config --> collection
    discovery() --> collection
    header_oracle() --> collection
    utp_listener() --> collection

    state portalnet {
        portalnet_config
        storage_config
        discovery()
    }
    state utp {
        utp_listener()
    }
    state validation {
        header_oracle()
    }

Once the initial collection of important configs and services have been aggregated, they are passed to the crates for each sub-protocol (trin-history shown here). The received data structures are then used to start the JSON-RPC server.

An events listener awaits network activity that can be actioned.

stateDiagram-v2
    trinhistory: trin-history
    jsonrpchistory: JSON-RPC History details
    historyhandler: History handler
    collection: configs and services

    state trin {
        collection --> initialize_history_network()
        collection --> HistoryRequestHandler
        initialize_history_network() --> jsonrpchistory
        jsonrpchistory --> launch_jsonrpc_server()
        HistoryRequestHandler --> historyhandler
        collection --> events()
        historyhandler --> events()
    }

    state portalnet {
        events()
    }

    state trinhistory {
        initialize_history_network()
        state jsonrpc {
            HistoryRequestHandler
        }
    }
    state rpc {
        launch_jsonrpc_server()
    }

Then ./portalnet/events.rs is handles events at the level of the Portal Wire Protocol. These are defined messages that are compliant with the Discv5 protocol, and specific to the Portal Network.

Database

The database related code is located in ./portalnet/storage.rs.

There are three main database kinds:

DB Name	Kind	Location	Purpose	Keys	Values
Main	SQLite	Disk	Data store	Content ID	Content key, content value, content size
Memory	HashMap	Memory	Kademlia cache	Content key	Content data bytes

Main content database

This is an SQLite database that stores content data. For a piece of content, this includes the content ID, content key, content value and the size of the content. It makes assessing the size of the database quicker by avoiding the need to repeatedly compute the size of each content.

Memory content database

This uses is an in-memory hashmap to keep content that may not be required for long term storage. An overlay service uses this database when receiving data from a peer as part of Kademlia-related actions. If required, data is later moved to disk in the main content database.

Testing

Testing occurs at different levels of abstraction.

Unit testing

Unit tests are for checking individual data structures and methods. These tests are included within each workspace, at the bottom the file that contains the code being tested. Tests are run by CI tasks on pull requests to the Trin repository.

Integration testing

Tests that involve testing different parts of a crate at the same time are included in a /tests directory within the relevant module or crate. They are also run by CI tasks on pull requests to the Trin repository.

Network simulation

The test-utp crate is part of continuous integration (CI). This sets up client and server infrastructure on a single machine to test data streaming with simulated packet loss.

Hive

Hive testing runs Trin as a node and challenges it in a peer to peer environment. This involves creating a docker image with the Trin binary and passing it to Hive.

Hive itself is a fork of Ethereum hive testing and exists as portal-hive, an external repository (here). It can be started with docker images of other clients for cross-client testing. The nodes are started, fed a small amount of data and then challenged with RPC requests related to that data.

Testing is automated, using docker configurations in the Trin repository to build test Trin and other clients at a regular cadence. Results of the latest test are displayed at https://portal-hive.ethdevops.io/.

Protocols

This section contains summaries of important protocols for the Portal Network. The purpose is to distill important concepts to quickly see how Trin works.

See the relevant specifications for deeper explanations.

JSON-RPC

This is a document for all JSON-RPC API endpoints currently supported by Trin. Trin plans to eventually support the entire Portal Network JSON-RPC API and Ethereum JSON-RPC API.

Currently supported endpoints

Portal Network

The specification for these endpoints can be found here.

discv5_nodeInfo
discv5_routingTableInfo
portal_historyFindContent
portal_historyFindNodes
portal_historyGossip
portal_historyLocalContent
portal_historyPing
portal_historyOffer
portal_historyGetContent
portal_historyStore
portal_stateFindContent
portal_stateFindNodes
portal_stateLocalContent
portal_stateGossip
portal_stateOffer
portal_stateStore
portal_statePing

Custom Trin JSON-RPC endpoints

The following endpoints are not part of the Portal Network specification and are defined in subsequent sections:

History Overlay Network

`portal_historyRadius`

Returns the current data storage radius being used for the History network.

Parameters

None

Returns

Data storage radius.

Example

{
  "id": 1,
  "jsonrpc": "2.0",
  "result": "18446744073709551615"
}

`portal_historyTraceGetContent`

Same as portal_historyGetContent, but will also return a "route" with the content. The "route" contains all of the ENR's contacted during the lookup, and their respective distance to the target content. If the content is available in local storage, the route will contain an empty array.

Parameters

content_key: Target content key.

Returns

Target content value, or 0x if the content was not found.
Network ENRs traversed to find the target content along with their base-2 log distance from the content. If the target content was found in local storage, this will be an empty array.

Example

{
  "id": 1,
  "jsonrpc": "2.0",
  "result": {
	  "content": "0xf90217a06add1c183f1194eb132ca8079197c7f2bc43f644f96bf5ab00a93aa4be499360a01dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347942a65aca4d5fc5b5c859090a6c34d164135398226a05ae233f6377f0671c612ec2a8bd15c20e428094f2fafc79bead9c55a989294dda064183d9f805f4aecbf532de75e6ad276dc281ba90947ff706beeaecc14eec6f5a059cf53b2f956a914b8360ea6fe271ebe7b10461c736eb16eb1a4121ba3abbb85b9010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000860710895564a08309a92a832fefd882520884565fc3be98d783010302844765746887676f312e352e31856c696e7578a0c5e99c6e90fbdee5650ff9b6dd41198655872ba32f810de58acb193a954e15898840f1ce50d18d7fdc",
	  "route": [{
            "enr": "enr:-IS4QFoKx0TNU0i-O2Bg7qf4Ohypb14-jb7Osuotnm74UVgfXjF4ohvk55ijI_UiOyStfLjpWUZsjugayK-k8WFxhzkBgmlkgnY0gmlwhISdQv2Jc2VjcDI1NmsxoQOuY9X8mZHUYbjqVTV4dXA4LYZarOIxnhcAqb40vMU9-YN1ZHCCZoU",
            "distance": 256
	  }]
  }
}

State Overlay Network

`portal_stateRadius`

Returns the current data storage radius being used for the State network.

Parameters

None

Returns

Data storage radius.

Example

{
  "id": 1,
  "jsonrpc": "2.0",
  "result": "18446744073709551615"
}

General

`portal_paginateLocalContentKeys`

Return a paginated list of all of the content keys (from every subnetwork) currently available in local storage.

Parameters

offset: The number of records that need to be skipped.
limit: Number of entries to return.

Returns

content_keys: List of content keys.
total_entries: Total number of content keys in local storage.

Example

{
  "id": 1,
  "jsonrpc": "2.0",
  "result": {
    "content_keys": ["0x0055b11b918355b1ef9c5db810302ebad0bf2544255b530cdce90674d5887bb286"],
    "total_entries": 1
  }
}

Core concepts

This section contains specific concepts that are common, important or that have an interesting facet to understand.

Finding peers

If a peer is in a network behind a NAT (Network Address Translation) table, the process for finding a peer is more complicated.

These diagrams are intended as a rough-guide.

Non-NAT simple case

The bootnode can gossip to Charlie who can then directly contact Alice.

sequenceDiagram
    Alice IP1 PORT1-->>Bootnode: Hello (ENR with no IP)
    Bootnode-->>Alice IP1 PORT1: Hi, I notice your address is <IP1>:<PORT1>
    Alice IP1 PORT1-->>Alice IP1 PORT1: Updates ENR (<IP1>:<PORT1>)
    Bootnode-->>Charlie: Meet Alice (ENR with <IP1>:<PORT1>)
    Charlie->>Alice IP1 PORT1: Hello Alice at <IP1>:<PORT1>
    Alice IP1 PORT1->>Charlie: Hello (ENR <IP1>:<PORT1>)

NAT problem

The bootnode can gossip to Charlie, but Charlie is a stranger from the NAT's perspective. It doesn't know who on the internal network is the recipient.

The NAT remembers who it has spoken to.
Messages from the bootnode are expected.
Messages from Charlie are not expected, and its not clear who they are for. Perhaps the smart fridge?

sequenceDiagram
    Alice IP1 PORT1-->>NAT IP2 PORT2: Hello bootnode (ENR with no IP)
    Note right of NAT IP2 PORT2: Stores Bootnode
    NAT IP2 PORT2-->>NAT IP2 PORT2: Maps from internal IP
    NAT IP2 PORT2-->>Bootnode: Hello bootnode (ENR with no IP)
    Bootnode-->>NAT IP2 PORT2: Hi, I notice your address is <IP2>:<PORT2>
    NAT IP2 PORT2-->>NAT IP2 PORT2: Maps to internal IP
    NAT IP2 PORT2-->>Alice IP1 PORT1: Hi, I notice your address is <IP2>:<PORT2>
    Alice IP1 PORT1-->>Alice IP1 PORT1: Updates ENR (<IP2>:<PORT2>)
    Alice IP1 PORT1-->>NAT IP2 PORT2: Thanks bootnode (ENR with <IP2>:<PORT2>)
    NAT IP2 PORT2-->>Bootnode: Thanks boodnode (ENR with <IP2>:<PORT2>)
    Bootnode-->>Charlie: Meet Alice (ENR with <IP2>:<PORT2>)
    Charlie->>NAT IP2 PORT2: Hello Alice at <IP2>:<PORT2>
    Note right of NAT IP2 PORT2: No map on record. Who is this for?
    Note right of Charlie: Hmm Alice didn't respond.

The NAT solution

If Alice knows she is behind a NAT, she can pass a message which goes:

"I'm behind a NAT. Send your requests via peers and I'll reach out to you."

The bootnode gossips to Charlie
Charlie sees "NAT" in Alices ENR
Charlie asks the bootnode to introduce him to Alice
Alice reaches out to Charlie
The NAT now has a mapping for Charlie-Alice messages.

Part 1: NAT detection

Alice can suspect that she is behind a NAT probabilistically. If 2 minutes after connecting with a bootnode, no strangers (like Charlie) have reached out, a NAT is likely.

sequenceDiagram
    Alice IP1 PORT1-->>NAT IP2 PORT2: Hello bootnode (ENR with no IP)
    Note right of NAT IP2 PORT2: Stores Bootnode
    NAT IP2 PORT2-->>NAT IP2 PORT2: Maps from internal IP
    NAT IP2 PORT2-->>Bootnode: Hello bootnode (ENR with no IP)
    Bootnode-->>NAT IP2 PORT2: Hi, I notice your address is <IP2>:<PORT2>
    NAT IP2 PORT2-->>NAT IP2 PORT2: Maps to internal IP
    NAT IP2 PORT2-->>Alice IP1 PORT1: Hi, I notice your address is <IP2>:<PORT2>
    Alice IP1 PORT1-->>Alice IP1 PORT1: Updates ENR (<IP2>:<PORT2>)
    Alice IP1 PORT1-->>NAT IP2 PORT2: Thanks bootnode (ENR with <IP2>:<PORT2>)
    NAT IP2 PORT2-->>Bootnode: Thanks boodnode (ENR with <IP2>:<PORT2>)
    Note right of Alice IP1 PORT1: ... Hmm no strangers. Must be a NAT.

Part 2: NAT communication

Alice can put "NAT" in her ENR. Now when Charlie tries to get in touch, he knows to go via a peer.

Continued from above, skipping Charlie's failed attempt to contact Alice directly.

sequenceDiagram
    Note right of Alice IP1 PORT1: ... Hmm no strangers. Must be a NAT.
    Alice IP1 PORT1-->>NAT IP2 PORT2: Update: NAT (ENR with NAT <IP2>:<PORT2>)
    NAT IP2 PORT2-->>Bootnode: Update: NAT (ENR with NAT <IP2>:<PORT2>)
    Bootnode-->>Charlie: Meet Alice (ENR with NAT <IP2>:<PORT2>)
    Charlie->>Bootnode: Hello Alice (From Charlie ENR(<charlie>))
    Note right of Bootnode: To Alice via Bootnode
    Bootnode->>NAT IP2 PORT2: Hello Alice (From Charlie ENR(<charlie>))
    NAT IP2 PORT2-->>NAT IP2 PORT2: Maps to internal IP
    NAT IP2 PORT2-->>Alice IP1 PORT1:  Hello Alice (From Charlie ENR(<charlie>))
    Alice IP1 PORT1-->>NAT IP2 PORT2: Hello Charlie (ENR with NAT <IP2>:<PORT2>)
    Note right of NAT IP2 PORT2: Stores Charlie
    NAT IP2 PORT2-->>NAT IP2 PORT2: Maps from internal IP
    NAT IP2 PORT2-->>Charlie: Hello Charlie (ENR with NAT <IP2>:<PORT2>)
    Charlie-->>NAT IP2 PORT2: Hi Alice
    NAT IP2 PORT2-->>NAT IP2 PORT2: Maps to internal IP
    Note right of NAT IP2 PORT2: Finally has a mapping for Charlie!
    NAT IP2 PORT2-->>Alice IP1 PORT1: Hello Alice

Chain tip

A Trin node can serve information about the chain tip, such as the latest block number. A Trin node knows about the beacon chain protocol that is creating the chain tip.

By listening to activity on the beacon chain network, it can follow the activities of members of the sync committee. If a certain fraction of the sync committee have signed off on a certain beacon block, the Trin node can be confident that this is likely to be the chain tip.

Beacon blocks contain references to Ethereum blocks, and so the node can see the tip of the Execution chain.

Cryptographic accumulator

A cryptographic accumulator is a structure that allows verification of a specific block header in the past is part of the canonical chain.

The History sub-protocol is responsible for accumulator-related data.

An accumulator has been constructed for the Portal Network, because it is too burdensome to keep all the headers on disk. This applies to pre-merge blocks. For post-merge blocks, the Beacon Chain already maintains an accumulator that Trin can use via a Beacon Chain light client.

Canonicality

A block can be valid but not canonical if it is an Uncle. Blocks A-F are canonical, with F being the latest block.

While Uncle_1 may have a valid block difficulty and parent, it was not built upon.

flowchart RL
    Uncle_3-.->D
    F--->E
    E--->D
    Uncle_4-.->D
    Uncle_1-.->C
    D--->C
    Uncle_2-.->C;
    C--->B;
    B--->A;

If a Trin node is presented with such a block, it can check the accumulator, which only processes non-uncle blocks A-F.

Tip knowledge

First, the most recent block hash at the tip of the accumulator must be known.

This is easy, as the accumulator only needs to cover pre-merge blocks. The final pre-merge block (last Proof of Work block) hash is known and never needs to be updated.

Proofs

A Merkle proof can be constructed for any given historical block hash. The proof asserts that a given hash (from a peer) is part of the accumulator (valid based on knowledge of the current chain tip).

A proof cannot be constructed for any other sort of block (Uncle block, fabricated block).

Accumulator construction

The accumulator is specifically a double-batched merkle log accumulator.

First historical blocks are processed in batches called Epochs (unrelated to the concept of a 32-slot epoch in the Beacon Chain).

The accumulator constructor consists of two lists:

One cache for holding blocks (header and difficulty).
One final store (Master Accumulator) that the cache roots are added to.

flowchart TD
    Genesis-->new_epoch[Start new Epoch Accumulator]
    new_epoch-->append
    append[Append block header and difficulty to Epoch Accumulator]
    append--> epoch_done{Done 8192 yet?}
    epoch_done -.->|Yes| get_epoch_root
    epoch_done -->|No| PoS{PoS reached?}
    PoS --> |No| append
    PoS -.-> |Yes| finished[Finished]
    add_root[Append root to Master Accumulator] -.-> new_epoch
    get_epoch_root[Compute hash tree root of epoch] -.-> add_root
    finished -.-> save[Save incomplete final epoch and Master Accumulator]

Thus the final output is a list of roots called the Master Accumulator.

Constructing proofs

If you have a block and you know the block number, then you know which epoch root is relevant. You also know which part of the epoch it came from. That is, you know the index of the leaf in the Merkle tree.

With the root of the tree, the index and the data (the block hash in question), a proof can be constructed showing that this leaf was part of the tree.

Proof use

A proof can be to a peer alongside the data. That way, a peer can quickly and check that the data is canonical.

Accumulator distribution

The Accumulator is built once and then distributed in Trin (and other clients). It does not change over time and so can be incorporated into the trin-validation (./trin-validation/src/assets) and included in binary releases.

The History network contains individual epoch hashes from the Master Accumulator and refers to them with the terms: epoch_accumulator and epoch_accumulator_key (includes selector). See the History sub-protocol section of the Portal Network spec.

Master accumulator details

The Master Accumulator consists of:

1895 complete epoch roots
1 incomplete epoch root (a partial epoch with 5362 records (block headers))

epoch,index
8191,0x5ec1ffb8c3b146f42606c74ced973dc16ec5a107c0345858c343fc94780b4218 // first epoch
16383,0xa5364e9a9bc513c4601f0d62e6b46dbdedf3200bbfae54d6350f46f2c7a01938
...
15523839,0x804008940c025a4e8a00ea42a659b484ba32c14dff133e9d3b7bf3685c1e54de // penultimate epoch (full)
15532031,0x3f81607c8cb3f0448a11cab8df0e504b605581f4891a9a35bd9c0dd37a71834f // final epoch (incomplete)

Final PoW block: 15537394

The hash tree root of the Master Accumulator is:

0x8eac399e24480dce3cfe06f4bdecba51c6e5d0c46200e3e8611a0b44a3a69ff9

Bridge

Blocks are produced by Ethereum Execution clients which use a different network to Portal Network nodes. A Bridge node is responsible for taking data from the external network and passing it to the Portal Network.

flowchart LR
    eth[Ethereum Execution node]-->bridge[Portal Network Bridge node]
    bridge-->portal[Portal network node]

This operates as follows:

sequenceDiagram
    Bridge-->>Execution: eth_getBlock
    Execution-->>Bridge: block
    Bridge-->>Portal: block

Currently the bridge functionality exists as a separate executable under portal-bridge.

Archive nodes

A Portal Network node is not an archival node. This page explores the reason for this and some considerations on the topic.

An archive node is one that can know the history at a certain block in the past.

A non-archive node has this information until a block is 128 blocks old. After this point the data is forgotten.

Old state

Archive nodes store old states

What was the balance of token x at block y?
What was in storage slot x at block y?

Old traces

Archive nodes store old traces. This means that they can re-execute old transactions and show everything that the EVM did.

What events were emitted during transaction x?
How much gas did transaction x use?

Requirements

Consider an archive node that is going to trace the 100th transaction in an old block.

The transaction may call a contract, which may in turn call another contract (etc., ). The state of the contracts must be known (balance, nonce, bytecode, storage)
The transaction may reference the hash of a preceding block (up to depth of 256 blocks)
The transaction may modify state that has already been modified in the preceding 99 transactions.

Would an Archive sub-protocol be of use?

Not for sequential data analysis

Archival nodes are great for data science because they allow traversing a large number of sequential blocks and tracking changes over time.

A portal node would not be suited for this activity because it requires sequential blocks rather than possession of data based on the nodes ID. Hence a Portal Node has a disperse subset of content and would need to ask peers for data for sequential blocks. Asking for all sequential blocks would cause an infeasible burden on peers.

Possibly for personal wallet history

A user with access to an index of address appearances (such as the Unchained Index) could make queries about their historical transactions. This could be for a wallet, multisig contract or any contract.

After retrieving the traces for these transactions, they could be used to create a display of activity. E.g., A graph of token balances changing over time, or a log of on-chain activity (trades, loans, transfers, NFT activity).

Could an Archive sub-protocol exist?

It is not impossible. However, the goal of the Portal Network is to provide the function of a non-tracing node. Some considerations are explored below.

Intra-block state

To trace the last transaction in a block, all preceding transaction final states must be known. Hence single request for a transaction trace could result in requiring many transactions in a single block to be obtained. This applies to popular contracts that appear frequently in a block (e.g., exchanges, and popular tokens).

Consequence of a request for a transaction at the end of a block involving popular contracts:

It would be very slow to get a response
It could be used as a denial of service (DoS) attack on the network. For instance, by finding the final transactions in blocks and requesting them from different nodes.

Nested contract calls

A contract could start a chain of nested calls to other contracts. If a node does not have the state of these contracts, it would have to request them. Hence, the time to trace such a transaction would be very slow. Every nested call would take the time that a single Portal Network request takes.

Consequences of a request for a transaction with deeply nested contract calls:

It would be very slow to get a response
It could be used as a denial of service (DoS) attack on the network. For instance, by finding many nested transactions and requesting them from different nodes.

Duplication of data

If Archive was a sub-protocol there may be some data that is required to be duplicated on the History or State sub-protocols. This implies that the sub-protocol is inefficient with respect to disk space but may not be a significant problem.

Medium-sized portal nodes

There is a always a moderate amount of interest in archive nodes, for many parties find historical Ethereum data valuable. As archive nodes require minimum ~2TB of storage, many people choose not to run one.

Perhaps there is a large enough appetite to run a "medium-sized portal archive node", such that many users contribute ~100GB. In this scenario, the DoS attacks are reduced as these medium-sized nodes would cause less amplification of network traffic.

Appetite for lags

If the desire for the results of an archive node are large enough, applications and users could be tolerant of slow lookup times. For example, a wallet connected to a portal archive node could display current wallet state quickly, but under a "history" tab could show: "performing deep search... Estimated time 24 hours". Once the information has been retrieved it could then be stored for fast access.

Contributor guidelines

These guidelines are heavily influenced by the Snake-Charmer Tactical Manual. While the manual is written with a focus on Python projects, there is tons of relevant information in there for how to effectively contribute to open-source projects, and it's recommended that you look through the manual before contributing.

Rust

Trin is written in Rust. This section includes guidelines for Rust-specific patterns and principles.

Comments

Any datatype of significance should have an accompanying comment briefly describing its role and responsibilities. Comments are an extremely valuable tool in open-source projects with many different contributors, and can greatly improve development speed. Explain your assumptions clearly so others don't need to dig through the code.

Rust doc comments are the most best way to comment your code.

Imports

In *.rs files, imports should be split into 3 groups src and separated by a single line. Within a single group, imported items should be sorted alphabetically.
1. std, core and alloc,
2. external crates,
3. self, super and crate imports.
Alphabetize imports in Cargo.toml

use alloc::alloc::Layout;
use core::f32;
use std::sync::Arc;

use broker::database::PooledConnection;
use chrono::Utc;
use juniper::{FieldError, FieldResult};
use uuid::Uuid;

use super::schema::{Context, Payload};
use super::update::convert_publish_payload;
use crate::models::Event;

Logging

All logging should be done with the log library and not println!() statements.
Appropriate log levels (debug, warn, info, etc.) should be used with respect to their content.
Log statements should be declarative, useful, succinct and formatted for readability.

Bad:

Oct 25 23:42:11.079 DEBUG trin_core::portalnet::events: Got discv5 event TalkRequest(TalkRequest { id: RequestId([226, 151, 109, 239, 115, 223, 116, 109]), node_address: NodeAddress { socket_addr: 127.0.0.1:4568, node_id: NodeId { raw: [5, 208, 240, 167, 153, 116, 216, 224, 160, 101, 80, 229, 154, 206, 113, 239, 182, 109, 181, 137, 16, 96, 251, 63, 85, 223, 235, 208, 3, 242, 175, 11] } }, protocol: [115, 116, 97, 116, 101], body: [1, 1, 0, 0, 0, 0, 0, 0, 0, 12, 0, 0, 0, 1, 0, 0, 0, 255, 255, 255, 255, 255, 255, 255, 255, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 36, 0, 0, 0, 0, 0, 0, 0], sender: Some(UnboundedSender { chan: Tx { inner: Chan { tx: Tx { block_tail: 0x55c4fe611290, tail_position: 1 }, semaphore: 0, rx_waker: AtomicWaker, tx_count: 2, rx_fields: "..." } } }) })

Good:

Oct 25 23:43:02.373 DEBUG trin_core::portalnet::overlay: Received Ping(enr_seq=1, radius=18446744073709551615)

Error handling

Handle errors. Naked .unwrap()s aren't allowed, except for in unit tests. Exceptions must be accompanied by a note justifying usage.
- In most cases where an exception can be made (E.g., parsing a static value) .expect() with a relevant message should be used over a naked unwrap.
Write descriptive error messages that give context of the problem that occurred. Error messages should be unique, to aid with debugging.
Meaningful error types should be used in place of Result< _, String>.
- General errors should use the anyhow crate.
- Custom / typed errors should derive from the std::error::Error trait. The thiserror crate provides a useful macro to simplify creating custom error types.

Style

Clone

Minimize the amount of .clone()s used. Cloning can be a useful mechanism, but should be used with discretion. When leaned upon excessively to satisfy the borrow checker it can lead to unintended consequences.

String interpolation

Use interpolated string formatting when possible.

Do format!("words: {var:?}") not format!("words: {:?}", var)

Git

This section covers guidelines and common scenarios encountered with using git and github for Trin development.

Commit messages

Commit Hygiene

We do not have any stringent requirements on how you commit your work, however you should work towards the following with your git habits.

Logical Commits

This means that each commit contains one logical change to the code. For example:

commit A introduces new API
commit B deprecates or removes the old API being replaced.
commit C modifies the configuration for CI.

This approach is sometimes easier to do after all of the code has been written. Once things are complete, you can git reset master to unstage all of the changes you've made, and then re-commit them in small chunks using git add -p.

Commit Messages

We use conventional commits for our commit messages. This means that your commit messages should be of the form:

<type>[optional scope]: <description>

To learn more about conventional commits please check out the conventional commits website.

Examples:

fix: Update metrics strategy to support multiple subnetworks
refactor(light-client): Refactor light-client crate to use ethportal-api consensus types
feat(rpc): Return header to eth_getBlockByHash

One way to test whether you have it right is to complete the following sentence.

If you apply this commit it will ________________.

Submodules

This project uses Git Submodules. If you just cloned the project, be sure to run:

$ git submodule update --init

This page provides a short overview of the most common use cases.

Pulling in Upstream Changes from the Submodule Remote

You want to do this when the remote version of submodule is updated. The simplest way to resolve this is to run:

$ git submodule update --remote --rebase

If you modified your local submodule, you might want to use different flags.

If you run git status, you should see that submodule is updated. Commit and push the changes so others can use the same version.

Pulling Upstream Changes from the Project Remote

If somebody else updated the submodule and you pulled the changes, you have to update your local clone as well. The message "Submodules changed but not updated" will show when running git status. To update local submodule, run:

$ git submodule update --init

Rebasing

You should be using git rebase when there are upstream changes that you need in your branch. You should not use git merge to pull in these changes.

Release notes

We use conventional commits to generate release notes.

Check the commits guide for more information on how to write commit messages.

Pull requests

We are a distributed team. The primary way we communicate about our code is through github via pull requests.

When you start work on something you should have a pull request opened that same day.
Mark unfinished pull requests with the "Work in Progress" label.
Before submitting a pr for review, you should run the following commands locally and make sure they are passing, otherwise CI will raise an error.
- cargo +nightly fmt --all -- --check and cargo clippy --all --all-targets --all-features -- --deny warnings for linting checks
- RUSTFLAGS='-D warnings' cargo test --workspace to run all tests
Pull requests should always be reviewed by another member of the team prior to being merged.
- Obvious exceptions include very small pull requests.
- Less obvious examples include things like time-sensitive fixes.
You should not expect feedback on a pull request that is not passing CI.
- Obvious exceptions include soliciting high-level feedback on your approach.

Large pull requests (above 200-400 lines of code changed) cannot be effectively reviewed. If your pull request exceeds this threshold you should make every effort to divide it into smaller pieces.

You as the person opening the pull request should assign a reviewer.

Code review

Reviewing

Every team member is responsible for reviewing code. The designations :speech_balloon:, :heavy_check_mark:, and :x: should be left by a reviewer as follows:

:speech_balloon: (Comment) should be used when there is not yet an opinion on the overall validity of complete PR, for example:
- comments from a partial review
- comments from a complete review on a Work in Progress PR
- questions or non-specific concerns, where the answer might trigger an expected change before merging
:heavy_check_mark: (Approve) should be used when the reviewer would consider it acceptable for the contributor to merge, after addressing all the comments. For example:
- style nitpick comments
- compliments or highlights of excellent patterns ("addressing" might be in the form of a reply that defines scenarios where the pattern could be used more in the code, or a simple :+1:)
- a specific concern, where multiple reasonable solutions can adequately resolve the concern
- a Work in Progress PR that is far enough along
:x: (Request changes) should be used when the reviewer considers it unacceptable to merge without another review of changes that address the comments. For example:
- a specific concern, without a satisfactory solution in mind
- a specific concern with a satisfactory solution provided, but alternative solutions may be unacceptable
- any concern with significant subtleties

Responding

Contributors should react to reviews as follows:

:x: if any review is marked as "Request changes":
- make changes and/or request clarification
- should not merge until reviewer has reviewed again and changed the status
(none) if there are no reviews, contributor should not merge.
:speech_balloon: if all reviews are comments, then address the comments. Otherwise, treat as if no one has reviewed the PR.
:heavy_check_mark: if at least one review is Approved, contributor should do these things before merging:
- make requested changes
- if any concern is unclear in any way, ask the reviewer for clarification before merging
- solve a concern with suggested, or alternative, solution
- if the reviewer's concern is clearly a misunderstanding, explain and merge. Contributor should be on the lookout for followup clarifications on the closed PR
- if the contributor simply disagrees with the concern, it would be best to communicate with the reviewer before merging
- if the PR is approved as a work-in-progress: consider reducing the scope of the PR to roughly the current state, and merging. (multiple smaller PRs is better than one big one)

It is also recommended to use the emoji responses to signal agreement or that you've seen a comment and will address it rather than replying. This reduces github inbox spam.

Everyone is free to review any pull request.

Fetching a pull request

We often want or need to run code that someone proposes in a PR. Typically this involves adding the remote of the PR author locally and then fetching their branches.

Example:

git remote add someone https://github.com/someone/reponame.git
git fetch someone
git checkout someone/branch-name

With an increasing number of different contributors this workflow becomes tedious. Luckily, there's a little trick that greatly improves the workflow as it lets us pull down any PR without adding another remote.

To do this, we just have to add the following line in the [remote "origin"] section of the .git/config file in our local repository.

fetch = +refs/pull/*/head:refs/remotes/origin/pr/*

Then, checking out a PR locally becomes as easy as:

git fetch origin
git checkout origin/pr/<number>

Replace origin ☝ with the actual name (e.g. upstream) that we use for the remote that we want to fetch PRs from.

Notice that fetching PRs this way is read-only which means that in case we do want to contribute back to the PR (and the author has this enabled), we would still need to add their remote explicitly.

Merging

Once your pull request has been Approved it may be merged at your discretion. In most cases responsibility for merging is left to the person who opened the pull request, however for simple pull requests it is fine for anyone to merge.

If substantive changes are made after the pull request has been marked Approved you should ask for an additional round of review.

Team Rotation - aka "Flamingo" 🦩

Recently, we started a new process. It is evolving, so this doc will evolve with it.

Every Monday, a full-time member of trin rotates into a maintenance position. The person in that role is called the "Flamingo" (truly amazing animals).

Some responsibilities for the Flamingo are:

monitor glados & hive to watch for regressions; investigate & resolve them
answer questions from other Portal teams
run releases & deployments
review PRs that are stuck
add documentation & build tools
fun integrations, like Discord bots

The idea is to set aside normal coding progress for the week, to take care of these important and often overlooked tasks.

Responsibilities

There are some daily tasks, and some transition tasks: Kickoff on Monday & Handoff on Friday.

Unlike your code-heavy weeks, this can be an interruption-heavy week. Although it's often good practice to not get distracted by inbound messages while coding, this week is the opposite: as soon as you see new messages (like on Discord), jump into it.

Kickoff

Have a discussion with the previous Flamingo on any ongoing issues or in-progress tasks. This is crucial to get a quick understanding of the state of the network and any in-progress tasks to resume.

Make sure your status is "online" in Discord. Make sure you're tagged under the trin-flamingo role (ping discord Admin). Put on your favorite pink shirt. Watch a silly flamingo video. Fly.

First

Read through the "Setup" section of the Deployment Instructions and follow the steps to make sure that your PGP and SSH keys are in place and ready for a deployment.

Daily

Checklist

Every day, go down the daily and items for that day of the week. If you think of new daily things to add to the checklist, create a PR.

Daily
- Read Discord, especially for help requests or signs of network issues
- Monitor Glados changes, and correlate with releases (trin, glados, other clients?)
- Monitor portal-hive changes
  - Check the dates, did all test suites run on the previous cycle?
  - For each suite, did the expected number of tests run?
  - Did clients start failing any new tests?
    - If trin failing, create rotation issue to pursue
    - If other client failing, notify the team
- Look for inspiration for Flamingo projects
Monday - kickoff
- Announce that you are on rotation in Discord
- Give yourself the Discord role @trin-flamingo
- Discuss ongoing issues and in-progress tasks with previous Flamingo
- Give weekly summary update in all-Portal call
- Pick day of week for deployment (usually Wednesday or Thursday), discuss in #trin
Wednesday or Thursday - release
- Release and deploy new version of trin
Friday - wrap-up
- Haven't deployed yet? Oops, a bit late. Get it done as early as possible.
- Comment on the checklist to add/update/delete anything?
- Identify the next Flamingo and prepare for handoff

As long as there aren't any major incidents, you should finish the checklist with plenty of time left in your day. See Maintenance Inspiration for what to do for the rest of the day.

Maintenance Inspiration

When you get to the end of your checklist, here are ideas for what to work on next:

Respond to Github Participating Notifications
Review PRs that have been stuck for >24 hours
Find a Flamingo Issue that seems promising, and assign it to yourself (and the project dashboard)
grep code for TODOs. For each one you find:
- write up issue to explain what needs to be done, and a plan
- link to the TODO in the code, and any relevant context
- label issue as Flamingo. If it is not urgent and a good fit, add "Good First Issue"
- Post a PR to remove the TODO from the code, in favor of the issue.
- git grep -iE "(todo)|(fixme)"
Look through all open trin issues
- Close outdated issues, with a short explanation
- If appropriate, add a Flamingo tag
- Sort by "Least Recently Updated" and tackle the most stale issues, until the stale issue bot is active & we've caught up to date with 0 stale issues.
Write new portal-hive tests
- The first step is to make an issue with a list of test ideas, or take over this one
- Add new portal-hive test ideas to the list
- Claim one of the ideas of the list to work on
Respond to all trin repo activity
Add new integration tools (eg~ portal-hive -> Discord bot?)
Scratch your own itch! What do you wish was easier? Now is the time to build it.

Handoff

Look back at your week as Flamingo and summarize it in notes if needed. Prepare for a kickoff discussion with the next Flamingo and update them on your work from previous week.

Especially while we continue to develop the procedure, try to be available the following Monday to help the subsequent Flamingo transition in, and get started.

Think through: what kinds of things do you think should be on the checklist?

Double check the Flamingo schedule and make sure you're available for your next rotation. If not, please switch with somebody asap.

Being the Announcer

The Flamingo is often the first to notice downtime. Whether it's something that affects users or developers, announce the downtime and link to status updates as soon as possible. Also, announce when the downtime is over.

Not Flamingo

If you're not Flamingo currently, there are some situations where it is important to ping the Flamingo in Discord with @flamingo-trin. For example:

You notice the network behaving abnormally
You notice any of our tooling behaving abnormally (github, Circle, glados, etc)
You have a PR that has had no review activity after asking more than 24 hours ago
You are doing anything that affects other people, like:
- planned downtime
- a team-wide configuration change in CircleCI
- deploying a new tool

Naturally, before writing it in chat, look for a pre-existing announcement from the Flamingo. Maybe they already figured it out, which means they have already announced it, and we don't need the duplicate message.

Build Instructions

The following are guides for building Trin on different platforms. Other methods may be used as appropriate.

Mac Os

Clone trin and run.

$ cd ~
$ git clone https://github.com/ethereum/trin.git
$ cd trin
$ cargo run -p trin --release

Linux

Installing Trin on Arch Linux

There's an AUR package for Trin. You can install it with your favorite AUR helper, e.g. yay:

yay -S trin-git

Trin as a service on Ubuntu

These steps are for setting up a Trin node as a service on Ubuntu.

Installation

$ sudo apt install libclang-dev pkg-config build-essential

Install Trin:

Tip: If you intend to submit code changes to trin, first fork the repo and then clone that url.

$ cd ~
$ git clone https://github.com/ethereum/trin.git
$ cd trin
$ cargo build --workspace --release

Now the executable is located in trin/target/release and can be called by systemd. Move that binary to the standard location for binaries:

$ sudo cp -a ~/trin/target/release/trin /usr/local/bin/trin

Tip: If you make changes to these steps, keep a record for future reference.

Make a new user for the Trin service:

$ sudo useradd --no-create-home --shell /bin/false trin

Make a directory for Trin data and give the Trin user permission to access it:

$ sudo mkdir -p /var/lib/trin
$ sudo chown -R trin:trin /var/lib/trin

Check that the binary works:

$ /usr/local/bin/trin --version

Example response:

> Launching trin
> trin 0.0.1

Configuration

Before setting up the service, look at the flags that can be set when starting Trin:

$ /usr/local/bin/trin --help

Some selected flags are described below.

Optional flag for database size

--mb 200. Trin lets you control how much storage the node takes up (e.g., 200MB). The default is 100 megabytes and can be changed.

Optional flag for no connection to external server

--no-stun. A third party server connection is configured by default to assist in testing. This is a Session Traversal Utilities for NAT (STUN) server and may be disabled as a flag. The docs state: "Do not use STUN to determine an external IP. Leaves ENR entry for IP blank. Some users report better connections over VPN."

Optional flags for conflicting nodes

The discovery and JSON-RPC ports may conflict with an existing an Ethereum client or other software on the same machine.

--discovery-port <port>. The default port is 9009. Pick something else if in use.

--web3-http-address <ip_address>:<port>. If an Ethereum execution client is already running, it may be using the default port 8545. The localhost IP address (127.0.0.1) is recommended here.

--web3-transport http. If a new http port is specified using --web3-http-address (as above), the transport must also be changed to http from the default (ipc).

To pick a new port, select a number in the range 1024–49151 and test if it is in use (no response indicates it is ok to use):

$ sudo ss -tulpn | grep ':9008'

Optional flag to enable websocket rpc

--ws. This will allow you to run a websocket on port 8546 --ws --ws-port 3334. A custom websocket port can be configured like this

Create the node service

Create a service to run the Trin node:

$ sudo nano /etc/systemd/system/trin.service

Paste the following, modifying flags as appropriate:

Tip: Note that backslash is needed if starting a flag on a new line.

[Unit]
Description=Trin Portal Network client
After=network.target
Wants=network.target
[Service]
User=trin
Group=trin
Type=simple
Restart=always
RestartSec=5
ExecStart=/usr/local/bin/trin \
    --discovery-port 9008 \
    --web3-http-address 127.0.0.1:8543 \
    --web3-transport http \
    --bootnodes default \
    --mb 200 \
    --no-stun
[Install]
WantedBy=default.target

CTRL-X then CTRL-Y to exit and save.

Tip: Note that we are not using the default discovery (9009) and HTTP (8545) ports. This is done on purpose with the assumption that we will run Trin from the source code as well. See section below.

Add environment variables

The environment variables are going in a different file so they are not accidentally copy-pasted to public places. Create the override.conf file, which will be placed in a new trin.service.d directory beside the trin.service file:

$ sudo systemctl edit trin

Open the file:

$ sudo nano /etc/systemd/system/trin.service.d/override.conf

Tip: The 'info' level of logs is a good starting value.

[Service]
# (optional) Rust log level: <error/warn/info/debug/trace>
Environment="RUST_LOG=info"
# (optional) This flag sets the data directory to the location we created earlier.
Environment="TRIN_DATA_PATH=/var/lib/trin"

Configure firewall

Ensure that the discovery port (custom or default 9009) is not blocked by the firewall:

$ sudo ufw allow 9009

Check the configuration:

$ sudo ufw status numbered

Tip: use sudo ufw delete <number> to remove a particular rule.

Start the service

Start the Trin node service and enable it to start on reboot:

$ sudo systemctl daemon-reload
$ sudo systemctl start trin
$ sudo systemctl status trin
$ sudo systemctl enable trin

Follow Trin's logs:

$ sudo journalctl -fu trin

CTRL-C to to exit.

Logs can be searched for an "exact phrase":

$ grep "trin" /var/log/syslog | grep "exact phrase"

To stop Trin and disable it from starting on reboot:

$ sudo systemctl stop trin
$ sudo systemctl disable trin

Keeping service up to date

To get upstream updates, sync your fork with upstream on Github. To move any changes from the codebase to the service, rebuild and move the binary as before:

$ git pull
$ cd trin
$ cargo build --workspace --release
$ sudo systemctl stop trin
$ sudo cp -a ~/trin/target/release/trin /usr/local/bin/trin

Restart the service to use the new binary:

$ sudo systemctl daemon-reload
$ sudo systemctl start trin

Trin from the source

See getting started notes for more tips including setting environment variables during testing.

Tip: If Trin service is using non-default ports (as suggested earlier), you don't have to set ports now. Either way, make sure ports are not already in use.

$ cargo test --workspace
$ cargo run -- --discovery-port 9009 \
    --web3-http-address 127.0.0.1:8545 \
    --web3-transport http \
    --bootnodes default \
    --mb 200 \
    --no-stun

Raspberry Pi

Not yet attempted, but experiments are encouraged.

Windows

These are instructions for Native and cross compiling Windows builds

Native compilation of Windows

If you don't already have Rust install it

$ winget install Rustlang.Rustup

Install clang/llvm as it is required to compile c-kzg

$ winget install LLVM.LLVM

Add Rust's msvc target

$ rustup target add x86_64-pc-windows-msvc

Install target toolchain

$ rustup toolchain install stable-x86_64-pc-windows-msvc

Build Trin

$ cargo build -p trin

Cross-compilation for Ubuntu compiling to Windows

This is assuming you already have rust installed on Linux

Install required dependencies

$ sudo apt update
$ sudo apt upgrade
$ sudo apt install git clang g++-mingw-w64-x86-64-posix

Clone trin and build.

$ git clone https://github.com/ethereum/trin.git
$ cd trin
$ rustup target add x86_64-pc-windows-gnu
$ rustup toolchain install stable-x86_64-pc-windows-gnu
$ cargo build -p trin --target x86_64-pc-windows-gnu

Releases

This section covers the process of making & deploying a Trin release.

Release checklist

First

Ensure that the CI for the latest commit to master is passing. This ensures that trin itself is working, and that the latest docker image is working and published.

Communicate

Announce in #trin chat the upcoming release. Aim for a day or more notice, but announcing a few minutes before releasing is still better than not saying anything.

Choosing a version

Make sure that version follows semver rules e.g (0.2.0-alpha.3).

Since trin is now stable, but v0, breaking changes are a minor version bump, and all other changes are a patch version bump. Updating to v1 would require a group discussion and decision.

Release dependencies

For now, that's just ethportal-api. Manually bump the version in the Cargo.toml, and run cargo update to update the workspace lockfile. Commit and merge these changes to trin. Then run:

cd ethportal-api
cargo publish --no-verify

We would like to get rid of the no-verify ASAP, but for now Cargo.lock is causing issues that we have no other workaround for. cargo publish generates a new lock file, and then complains that the lockfile is new. See this StackOverflow post.

Release Trin

We use automated github release workflow to create a new release. This will create a new release draft with the new tag and will build all the binaries and attach them to the release.

Checkout and rebase local master to upstream

git checkout master
git pull --ff-only upstream master

Create a new git tag with the chosen version, for example:

git tag v0.1.0-alpha.15

Push the tag to the upstream repository:

git push upstream v0.1.0-alpha.15

Wait for the github actions release job to finish. It will create automatically a draft release with all precompiled binaries included. This should take 15-20 min to complete.
Find the draft release generated by the github bot in releases and edit the template by completing and deleting all checklists. Write a short summary if available. Add any clarifying information that's helpful about the release.
Scroll to the bottom, check the Set as a pre-release box and click Publish release.

Build Instructions

Deploy

Push these changes out to the nodes we run in the network. See next page for details.

Communicate

Notify in Discord chat about the new release being complete.

As trin stabilizes, more notifications will be necessary (twitter, blog post, etc). Though we probably want to do at least a small network deployment before publicizing.

Deploy trin to network

First time Setup

Get access to cluster repo (add person to @trin-deployments)
git clone the cluster repo: https://github.com/ethereum/cluster.git

Install dependencies within cluster virtualenv:

cd cluster
python3 -m venv venv
. venv/bin/activate
pip install ansible
pip install docker
sudo apt install ansible-core

On mac you can do brew install ansible instead of apt.

Install keybase
Publish your pgp public key with keybase, using: keybase pgp select --import
- This fails if you don't have a pgp key yet. If so, create one with gpg --generate-key
Install sops
Contact @paulj, get public pgp key into cluster repo
Contact @paulj, get public ssh key onto cluster nodes
Make sure your pgp key is working by running: sops portal-network/trin/ansible/inventories/dev/group_vars/secrets.sops.yml
Log in to Docker with: docker login
Ask Nick to be added as collaborator on Docker repo
Needed for rebooting nodes
- Install doctl
- Contact @paulj to get doctl API key
- Make sure API key works by running: doctl auth init

Each Deployment

Prepare

Generally we want to cut a new release before deployment, see previous page for instructions.
Announce in Discord #trin that you're about to run the deployment
Make sure to schedule plenty of time to react to deployment issues

Update Docker images

Docker images are how Ansible moves the binaries to the nodes. Update the Docker tags with:

docker pull portalnetwork/trin:latest
docker pull portalnetwork/trin:latest-bridge
docker image tag portalnetwork/trin:latest portalnetwork/trin:testnet
docker image tag portalnetwork/trin:latest-bridge portalnetwork/trin:bridge
docker push portalnetwork/trin:testnet
docker push portalnetwork/trin:bridge

This step directs Ansible to use the current master version of trin. Read about the tags to understand more.

Run ansible

Check monitoring tools to understand network health, and compare against post-deployment, eg~
- Glados
- Grafana
Activate the virtual environment in the cluster repo: . venv/bin/activate
Make sure you've pulled the latest master branch of the deployment scripts, to include any recent changes: git pull origin master
Go into the Portal section of Ansible: cd portal-network/trin/ansible/
Run the deployment:
- Trin nodes:
  - ansible-playbook playbook.yml --tags trin
- State network nodes (check with the team if there is a reason not to update them):
  - Recently, we don't regularly deploy state bridge nodes (because they run for a long time and we don't want to restart them). To deploy all other state nodes, use following command:
    - ansible-playbook playbook.yml --tags state-network --limit state_stun,state_bootnode,state_regular
  - To deploy to all state network nodes:
    - ansible-playbook playbook.yml --tags state-network
Run Glados deployment: updates glados + portal client (currently configured as trin, but this could change)
- cd ../../glados/ansible
- ansible-playbook playbook.yml --tags glados
if you experience "couldn't resolve module/action 'community.docker.docker_compose_v2'" error, you might need to re-install the community.docker collection:
- ansible-galaxy collection install community.docker --force
Wait for completion
Launch a fresh trin node, check it against the bootnodes
ssh into random nodes, one of each kind, to check the logs:
- find an IP address
- node types
  - bootnode: trin-*-1
  - bridge node: trin-*-2
  - backfill node: trin-*-3
  - regular nodes: all remaining ips
- ssh ubuntu@$IP_ADDR
- check logs, ignoring DEBUG: sudo docker logs trin -n 1000 | grep -v DEBUG
Check monitoring tools to see if network health is the same or better as before deployment. Glados might lag for 10-15 minutes, so keep checking back.

Communicate

Notify in Discord chat about the network nodes being updated.

Update these docs

Immediately after a release is the best time to improve these docs:

add a line of example code
fix a typo
add a warning about a common mistake
etc.

For more about generally working with mdbook see the guide to Contribute to the book.

Celebrate

Another successful release! 🎉

FAQ

What do the Docker tags mean?

latest: This image with trin is built on every push to master
latest-bridge: This image with portal-bridge is built on every push to master
angelfood: This tag is used by Ansible to load trin onto the nodes we host
bridge: This tag is used by Ansible to load portal-bridge onto the nodes we host

Note that building the Docker image on git's master takes some time. If you merge to master and immediately pull the latest Docker image, you won't be getting the build of that latest commit. You have to wait for the Docker build to complete. You should be able to see on github when the Docker build has finished.

Why can't I decrypt the SOPS file?

You might see this when running ansible, or the sops check:

Failed to get the data key required to decrypt the SOPS file.

Group 0: FAILED
  32F602D86B61912D7367607E6D285A1D2652C16B: FAILED
    - | could not decrypt data key with PGP key:
      | github.com/ProtonMail/go-crypto/openpgp error: Could not
      | load secring: open ~/.gnupg/secring.gpg: no such
      | file or directory; GPG binary error: exit status 2

  81550B6FE9BC474CA9FA7347E07CEA3BE5D5AB60: FAILED
    - | could not decrypt data key with PGP key:
      | github.com/ProtonMail/go-crypto/openpgp error: Could not
      | load secring: open ~/.gnupg/secring.gpg: no such
      | file or directory; GPG binary error: exit status 2

Recovery failed because no master key was able to decrypt the file. In
order for SOPS to recover the file, at least one key has to be successful,
but none were.

It means your key isn't working. Check with @paulj.

If using gpg and decryption problems persist, see this potential fix.

What do I do if Ansible says a node is unreachable?

You might see this during a deployment:

fatal: [trin-ams3-1]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: ssh: connect to host XXX.XXX.XXX.XXX port XX: Connection timed out", "unreachable": true}

Retry once more. If it times out again, run reboot script (check First time Setup chapter for setup):

./reboot_node.sh <host name1>,<host name2>,...,<host nameN>

What if everything breaks and I need to rollback the deployment?

If you observe things breaking or (significantly) degraded network performance after a deployment, you might want to rollback the changes to a previously working version until the breaking change can be identified and fixed. Keep in mind that you might want to rollback just the bridge nodes, or the backfill nodes, as opposed to every node on the network.

Go to the commit from the previously released version tag. Click into the CI workflows for that commit and look for the docker-publish or docker-publish-bridge flow, depending on what images you want to rollback.
In the logs for these flows, find the sha256 digest from the Publish docker image to Docker Hub step.
Pull this specific image locally, using docker pull portalnetwork/trin@sha256:<HASH>
Retag the target image to this version, for example, if you want to re-deploy the bridges, do: docker image tag portalnetwork/trin@sha256:6dc0577a2121b711ae0e43cd387df54c8f69c8671abafb9f83df23ae750b9f14 portalnetwork/trin:bridge
Push the newly tagged bridge image to Docker Hub. eg. docker push portalnetwork/trin:bridge
Re-run the ansible script, which will use the newly updated image. Use the --limit cli flag if you only want to redeploy a subset of nodes. eg: ansible-playbook playbook.yml --tags trin --limit backfill_nodes.
Verify that the network is back to regular operation.

Tests

Testing is essential to the production of software with minimal flaws. The default should always be writing tests for the code you produce.

Testing also introduces overhead into our workflow. If a test suite takes a long time to run, it slows down our iteration cycle. This means finding a pragmatic balance between thorough testing, and the speed of our test suite, as well as always iterating on our testing infrastructure.

Unit test names should unambiguously identify the functionality being tested. Omit any "test" prefix from the name to avoid redundancy.

Contribute to trin Book

Using the book

The book can be built and served locally.

Installing book tools

The first time you work on the book, you need to install mdbook and the mermaid tool to generate diagrams:

cargo install mdbook mdbook-mermaid
cd book/
mdbook-mermaid install
cd ..

This will create mermaid.min.js and mermaid-init.js files.

Running the book server

Then run the book from the book crate:

cd book/
mdbook serve --open

Or, from the project root:

mdbook serve --open ./book

Adding new pages

Add a new entry to ./book/src/SUMMARY.md. Follow the style there, which follows strict formatting. There are two kinds of additions:

New single section
- Tab to the appropriate depth
- Add a [Section name](section_name.md)
New nested section
- Tab to the appropriate depth
- Add a [Section name](section_name/README.md)
  - Add [Subsection one](section_name/subsection_one.md)
  - Add [Subsection two](section_name/subsection_two.md)

Don't create these pages, the ./book/src/SUMMARY.md file is parsed and any missing pages are generated when mdbook serve is run. Content can then be added to the generated pages.

Then run serve:

mdbook serve --open

Test

To test the code within the book run:

mdbook test

Diagrams

Diagrams can be added using mermaid annotations on a normal code block:

    ```mermaid
    graph TD;
        A-->B;
        A-->C;
        B-->D;
        C-->D;
    ```

The above be converted to the following during book-building:

graph TD;
    A-->B;
    A-->C;
    B-->D;
    C-->D;

Crate documentation location

Workspace crates are published to crates.io and include the README.md in the root of the crate. This is valuable to have when using the crates outside of the context of Trin E.g., ethportal-api.

Any documentation of workspace crates in the book should therefore be limited to explaining how the crate interacts with the other workspaces in the context of Trin. Rather than moving the workspace README.md's to the book.