# Monitoring your validator with Grafana and Prometheus

Prometheus is a monitoring platform that collects metrics from monitored targets by scraping metrics HTTP endpoints on these targets.

Grafana is a dashboard used to visualize the collected data.

[Official documentation is available here.](https://prometheus.io/docs/introduction/overview/) :book:

### 1. Install Prometheus and Node Exporter

```bash
sudo apt-get install -y prometheus prometheus-node-exporter
```

### 2. Install Grafana

```bash
sudo apt-get install -y apt-transport-https
sudo apt-get install -y software-properties-common wget
sudo wget -q -O /usr/share/keyrings/grafana.key https://apt.grafana.com/gpg.key
```

```bash
echo "deb [signed-by=/usr/share/keyrings/grafana.key] https://apt.grafana.com stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list
sudo apt-get update && sudo apt-get install -y grafana
```

### 3. Enable services so they start automatically

```bash
sudo systemctl enable grafana-server prometheus prometheus-node-exporter
```

### 4. Create the **prometheus.yml** config file

Remove the default **prometheus.yml** configuration file and edit a new one.

```bash
sudo rm /etc/prometheus/prometheus.yml
sudo nano /etc/prometheus/prometheus.yml
```

Choose the tab for your consensus client. Paste the following configuration into the file.

{% tabs %}
{% tab title="Lighthouse" %}

```bash
global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
    monitor: 'codelab-monitor'

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
   - job_name: 'node_exporter'
     static_configs:
       - targets: ['localhost:9100']
   - job_name: 'lighthouse'
     metrics_path: /metrics
     static_configs:
       - targets: ['localhost:8008']
   - job_name: 'lighthouse_validator'
     metrics_path: /metrics
     static_configs:
       - targets: ['localhost:8009']
```

{% endtab %}

{% tab title="Nimbus" %}

```bash
global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
    monitor: 'codelab-monitor'

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
   - job_name: 'node_exporter'
     static_configs:
       - targets: ['localhost:9100']
   - job_name: 'Nimbus'
     metrics_path: /metrics
     static_configs:
       - targets: ['localhost:8008']
   - job_name: 'Nimbus_Validator'
     metrics_path: /metrics
     static_configs:
       - targets: ['localhost:8009']
```

{% endtab %}

{% tab title="Teku" %}

```bash
global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
    monitor: 'codelab-monitor'

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
   - job_name: 'node_exporter'
     static_configs:
       - targets: ['localhost:9100']
   - job_name: 'Teku'
     metrics_path: /metrics
     static_configs:
       - targets: ['localhost:8008']
   - job_name: 'Teku_Validator'
     metrics_path: /metrics
     static_configs:
       - targets: ['localhost:8009']
```

{% endtab %}

{% tab title="Prysm" %}

```bash
global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
    monitor: 'codelab-monitor'

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
   - job_name: 'node_exporter'
     static_configs:
       - targets: ['localhost:9100']
   - job_name: 'validator'
     static_configs:
       - targets: ['localhost:8009']
   - job_name: 'Prysm'
     static_configs:
       - targets: ['localhost:8008']
```

{% endtab %}

{% tab title="Lodestar" %}

<pre class="language-bash"><code class="lang-bash"><strong>global:
</strong>  scrape_interval:     15s # By default, scrape targets every 15 seconds.

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
    monitor: 'codelab-monitor'

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
   - job_name: 'node_exporter'
     static_configs:
       - targets: ['localhost:9100']
   - job_name: 'beacon'
     metrics_path: /metrics    
     static_configs:
       - targets: ['localhost:8008']
   - job_name: 'validator'
     metrics_path: /metrics    
     static_configs:
       - targets: ['localhost:8009']
</code></pre>

{% endtab %}
{% endtabs %}

### 5. Setup prometheus for your execution client

Append the applicable job snippet for your execution client to the end of **prometheus.yml**.

{% hint style="warning" %}
**Spacing matters**. Ensure all `job_name` snippets are in alignment.
{% endhint %}

{% tabs %}
{% tab title="Geth" %}

```bash
   - job_name: 'geth'
     scrape_interval: 15s
     scrape_timeout: 10s
     metrics_path: /debug/metrics/prometheus
     scheme: http
     static_configs:
       - targets: ['localhost:6060']
```

{% endtab %}

{% tab title="Besu" %}

```bash
   - job_name: 'besu'
     scrape_interval: 15s
     scrape_timeout: 10s
     metrics_path: /metrics
     scheme: http
     static_configs:
     - targets:
       - localhost:6060
```

{% endtab %}

{% tab title="Nethermind" %}

```bash
   - job_name: 'nethermind'
     static_configs:
       - targets: ['localhost:6060']
```

{% endtab %}

{% tab title="Erigon" %}

```bash
   - job_name: 'erigon'
     scrape_interval: 10s
     scrape_timeout: 3s
     metrics_path: /debug/metrics/prometheus
     scheme: http
     static_configs:
       - targets: ['localhost:6060']
```

{% endtab %}

{% tab title="Reth" %}

```bash
   - job_name: 'reth'
     metrics_path: "/"
     scrape_interval: 10s
     static_configs:
       - targets: ['localhost:6060']
```

{% endtab %}
{% endtabs %}

Here's an example of a Lighthouse-Nethermind config:

```bash
global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
    monitor: 'codelab-monitor'

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
   - job_name: 'node_exporter'
     static_configs:
       - targets: ['localhost:9100']
   - job_name: 'Lighthouse'
     metrics_path: /metrics
     static_configs:
       - targets: ['localhost:5054']
   - job_name: 'validators'
     metrics_path: /metrics
     static_configs:
       - targets: ['localhost:5064']
   - job_name: 'nethermind'
     static_configs:
       - targets: ['localhost:6060']
```

To exit and save, press `Ctrl` + `X`, then `Y`, then `Enter`.

Update file permissions.

```bash
sudo chmod 644 /etc/prometheus/prometheus.yml
```

Restart the services.

```bash
sudo systemctl restart grafana-server prometheus prometheus-node-exporter
```

Verify that the services are running.

```bash
sudo systemctl status grafana-server prometheus prometheus-node-exporter
```

### 6. Create a SSH Tunnel to Grafana

Each time you want to access Grafana, create a SSH tunnel with port 3000 forwarded.

{% tabs %}
{% tab title="Linux or MacOS" %}
Example of how to create a SSH tunnel in Linux or MacOS:

```bash
ssh -N -v <user>@<staking.node.ip.address> -L 3000:localhost:3000

#Full Example
ssh -N -v ethereum@192.168.1.69 -L 3000:localhost:3000
```

{% endtab %}

{% tab title="Windows" %}
Example of how to create a SSH tunnel in Windows with [Putty](https://putty.org/):

Navigate to Connection > SSH > Tunnels > Enter Source Port `3000` > Enter Destination `localhost:3000` > Click Add

![](https://3796256119-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FG9WtRptWgpOtES8UPFKA%2Fuploads%2Fgit-blob-fcc650562c4d69f1dfa5f1540b3b89d38077814f%2Fimage.png?alt=media)

Now save your configuration. Navigate to Session > Enter a session name > Save

Click Open to open a connection
{% endtab %}
{% endtabs %}

Now you can access Grafana on your local machine by pointing a web browser to [http://localhost:3000](http://localhost:3000/)

### 7. Setup Grafana Dashboards

1. Open <http://localhost:3000>
2. Login with **admin** / **admin**
3. Change password
4. Click the **configuration gear** icon, then **Add data Source**
5. Select **Prometheus**
6. Set **Name** to **"Prometheus**"
7. Set **URL** to <http://localhost:9090>
8. Click **Save & Test**
9. **Download and save** your consensus client's json file. More json dashboard options available below. \[ [Lighthouse](https://raw.githubusercontent.com/Yoldark34/lighthouse-staking-dashboard/main/Yoldark_ETH_staking_dashboard.json) | [Teku ](https://grafana.com/api/dashboards/13457/revisions/2/download)| [Nimbus ](https://raw.githubusercontent.com/status-im/nimbus-eth2/master/grafana/beacon_nodes_Grafana_dashboard.json)| [Prysm ](https://raw.githubusercontent.com/GuillaumeMiralles/prysm-grafana-dashboard/master/less_10_validators.json)| [Prysm > 10 Validators](https://raw.githubusercontent.com/GuillaumeMiralles/prysm-grafana-dashboard/master/more_10_validators.json) | [Lodestar](https://raw.githubusercontent.com/ChainSafe/lodestar/unstable/dashboards/lodestar_summary.json) ]
10. **Download and save** your execution client's json file \[ [Geth](https://github.com/ethereum/go-ethereum/files/14211070/Geth-Cancun-Prometheus.json) | [Besu ](https://grafana.com/api/dashboards/10273/revisions/5/download)| [Nethermind](https://raw.githubusercontent.com/NethermindEth/metrics-infrastructure/master/grafana/provisioning/dashboards/nethermind.json) | [Erigon](https://raw.githubusercontent.com/ledgerwatch/erigon/devel/cmd/prometheus/dashboards/erigon.json) | [Reth](https://raw.githubusercontent.com/paradigmxyz/reth/main/etc/grafana/dashboards/overview.json) ]
11. **Download and save** a [node-exporter dashboard](https://grafana.com/api/dashboards/11074/revisions/9/download) for general system monitoring
12. Click **Create +** icon > **Import**
13. Add the consensus client dashboard via **Upload JSON file**
14. If needed, select Prometheus as **Data Source**.
15. Click the **Import** button.
16. Repeat steps 12-15 for the execution client dashboard.
17. Repeat steps 12-15 for the node-exporter dashboard.

{% hint style="warning" %}
:fire: **Troubleshooting common Grafana issues**

**Symptom 1**: Your dashboard is missing some data\_.\_

**Solution 1***:* Ensure that the execution or consensus client has enabled the appropriate metrics flag.

```bash
cat /etc/systemd/system/execution.service
cat /etc/systemd/system/consensus.service
```

* **Geth**: --http --metrics --pprof
* **Besu**: --metrics-enabled=true
* **Nethermind**: --Metrics.Enabled true
* **Erigon**: --metrics
* **Reth**: --metrics 127.0.0.1:9001
* **Lighthouse**: --validator-monitor-auto
* **Nimbus**: --metrics --metrics-port=8008
* **Teku**: --metrics-enabled=true --metrics-port=8008
* **Lodestar**: --metrics true

**Symptom 2**: Don't want to use SSH tunnels and you want to expose port 3000 to access Grafana, but understand the security concerns.

**Solution 2**: Open port 3000 in your ufw firewall. Access grafana at http\://\<node ipaddress>:3000

```
sudo ufw allow 3000
```

{% endhint %}

#### Example of Grafana Dashboards for each consensus client.

{% tabs %}
{% tab title="Lighthouse" %}
![Beacon Chain dashboard by sigp](https://3796256119-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FG9WtRptWgpOtES8UPFKA%2Fuploads%2Fgit-blob-f09f9bc14d99b7243f627c5e6005d06049cf6874%2Flhm.png?alt=media)

![Validator Client dashboard by sigp](https://3796256119-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FG9WtRptWgpOtES8UPFKA%2Fuploads%2Fgit-blob-92f4787777d721384cb3211f94aa022bb2297af2%2Flighthouse-validator.png?alt=media)

Beacon Chain JSON Download link: <https://raw.githubusercontent.com/sigp/lighthouse-metrics/master/dashboards/Summary.json>

Validator Client JSON download link: <https://raw.githubusercontent.com/sigp/lighthouse-metrics/master/dashboards/ValidatorClient.json>

Credits: <https://github.com/sigp/lighthouse-metrics/>

![LH dashboard by Yoldark](https://3796256119-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FG9WtRptWgpOtES8UPFKA%2Fuploads%2Fgit-blob-3ee0a9af2def97bf0e0033a0e6662352a90ebb63%2Fyoldark-lighthouse.png?alt=media)

JSON Download link: <https://raw.githubusercontent.com/Yoldark34/lighthouse-staking-dashboard/main/Yoldark_ETH_staking_dashboard.json>

Credits: <https://github.com/Yoldark34/lighthouse-staking-dashboard>
{% endtab %}

{% tab title="Nimbus" %}
![Dashboard by status-im](https://3796256119-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FG9WtRptWgpOtES8UPFKA%2Fuploads%2Fgit-blob-96b24f468d921ec86a04f1ed3305fec75e4719e7%2Fnim_dashboard.png?alt=media)

Credits: <https://github.com/status-im/nimbus-eth2/>

![Nimbus dashboard by metanull-operator](https://github.com/nodebridgeafric/nodebridgeafrica/blob/main/.gitbook/assets/eth2-grafana-dashboard-prysm.jpg)

JSON download link:

Credits: <https://github.com/metanull-operator/eth2-grafana/>
{% endtab %}

{% tab title="Teku" %}
![Teku by PegaSys Engineering](https://3796256119-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FG9WtRptWgpOtES8UPFKA%2Fuploads%2Fgit-blob-be6b5b3da4c7c5d50ee97b165c98f74526f3995c%2Fteku.dash.png?alt=media)

Credits: <https://grafana.com/grafana/dashboards/13457>
{% endtab %}

{% tab title="Prysm" %}
![Prysm dashboard by GuillaumeMiralles](https://3796256119-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FG9WtRptWgpOtES8UPFKA%2Fuploads%2Fgit-blob-505d16cba4fc042885f96c49652e5ae7e59f46a7%2Fprysm_dash.png?alt=media)

Credits: <https://github.com/GuillaumeMiralles/prysm-grafana-dashboard>

![Prysm dashboard by metanull-operator](https://3796256119-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FG9WtRptWgpOtES8UPFKA%2Fuploads%2Fgit-blob-f709d19b3bc3b2ccb987197c50fbd4b7a6a0df5a%2Feth2-grafana-dashboard-prysm.jpg?alt=media)

JSON download link: <https://github.com/metanull-operator/eth2-grafana/raw/master/eth2-grafana-dashboard-single-source.json>

Credits: <https://github.com/metanull-operator/eth2-grafana/>
{% endtab %}

{% tab title="Lodestar" %}

<figure><img src="https://3796256119-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FG9WtRptWgpOtES8UPFKA%2Fuploads%2Fgit-blob-966d310f5c29063a4138cf392059fd7baa66003a%2Flode-dash.png?alt=media" alt=""><figcaption></figcaption></figure>

Credits: <https://raw.githubusercontent.com/ChainSafe/lodestar/unstable/dashboards/lodestar_summary.json>
{% endtab %}
{% endtabs %}

#### Example of Grafana Dashboards for each execution client.

{% tabs %}
{% tab title="Geth" %}
![Dashboard by karalabe](https://3796256119-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FG9WtRptWgpOtES8UPFKA%2Fuploads%2Fgit-blob-76a2bd1c1e66219b7d79d3a8719c34695e0a7ad5%2Fgeth-dash.png?alt=media)

Credits: <https://gist.github.com/karalabe/e7ca79abdec54755ceae09c08bd090cd>

Blob enabled dashboard: <https://github.com/ethereum/go-ethereum/files/14211070/Geth-Cancun-Prometheus.json>
{% endtab %}

{% tab title="Besu" %}
![](https://3796256119-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FG9WtRptWgpOtES8UPFKA%2Fuploads%2Fgit-blob-add044c5af11071b4ce61b8f494194fdf19a4520%2Fbesu-dash.png?alt=media)

Credits: <https://grafana.com/dashboards/10273>
{% endtab %}

{% tab title="Nethermind" %}
![](https://3796256119-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FG9WtRptWgpOtES8UPFKA%2Fuploads%2Fgit-blob-6b21f222be74dc666e09aba196ba5837334fb0c1%2Fnethermind-dash.png?alt=media)

Credits: <https://github.com/NethermindEth/metrics-infrastructure>
{% endtab %}

{% tab title="Erigon" %}
![](https://3796256119-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FG9WtRptWgpOtES8UPFKA%2Fuploads%2Fgit-blob-7a96f8046b45a4287e09a76b44d4fd6cfcfe547d%2Ferigon-grafana.png?alt=media)

Credits: <https://github.com/ledgerwatch/erigon/tree/devel/cmd/prometheus/dashboards>
{% endtab %}

{% tab title="Reth" %}

<figure><img src="https://3796256119-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FG9WtRptWgpOtES8UPFKA%2Fuploads%2Fgit-blob-642b14910f619eb6382df4a7620869ee53e8aa0a%2Freth.png?alt=media" alt=""><figcaption></figcaption></figure>

Credits: <https://github.com/paradigmxyz/reth/blob/main/etc/grafana/dashboards/overview.json>
{% endtab %}
{% endtabs %}

#### Example of Node-Exporter Dashboard

{% tabs %}
{% tab title="Node-Exporter Dashboard by starsliao" %}
**General system monitoring**

Includes: CPU, memory, disk IO, network, temperature and other monitoring metrics。

![](https://3796256119-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FG9WtRptWgpOtES8UPFKA%2Fuploads%2Fgit-blob-8cea65eee2d2cf3423bf63dab2145c83c08c874d%2Fgrafana%20\(1\).png?alt=media)

![](https://3796256119-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FG9WtRptWgpOtES8UPFKA%2Fuploads%2Fgit-blob-6615e99f37b81599f5ea7f71118e925a0a2ac245%2Fnode-exporter2.png?alt=media)

Credits: [starsliao](https://grafana.com/grafana/dashboards/11074)
{% endtab %}
{% endtabs %}

### 8. Setup Alert Notifications

{% hint style="info" %}
Setup alerts to get notified if your validators go offline.
{% endhint %}

Get notified of problems with your validators. Choose between email, telegram, discord or slack.

<details>

<summary>Option 1: Email Notifications</summary>

1. Visit [https://beaconcha.in/](https://beaconcha.in)
2. Sign up for an account
3. Verify your **email**
4. Search for your **validator's public address**
5. Add validators to your watchlist by clicking the **bookmark symbol**.

</details>

<details>

<summary>Option 2: Telegram Notifications</summary>

1. On the menu of Grafana, select **Notification channels** under the bell icon.
2. Click on **Add channel**.
3. Give the notification channel a **name**.
4. Select **Telegram** from the Type list.
5. To complete the **Telegram API settings**, a **Telegram channel** and \*\*bot \*\*are required. For instructions on setting up a bot with `@Botfather`, see [this section](https://core.telegram.org/bots#6-botfather) of the Telegram documentation. You need to create a BOT API token.
6. Create a new telegram group.
7. Invite the bot to your new group.
8. Type at least 1 message into the group to initialize it.
9. Visit [`https://api.telegram.org/botXXX:YYY/getUpdates`](https://api.telegram.org/botXXX:YYY/getUpdates) where `XXX:YYY` is your BOT API Token.
10. In the JSON response, find and copy the **Chat ID**. Find it between \*\*chat \*\*and **title**. *Example of Chat ID*: `-1123123123`

    ```
    "chat":{"id":-123123123,"title":
    ```
11. Paste the **Chat ID** into the corresponding field in **Grafana**.
12. **Save and test** the notification channel for your alerts.
13. Now you can create custom alerts from your dashboards. [Visit here to learn how to create alerts.](https://grafana.com/docs/grafana/latest/alerting/create-alerts/)

</details>

<details>

<summary>Option 3: Discord Notifications</summary>

1. On the menu of Grafana, select **Notification channels** under the bell icon.
2. Click on **Add channel**.
3. Add a **name** to the notification channel.
4. Select **Discord** from the Type list.
5. To complete the set up, a Discord server (and a text channel available) as well as a Webhook URL are required. For instructions on setting up a Discord's Webhooks, see [this section](https://support.discord.com/hc/en-us/articles/228383668-Intro-to-Webhooks) of their documentation.
6. Enter the Webhook **URL** in the Discord notification settings panel.
7. Click **Send Test**, which will push a confirmation message to the Discord channel.

</details>

<details>

<summary>Option 4: Slack Notifications</summary>

1. On the menu of Grafana, select **Notification channels** under the bell icon.
2. Click on **Add channel**.
3. Add a **name** to the notification channel.
4. Select **Slack** from the Type list.
5. For instructions on setting up a Slack's Incoming Webhooks, see [this section](https://api.slack.com/messaging/webhooks) of their documentation.
6. Enter the Slack Incoming Webhook URL in the **URL** field.
7. Click **Send Test**, which will push a confirmation message to the Slack channel.

</details>