From 7af0b76f50964b7a47c0078ee71c04bbc0181f19 Mon Sep 17 00:00:00 2001 From: Paradigm AI Date: Thu, 16 Apr 2026 13:42:20 +0000 Subject: [PATCH 01/54] docs: restructure validator operator docs based on feedback Amp-Thread-ID: https://ampcode.com/threads/T-019d9682-7698-7147-8112-2da300236ab0 Co-authored-by: Amp --- src/pages/guide/node/operate-validator.mdx | 227 ++++++++++++++------- src/pages/guide/node/validator.mdx | 63 ++++-- 2 files changed, 207 insertions(+), 83 deletions(-) diff --git a/src/pages/guide/node/operate-validator.mdx b/src/pages/guide/node/operate-validator.mdx index 0dc4bccf..2724982b 100644 --- a/src/pages/guide/node/operate-validator.mdx +++ b/src/pages/guide/node/operate-validator.mdx @@ -7,20 +7,63 @@ description: Day-to-day operations for Tempo validators. Node lifecycle, monitor This guide covers day-to-day operations for running a Tempo validator in production. +:::danger +**You cannot reset a validator's data and continue with the same identity.** Doing so risks inconsistent voting, which can cause irrecoverable network safety failures. If you need to reset your validator's data, you must rotate to a new validator identity — see [Signing key rotation](#signing-key-rotation). +::: + - **[Validator Configuration](/guide/node/validator-config-v2)** — self-service key rotation, IP updates, and ownership transfers +## Validator states + +Your validator moves through different states during operation. The transitions will happen only once on your validator's creation. Every state transition will happen on epoch boundaries in the happy case. + +```mermaid {style: {maxWidth: '480px', margin: '0 auto'}} +flowchart TD + A[Inactive] -->|epoch ends| B["Syncer
The validator starts syncing"] + B -->|+1 epoch| C["Syncer and Player"] + C -->|"+1 epoch, receives share"| D["Syncer
Player
Dealer"] + D -->|"once fully synced"| E["Voter ↔ Proposer"] + + click A "#not-a-participant-e" + click B "#syncer-epoch-e1" + click C "#player-epoch-e2" + click D "#dealer-epoch-e3" + click E "#dealer-epoch-e3" +``` + +Currently, on mainnet and testnet, the epoch length is around 3 hours, which means that your validator will transition through these states approximately every 3 hours. If the epoch length ever changes, the transition times will also change. + +#### Not a participant (E) + +Epoch E marks the addition of your validator to the on-chain validator configuration smart contract. Your validator isn't considered a peer by validators yet. This is because the validator hasn't been refreshed in the current epoch yet. It is normal that no height metrics progress during this period — your node has to be considered a syncer to receive blocks. + +#### Syncer (epoch E+1) + +Your validator is now considered a peer by validators. It's syncing with the network and will be considered a player in the next epoch. + +#### Player (epoch E+2) + +Your validator is receiving consensus signing shares from dealers during the ceremony. + +#### Dealer (epoch E+3) + +Your validator is distributing consensus signing shares to other validators during the ceremony. Once your node is fully synced up, it will also be able to propose blocks and vote for other validators' proposed blocks. + ## Checking your validator's state Monitor these metrics to track your validator's state: ```bash -# Is your validator connected to other peers (i.e. is it considered a syncer)? This should be >0 in at most 3 hours after your validator's addition +# Is your validator connected to other peers (i.e. is it considered a syncer)? +# This should be >0 in at most 3 hours after your validator's addition. curl -s localhost:8002/metrics | grep consensus_network_spawner_connections -# How many times YOUR node has been a dealer (distributing shares) - this metric should be >0 over 6 hours +# How many times YOUR node has been a dealer (distributing shares) +# This metric should be >0 over 6 hours. curl -s localhost:8002/metrics | grep consensus_engine_dkg_manager_how_often_dealer -# How many times YOUR node has been a player (receiving shares) - this metric should be >0 over 6 hours +# How many times YOUR node has been a player (receiving shares) +# This metric should be >0 over 6 hours. curl -s localhost:8002/metrics | grep consensus_engine_dkg_manager_how_often_player # Successful ceremonies (should increase every ~3 hours) @@ -32,14 +75,86 @@ curl -s localhost:8002/metrics | grep consensus_engine_dkg_manager_ceremony_fail If `how_often_dealer` or `how_often_player` is increasing, your node is actively participating in DKG ceremonies. After your validator has been added to the network, you should alert on these metrics, as they indicate that your validator is actively participating in the network. -If your validator is not connected to other peers, but at least 3 epochs have passed, please check that your node is properly configured - e.g. firewall settings are open to other peers. If you have reset your validator's state, -your validator might've been blocked due to double-signing a block. In that case, please reach out to the Tempo team. +If your validator is not connected to other peers, but at least 3 epochs have passed, check that your node is properly configured — e.g. firewall settings are open to other peers. If you have reset your validator's state, your validator might have been blocked due to double-signing a block. In that case, please reach out to the Tempo team — even with the Tempo team, resolving this requires coordinating a new validator identity. + +You can also check your validator's on-chain status using the CLI: + +::::code-group +```bash [Mainnet] +tempo consensus validators-info --rpc-url https://rpc.tempo.xyz +``` +```bash [Testnet] +tempo consensus validators-info --rpc-url https://rpc.testnet.tempo.xyz +``` +:::: + +## Validator lifecycle + +### Starting and stopping + +Use `SIGINT` (Ctrl+C) or `SIGTERM` to gracefully stop the node: + +```bash +# If running directly +kill -INT + +# If running via systemd +sudo systemctl stop tempo +``` + +The node will finish processing the current block before shutting down. Avoid using `SIGKILL` as it may corrupt the database. + +### Resetting your validator's data + +:::danger +**You cannot reset a validator's data and continue with the same identity.** Doing so risks inconsistent voting, which can cause irrecoverable network safety failures. +::: + +If you need to reset your validator's data, you must rotate to a new validator identity. This requires coordinating with the Tempo team to deactivate your old identity and register a new one. Don't hesitate to reach out — even with the Tempo team, this is a routine operation. + +## Managing validator keys + +### Signing key rotation + +:::warning +The old validator identity must be deactivated before the new one is activated. +::: + +To rotate your validator's signing key: + +1. Generate a new keypair: + +::::code-group +```bash [Generate key] +tempo consensus generate-private-key --output +``` +```bash [Verify public key] +tempo consensus calculate-public-key --private-key +``` +:::: + +2. Contact the Tempo team to update your validator's public key on-chain. + +3. Once confirmed, update your node configuration to use the new key and restart. Your validator will go through the [validator state transitions](#validator-states) again. -### Consensus +### Signing share recovery + +:::danger +**You cannot reset a validator's data and continue with the same identity.** Doing so risks inconsistent voting, which can cause irrecoverable network safety failures. +::: + +If you lose your signing share (stored on the database in `/consensus/`), you will need to rotate to a new validator identity. This requires coordinating with the Tempo team to deactivate your old identity and register a new one. + +We're planning to release a high-availability feature that allows storing consensus data in an external database, which will enable signing share recovery without the need for key rotation. + +## Monitoring your validator + +### Consensus states #### Proposer Your validator is currently the leader and proposing blocks. Check proposal activity: + ```bash # Number of blocks your node has built and resolved curl -s localhost:9000/metrics | grep reth_payloads_resolved_block @@ -58,7 +173,7 @@ curl -s localhost:8002/metrics | grep consensus_engine_epoch_manager_simplex_bat This counter should increase steadily when your node is participating in consensus. -### Execution +### Execution states #### Catching up @@ -82,39 +197,52 @@ curl -s localhost:8002/metrics | grep -E "marshal_finalized_height|marshal_proce Both values should be nearly equal and increasing together. -## Node Lifecycle +### Metrics glossary -### Starting and Stopping +This is a table of the most important metrics to watch on your validator node: -Use `SIGINT` (Ctrl+C) or `SIGTERM` to gracefully stop the node: +| Metric Name | Description | When to alert? | Meaning | +| --- | --- | --- | --- | +| `consensus_engine_dkg_manager_ceremony_successes_total` | Number of successful DKG ceremonies | Critical when it hasn't increased in 12 hours | Your node has participated in a successful DKG ceremony | +| `consensus_engine_dkg_manager_ceremony_failures_total` | Number of failed DKG ceremonies | Warning when it increases | Your node has failed to participate in a DKG ceremony | +| `consensus_engine_marshal_finalized_height` | Latest finalized height your node is aware of | Warning when it hasn't increased in 1 hour, critical when it hasn't increased in 3 hours | Your node is aware of the latest finalized height | +| `consensus_engine_marshal_processed_height` | Latest height your node has processed | Critical when it hasn't increased in an hour, warning when it's behind finalized height | Your node is processing blocks | +| `consensus_network_spawner_connections` | Number of active peers on the consensus layer | Warning when it's below the expected validator count, critical when it's 0 | Your node is connected to the consensus layer | +| `consensus_engine_epoch_manager_simplex_batcher_inbound_messages_total` | Number of inbound messages related to voting on the consensus layer | Warning when it's not increasing and your node is synced | Your node is receiving voting messages from the consensus layer | +| `reth_sync_checkpoint` | Current sync progress | Warning on when there are no changes in the `Finish` stage | Your node is syncing with the network | +| `reth_payloads_resolved_block` | Number of built and resolved payloads | Warning when it hasn't increased in 12 hours | Your node has built and resolved blocks | -```bash -# If running directly -kill -INT +### Grafana dashboard -# If running via systemd -sudo systemctl stop tempo -``` +We provide a pre-built Grafana dashboard for monitoring your validator. It visualizes key metrics including node status, sync progress, voting activity, consensus latency, and execution performance. -The node will finish processing the current block before shutting down. Avoid using `SIGKILL` as it may corrupt the database. +#### Importing the dashboard -### Resetting your validator's data +1. Download the dashboard JSON from the [Tempo repository](https://github.com/tempoxyz/tempo/blob/main/contrib/grafana/dashboards/validator-health.json) -:::danger -**You cannot reset a validator's data and continue with the same identity.** Doing so risks inconsistent voting, which can cause irrecoverable network safety failures. -::: +2. In Grafana, go to **Dashboards → Import** -If you need to reset your validator's data, you must rotate to a new validator identity. See key rotation under [Validator Configuration](/guide/node/validator-config-v2#rotate-validator-identity). +3. Either paste the JSON content or upload the file -## Key Management +4. Select your Prometheus and Loki datasources when prompted -For signing key rotation, IP address updates, signing share recovery, and ownership transfers, see: +5. Click **Import** -- **[Validator Configuration](/guide/node/validator-config-v2)** (post-T2) — self-service operations +#### Dashboard variables -## Log Management +The dashboard uses these template variables: -### Parsing Logs +| Variable | Description | +| --- | --- | +| `datasource` | Prometheus datasource | +| `loki_ds` | Loki datasource (for log-based panels) | +| `network_name` | Filter by network/validator job name | + +Make sure your Prometheus is scraping metrics from your validator's metrics endpoint (default: `localhost:8002/metrics`). + +## Log management + +### Parsing logs Tempo logs include ANSI escape codes for colors. To strip them for grep/awk: @@ -132,7 +260,7 @@ If you're using Loki, you can also use the `decolorize` filter to strip colors: {job="tempo-node"} | decolorize ``` -### Log Levels +### Log levels Control verbosity with `RUST_LOG`: @@ -146,46 +274,3 @@ RUST_LOG=info,tempo_commonware_node::consensus=debug # Quiet mode (warnings and errors only) RUST_LOG=warn ``` - -## Metrics glossary - -This is a table of the most important metrics to watch on your validator node: - -| Metric Name | Description | When to alert? | Meaning | -| --- | --- | --- | --- | -| `consensus_engine_dkg_manager_ceremony_successes_total` | Number of successful DKG ceremonies | Critical when it hasn't increased in 12 hours | Your node has participated in a successful DKG ceremony | -| `consensus_engine_dkg_manager_ceremony_failures_total` | Number of failed DKG ceremonies | Warning when it increases | Your node has failed to participate in a DKG ceremony | -| `consensus_engine_marshal_finalized_height` | Latest finalized height your node is aware of | Warning when it hasn't increased in 1 hour, critical when it hasn't increased in 3 hours | Your node is aware of the latest finalized height | -| `consensus_engine_marshal_processed_height` | Latest height your node has processed | Critical when it hasn't increased in an hour, warning when it's behind finalized height | Your node is processing blocks | -| `consensus_network_spawner_connections` | Number of active peers on the consensus layer | Warning when it's below the expected validator count, critical when it's 0 | Your node is connected to the consensus layer | -| `consensus_engine_epoch_manager_simplex_batcher_inbound_messages_total` | Number of inbound messages related to voting on the consensus layer | Warning when it's not increasing and your node is synced | Your node is receiving voting messages from the consensus layer | -| `reth_sync_checkpoint` | Current sync progress | Warning on when there are no changes in the `Finish` stage | Your node is syncing with the network | -| `reth_payloads_resolved_block` | Number of built and resolved payloads | Warning when it hasn't increased in 12 hours | Your node has built and resolved blocks | - -## Grafana Dashboard - -We provide a pre-built Grafana dashboard for monitoring your validator. It visualizes key metrics including node status, sync progress, voting activity, consensus latency, and execution performance. - -### Importing the dashboard - -1. Download the dashboard JSON from the [Tempo repository](https://github.com/tempoxyz/tempo/blob/main/contrib/grafana/dashboards/validator-health.json) - -2. In Grafana, go to **Dashboards → Import** - -3. Either paste the JSON content or upload the file - -4. Select your Prometheus and Loki datasources when prompted - -5. Click **Import** - -### Dashboard variables - -The dashboard uses these template variables: - -| Variable | Description | -| --- | --- | -| `datasource` | Prometheus datasource | -| `loki_ds` | Loki datasource (for log-based panels) | -| `network_name` | Filter by network/validator job name | - -Make sure your Prometheus is scraping metrics from your validator's metrics endpoint (default: `localhost:8002/metrics`). diff --git a/src/pages/guide/node/validator.mdx b/src/pages/guide/node/validator.mdx index d0957772..885e9073 100644 --- a/src/pages/guide/node/validator.mdx +++ b/src/pages/guide/node/validator.mdx @@ -13,24 +13,58 @@ The active validator set is currently permissioned. If you are interested in bec ## Cryptographic key hierarchy All validators have two kinds of keys: -* signing key - this is an ED25519 keypair used to identify your validator amongst others -* signing share - this is a share of a BLS12-381 keypair used to sign block notarizations and finalizations +* **Signing key** — an ED25519 keypair used to identify your validator amongst others. +* **Signing share** — a share of a BLS12-381 keypair used to sign block notarizations and finalizations. -The signing key is static. Pre-T2, it can only be rotated by removing and re-creating a validator on-chain with the Tempo team. Post-T2, [ValidatorConfig V2](/guide/node/validator-config-v2) enables self-service rotation via the `rotateValidator` precompile. The signing share is dynamic and is updated periodically on each DKG ceremony. Once updated, it will be saved to disk. Currently, the DKG ceremony is scheduled to run on-chain every ~3 hours. +The signing key can be rotated while preserving your validator index — see [Signing key rotation](/guide/node/operate-validator#signing-key-rotation) for details. The signing share is dynamic and is updated periodically on each DKG ceremony. Once updated, it will be saved to disk. Currently, the DKG ceremony is scheduled to run on-chain every ~3 hours. + +## Generating a signing key + +You can generate an ED25519 keypair using the following command: + +```bash +tempo consensus generate-private-key --output +``` + +Once the private key is generated, you can calculate/verify the public key by running: + +```bash +tempo consensus calculate-public-key --private-key +``` + +The public key should match the output of the `generate-private-key` command. ## Running the validator The process for running a validator node is very similar to [running a full node](/guide/node/rpc). -You should start by downloading the latest snapshot using `tempo download --minimal`. Downloading the snapshot allows your validator to start participating in consensus much faster. -Once you've downloaded the snapshot and have been whitelisted on-chain, you can proceed to run the validator node as such: +You should start by downloading the latest snapshot. Downloading the snapshot allows your validator to start participating in consensus much faster. -```bash +::::code-group +```bash [Mainnet] +tempo download --chain mainnet +``` +```bash [Testnet] +tempo download --chain moderato +``` +:::: + +Once you've downloaded the snapshot and have been whitelisted on-chain, you can proceed to run the validator node: + +::::code-group +```bash [Mainnet] +tempo node --datadir \ + --chain mainnet \ + --consensus.signing-key \ + --telemetry-url +``` +```bash [Testnet] tempo node --datadir \ - --chain \ + --chain moderato \ --consensus.signing-key \ --telemetry-url ``` +:::: :::warning `--consensus.fee-recipient` is deprecated as of `v1.5.2` and will be removed in an upcoming release. Validators should migrate fee-recipient management to [Validator Config V2](/guide/node/validator-config-v2). @@ -90,10 +124,15 @@ INFO handle_propose{epoch=18 view=387213 parent.view=387212 parent.digest=0x4388 ``` If you do not see logs like these: -- please check that your validator is part of the active set, and is both a player and a dealer. See [validator states](/guide/node/validator-config-v2#validator-states) -- please check that your node is synced up to the [latest block height](https://explorer.tempo.xyz/). -- please check that your outgoing IP address matches the one whitelisted on the validator smart contract: +- Check that your validator is part of the active set, and is both [a player and a dealer](/guide/node/operate-validator#dealer-epoch-e3). +- Check that your node is synced up to the [latest block height](https://explorer.tempo.xyz/). +- Check that your outgoing IP address matches the one whitelisted on the validator smart contract: -```bash -tempo consensus validators-info --rpc-url +::::code-group +```bash [Mainnet] +tempo consensus validators-info --rpc-url https://rpc.tempo.xyz +``` +```bash [Testnet] +tempo consensus validators-info --rpc-url https://rpc.testnet.tempo.xyz ``` +:::: From 7346e2377eb1718b81f356e2d1dfd13adae6d3cf Mon Sep 17 00:00:00 2001 From: Paradigm AI Date: Fri, 17 Apr 2026 10:33:33 +0000 Subject: [PATCH 02/54] docs: split operator docs into actionable pages Replaces 'Operation' and 'ValidatorConfig V2' with four focused pages: - Checking validator state (states diagram, metrics, on-chain queries) - Controlling validator lifecycle (start/stop, registration, rotation, deactivation) - Managing validator keys (key hierarchy, generation, rotation, share recovery) - Observing a validator (monitoring, metrics glossary, Grafana, logs) All cross-references updated. Mainnet/testnet code tabs added to all RPC commands. Amp-Thread-ID: https://ampcode.com/threads/T-019d9aa2-72ab-7528-b44d-d82c81cc98ef Co-authored-by: Amp --- src/pages/cli/node.mdx | 2 +- src/pages/guide/node/network-upgrades.mdx | 2 +- src/pages/guide/node/operate-validator.mdx | 276 ------------------ src/pages/guide/node/validator-config-v2.mdx | 215 -------------- src/pages/guide/node/validator-keys.mdx | 57 ++++ src/pages/guide/node/validator-lifecycle.mdx | 212 ++++++++++++++ src/pages/guide/node/validator-monitoring.mdx | 132 +++++++++ src/pages/guide/node/validator-state.mdx | 119 ++++++++ src/pages/guide/node/validator.mdx | 8 +- src/pages/protocol/upgrades/t2.mdx | 2 +- vocs.config.ts | 16 +- 11 files changed, 539 insertions(+), 502 deletions(-) delete mode 100644 src/pages/guide/node/operate-validator.mdx delete mode 100644 src/pages/guide/node/validator-config-v2.mdx create mode 100644 src/pages/guide/node/validator-keys.mdx create mode 100644 src/pages/guide/node/validator-lifecycle.mdx create mode 100644 src/pages/guide/node/validator-monitoring.mdx create mode 100644 src/pages/guide/node/validator-state.mdx diff --git a/src/pages/cli/node.mdx b/src/pages/cli/node.mdx index 3d0e3b42..0600ae69 100644 --- a/src/pages/cli/node.mdx +++ b/src/pages/cli/node.mdx @@ -37,7 +37,7 @@ Flags grouped by function: | `--consensus.datadir ` | Separate volume for consensus data | :::warning -`--consensus.fee-recipient` is deprecated as of `v1.5.2` and will be removed in an upcoming release. See [Validator Config V2](/guide/node/validator-config-v2). +`--consensus.fee-recipient` is deprecated as of `v1.5.2` and will be removed in an upcoming release. See [Update the fee recipient](/guide/node/validator-lifecycle#update-the-fee-recipient). ::: ### Observability diff --git a/src/pages/guide/node/network-upgrades.mdx b/src/pages/guide/node/network-upgrades.mdx index 0856818c..1be7a118 100644 --- a/src/pages/guide/node/network-upgrades.mdx +++ b/src/pages/guide/node/network-upgrades.mdx @@ -27,7 +27,7 @@ For detailed release notes and binaries, see the [Changelog](/changelog). | [v1.2.0](https://github.com/tempoxyz/tempo/releases/tag/v1.2.0) | Feb 13, 2026 | Mainnet only | Fixes validation bug rejecting transactions with gas limits above ~16.7M, blocking large contract deployments | Required | :::warning -`--consensus.fee-recipient` is deprecated as of `v1.5.2` and will be removed in an upcoming release. Validators should migrate fee-recipient management to [Validator Config V2](/guide/node/validator-config-v2). +`--consensus.fee-recipient` is deprecated as of `v1.5.2` and will be removed in an upcoming release. Validators should migrate fee-recipient management to [Update the fee recipient](/guide/node/validator-lifecycle#update-the-fee-recipient). ::: --- diff --git a/src/pages/guide/node/operate-validator.mdx b/src/pages/guide/node/operate-validator.mdx deleted file mode 100644 index 2724982b..00000000 --- a/src/pages/guide/node/operate-validator.mdx +++ /dev/null @@ -1,276 +0,0 @@ ---- -title: Operate your validator node -description: Day-to-day operations for Tempo validators. Node lifecycle, monitoring, metrics, log management, and Grafana dashboards. ---- - -# Operate your validator - -This guide covers day-to-day operations for running a Tempo validator in production. - -:::danger -**You cannot reset a validator's data and continue with the same identity.** Doing so risks inconsistent voting, which can cause irrecoverable network safety failures. If you need to reset your validator's data, you must rotate to a new validator identity — see [Signing key rotation](#signing-key-rotation). -::: - -- **[Validator Configuration](/guide/node/validator-config-v2)** — self-service key rotation, IP updates, and ownership transfers - -## Validator states - -Your validator moves through different states during operation. The transitions will happen only once on your validator's creation. Every state transition will happen on epoch boundaries in the happy case. - -```mermaid {style: {maxWidth: '480px', margin: '0 auto'}} -flowchart TD - A[Inactive] -->|epoch ends| B["Syncer
The validator starts syncing"] - B -->|+1 epoch| C["Syncer and Player"] - C -->|"+1 epoch, receives share"| D["Syncer
Player
Dealer"] - D -->|"once fully synced"| E["Voter ↔ Proposer"] - - click A "#not-a-participant-e" - click B "#syncer-epoch-e1" - click C "#player-epoch-e2" - click D "#dealer-epoch-e3" - click E "#dealer-epoch-e3" -``` - -Currently, on mainnet and testnet, the epoch length is around 3 hours, which means that your validator will transition through these states approximately every 3 hours. If the epoch length ever changes, the transition times will also change. - -#### Not a participant (E) - -Epoch E marks the addition of your validator to the on-chain validator configuration smart contract. Your validator isn't considered a peer by validators yet. This is because the validator hasn't been refreshed in the current epoch yet. It is normal that no height metrics progress during this period — your node has to be considered a syncer to receive blocks. - -#### Syncer (epoch E+1) - -Your validator is now considered a peer by validators. It's syncing with the network and will be considered a player in the next epoch. - -#### Player (epoch E+2) - -Your validator is receiving consensus signing shares from dealers during the ceremony. - -#### Dealer (epoch E+3) - -Your validator is distributing consensus signing shares to other validators during the ceremony. Once your node is fully synced up, it will also be able to propose blocks and vote for other validators' proposed blocks. - -## Checking your validator's state - -Monitor these metrics to track your validator's state: - -```bash -# Is your validator connected to other peers (i.e. is it considered a syncer)? -# This should be >0 in at most 3 hours after your validator's addition. -curl -s localhost:8002/metrics | grep consensus_network_spawner_connections - -# How many times YOUR node has been a dealer (distributing shares) -# This metric should be >0 over 6 hours. -curl -s localhost:8002/metrics | grep consensus_engine_dkg_manager_how_often_dealer - -# How many times YOUR node has been a player (receiving shares) -# This metric should be >0 over 6 hours. -curl -s localhost:8002/metrics | grep consensus_engine_dkg_manager_how_often_player - -# Successful ceremonies (should increase every ~3 hours) -curl -s localhost:8002/metrics | grep consensus_engine_dkg_manager_ceremony_successes_total - -# Failed ceremonies (should stay at 0 or increase rarely) -curl -s localhost:8002/metrics | grep consensus_engine_dkg_manager_ceremony_failures_total -``` - -If `how_often_dealer` or `how_often_player` is increasing, your node is actively participating in DKG ceremonies. After your validator has been added to the network, you should alert on these metrics, as they indicate that your validator is actively participating in the network. - -If your validator is not connected to other peers, but at least 3 epochs have passed, check that your node is properly configured — e.g. firewall settings are open to other peers. If you have reset your validator's state, your validator might have been blocked due to double-signing a block. In that case, please reach out to the Tempo team — even with the Tempo team, resolving this requires coordinating a new validator identity. - -You can also check your validator's on-chain status using the CLI: - -::::code-group -```bash [Mainnet] -tempo consensus validators-info --rpc-url https://rpc.tempo.xyz -``` -```bash [Testnet] -tempo consensus validators-info --rpc-url https://rpc.testnet.tempo.xyz -``` -:::: - -## Validator lifecycle - -### Starting and stopping - -Use `SIGINT` (Ctrl+C) or `SIGTERM` to gracefully stop the node: - -```bash -# If running directly -kill -INT - -# If running via systemd -sudo systemctl stop tempo -``` - -The node will finish processing the current block before shutting down. Avoid using `SIGKILL` as it may corrupt the database. - -### Resetting your validator's data - -:::danger -**You cannot reset a validator's data and continue with the same identity.** Doing so risks inconsistent voting, which can cause irrecoverable network safety failures. -::: - -If you need to reset your validator's data, you must rotate to a new validator identity. This requires coordinating with the Tempo team to deactivate your old identity and register a new one. Don't hesitate to reach out — even with the Tempo team, this is a routine operation. - -## Managing validator keys - -### Signing key rotation - -:::warning -The old validator identity must be deactivated before the new one is activated. -::: - -To rotate your validator's signing key: - -1. Generate a new keypair: - -::::code-group -```bash [Generate key] -tempo consensus generate-private-key --output -``` -```bash [Verify public key] -tempo consensus calculate-public-key --private-key -``` -:::: - -2. Contact the Tempo team to update your validator's public key on-chain. - -3. Once confirmed, update your node configuration to use the new key and restart. Your validator will go through the [validator state transitions](#validator-states) again. - -### Signing share recovery - -:::danger -**You cannot reset a validator's data and continue with the same identity.** Doing so risks inconsistent voting, which can cause irrecoverable network safety failures. -::: - -If you lose your signing share (stored on the database in `/consensus/`), you will need to rotate to a new validator identity. This requires coordinating with the Tempo team to deactivate your old identity and register a new one. - -We're planning to release a high-availability feature that allows storing consensus data in an external database, which will enable signing share recovery without the need for key rotation. - -## Monitoring your validator - -### Consensus states - -#### Proposer - -Your validator is currently the leader and proposing blocks. Check proposal activity: - -```bash -# Number of blocks your node has built and resolved -curl -s localhost:9000/metrics | grep reth_payloads_resolved_block -``` - -If this counter is increasing, your validator is actively proposing blocks. - -#### Voter - -Your validator is voting on blocks proposed by others (notarization and finalization). This is the most common state. - -```bash -# Inbound voting messages (should increase steadily) -curl -s localhost:8002/metrics | grep consensus_engine_epoch_manager_simplex_batcher_inbound_messages_total | grep data_0 -``` - -This counter should increase steadily when your node is participating in consensus. - -### Execution states - -#### Catching up - -Your node is syncing historical blocks. - -```bash -# Check sync stage -curl -s localhost:6060/metrics | grep reth_sync_checkpoint -``` - -If `reth_sync_checkpoint` shows stages other than `Finish`, you're still syncing. - -#### Up to sync - -Your node is fully synced and processing new blocks in real-time. - -```bash -# Processed height should match or be close to finalized height -curl -s localhost:8002/metrics | grep -E "marshal_finalized_height|marshal_processed_height" -``` - -Both values should be nearly equal and increasing together. - -### Metrics glossary - -This is a table of the most important metrics to watch on your validator node: - -| Metric Name | Description | When to alert? | Meaning | -| --- | --- | --- | --- | -| `consensus_engine_dkg_manager_ceremony_successes_total` | Number of successful DKG ceremonies | Critical when it hasn't increased in 12 hours | Your node has participated in a successful DKG ceremony | -| `consensus_engine_dkg_manager_ceremony_failures_total` | Number of failed DKG ceremonies | Warning when it increases | Your node has failed to participate in a DKG ceremony | -| `consensus_engine_marshal_finalized_height` | Latest finalized height your node is aware of | Warning when it hasn't increased in 1 hour, critical when it hasn't increased in 3 hours | Your node is aware of the latest finalized height | -| `consensus_engine_marshal_processed_height` | Latest height your node has processed | Critical when it hasn't increased in an hour, warning when it's behind finalized height | Your node is processing blocks | -| `consensus_network_spawner_connections` | Number of active peers on the consensus layer | Warning when it's below the expected validator count, critical when it's 0 | Your node is connected to the consensus layer | -| `consensus_engine_epoch_manager_simplex_batcher_inbound_messages_total` | Number of inbound messages related to voting on the consensus layer | Warning when it's not increasing and your node is synced | Your node is receiving voting messages from the consensus layer | -| `reth_sync_checkpoint` | Current sync progress | Warning on when there are no changes in the `Finish` stage | Your node is syncing with the network | -| `reth_payloads_resolved_block` | Number of built and resolved payloads | Warning when it hasn't increased in 12 hours | Your node has built and resolved blocks | - -### Grafana dashboard - -We provide a pre-built Grafana dashboard for monitoring your validator. It visualizes key metrics including node status, sync progress, voting activity, consensus latency, and execution performance. - -#### Importing the dashboard - -1. Download the dashboard JSON from the [Tempo repository](https://github.com/tempoxyz/tempo/blob/main/contrib/grafana/dashboards/validator-health.json) - -2. In Grafana, go to **Dashboards → Import** - -3. Either paste the JSON content or upload the file - -4. Select your Prometheus and Loki datasources when prompted - -5. Click **Import** - -#### Dashboard variables - -The dashboard uses these template variables: - -| Variable | Description | -| --- | --- | -| `datasource` | Prometheus datasource | -| `loki_ds` | Loki datasource (for log-based panels) | -| `network_name` | Filter by network/validator job name | - -Make sure your Prometheus is scraping metrics from your validator's metrics endpoint (default: `localhost:8002/metrics`). - -## Log management - -### Parsing logs - -Tempo logs include ANSI escape codes for colors. To strip them for grep/awk: - -```bash -# Strip colors when searching -sudo journalctl -u tempo | sed 's/\x1b\[[0-9;]*m//g' | grep "error" - -# Or disable colors at runtime -RUST_LOG_STYLE=never tempo node ... -``` - -If you're using Loki, you can also use the `decolorize` filter to strip colors: - -``` -{job="tempo-node"} | decolorize -``` - -### Log levels - -Control verbosity with `RUST_LOG`: - -```bash -# Default (info) -RUST_LOG=info - -# Debug consensus only -RUST_LOG=info,tempo_commonware_node::consensus=debug - -# Quiet mode (warnings and errors only) -RUST_LOG=warn -``` diff --git a/src/pages/guide/node/validator-config-v2.mdx b/src/pages/guide/node/validator-config-v2.mdx deleted file mode 100644 index 3e41d115..00000000 --- a/src/pages/guide/node/validator-config-v2.mdx +++ /dev/null @@ -1,215 +0,0 @@ ---- -title: ValidatorConfig V2 -description: Manage your validator with ValidatorConfig V2. Self-service key rotation, IP updates, and ownership transfer. ---- - -import { StaticMermaidDiagram } from '../../../components/StaticMermaidDiagram' - -# ValidatorConfig V2 - -ValidatorConfig V2 ([TIP-1017](/protocol/tips/tip-1017)) is the precompile that manages consensus participants. This config was [activated on mainnet](https://explore.mainnet.tempo.xyz/receipt/0x4716147e3c2bf5c8d014b8c27d6e2af0042d5a5f29bdead256d6f33038702d64) after the T2 hardfork. - -## Validator states - -Once added, a validator immediately becomes a player in the next epoch. - -|epoch ends| B["Player
Receives signing shares"] - B -->|"+1 epoch, receives share"| C["Player
Dealer"] - C -->|"once fully synced"| D["Voter ↔ Proposer"] -`} /> -## What changes for operators - -Validators can perform several operations themselves without coordinating with the Tempo team: - -- **Rotate your validator identity** — swap to a new ed25519 key without losing your committee slot -- **Update IP addresses** — change ingress and egress endpoints independently (V2 supports asymmetric ingress/egress IPs) -- **Transfer ownership** — rebind your validator entry to a new address - -All write operations require either the contract owner or the validator's own address. - -## Precompile address - -```solidity -address constant VALIDATOR_CONFIG_V2 = 0xCCCCCCCC00000000000000000000000000000001; -``` - -## Reading validator state - -### Query active validators - -Use `validators-info` to see the current state of all validators: - -```bash -tempo consensus validators-info --rpc-url https://rpc.tempo.xyz -``` - -This returns the committee of the current epoch (validators with `in_committee = true`) as well as validators that have been added but will only become active in a future epoch. Use this to verify your validator's status after registration or rotation. - -:::info -The on-chain contract represents which validators should eventually be members of the committee, not which ones currently are. Every epoch, the Tempo network performs a distributed key generation ceremony, distributing keys to validators that should be -committee members in the next epoch. So adding, deactivating, or rotating validators entries in epoch `E` can only be taken into account for the next DKG ceremony that runs during epoch `E+1`, and take effect in epoch `E+2`. -::: - -### Look up your validator - -You can query your validator by ethereum address, ed25519 public key, or index: - -```bash -tempo consensus validator
--rpc-url https://rpc.tempo.xyz -``` - -The returned `Validator` struct fields are: - -| Field | Type | Description | -|-------|------|-------------| -| `publicKey` | `bytes32` | Ed25519 communication public key | -| `validatorAddress` | `address` | Validator control address | -| `ingress` | `string` | Inbound address (`IP:port`) | -| `egress` | `string` | Outbound address (`IP`) | -| `index` | `uint64` | Index of the validator (constant under rotation)| -| `addedAtHeight` | `uint64` | Block height when added | -| `deactivatedAtHeight` | `uint64` | Block height when deactivated (`0` = active) | -| `feeRecipient` | `address` | Address that receives block proposal fees | - -## Operator guide - -### Initial registration - -To register a new validator, you must provide the following values to the Tempo team: - -| Value | Format | Description | -|-------|--------|-------------| -| **Validator address** | Ethereum address (`0x…`) | The control address for your validator. Used to authorize on-chain operations (IP updates, rotation, ownership transfer). | -| **Public key** | `0x`-prefixed 32-byte hex | Your ed25519 identity key. Generate one with `tempo consensus generate-private-key` (see [below](#generating-a-signing-key)). | -| **Ingress** | `IP:port` | The inbound address other validators use to reach your node. Must be unique across all active validators. | -| **Egress** | `IP` | The outbound IP address your node uses to connect to other validators. | -| **Fee recipient** | Ethereum address (`0x…`) | The address that receives transaction fees from blocks your validator proposes. If you are not prepared to accept fees, use `0x0000000000000000000000000000000000000000`. | -| **Signature** | `0x`-prefixed hex | An ed25519 signature proving you control the signing key. See below. | - -#### Generating a signing key - -:::warning -Never share your private signing key. Anyone with access to it can impersonate your validator. -::: - -Generate an ed25519 keypair: - -```bash -tempo consensus generate-private-key --output -``` - -Verify the public key: - -```bash -tempo consensus calculate-public-key --private-key -``` - -The public key should match the output of the `generate-private-key` command. - -#### Creating the add-validator signature - -The signature proves ownership of the ed25519 key being registered. Generate it with: - -```bash -tempo consensus create-add-validator-signature \ - --signing-key \ - --validator-address \ - --public-key \ - --ingress \ - --egress \ - --fee-recipient \ - --chain-id-from-rpc-url https://rpc.tempo.xyz -``` - -This outputs a hex-encoded signature. Provide this signature along with the values above to the Tempo team to complete registration. - -Once the Tempo team adds your validator on-chain, it will enter the active set in the [next epoch](#validator-states). - -### Update the Fee Recipient - -The fee recipient used by your validator when constructing block proposals is managed on-chain. This can be updated and takes effect on the next finalized block - -```bash -tempo consensus set-validator-fee-recipient
- --fee-recipient - --rpc-url https://rpc.tempo.xyz - --private-key -``` -### Update IP addresses - -If your node's network endpoints change, update them on-chain. The change takes effect at the next finalized block. - -```bash -tempo consensus set-validator-ip-address
- --ingress :NEW_PROT> - --egress - --rpc-url https://rpc.tempo.xyz - --private-key -``` - -:::warning -Ingress addresses must be unique across all active validators. The transaction will revert if another active validator already uses the same `IP:port`. -::: - -### Rotate validator identity - -The ed25519 key can be changed while keeping your validator index stable. This is useful for key rotation or recovery without leaving and re-joining the committee. [Generate a new signing key](#generating-a-signing-key) first. - -The simplest way to rotate is using the `tempo` CLI, which handles signature creation and the on-chain transaction in one step: - -```bash -tempo consensus rotate-validator \ - --validator-address \ - --public-key \ - --ingress \ - --egress \ - --signing-key \ - --private-key \ - --rpc-url https://rpc.tempo.xyz -``` - -If self-service rotation is not yet enabled for your validator, use `tempo consensus create-rotate-validator-signature` to generate the signature and provide it to the Tempo team. - -:::info -Rotation preserves your validator index and active validator count. The old entry is appended to history as deactivated, and the entry at your index is updated in place. You must use a different ingress address (changing the port is sufficient). -::: - -After rotation, your validator goes through the [standard lifecycle](#validator-states) with the new identity. - -:::danger[Do not shut down the old validator immediately] -After rotation, the rotated-out validator is still a member of the current committee until the next successful DKG ceremony completes. Shutting it down early will degrade network liveness. - -Keep the old validator running and use `validators-info` to monitor its status: - -```bash -tempo consensus validators-info --rpc-url https://rpc.tempo.xyz -``` - -Once the old validator shows `in_committee = false`, it is safe to shut down. -::: - -### Transfer validator ownership - -Rebind your validator entry to a new control address: - -```bash -tempo consensus transfer-validator-ownership
- --rpc-url https://rpc.tempo.xyz \ - --private-key - --new-private-key -``` - -The new address must not already be used by another active validator. - -### Deactivate your validator - -Deactivate your validator. Once deactivated, please keep your the node. The validator is only removed from the commitee once reaching the end of an epoch where the DKG was successful, thus the validator is no longer dealt any new shares. - -See the [validators-info](#query-active-validators) command which lists active validators. It is safe to shut down the node once the deactivated validator is no longer present in this list. - -```bash -tempo consensus deactivate-validator
- --rpc-url https://rpc.tempo.xyz \ - --private-key -``` diff --git a/src/pages/guide/node/validator-keys.mdx b/src/pages/guide/node/validator-keys.mdx new file mode 100644 index 00000000..7e1a2d15 --- /dev/null +++ b/src/pages/guide/node/validator-keys.mdx @@ -0,0 +1,57 @@ +--- +title: Managing validator keys +description: Generate, rotate, and recover validator signing keys and shares. +--- + +# Managing validator keys + +Tempo validators use two kinds of cryptographic keys. This page covers how to generate, rotate, and recover them. + +## Key hierarchy + +* **Signing key** — an ED25519 keypair used to identify your validator amongst others. Can be rotated while preserving your validator index. +* **Signing share** — a share of a BLS12-381 keypair used to sign block notarizations and finalizations. Updated automatically every DKG ceremony (~3 hours). + +## Generating a signing key + +:::warning +Never share your private signing key. Anyone with access to it can impersonate your validator. +::: + +Generate an ed25519 keypair: + +```bash +tempo consensus generate-private-key --output +``` + +Verify the public key: + +```bash +tempo consensus calculate-public-key --private-key +``` + +The public key should match the output of the `generate-private-key` command. + +## Signing key rotation + +The ed25519 signing key can be rotated while preserving your validator index and committee slot. See [Rotate validator identity](/guide/node/validator-lifecycle#rotate-validator-identity) for the full procedure. + +## Signing share recovery + +:::danger +**You cannot reset a validator's data and continue with the same identity.** Doing so risks inconsistent voting, which can cause irrecoverable network safety failures. +::: + +If you lose your signing share (stored in `/consensus/`), you will need to rotate to a new validator identity. This requires coordinating with the Tempo team to deactivate your old identity and register a new one. + +We're planning to release a high-availability feature that allows storing consensus data in an external database, which will enable signing share recovery without the need for key rotation. + +## ValidatorConfig V2 precompile + +All key and identity operations are executed through the ValidatorConfig V2 precompile ([TIP-1017](/protocol/tips/tip-1017)): + +```solidity +address constant VALIDATOR_CONFIG_V2 = 0xCCCCCCCC00000000000000000000000000000001; +``` + +All write operations require either the contract owner or the validator's own address. diff --git a/src/pages/guide/node/validator-lifecycle.mdx b/src/pages/guide/node/validator-lifecycle.mdx new file mode 100644 index 00000000..1fd6a0b4 --- /dev/null +++ b/src/pages/guide/node/validator-lifecycle.mdx @@ -0,0 +1,212 @@ +--- +title: Controlling validator lifecycle +description: Start, stop, register, rotate, deactivate, and transfer ownership of your Tempo validator. +--- + +# Controlling validator lifecycle + +This guide covers operational actions for managing a Tempo validator — from initial registration to deactivation. + +## Starting and stopping + +Use `SIGINT` (Ctrl+C) or `SIGTERM` to gracefully stop the node: + +```bash +# If running directly +kill -INT + +# If running via systemd +sudo systemctl stop tempo +``` + +The node will finish processing the current block before shutting down. Avoid using `SIGKILL` as it may corrupt the database. + +## Resetting your validator's data + +:::danger +**You cannot reset a validator's data and continue with the same identity.** Doing so risks inconsistent voting, which can cause irrecoverable network safety failures. +::: + +If you need to reset your validator's data, you must rotate to a new validator identity. This requires coordinating with the Tempo team to deactivate your old identity and register a new one. Don't hesitate to reach out — even with the Tempo team, this is a routine operation. + +## Initial registration + +To register a new validator, you must provide the following values to the Tempo team: + +| Value | Format | Description | +|-------|--------|-------------| +| **Validator address** | Ethereum address (`0x…`) | The control address for your validator. Used to authorize on-chain operations (IP updates, rotation, ownership transfer). | +| **Public key** | `0x`-prefixed 32-byte hex | Your ed25519 identity key. Generate one with `tempo consensus generate-private-key` (see [Managing validator keys](/guide/node/validator-keys#generating-a-signing-key)). | +| **Ingress** | `IP:port` | The inbound address other validators use to reach your node. Must be unique across all active validators. | +| **Egress** | `IP` | The outbound IP address your node uses to connect to other validators. | +| **Fee recipient** | Ethereum address (`0x…`) | The address that receives transaction fees from blocks your validator proposes. If you are not prepared to accept fees, use `0x0000000000000000000000000000000000000000`. | +| **Signature** | `0x`-prefixed hex | An ed25519 signature proving you control the signing key. See below. | + +### Creating the add-validator signature + +The signature proves ownership of the ed25519 key being registered. Generate it with: + +::::code-group +```bash [Mainnet] +tempo consensus create-add-validator-signature \ + --signing-key \ + --validator-address \ + --public-key \ + --ingress \ + --egress \ + --fee-recipient \ + --chain-id-from-rpc-url https://rpc.tempo.xyz +``` +```bash [Testnet] +tempo consensus create-add-validator-signature \ + --signing-key \ + --validator-address \ + --public-key \ + --ingress \ + --egress \ + --fee-recipient \ + --chain-id-from-rpc-url https://rpc.testnet.tempo.xyz +``` +:::: + +Provide this signature along with the values above to the Tempo team to complete registration. + +Once the Tempo team adds your validator on-chain, it will enter the active set in the [next epoch](/guide/node/validator-state#state-transitions). + +## Rotate validator identity + +The ed25519 key can be changed while keeping your validator index stable. This is useful for key rotation or recovery without leaving and re-joining the committee. [Generate a new signing key](/guide/node/validator-keys#generating-a-signing-key) first. + +The simplest way to rotate is using the `tempo` CLI, which handles signature creation and the on-chain transaction in one step: + +::::code-group +```bash [Mainnet] +tempo consensus rotate-validator \ + --validator-address \ + --public-key \ + --ingress \ + --egress \ + --signing-key \ + --private-key \ + --rpc-url https://rpc.tempo.xyz +``` +```bash [Testnet] +tempo consensus rotate-validator \ + --validator-address \ + --public-key \ + --ingress \ + --egress \ + --signing-key \ + --private-key \ + --rpc-url https://rpc.testnet.tempo.xyz +``` +:::: + +If self-service rotation is not yet enabled for your validator, use `tempo consensus create-rotate-validator-signature` to generate the signature and provide it to the Tempo team. + +:::info +Rotation preserves your validator index and active validator count. The old entry is appended to history as deactivated, and the entry at your index is updated in place. You must use a different ingress address (changing the port is sufficient). +::: + +After rotation, your validator goes through the [standard state transitions](/guide/node/validator-state#state-transitions) with the new identity. + +:::danger[Do not shut down the old validator immediately] +After rotation, the rotated-out validator is still a member of the current committee until the next successful DKG ceremony completes. Shutting it down early will degrade network liveness. + +Keep the old validator running and use `validators-info` to monitor its status: + +::::code-group +```bash [Mainnet] +tempo consensus validators-info --rpc-url https://rpc.tempo.xyz +``` +```bash [Testnet] +tempo consensus validators-info --rpc-url https://rpc.testnet.tempo.xyz +``` +:::: + +Once the old validator shows `in_committee = false`, it is safe to shut down. +::: + +## Update IP addresses + +If your node's network endpoints change, update them on-chain. The change takes effect at the next finalized block. + +::::code-group +```bash [Mainnet] +tempo consensus set-validator-ip-address
\ + --ingress \ + --egress \ + --rpc-url https://rpc.tempo.xyz \ + --private-key +``` +```bash [Testnet] +tempo consensus set-validator-ip-address
\ + --ingress \ + --egress \ + --rpc-url https://rpc.testnet.tempo.xyz \ + --private-key +``` +:::: + +:::warning +Ingress addresses must be unique across all active validators. The transaction will revert if another active validator already uses the same `IP:port`. +::: + +## Update the fee recipient + +The fee recipient used by your validator when constructing block proposals is managed on-chain. This can be updated and takes effect on the next finalized block. + +::::code-group +```bash [Mainnet] +tempo consensus set-validator-fee-recipient
\ + --fee-recipient \ + --rpc-url https://rpc.tempo.xyz \ + --private-key +``` +```bash [Testnet] +tempo consensus set-validator-fee-recipient
\ + --fee-recipient \ + --rpc-url https://rpc.testnet.tempo.xyz \ + --private-key +``` +:::: + +## Transfer validator ownership + +Rebind your validator entry to a new control address: + +::::code-group +```bash [Mainnet] +tempo consensus transfer-validator-ownership
\ + --rpc-url https://rpc.tempo.xyz \ + --private-key \ + --new-private-key +``` +```bash [Testnet] +tempo consensus transfer-validator-ownership
\ + --rpc-url https://rpc.testnet.tempo.xyz \ + --private-key \ + --new-private-key +``` +:::: + +The new address must not already be used by another active validator. + +## Deactivate your validator + +Deactivate your validator when you want to leave the active set. The validator is only removed from the committee once reaching the end of an epoch where the DKG was successful. + +::::code-group +```bash [Mainnet] +tempo consensus deactivate-validator
\ + --rpc-url https://rpc.tempo.xyz \ + --private-key +``` +```bash [Testnet] +tempo consensus deactivate-validator
\ + --rpc-url https://rpc.testnet.tempo.xyz \ + --private-key +``` +:::: + +Keep your node running after deactivation. Use [validators-info](/guide/node/validator-state#querying-on-chain-state) to check when your validator is no longer listed — it is safe to shut down the node at that point. diff --git a/src/pages/guide/node/validator-monitoring.mdx b/src/pages/guide/node/validator-monitoring.mdx new file mode 100644 index 00000000..480129c4 --- /dev/null +++ b/src/pages/guide/node/validator-monitoring.mdx @@ -0,0 +1,132 @@ +--- +title: Observing a validator +description: Monitor validator health with metrics, Grafana dashboards, and log management. +--- + +# Observing a validator + +This guide covers how to monitor your Tempo validator's health, diagnose issues, and manage logs. + +## Consensus states + +### Proposer + +Your validator is currently the leader and proposing blocks. Check proposal activity: + +```bash +# Number of blocks your node has built and resolved +curl -s localhost:9000/metrics | grep reth_payloads_resolved_block +``` + +If this counter is increasing, your validator is actively proposing blocks. + +### Voter + +Your validator is voting on blocks proposed by others (notarization and finalization). This is the most common state. + +```bash +# Inbound voting messages (should increase steadily) +curl -s localhost:8002/metrics | grep consensus_engine_epoch_manager_simplex_batcher_inbound_messages_total | grep data_0 +``` + +This counter should increase steadily when your node is participating in consensus. + +## Execution states + +### Catching up + +Your node is syncing historical blocks. + +```bash +# Check sync stage +curl -s localhost:6060/metrics | grep reth_sync_checkpoint +``` + +If `reth_sync_checkpoint` shows stages other than `Finish`, you're still syncing. + +### Up to sync + +Your node is fully synced and processing new blocks in real-time. + +```bash +# Processed height should match or be close to finalized height +curl -s localhost:8002/metrics | grep -E "marshal_finalized_height|marshal_processed_height" +``` + +Both values should be nearly equal and increasing together. + +## Metrics glossary + +| Metric Name | Description | When to alert? | Meaning | +| --- | --- | --- | --- | +| `consensus_engine_dkg_manager_ceremony_successes_total` | Number of successful DKG ceremonies | Critical when it hasn't increased in 12 hours | Your node has participated in a successful DKG ceremony | +| `consensus_engine_dkg_manager_ceremony_failures_total` | Number of failed DKG ceremonies | Warning when it increases | Your node has failed to participate in a DKG ceremony | +| `consensus_engine_marshal_finalized_height` | Latest finalized height your node is aware of | Warning when it hasn't increased in 1 hour, critical when it hasn't increased in 3 hours | Your node is aware of the latest finalized height | +| `consensus_engine_marshal_processed_height` | Latest height your node has processed | Critical when it hasn't increased in an hour, warning when it's behind finalized height | Your node is processing blocks | +| `consensus_network_spawner_connections` | Number of active peers on the consensus layer | Warning when it's below the expected validator count, critical when it's 0 | Your node is connected to the consensus layer | +| `consensus_engine_epoch_manager_simplex_batcher_inbound_messages_total` | Number of inbound messages related to voting on the consensus layer | Warning when it's not increasing and your node is synced | Your node is receiving voting messages from the consensus layer | +| `reth_sync_checkpoint` | Current sync progress | Warning on when there are no changes in the `Finish` stage | Your node is syncing with the network | +| `reth_payloads_resolved_block` | Number of built and resolved payloads | Warning when it hasn't increased in 12 hours | Your node has built and resolved blocks | + +## Grafana dashboard + +We provide a pre-built Grafana dashboard for monitoring your validator. It visualizes key metrics including node status, sync progress, voting activity, consensus latency, and execution performance. + +### Importing the dashboard + +1. Download the dashboard JSON from the [Tempo repository](https://github.com/tempoxyz/tempo/blob/main/contrib/grafana/dashboards/validator-health.json) + +2. In Grafana, go to **Dashboards → Import** + +3. Either paste the JSON content or upload the file + +4. Select your Prometheus and Loki datasources when prompted + +5. Click **Import** + +### Dashboard variables + +The dashboard uses these template variables: + +| Variable | Description | +| --- | --- | +| `datasource` | Prometheus datasource | +| `loki_ds` | Loki datasource (for log-based panels) | +| `network_name` | Filter by network/validator job name | + +Make sure your Prometheus is scraping metrics from your validator's metrics endpoint (default: `localhost:8002/metrics`). + +## Log management + +### Parsing logs + +Tempo logs include ANSI escape codes for colors. To strip them for grep/awk: + +```bash +# Strip colors when searching +sudo journalctl -u tempo | sed 's/\x1b\[[0-9;]*m//g' | grep "error" + +# Or disable colors at runtime +RUST_LOG_STYLE=never tempo node ... +``` + +If you're using Loki, you can also use the `decolorize` filter to strip colors: + +``` +{job="tempo-node"} | decolorize +``` + +### Log levels + +Control verbosity with `RUST_LOG`: + +```bash +# Default (info) +RUST_LOG=info + +# Debug consensus only +RUST_LOG=info,tempo_commonware_node::consensus=debug + +# Quiet mode (warnings and errors only) +RUST_LOG=warn +``` diff --git a/src/pages/guide/node/validator-state.mdx b/src/pages/guide/node/validator-state.mdx new file mode 100644 index 00000000..51d791f6 --- /dev/null +++ b/src/pages/guide/node/validator-state.mdx @@ -0,0 +1,119 @@ +--- +title: Checking validator state +description: Understand validator state transitions, check your validator's participation status, and query on-chain state. +--- + +# Checking validator state + +Your validator moves through different states after registration. Understanding these states helps you verify that your node is healthy and participating in consensus. + +:::danger +**You cannot reset a validator's data and continue with the same identity.** Doing so risks inconsistent voting, which can cause irrecoverable network safety failures. If you need to reset your validator's data, you must rotate to a new validator identity — see [Signing key rotation](/guide/node/validator-keys#signing-key-rotation). +::: + +## State transitions + +Every state transition happens on epoch boundaries. Currently on mainnet and testnet, the epoch length is around 3 hours. + +```mermaid {style: {maxWidth: '480px', margin: '0 auto'}} +flowchart TD + A[Inactive] -->|epoch ends| B["Syncer
The validator starts syncing"] + B -->|+1 epoch| C["Syncer and Player"] + C -->|"+1 epoch, receives share"| D["Syncer
Player
Dealer"] + D -->|"once fully synced"| E["Voter ↔ Proposer"] + + click A "#not-a-participant-e" + click B "#syncer-epoch-e1" + click C "#player-epoch-e2" + click D "#dealer-epoch-e3" + click E "#dealer-epoch-e3" +``` + +#### Not a participant (E) + +Epoch E marks the addition of your validator to the on-chain validator configuration smart contract. Your validator isn't considered a peer by validators yet. This is because the validator hasn't been refreshed in the current epoch yet. It is normal that no height metrics progress during this period — your node has to be considered a syncer to receive blocks. + +#### Syncer (epoch E+1) + +Your validator is now considered a peer by validators. It's syncing with the network and will be considered a player in the next epoch. + +#### Player (epoch E+2) + +Your validator is receiving consensus signing shares from dealers during the ceremony. + +#### Dealer (epoch E+3) + +Your validator is distributing consensus signing shares to other validators during the ceremony. Once your node is fully synced up, it will also be able to propose blocks and vote for other validators' proposed blocks. + +:::info +The on-chain contract represents which validators should eventually be members of the committee, not which ones currently are. Every epoch, the Tempo network performs a distributed key generation ceremony, distributing keys to validators that should be committee members in the next epoch. So adding, deactivating, or rotating validator entries in epoch `E` can only be taken into account for the next DKG ceremony that runs during epoch `E+1`, and take effect in epoch `E+2`. +::: + +## Checking state via metrics + +Monitor these metrics to track your validator's state: + +```bash +# Is your validator connected to other peers (i.e. is it considered a syncer)? +# This should be >0 in at most 3 hours after your validator's addition. +curl -s localhost:8002/metrics | grep consensus_network_spawner_connections + +# How many times YOUR node has been a dealer (distributing shares) +# This metric should be >0 over 6 hours. +curl -s localhost:8002/metrics | grep consensus_engine_dkg_manager_how_often_dealer + +# How many times YOUR node has been a player (receiving shares) +# This metric should be >0 over 6 hours. +curl -s localhost:8002/metrics | grep consensus_engine_dkg_manager_how_often_player + +# Successful ceremonies (should increase every ~3 hours) +curl -s localhost:8002/metrics | grep consensus_engine_dkg_manager_ceremony_successes_total + +# Failed ceremonies (should stay at 0 or increase rarely) +curl -s localhost:8002/metrics | grep consensus_engine_dkg_manager_ceremony_failures_total +``` + +If `how_often_dealer` or `how_often_player` is increasing, your node is actively participating in DKG ceremonies. After your validator has been added to the network, you should alert on these metrics, as they indicate that your validator is actively participating in the network. + +If your validator is not connected to other peers, but at least 3 epochs have passed, check that your node is properly configured — e.g. firewall settings are open to other peers. If you have reset your validator's state, your validator might have been blocked due to double-signing a block. In that case, please reach out to the Tempo team — even with the Tempo team, resolving this requires coordinating a new validator identity. + +## Querying on-chain state + +Check your validator's on-chain status using the CLI: + +::::code-group +```bash [Mainnet] +tempo consensus validators-info --rpc-url https://rpc.tempo.xyz +``` +```bash [Testnet] +tempo consensus validators-info --rpc-url https://rpc.testnet.tempo.xyz +``` +:::: + +This returns the committee of the current epoch (validators with `in_committee = true`) as well as validators that have been added but will only become active in a future epoch. Use this to verify your validator's status after registration or rotation. + +### Look up your validator + +You can query your validator by ethereum address, ed25519 public key, or index: + +::::code-group +```bash [Mainnet] +tempo consensus validator
--rpc-url https://rpc.tempo.xyz +``` +```bash [Testnet] +tempo consensus validator
--rpc-url https://rpc.testnet.tempo.xyz +``` +:::: + +The returned `Validator` struct fields are: + +| Field | Type | Description | +|-------|------|-------------| +| `publicKey` | `bytes32` | Ed25519 communication public key | +| `validatorAddress` | `address` | Validator control address | +| `ingress` | `string` | Inbound address (`IP:port`) | +| `egress` | `string` | Outbound address (`IP`) | +| `index` | `uint64` | Index of the validator (constant under rotation)| +| `addedAtHeight` | `uint64` | Block height when added | +| `deactivatedAtHeight` | `uint64` | Block height when deactivated (`0` = active) | +| `feeRecipient` | `address` | Address that receives block proposal fees | diff --git a/src/pages/guide/node/validator.mdx b/src/pages/guide/node/validator.mdx index 885e9073..0d18f7a4 100644 --- a/src/pages/guide/node/validator.mdx +++ b/src/pages/guide/node/validator.mdx @@ -8,7 +8,7 @@ Validator nodes secure Tempo by validating blocks and participating in consensus ## Deploying a validator -The active validator set is currently permissioned. If you are interested in becoming a validator, please [get in touch](mailto:partners@tempo.xyz) with the Tempo team. See [Initial registration](/guide/node/validator-config-v2#initial-registration) for technical details. +The active validator set is currently permissioned. If you are interested in becoming a validator, please [get in touch](mailto:partners@tempo.xyz) with the Tempo team. See [Initial registration](/guide/node/validator-lifecycle#initial-registration) for technical details. ## Cryptographic key hierarchy @@ -16,7 +16,7 @@ All validators have two kinds of keys: * **Signing key** — an ED25519 keypair used to identify your validator amongst others. * **Signing share** — a share of a BLS12-381 keypair used to sign block notarizations and finalizations. -The signing key can be rotated while preserving your validator index — see [Signing key rotation](/guide/node/operate-validator#signing-key-rotation) for details. The signing share is dynamic and is updated periodically on each DKG ceremony. Once updated, it will be saved to disk. Currently, the DKG ceremony is scheduled to run on-chain every ~3 hours. +The signing key can be rotated while preserving your validator index — see [Signing key rotation](/guide/node/validator-keys#signing-key-rotation) for details. The signing share is dynamic and is updated periodically on each DKG ceremony. Once updated, it will be saved to disk. Currently, the DKG ceremony is scheduled to run on-chain every ~3 hours. ## Generating a signing key @@ -67,7 +67,7 @@ tempo node --datadir \ :::: :::warning -`--consensus.fee-recipient` is deprecated as of `v1.5.2` and will be removed in an upcoming release. Validators should migrate fee-recipient management to [Validator Config V2](/guide/node/validator-config-v2). +`--consensus.fee-recipient` is deprecated as of `v1.5.2` and will be removed in an upcoming release. Validators should migrate fee-recipient management to [Update the fee recipient](/guide/node/validator-lifecycle#update-the-fee-recipient). ::: ### Optional flags @@ -124,7 +124,7 @@ INFO handle_propose{epoch=18 view=387213 parent.view=387212 parent.digest=0x4388 ``` If you do not see logs like these: -- Check that your validator is part of the active set, and is both [a player and a dealer](/guide/node/operate-validator#dealer-epoch-e3). +- Check that your validator is part of the active set, and is both [a player and a dealer](/guide/node/validator-state#dealer-epoch-e3). - Check that your node is synced up to the [latest block height](https://explorer.tempo.xyz/). - Check that your outgoing IP address matches the one whitelisted on the validator smart contract: diff --git a/src/pages/protocol/upgrades/t2.mdx b/src/pages/protocol/upgrades/t2.mdx index 4b6ec8c8..aad74f71 100644 --- a/src/pages/protocol/upgrades/t2.mdx +++ b/src/pages/protocol/upgrades/t2.mdx @@ -44,7 +44,7 @@ T2 is Tempo's next network upgrade, building on T1. It introduces the following **Spec:** [TIP-1017](/protocol/tips/tip-1017) -**Operator guide:** [Validator Config V2](/guide/node/validator-config-v2) +**Operator guide:** [Controlling validator lifecycle](/guide/node/validator-lifecycle) **TLDR:** New precompile for managing consensus validators. Adds Ed25519 signature verification at registration, append-only history and stable validator indices. diff --git a/vocs.config.ts b/vocs.config.ts index c65bb9b5..923743d9 100644 --- a/vocs.config.ts +++ b/vocs.config.ts @@ -815,12 +815,20 @@ export default defineConfig({ link: '/guide/node/validator', }, { - text: 'Operation', - link: '/guide/node/operate-validator', + text: 'Checking validator state', + link: '/guide/node/validator-state', }, { - text: 'ValidatorConfig V2', - link: '/guide/node/validator-config-v2', + text: 'Controlling validator lifecycle', + link: '/guide/node/validator-lifecycle', + }, + { + text: 'Managing validator keys', + link: '/guide/node/validator-keys', + }, + { + text: 'Observing a validator', + link: '/guide/node/validator-monitoring', }, ], }, From b6ec6d7dda960b34edd76669025f08b6c06a352a Mon Sep 17 00:00:00 2001 From: Paradigm AI Date: Fri, 17 Apr 2026 10:56:24 +0000 Subject: [PATCH 03/54] docs: simplify overview, extract initial setup and troubleshooting pages - Overview now just links to all sub-pages with short descriptions - 'Deploying a validator' converted to info box - Removed duplicate key hierarchy section - 'Generating a signing key' + 'Running the validator' moved to 'Initial setup' - 'Troubleshooting' moved to 'Troubleshooting and FAQ' Amp-Thread-ID: https://ampcode.com/threads/T-019d9aa2-72ab-7528-b44d-d82c81cc98ef Co-authored-by: Amp --- src/pages/guide/node/validator-setup.mdx | 103 +++++++++++++ .../guide/node/validator-troubleshooting.mdx | 28 ++++ src/pages/guide/node/validator.mdx | 136 ++---------------- vocs.config.ts | 8 ++ 4 files changed, 148 insertions(+), 127 deletions(-) create mode 100644 src/pages/guide/node/validator-setup.mdx create mode 100644 src/pages/guide/node/validator-troubleshooting.mdx diff --git a/src/pages/guide/node/validator-setup.mdx b/src/pages/guide/node/validator-setup.mdx new file mode 100644 index 00000000..c3f7849e --- /dev/null +++ b/src/pages/guide/node/validator-setup.mdx @@ -0,0 +1,103 @@ +--- +title: Initial setup +description: Generate signing keys and run your Tempo validator node for the first time. +--- + +# Initial setup + +This guide walks through generating your signing key and starting your validator node for the first time. Before proceeding, make sure you have completed [system requirements](/guide/node/system-requirements) and [installation](/guide/node/installation). + +## Generating a signing key + +You can generate an ED25519 keypair using the following command: + +```bash +tempo consensus generate-private-key --output +``` + +Once the private key is generated, you can calculate/verify the public key by running: + +```bash +tempo consensus calculate-public-key --private-key +``` + +The public key should match the output of the `generate-private-key` command. + +## Running the validator + +The process for running a validator node is very similar to [running a full node](/guide/node/rpc). + +You should start by downloading the latest snapshot. Downloading the snapshot allows your validator to start participating in consensus much faster. + +::::code-group +```bash [Mainnet] +tempo download --chain mainnet +``` +```bash [Testnet] +tempo download --chain moderato +``` +:::: + +Once you've downloaded the snapshot and have been whitelisted on-chain, you can proceed to run the validator node: + +::::code-group +```bash [Mainnet] +tempo node --datadir \ + --chain mainnet \ + --consensus.signing-key \ + --telemetry-url +``` +```bash [Testnet] +tempo node --datadir \ + --chain moderato \ + --consensus.signing-key \ + --telemetry-url +``` +:::: + +:::warning +`--consensus.fee-recipient` is deprecated as of `v1.5.2` and will be removed in an upcoming release. Validators should migrate fee-recipient management to [Update the fee recipient](/guide/node/validator-lifecycle#update-the-fee-recipient). +::: + +The notable difference between RPC nodes and validator nodes is the omission of the `--follow` argument and the addition of the `--consensus.signing-key` argument. + +Once your node is up, it may not start syncing immediately. This is because your node might not be part of the active set. In most cases, your validator will enter the active set in under 6 hours after the on-chain addition of the validator identity. + +### Optional flags + +| Flag | Description | +|------|-------------| +| `--telemetry-url ` | Unified metrics and logs export. See [Telemetry endpoint](#telemetry-endpoint) below for details. **We ask all validators to configure this so we can support troubleshooting.** | +| `--telemetry-metrics-interval ` | Interval for pushing metrics (default: `10s`). | +| `--consensus.datadir ` | Store consensus data on a separate volume (e.g., AWS EBS) while keeping execution state on high-performance local disks. Migrate by copying `/consensus` to the new location. | + +### Telemetry endpoint + +The `--telemetry-url` flag enables unified telemetry export. It pushes two types of data to a Tempo-operated metrics backend: + +- **Prometheus metrics** — execution-layer (reth) and consensus-layer metrics in Prometheus text format, pushed at a configurable interval (default every 10 seconds). +- **Structured logs** — operational logs exported via OTLP at `debug` level, covering consensus events, block processing, and sync progress. + +The URL must include credentials: `--telemetry-url https://user:pass@metrics.example.com` + +#### What is collected + +| Category | Examples | +|----------|---------| +| Execution metrics | Block processing times, transaction pool size, peer count, sync status, database stats | +| Consensus metrics | Epoch and view progress, DKG ceremony status, proposal and finalization counts | +| Operational logs | Consensus state transitions, block proposals, sync progress, error events | + +All consensus metrics are namespaced under a `consensus` prefix. + +#### What is **not** collected + +The telemetry endpoint does **not** collect any ambient information about the host machine. Specifically: + +- No hostname, IP address, or machine identifiers +- No operating system, CPU, memory, or disk information +- No information about other processes or services running on the machine +- No file paths or directory structures beyond the node's own data directory references in logs +- No network topology or firewall configuration + +The data is strictly limited to the node's own operational metrics and logs. diff --git a/src/pages/guide/node/validator-troubleshooting.mdx b/src/pages/guide/node/validator-troubleshooting.mdx new file mode 100644 index 00000000..130dbe66 --- /dev/null +++ b/src/pages/guide/node/validator-troubleshooting.mdx @@ -0,0 +1,28 @@ +--- +title: Troubleshooting and FAQ +description: Common issues and solutions for Tempo validator operators. +--- + +# Troubleshooting and FAQ + +## My node is not proposing blocks + +Once your node is proposing blocks, it will start emitting logs like these: + +``` +INFO handle_propose{epoch=18 view=387213 parent.view=387212 parent.digest=0x43885416b4a7ae7550c615ad4dc702045cd26540b78fa2bd75abdcbfbef02d9d}: tempo_commonware_node::consensus::application::actor: constructed proposal proposal.digest=0x6baa8fa813beea491cf598024d8b31cb2927e7ba1b92a29899a795dbafd682c6 proposal.height=5733680 +``` + +If you do not see logs like these: +- Check that your validator is part of the active set, and is both [a player and a dealer](/guide/node/validator-state#dealer-epoch-e3). +- Check that your node is synced up to the [latest block height](https://explorer.tempo.xyz/). +- Check that your outgoing IP address matches the one whitelisted on the validator smart contract: + +::::code-group +```bash [Mainnet] +tempo consensus validators-info --rpc-url https://rpc.tempo.xyz +``` +```bash [Testnet] +tempo consensus validators-info --rpc-url https://rpc.testnet.tempo.xyz +``` +:::: diff --git a/src/pages/guide/node/validator.mdx b/src/pages/guide/node/validator.mdx index 0d18f7a4..f2729b3f 100644 --- a/src/pages/guide/node/validator.mdx +++ b/src/pages/guide/node/validator.mdx @@ -1,138 +1,20 @@ --- -description: Configure and run a Tempo validator node. Generate signing keys, participate in DKG ceremonies, and troubleshoot consensus issues. +description: Overview of running a Tempo validator node. --- # Running a validator node Validator nodes secure Tempo by validating blocks and participating in consensus. -## Deploying a validator - +:::info The active validator set is currently permissioned. If you are interested in becoming a validator, please [get in touch](mailto:partners@tempo.xyz) with the Tempo team. See [Initial registration](/guide/node/validator-lifecycle#initial-registration) for technical details. - -## Cryptographic key hierarchy - -All validators have two kinds of keys: -* **Signing key** — an ED25519 keypair used to identify your validator amongst others. -* **Signing share** — a share of a BLS12-381 keypair used to sign block notarizations and finalizations. - -The signing key can be rotated while preserving your validator index — see [Signing key rotation](/guide/node/validator-keys#signing-key-rotation) for details. The signing share is dynamic and is updated periodically on each DKG ceremony. Once updated, it will be saved to disk. Currently, the DKG ceremony is scheduled to run on-chain every ~3 hours. - -## Generating a signing key - -You can generate an ED25519 keypair using the following command: - -```bash -tempo consensus generate-private-key --output -``` - -Once the private key is generated, you can calculate/verify the public key by running: - -```bash -tempo consensus calculate-public-key --private-key -``` - -The public key should match the output of the `generate-private-key` command. - -## Running the validator - -The process for running a validator node is very similar to [running a full node](/guide/node/rpc). - -You should start by downloading the latest snapshot. Downloading the snapshot allows your validator to start participating in consensus much faster. - -::::code-group -```bash [Mainnet] -tempo download --chain mainnet -``` -```bash [Testnet] -tempo download --chain moderato -``` -:::: - -Once you've downloaded the snapshot and have been whitelisted on-chain, you can proceed to run the validator node: - -::::code-group -```bash [Mainnet] -tempo node --datadir \ - --chain mainnet \ - --consensus.signing-key \ - --telemetry-url -``` -```bash [Testnet] -tempo node --datadir \ - --chain moderato \ - --consensus.signing-key \ - --telemetry-url -``` -:::: - -:::warning -`--consensus.fee-recipient` is deprecated as of `v1.5.2` and will be removed in an upcoming release. Validators should migrate fee-recipient management to [Update the fee recipient](/guide/node/validator-lifecycle#update-the-fee-recipient). ::: -### Optional flags - -| Flag | Description | -|------|-------------| -| `--telemetry-url ` | Unified metrics and logs export. See [Telemetry endpoint](#telemetry-endpoint) below for details. **We ask all validators to configure this so we can support troubleshooting.** | -| `--telemetry-metrics-interval ` | Interval for pushing metrics (default: `10s`). | -| `--consensus.datadir ` | Store consensus data on a separate volume (e.g., AWS EBS) while keeping execution state on high-performance local disks. Migrate by copying `/consensus` to the new location. | - -### Telemetry endpoint - -The `--telemetry-url` flag enables unified telemetry export. It pushes two types of data to a Tempo-operated metrics backend: - -- **Prometheus metrics** — execution-layer (reth) and consensus-layer metrics in Prometheus text format, pushed at a configurable interval (default every 10 seconds). -- **Structured logs** — operational logs exported via OTLP at `debug` level, covering consensus events, block processing, and sync progress. - -The URL must include credentials: `--telemetry-url https://user:pass@metrics.example.com` - -#### What is collected - -| Category | Examples | -|----------|---------| -| Execution metrics | Block processing times, transaction pool size, peer count, sync status, database stats | -| Consensus metrics | Epoch and view progress, DKG ceremony status, proposal and finalization counts | -| Operational logs | Consensus state transitions, block proposals, sync progress, error events | - -All consensus metrics are namespaced under a `consensus` prefix. - -#### What is **not** collected - -The telemetry endpoint does **not** collect any ambient information about the host machine. Specifically: - -- No hostname, IP address, or machine identifiers -- No operating system, CPU, memory, or disk information -- No information about other processes or services running on the machine -- No file paths or directory structures beyond the node's own data directory references in logs -- No network topology or firewall configuration - -The data is strictly limited to the node's own operational metrics and logs. - -The notable difference between RPC nodes and validator nodes is the omission of the `--follow` argument and the addition of the `--consensus.signing-key` argument. Validator fee recipients are moving to on-chain management via Validator Config V2, and the legacy `--consensus.fee-recipient` flag is deprecated. - -Once your node is up, it may not start syncing immediately. This is because your node might not be part of the active set. In most cases, your validator will enter the active set in under 6 hours after the on-chain addition of the validator identity. - -## Troubleshooting - -### Checking if your node is proposing blocks - -Once your node is proposing blocks, it will start emitting logs like these: - -``` -INFO handle_propose{epoch=18 view=387213 parent.view=387212 parent.digest=0x43885416b4a7ae7550c615ad4dc702045cd26540b78fa2bd75abdcbfbef02d9d}: tempo_commonware_node::consensus::application::actor: constructed proposal proposal.digest=0x6baa8fa813beea491cf598024d8b31cb2927e7ba1b92a29899a795dbafd682c6 proposal.height=5733680 -``` - -If you do not see logs like these: -- Check that your validator is part of the active set, and is both [a player and a dealer](/guide/node/validator-state#dealer-epoch-e3). -- Check that your node is synced up to the [latest block height](https://explorer.tempo.xyz/). -- Check that your outgoing IP address matches the one whitelisted on the validator smart contract: +## Guides -::::code-group -```bash [Mainnet] -tempo consensus validators-info --rpc-url https://rpc.tempo.xyz -``` -```bash [Testnet] -tempo consensus validators-info --rpc-url https://rpc.testnet.tempo.xyz -``` -:::: +- **[Initial setup](/guide/node/validator-setup)** — Generate your signing key, download a snapshot, and start your validator for the first time. +- **[Checking validator state](/guide/node/validator-state)** — Understand state transitions, check participation via metrics, and query on-chain status. +- **[Controlling validator lifecycle](/guide/node/validator-lifecycle)** — Start, stop, register, rotate, deactivate, and transfer ownership of your validator. +- **[Managing validator keys](/guide/node/validator-keys)** — Key hierarchy, generation, rotation, and signing share recovery. +- **[Observing a validator](/guide/node/validator-monitoring)** — Monitor consensus and execution health, metrics glossary, Grafana dashboards, and log management. +- **[Troubleshooting and FAQ](/guide/node/validator-troubleshooting)** — Common issues and solutions. diff --git a/vocs.config.ts b/vocs.config.ts index 923743d9..61455749 100644 --- a/vocs.config.ts +++ b/vocs.config.ts @@ -814,6 +814,10 @@ export default defineConfig({ text: 'Overview', link: '/guide/node/validator', }, + { + text: 'Initial setup', + link: '/guide/node/validator-setup', + }, { text: 'Checking validator state', link: '/guide/node/validator-state', @@ -830,6 +834,10 @@ export default defineConfig({ text: 'Observing a validator', link: '/guide/node/validator-monitoring', }, + { + text: 'Troubleshooting and FAQ', + link: '/guide/node/validator-troubleshooting', + }, ], }, { From 165a00edfca0b4875e516879f09d0ea282cbb00e Mon Sep 17 00:00:00 2001 From: Paradigm AI Date: Fri, 17 Apr 2026 12:26:39 +0000 Subject: [PATCH 04/54] docs: add key security warnings to initial setup and managing keys pages Amp-Thread-ID: https://ampcode.com/threads/T-019d9aa2-72ab-7528-b44d-d82c81cc98ef Co-authored-by: Amp --- src/pages/guide/node/validator-keys.mdx | 4 ++++ src/pages/guide/node/validator-setup.mdx | 4 ++++ 2 files changed, 8 insertions(+) diff --git a/src/pages/guide/node/validator-keys.mdx b/src/pages/guide/node/validator-keys.mdx index 7e1a2d15..eeb959eb 100644 --- a/src/pages/guide/node/validator-keys.mdx +++ b/src/pages/guide/node/validator-keys.mdx @@ -7,6 +7,10 @@ description: Generate, rotate, and recover validator signing keys and shares. Tempo validators use two kinds of cryptographic keys. This page covers how to generate, rotate, and recover them. +:::warning +Never share your private signing key. Anyone with access to it can impersonate your validator. Store keys securely and restrict file permissions. +::: + ## Key hierarchy * **Signing key** — an ED25519 keypair used to identify your validator amongst others. Can be rotated while preserving your validator index. diff --git a/src/pages/guide/node/validator-setup.mdx b/src/pages/guide/node/validator-setup.mdx index c3f7849e..00b86c1f 100644 --- a/src/pages/guide/node/validator-setup.mdx +++ b/src/pages/guide/node/validator-setup.mdx @@ -9,6 +9,10 @@ This guide walks through generating your signing key and starting your validator ## Generating a signing key +:::warning +Never share your private signing key. Anyone with access to it can impersonate your validator. +::: + You can generate an ED25519 keypair using the following command: ```bash From 5a660da4d1485a795945be0a124f19ed9ba6a4c8 Mon Sep 17 00:00:00 2001 From: Paradigm AI Date: Fri, 17 Apr 2026 13:24:47 +0000 Subject: [PATCH 05/54] docs: move initial registration to initial setup page - Moved registration values table and add-validator signature from validator-lifecycle to validator-setup - Updated overview info box to link to initial setup instead of lifecycle - Updated overview description Amp-Thread-ID: https://ampcode.com/threads/T-019d9aa2-72ab-7528-b44d-d82c81cc98ef Co-authored-by: Amp --- src/pages/guide/node/validator-lifecycle.mdx | 44 ------------------- src/pages/guide/node/validator-setup.mdx | 46 +++++++++++++++++++- src/pages/guide/node/validator.mdx | 4 +- 3 files changed, 47 insertions(+), 47 deletions(-) diff --git a/src/pages/guide/node/validator-lifecycle.mdx b/src/pages/guide/node/validator-lifecycle.mdx index 1fd6a0b4..8a7ee71f 100644 --- a/src/pages/guide/node/validator-lifecycle.mdx +++ b/src/pages/guide/node/validator-lifecycle.mdx @@ -29,50 +29,6 @@ The node will finish processing the current block before shutting down. Avoid us If you need to reset your validator's data, you must rotate to a new validator identity. This requires coordinating with the Tempo team to deactivate your old identity and register a new one. Don't hesitate to reach out — even with the Tempo team, this is a routine operation. -## Initial registration - -To register a new validator, you must provide the following values to the Tempo team: - -| Value | Format | Description | -|-------|--------|-------------| -| **Validator address** | Ethereum address (`0x…`) | The control address for your validator. Used to authorize on-chain operations (IP updates, rotation, ownership transfer). | -| **Public key** | `0x`-prefixed 32-byte hex | Your ed25519 identity key. Generate one with `tempo consensus generate-private-key` (see [Managing validator keys](/guide/node/validator-keys#generating-a-signing-key)). | -| **Ingress** | `IP:port` | The inbound address other validators use to reach your node. Must be unique across all active validators. | -| **Egress** | `IP` | The outbound IP address your node uses to connect to other validators. | -| **Fee recipient** | Ethereum address (`0x…`) | The address that receives transaction fees from blocks your validator proposes. If you are not prepared to accept fees, use `0x0000000000000000000000000000000000000000`. | -| **Signature** | `0x`-prefixed hex | An ed25519 signature proving you control the signing key. See below. | - -### Creating the add-validator signature - -The signature proves ownership of the ed25519 key being registered. Generate it with: - -::::code-group -```bash [Mainnet] -tempo consensus create-add-validator-signature \ - --signing-key \ - --validator-address \ - --public-key \ - --ingress \ - --egress \ - --fee-recipient \ - --chain-id-from-rpc-url https://rpc.tempo.xyz -``` -```bash [Testnet] -tempo consensus create-add-validator-signature \ - --signing-key \ - --validator-address \ - --public-key \ - --ingress \ - --egress \ - --fee-recipient \ - --chain-id-from-rpc-url https://rpc.testnet.tempo.xyz -``` -:::: - -Provide this signature along with the values above to the Tempo team to complete registration. - -Once the Tempo team adds your validator on-chain, it will enter the active set in the [next epoch](/guide/node/validator-state#state-transitions). - ## Rotate validator identity The ed25519 key can be changed while keeping your validator index stable. This is useful for key rotation or recovery without leaving and re-joining the committee. [Generate a new signing key](/guide/node/validator-keys#generating-a-signing-key) first. diff --git a/src/pages/guide/node/validator-setup.mdx b/src/pages/guide/node/validator-setup.mdx index 00b86c1f..de6740a7 100644 --- a/src/pages/guide/node/validator-setup.mdx +++ b/src/pages/guide/node/validator-setup.mdx @@ -5,7 +5,51 @@ description: Generate signing keys and run your Tempo validator node for the fir # Initial setup -This guide walks through generating your signing key and starting your validator node for the first time. Before proceeding, make sure you have completed [system requirements](/guide/node/system-requirements) and [installation](/guide/node/installation). +This guide walks through registering, generating your signing key, and starting your validator node for the first time. Before proceeding, make sure you have completed [system requirements](/guide/node/system-requirements) and [installation](/guide/node/installation). + +## Initial registration + +To register a new validator, you must provide the following values to the Tempo team: + +| Value | Format | Description | +|-------|--------|-------------| +| **Validator address** | Ethereum address (`0x…`) | The control address for your validator. Used to authorize on-chain operations (IP updates, rotation, ownership transfer). | +| **Public key** | `0x`-prefixed 32-byte hex | Your ed25519 identity key. Generate one with `tempo consensus generate-private-key` (see [below](#generating-a-signing-key)). | +| **Ingress** | `IP:port` | The inbound address other validators use to reach your node. Must be unique across all active validators. | +| **Egress** | `IP` | The outbound IP address your node uses to connect to other validators. | +| **Fee recipient** | Ethereum address (`0x…`) | The address that receives transaction fees from blocks your validator proposes. If you are not prepared to accept fees, use `0x0000000000000000000000000000000000000000`. | +| **Signature** | `0x`-prefixed hex | An ed25519 signature proving you control the signing key. See [below](#creating-the-add-validator-signature). | + +### Creating the add-validator signature + +The signature proves ownership of the ed25519 key being registered. Generate it with: + +::::code-group +```bash [Mainnet] +tempo consensus create-add-validator-signature \ + --signing-key \ + --validator-address \ + --public-key \ + --ingress \ + --egress \ + --fee-recipient \ + --chain-id-from-rpc-url https://rpc.tempo.xyz +``` +```bash [Testnet] +tempo consensus create-add-validator-signature \ + --signing-key \ + --validator-address \ + --public-key \ + --ingress \ + --egress \ + --fee-recipient \ + --chain-id-from-rpc-url https://rpc.testnet.tempo.xyz +``` +:::: + +Provide this signature along with the values above to the Tempo team to complete registration. + +Once the Tempo team adds your validator on-chain, it will enter the active set in the [next epoch](/guide/node/validator-state#state-transitions). ## Generating a signing key diff --git a/src/pages/guide/node/validator.mdx b/src/pages/guide/node/validator.mdx index f2729b3f..cd5b519f 100644 --- a/src/pages/guide/node/validator.mdx +++ b/src/pages/guide/node/validator.mdx @@ -7,12 +7,12 @@ description: Overview of running a Tempo validator node. Validator nodes secure Tempo by validating blocks and participating in consensus. :::info -The active validator set is currently permissioned. If you are interested in becoming a validator, please [get in touch](mailto:partners@tempo.xyz) with the Tempo team. See [Initial registration](/guide/node/validator-lifecycle#initial-registration) for technical details. +The active validator set is currently permissioned. If you are interested in becoming a validator, please [get in touch](mailto:partners@tempo.xyz) with the Tempo team. See [Initial setup](/guide/node/validator-setup) for technical details. ::: ## Guides -- **[Initial setup](/guide/node/validator-setup)** — Generate your signing key, download a snapshot, and start your validator for the first time. +- **[Initial setup](/guide/node/validator-setup)** — Register with the Tempo team, generate your signing key, download a snapshot, and start your validator for the first time. - **[Checking validator state](/guide/node/validator-state)** — Understand state transitions, check participation via metrics, and query on-chain status. - **[Controlling validator lifecycle](/guide/node/validator-lifecycle)** — Start, stop, register, rotate, deactivate, and transfer ownership of your validator. - **[Managing validator keys](/guide/node/validator-keys)** — Key hierarchy, generation, rotation, and signing share recovery. From f1f9dada0755720ba1c5f17ceeee7a785df034ae Mon Sep 17 00:00:00 2001 From: Paradigm AI Date: Fri, 17 Apr 2026 13:28:31 +0000 Subject: [PATCH 06/54] docs: add 'Tempo team will never ask for your private key' to all key warnings Amp-Thread-ID: https://ampcode.com/threads/T-019d9aa2-72ab-7528-b44d-d82c81cc98ef Co-authored-by: Amp --- src/pages/guide/node/validator-keys.mdx | 4 ++-- src/pages/guide/node/validator-setup.mdx | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/src/pages/guide/node/validator-keys.mdx b/src/pages/guide/node/validator-keys.mdx index eeb959eb..d814dbe4 100644 --- a/src/pages/guide/node/validator-keys.mdx +++ b/src/pages/guide/node/validator-keys.mdx @@ -8,7 +8,7 @@ description: Generate, rotate, and recover validator signing keys and shares. Tempo validators use two kinds of cryptographic keys. This page covers how to generate, rotate, and recover them. :::warning -Never share your private signing key. Anyone with access to it can impersonate your validator. Store keys securely and restrict file permissions. +Never share your private signing key. Anyone with access to it can impersonate your validator. The Tempo team will never ask for your private key. Store keys securely and restrict file permissions. ::: ## Key hierarchy @@ -19,7 +19,7 @@ Never share your private signing key. Anyone with access to it can impersonate y ## Generating a signing key :::warning -Never share your private signing key. Anyone with access to it can impersonate your validator. +Never share your private signing key. Anyone with access to it can impersonate your validator. The Tempo team will never ask for your private key. ::: Generate an ed25519 keypair: diff --git a/src/pages/guide/node/validator-setup.mdx b/src/pages/guide/node/validator-setup.mdx index de6740a7..e7c6516f 100644 --- a/src/pages/guide/node/validator-setup.mdx +++ b/src/pages/guide/node/validator-setup.mdx @@ -54,7 +54,7 @@ Once the Tempo team adds your validator on-chain, it will enter the active set i ## Generating a signing key :::warning -Never share your private signing key. Anyone with access to it can impersonate your validator. +Never share your private signing key. Anyone with access to it can impersonate your validator. The Tempo team will never ask for your private key. ::: You can generate an ED25519 keypair using the following command: From 44f3665ae541e92e5a85f6c6a60c023e9188e6b7 Mon Sep 17 00:00:00 2001 From: Paradigm AI Date: Mon, 20 Apr 2026 05:36:30 +0000 Subject: [PATCH 07/54] docs: copy permissioned validator info box to initial setup page Co-authored-by: Amp Amp-Thread-ID: https://ampcode.com/threads/T-019da962-06e0-7083-806a-3d46ee63e7b1 --- src/pages/guide/node/validator-setup.mdx | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/src/pages/guide/node/validator-setup.mdx b/src/pages/guide/node/validator-setup.mdx index e7c6516f..2320e58c 100644 --- a/src/pages/guide/node/validator-setup.mdx +++ b/src/pages/guide/node/validator-setup.mdx @@ -5,6 +5,10 @@ description: Generate signing keys and run your Tempo validator node for the fir # Initial setup +:::info +The active validator set is currently permissioned. If you are interested in becoming a validator, please [get in touch](mailto:partners@tempo.xyz) with the Tempo team. +::: + This guide walks through registering, generating your signing key, and starting your validator node for the first time. Before proceeding, make sure you have completed [system requirements](/guide/node/system-requirements) and [installation](/guide/node/installation). ## Initial registration From c494b8761d3bbecd13a3244f5f1cabbe457818b9 Mon Sep 17 00:00:00 2001 From: Paradigm AI Date: Mon, 20 Apr 2026 05:37:42 +0000 Subject: [PATCH 08/54] docs: restructure initial registration into 3 ordered steps 1. Generate signing keypair 2. Create add-validator signature 3. Submit registration details to Tempo team Co-authored-by: Amp Amp-Thread-ID: https://ampcode.com/threads/T-019da962-06e0-7083-806a-3d46ee63e7b1 --- src/pages/guide/node/validator-setup.mdx | 66 +++++++++++++----------- 1 file changed, 36 insertions(+), 30 deletions(-) diff --git a/src/pages/guide/node/validator-setup.mdx b/src/pages/guide/node/validator-setup.mdx index 2320e58c..e35d768a 100644 --- a/src/pages/guide/node/validator-setup.mdx +++ b/src/pages/guide/node/validator-setup.mdx @@ -13,18 +13,33 @@ This guide walks through registering, generating your signing key, and starting ## Initial registration -To register a new validator, you must provide the following values to the Tempo team: +Registering a validator takes three steps: -| Value | Format | Description | -|-------|--------|-------------| -| **Validator address** | Ethereum address (`0x…`) | The control address for your validator. Used to authorize on-chain operations (IP updates, rotation, ownership transfer). | -| **Public key** | `0x`-prefixed 32-byte hex | Your ed25519 identity key. Generate one with `tempo consensus generate-private-key` (see [below](#generating-a-signing-key)). | -| **Ingress** | `IP:port` | The inbound address other validators use to reach your node. Must be unique across all active validators. | -| **Egress** | `IP` | The outbound IP address your node uses to connect to other validators. | -| **Fee recipient** | Ethereum address (`0x…`) | The address that receives transaction fees from blocks your validator proposes. If you are not prepared to accept fees, use `0x0000000000000000000000000000000000000000`. | -| **Signature** | `0x`-prefixed hex | An ed25519 signature proving you control the signing key. See [below](#creating-the-add-validator-signature). | +1. **[Generate a signing keypair](#step-1-generate-a-signing-keypair)** — Create an ed25519 private key and derive the public key. +2. **[Create the add-validator signature](#step-2-create-the-add-validator-signature)** — Prove ownership of the signing key by producing a signature over your registration details. +3. **[Submit registration details](#step-3-submit-registration-details)** — Provide the Tempo team with all required values to add your validator on-chain. + +### Step 1: Generate a signing keypair + +:::warning +Never share your private signing key. Anyone with access to it can impersonate your validator. The Tempo team will never ask for your private key. +::: + +Generate an ed25519 keypair: + +```bash +tempo consensus generate-private-key --output +``` + +Verify the public key: + +```bash +tempo consensus calculate-public-key --private-key +``` + +The public key should match the output of the `generate-private-key` command. -### Creating the add-validator signature +### Step 2: Create the add-validator signature The signature proves ownership of the ed25519 key being registered. Generate it with: @@ -51,29 +66,20 @@ tempo consensus create-add-validator-signature \ ``` :::: -Provide this signature along with the values above to the Tempo team to complete registration. +### Step 3: Submit registration details -Once the Tempo team adds your validator on-chain, it will enter the active set in the [next epoch](/guide/node/validator-state#state-transitions). - -## Generating a signing key - -:::warning -Never share your private signing key. Anyone with access to it can impersonate your validator. The Tempo team will never ask for your private key. -::: - -You can generate an ED25519 keypair using the following command: - -```bash -tempo consensus generate-private-key --output -``` - -Once the private key is generated, you can calculate/verify the public key by running: +Provide the following values along with the signature to the Tempo team: -```bash -tempo consensus calculate-public-key --private-key -``` +| Value | Format | Description | +|-------|--------|-------------| +| **Validator address** | Ethereum address (`0x…`) | The control address for your validator. Used to authorize on-chain operations (IP updates, rotation, ownership transfer). | +| **Public key** | `0x`-prefixed 32-byte hex | Your ed25519 identity key (from [Step 1](#step-1-generate-a-signing-keypair)). | +| **Ingress** | `IP:port` | The inbound address other validators use to reach your node. Must be unique across all active validators. | +| **Egress** | `IP` | The outbound IP address your node uses to connect to other validators. | +| **Fee recipient** | Ethereum address (`0x…`) | The address that receives transaction fees from blocks your validator proposes. If you are not prepared to accept fees, use `0x0000000000000000000000000000000000000000`. | +| **Signature** | `0x`-prefixed hex | The ed25519 signature proving you control the signing key (from [Step 2](#step-2-create-the-add-validator-signature)). | -The public key should match the output of the `generate-private-key` command. +Once the Tempo team adds your validator on-chain, it will enter the active set in the [next epoch](/guide/node/validator-state#state-transitions). ## Running the validator From 7d0e4e324fdd7a3e93749a1cc872fce16cead16c Mon Sep 17 00:00:00 2001 From: Paradigm AI Date: Mon, 20 Apr 2026 06:06:49 +0000 Subject: [PATCH 09/54] docs: remove deprecated --consensus.fee-recipient mention from initial setup Co-authored-by: Amp Amp-Thread-ID: https://ampcode.com/threads/T-019da962-06e0-7083-806a-3d46ee63e7b1 --- src/pages/guide/node/validator-setup.mdx | 5 ----- 1 file changed, 5 deletions(-) diff --git a/src/pages/guide/node/validator-setup.mdx b/src/pages/guide/node/validator-setup.mdx index e35d768a..f4cc4769 100644 --- a/src/pages/guide/node/validator-setup.mdx +++ b/src/pages/guide/node/validator-setup.mdx @@ -112,11 +112,6 @@ tempo node --datadir \ --telemetry-url ``` :::: - -:::warning -`--consensus.fee-recipient` is deprecated as of `v1.5.2` and will be removed in an upcoming release. Validators should migrate fee-recipient management to [Update the fee recipient](/guide/node/validator-lifecycle#update-the-fee-recipient). -::: - The notable difference between RPC nodes and validator nodes is the omission of the `--follow` argument and the addition of the `--consensus.signing-key` argument. Once your node is up, it may not start syncing immediately. This is because your node might not be part of the active set. In most cases, your validator will enter the active set in under 6 hours after the on-chain addition of the validator identity. From a050b9ac3400971cc1fd68ebecb28ba298cef987 Mon Sep 17 00:00:00 2001 From: Paradigm AI Date: Mon, 20 Apr 2026 06:11:09 +0000 Subject: [PATCH 10/54] =?UTF-8?q?docs:=20update=20signing=20share=20recove?= =?UTF-8?q?ry=20=E2=80=94=20node=20recovers=20from=20network=20automatical?= =?UTF-8?q?ly?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Amp Amp-Thread-ID: https://ampcode.com/threads/T-019da962-06e0-7083-806a-3d46ee63e7b1 --- src/pages/guide/node/validator-keys.mdx | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-) diff --git a/src/pages/guide/node/validator-keys.mdx b/src/pages/guide/node/validator-keys.mdx index d814dbe4..4ab9ef28 100644 --- a/src/pages/guide/node/validator-keys.mdx +++ b/src/pages/guide/node/validator-keys.mdx @@ -42,13 +42,7 @@ The ed25519 signing key can be rotated while preserving your validator index and ## Signing share recovery -:::danger -**You cannot reset a validator's data and continue with the same identity.** Doing so risks inconsistent voting, which can cause irrecoverable network safety failures. -::: - -If you lose your signing share (stored in `/consensus/`), you will need to rotate to a new validator identity. This requires coordinating with the Tempo team to deactivate your old identity and register a new one. - -We're planning to release a high-availability feature that allows storing consensus data in an external database, which will enable signing share recovery without the need for key rotation. +If the signing share is lost — for example by deleting `/consensus` — the node will automatically recover its share from the network when it restarts. ## ValidatorConfig V2 precompile From fd88c5bf998fc24c985a23e5d05ef60b673f2c06 Mon Sep 17 00:00:00 2001 From: Paradigm AI Date: Mon, 20 Apr 2026 06:25:12 +0000 Subject: [PATCH 11/54] docs: rename 'checking validator state' to 'checking validator status' - Rename page and file from validator-state to validator-status - Update 'Look up your validator' to use tempo consensus validator - Add example JSON output based on IValidatorConfig::Validator struct - Link pubkey param to initial setup and managing keys pages - Update all cross-references across docs Co-authored-by: Amp Amp-Thread-ID: https://ampcode.com/threads/T-019da962-06e0-7083-806a-3d46ee63e7b1 --- src/pages/guide/node/validator-lifecycle.mdx | 4 +- src/pages/guide/node/validator-setup.mdx | 2 +- ...lidator-state.mdx => validator-status.mdx} | 37 ++++++++++++------- .../guide/node/validator-troubleshooting.mdx | 2 +- src/pages/guide/node/validator.mdx | 2 +- vocs.config.ts | 4 +- 6 files changed, 30 insertions(+), 21 deletions(-) rename src/pages/guide/node/{validator-state.mdx => validator-status.mdx} (81%) diff --git a/src/pages/guide/node/validator-lifecycle.mdx b/src/pages/guide/node/validator-lifecycle.mdx index 8a7ee71f..62b5a854 100644 --- a/src/pages/guide/node/validator-lifecycle.mdx +++ b/src/pages/guide/node/validator-lifecycle.mdx @@ -64,7 +64,7 @@ If self-service rotation is not yet enabled for your validator, use `tempo conse Rotation preserves your validator index and active validator count. The old entry is appended to history as deactivated, and the entry at your index is updated in place. You must use a different ingress address (changing the port is sufficient). ::: -After rotation, your validator goes through the [standard state transitions](/guide/node/validator-state#state-transitions) with the new identity. +After rotation, your validator goes through the [standard state transitions](/guide/node/validator-status#state-transitions) with the new identity. :::danger[Do not shut down the old validator immediately] After rotation, the rotated-out validator is still a member of the current committee until the next successful DKG ceremony completes. Shutting it down early will degrade network liveness. @@ -165,4 +165,4 @@ tempo consensus deactivate-validator
\ ``` :::: -Keep your node running after deactivation. Use [validators-info](/guide/node/validator-state#querying-on-chain-state) to check when your validator is no longer listed — it is safe to shut down the node at that point. +Keep your node running after deactivation. Use [validators-info](/guide/node/validator-status#querying-on-chain-state) to check when your validator is no longer listed — it is safe to shut down the node at that point. diff --git a/src/pages/guide/node/validator-setup.mdx b/src/pages/guide/node/validator-setup.mdx index f4cc4769..ebef87a2 100644 --- a/src/pages/guide/node/validator-setup.mdx +++ b/src/pages/guide/node/validator-setup.mdx @@ -79,7 +79,7 @@ Provide the following values along with the signature to the Tempo team: | **Fee recipient** | Ethereum address (`0x…`) | The address that receives transaction fees from blocks your validator proposes. If you are not prepared to accept fees, use `0x0000000000000000000000000000000000000000`. | | **Signature** | `0x`-prefixed hex | The ed25519 signature proving you control the signing key (from [Step 2](#step-2-create-the-add-validator-signature)). | -Once the Tempo team adds your validator on-chain, it will enter the active set in the [next epoch](/guide/node/validator-state#state-transitions). +Once the Tempo team adds your validator on-chain, it will enter the active set in the [next epoch](/guide/node/validator-status#state-transitions). ## Running the validator diff --git a/src/pages/guide/node/validator-state.mdx b/src/pages/guide/node/validator-status.mdx similarity index 81% rename from src/pages/guide/node/validator-state.mdx rename to src/pages/guide/node/validator-status.mdx index 51d791f6..e66691e0 100644 --- a/src/pages/guide/node/validator-state.mdx +++ b/src/pages/guide/node/validator-status.mdx @@ -1,9 +1,9 @@ --- -title: Checking validator state -description: Understand validator state transitions, check your validator's participation status, and query on-chain state. +title: Checking validator status +description: Understand validator state transitions, check your validator's participation status, and query on-chain status. --- -# Checking validator state +# Checking validator status Your validator moves through different states after registration. Understanding these states helps you verify that your node is healthy and participating in consensus. @@ -94,26 +94,35 @@ This returns the committee of the current epoch (validators with `in_committee = ### Look up your validator -You can query your validator by ethereum address, ed25519 public key, or index: +You can query a single validator by its consensus public key (the ed25519 key generated during [initial setup](/guide/node/validator-setup#step-1-generate-a-signing-keypair) — see also [Managing validator keys](/guide/node/validator-keys#generating-a-signing-key)): ::::code-group ```bash [Mainnet] -tempo consensus validator
--rpc-url https://rpc.tempo.xyz +tempo consensus validator --rpc-url https://rpc.tempo.xyz ``` ```bash [Testnet] -tempo consensus validator
--rpc-url https://rpc.testnet.tempo.xyz +tempo consensus validator --rpc-url https://rpc.testnet.tempo.xyz ``` :::: -The returned `Validator` struct fields are: +Example output: + +```json +{ + "publicKey": "0x1a2b3c4d5e6f7a8b9c0d1e2f3a4b5c6d7e8f9a0b1c2d3e4f5a6b7c8d9e0f1a2b", + "active": true, + "index": 5, + "validatorAddress": "0x1234567890abcdef1234567890abcdef12345678", + "inboundAddress": "203.0.113.10:9000", + "outboundAddress": "203.0.113.10:9001" +} +``` | Field | Type | Description | |-------|------|-------------| -| `publicKey` | `bytes32` | Ed25519 communication public key | +| `publicKey` | `bytes32` | Ed25519 consensus public key | +| `active` | `bool` | Whether the validator is active in the current contract state | +| `index` | `uint64` | Index of the validator (constant under rotation) | | `validatorAddress` | `address` | Validator control address | -| `ingress` | `string` | Inbound address (`IP:port`) | -| `egress` | `string` | Outbound address (`IP`) | -| `index` | `uint64` | Index of the validator (constant under rotation)| -| `addedAtHeight` | `uint64` | Block height when added | -| `deactivatedAtHeight` | `uint64` | Block height when deactivated (`0` = active) | -| `feeRecipient` | `address` | Address that receives block proposal fees | +| `inboundAddress` | `string` | Inbound address (`IP:port`) for incoming connections | +| `outboundAddress` | `string` | Outbound IP address for firewall whitelisting | diff --git a/src/pages/guide/node/validator-troubleshooting.mdx b/src/pages/guide/node/validator-troubleshooting.mdx index 130dbe66..4726647b 100644 --- a/src/pages/guide/node/validator-troubleshooting.mdx +++ b/src/pages/guide/node/validator-troubleshooting.mdx @@ -14,7 +14,7 @@ INFO handle_propose{epoch=18 view=387213 parent.view=387212 parent.digest=0x4388 ``` If you do not see logs like these: -- Check that your validator is part of the active set, and is both [a player and a dealer](/guide/node/validator-state#dealer-epoch-e3). +- Check that your validator is part of the active set, and is both [a player and a dealer](/guide/node/validator-status#dealer-epoch-e3). - Check that your node is synced up to the [latest block height](https://explorer.tempo.xyz/). - Check that your outgoing IP address matches the one whitelisted on the validator smart contract: diff --git a/src/pages/guide/node/validator.mdx b/src/pages/guide/node/validator.mdx index cd5b519f..a70fa81a 100644 --- a/src/pages/guide/node/validator.mdx +++ b/src/pages/guide/node/validator.mdx @@ -13,7 +13,7 @@ The active validator set is currently permissioned. If you are interested in bec ## Guides - **[Initial setup](/guide/node/validator-setup)** — Register with the Tempo team, generate your signing key, download a snapshot, and start your validator for the first time. -- **[Checking validator state](/guide/node/validator-state)** — Understand state transitions, check participation via metrics, and query on-chain status. +- **[Checking validator status](/guide/node/validator-status)** — Understand state transitions, check participation via metrics, and query on-chain status. - **[Controlling validator lifecycle](/guide/node/validator-lifecycle)** — Start, stop, register, rotate, deactivate, and transfer ownership of your validator. - **[Managing validator keys](/guide/node/validator-keys)** — Key hierarchy, generation, rotation, and signing share recovery. - **[Observing a validator](/guide/node/validator-monitoring)** — Monitor consensus and execution health, metrics glossary, Grafana dashboards, and log management. diff --git a/vocs.config.ts b/vocs.config.ts index 61455749..d92e1618 100644 --- a/vocs.config.ts +++ b/vocs.config.ts @@ -819,8 +819,8 @@ export default defineConfig({ link: '/guide/node/validator-setup', }, { - text: 'Checking validator state', - link: '/guide/node/validator-state', + text: 'Checking validator status', + link: '/guide/node/validator-status', }, { text: 'Controlling validator lifecycle', From 29960deacc86dcef71ba4b3ac31df7bcef1f7afd Mon Sep 17 00:00:00 2001 From: Paradigm AI Date: Mon, 20 Apr 2026 06:28:19 +0000 Subject: [PATCH 12/54] docs: fix validator lookup example to match ValidatorInfoOutput/ValidatorEntry structs Example JSON now matches the actual serde output from tempo_cmd.rs: - Top-level: current_epoch, current_height, last_boundary, epoch_length, is_next_full_dkg, next_full_dkg_epoch - Per-validator: onchain_address, public_key, inbound_address, outbound_address, active, is_player, is_dealer Co-authored-by: Amp Amp-Thread-ID: https://ampcode.com/threads/T-019da962-06e0-7083-806a-3d46ee63e7b1 --- src/pages/guide/node/validator-status.mdx | 53 +++++++++++++++++------ 1 file changed, 39 insertions(+), 14 deletions(-) diff --git a/src/pages/guide/node/validator-status.mdx b/src/pages/guide/node/validator-status.mdx index e66691e0..42cb97d6 100644 --- a/src/pages/guide/node/validator-status.mdx +++ b/src/pages/guide/node/validator-status.mdx @@ -109,20 +109,45 @@ Example output: ```json { - "publicKey": "0x1a2b3c4d5e6f7a8b9c0d1e2f3a4b5c6d7e8f9a0b1c2d3e4f5a6b7c8d9e0f1a2b", - "active": true, - "index": 5, - "validatorAddress": "0x1234567890abcdef1234567890abcdef12345678", - "inboundAddress": "203.0.113.10:9000", - "outboundAddress": "203.0.113.10:9001" + "current_epoch": 18, + "current_height": 5733680, + "last_boundary": 5732400, + "epoch_length": 1280, + "is_next_full_dkg": false, + "next_full_dkg_epoch": 25, + "validators": [ + { + "onchain_address": "0x1234567890abcdef1234567890abcdef12345678", + "public_key": "1a2b3c4d5e6f7a8b9c0d1e2f3a4b5c6d7e8f9a0b1c2d3e4f5a6b7c8d9e0f1a2b", + "inbound_address": "203.0.113.10:9000", + "outbound_address": "203.0.113.10:9001", + "active": true, + "is_player": true, + "is_dealer": true + } + ] } ``` -| Field | Type | Description | -|-------|------|-------------| -| `publicKey` | `bytes32` | Ed25519 consensus public key | -| `active` | `bool` | Whether the validator is active in the current contract state | -| `index` | `uint64` | Index of the validator (constant under rotation) | -| `validatorAddress` | `address` | Validator control address | -| `inboundAddress` | `string` | Inbound address (`IP:port`) for incoming connections | -| `outboundAddress` | `string` | Outbound IP address for firewall whitelisting | +**Top-level fields:** + +| Field | Description | +|-------|-------------| +| `current_epoch` | The epoch at the time of the query | +| `current_height` | The block height at the time of the query | +| `last_boundary` | The boundary height from which the DKG outcome was read | +| `epoch_length` | The epoch length as set in the chain spec | +| `is_next_full_dkg` | Whether the next DKG ceremony is a full DKG (new polynomial) or a reshare | +| `next_full_dkg_epoch` | The epoch at which the next full DKG ceremony will be triggered | + +**Per-validator fields:** + +| Field | Description | +|-------|-------------| +| `onchain_address` | Validator control address | +| `public_key` | Ed25519 consensus public key (hex) | +| `inbound_address` | Inbound address (`IP:port`) for incoming connections | +| `outbound_address` | Outbound IP address for firewall whitelisting | +| `active` | Whether the validator is active in the current contract state | +| `is_player` | Whether the validator is a player (receiving shares) in the current epoch | +| `is_dealer` | Whether the validator is a dealer (distributing shares) in the current epoch | From 29180c33353fc9319c83834709b09f46c45ed52c Mon Sep 17 00:00:00 2001 From: Paradigm AI Date: Mon, 20 Apr 2026 06:39:42 +0000 Subject: [PATCH 13/54] docs: move single validator lookup to top of status page, remove validators-info Co-authored-by: Amp Amp-Thread-ID: https://ampcode.com/threads/T-019da962-06e0-7083-806a-3d46ee63e7b1 --- src/pages/guide/node/validator-lifecycle.mdx | 2 +- src/pages/guide/node/validator-status.mdx | 133 +++++++++---------- 2 files changed, 61 insertions(+), 74 deletions(-) diff --git a/src/pages/guide/node/validator-lifecycle.mdx b/src/pages/guide/node/validator-lifecycle.mdx index 62b5a854..12ae895a 100644 --- a/src/pages/guide/node/validator-lifecycle.mdx +++ b/src/pages/guide/node/validator-lifecycle.mdx @@ -165,4 +165,4 @@ tempo consensus deactivate-validator
\ ``` :::: -Keep your node running after deactivation. Use [validators-info](/guide/node/validator-status#querying-on-chain-state) to check when your validator is no longer listed — it is safe to shut down the node at that point. +Keep your node running after deactivation. Use [validator lookup](/guide/node/validator-status#look-up-your-validator) to check when your validator is no longer listed — it is safe to shut down the node at that point. diff --git a/src/pages/guide/node/validator-status.mdx b/src/pages/guide/node/validator-status.mdx index 42cb97d6..a9258b89 100644 --- a/src/pages/guide/node/validator-status.mdx +++ b/src/pages/guide/node/validator-status.mdx @@ -11,6 +11,66 @@ Your validator moves through different states after registration. Understanding **You cannot reset a validator's data and continue with the same identity.** Doing so risks inconsistent voting, which can cause irrecoverable network safety failures. If you need to reset your validator's data, you must rotate to a new validator identity — see [Signing key rotation](/guide/node/validator-keys#signing-key-rotation). ::: +## Look up your validator + +You can query a single validator by its consensus public key (the ed25519 key generated during [initial setup](/guide/node/validator-setup#step-1-generate-a-signing-keypair) — see also [Managing validator keys](/guide/node/validator-keys#generating-a-signing-key)): + +::::code-group +```bash [Mainnet] +tempo consensus validator --rpc-url https://rpc.tempo.xyz +``` +```bash [Testnet] +tempo consensus validator --rpc-url https://rpc.testnet.tempo.xyz +``` +:::: + +Example output: + +```json +{ + "current_epoch": 18, + "current_height": 5733680, + "last_boundary": 5732400, + "epoch_length": 1280, + "is_next_full_dkg": false, + "next_full_dkg_epoch": 25, + "validators": [ + { + "onchain_address": "0x1234567890abcdef1234567890abcdef12345678", + "public_key": "1a2b3c4d5e6f7a8b9c0d1e2f3a4b5c6d7e8f9a0b1c2d3e4f5a6b7c8d9e0f1a2b", + "inbound_address": "203.0.113.10:9000", + "outbound_address": "203.0.113.10:9001", + "active": true, + "is_player": true, + "is_dealer": true + } + ] +} +``` + +**Top-level fields:** + +| Field | Description | +|-------|-------------| +| `current_epoch` | The epoch at the time of the query | +| `current_height` | The block height at the time of the query | +| `last_boundary` | The boundary height from which the DKG outcome was read | +| `epoch_length` | The epoch length as set in the chain spec | +| `is_next_full_dkg` | Whether the next DKG ceremony is a full DKG (new polynomial) or a reshare | +| `next_full_dkg_epoch` | The epoch at which the next full DKG ceremony will be triggered | + +**Per-validator fields:** + +| Field | Description | +|-------|-------------| +| `onchain_address` | Validator control address | +| `public_key` | Ed25519 consensus public key (hex) | +| `inbound_address` | Inbound address (`IP:port`) for incoming connections | +| `outbound_address` | Outbound IP address for firewall whitelisting | +| `active` | Whether the validator is active in the current contract state | +| `is_player` | Whether the validator is a player (receiving shares) in the current epoch | +| `is_dealer` | Whether the validator is a dealer (distributing shares) in the current epoch | + ## State transitions Every state transition happens on epoch boundaries. Currently on mainnet and testnet, the epoch length is around 3 hours. @@ -77,77 +137,4 @@ If `how_often_dealer` or `how_often_player` is increasing, your node is actively If your validator is not connected to other peers, but at least 3 epochs have passed, check that your node is properly configured — e.g. firewall settings are open to other peers. If you have reset your validator's state, your validator might have been blocked due to double-signing a block. In that case, please reach out to the Tempo team — even with the Tempo team, resolving this requires coordinating a new validator identity. -## Querying on-chain state - -Check your validator's on-chain status using the CLI: - -::::code-group -```bash [Mainnet] -tempo consensus validators-info --rpc-url https://rpc.tempo.xyz -``` -```bash [Testnet] -tempo consensus validators-info --rpc-url https://rpc.testnet.tempo.xyz -``` -:::: - -This returns the committee of the current epoch (validators with `in_committee = true`) as well as validators that have been added but will only become active in a future epoch. Use this to verify your validator's status after registration or rotation. - -### Look up your validator - -You can query a single validator by its consensus public key (the ed25519 key generated during [initial setup](/guide/node/validator-setup#step-1-generate-a-signing-keypair) — see also [Managing validator keys](/guide/node/validator-keys#generating-a-signing-key)): - -::::code-group -```bash [Mainnet] -tempo consensus validator --rpc-url https://rpc.tempo.xyz -``` -```bash [Testnet] -tempo consensus validator --rpc-url https://rpc.testnet.tempo.xyz -``` -:::: - -Example output: -```json -{ - "current_epoch": 18, - "current_height": 5733680, - "last_boundary": 5732400, - "epoch_length": 1280, - "is_next_full_dkg": false, - "next_full_dkg_epoch": 25, - "validators": [ - { - "onchain_address": "0x1234567890abcdef1234567890abcdef12345678", - "public_key": "1a2b3c4d5e6f7a8b9c0d1e2f3a4b5c6d7e8f9a0b1c2d3e4f5a6b7c8d9e0f1a2b", - "inbound_address": "203.0.113.10:9000", - "outbound_address": "203.0.113.10:9001", - "active": true, - "is_player": true, - "is_dealer": true - } - ] -} -``` - -**Top-level fields:** - -| Field | Description | -|-------|-------------| -| `current_epoch` | The epoch at the time of the query | -| `current_height` | The block height at the time of the query | -| `last_boundary` | The boundary height from which the DKG outcome was read | -| `epoch_length` | The epoch length as set in the chain spec | -| `is_next_full_dkg` | Whether the next DKG ceremony is a full DKG (new polynomial) or a reshare | -| `next_full_dkg_epoch` | The epoch at which the next full DKG ceremony will be triggered | - -**Per-validator fields:** - -| Field | Description | -|-------|-------------| -| `onchain_address` | Validator control address | -| `public_key` | Ed25519 consensus public key (hex) | -| `inbound_address` | Inbound address (`IP:port`) for incoming connections | -| `outbound_address` | Outbound IP address for firewall whitelisting | -| `active` | Whether the validator is active in the current contract state | -| `is_player` | Whether the validator is a player (receiving shares) in the current epoch | -| `is_dealer` | Whether the validator is a dealer (distributing shares) in the current epoch | From f1d0ff23f7ace63890ec73efbbf5ea1bb875233f Mon Sep 17 00:00:00 2001 From: Paradigm AI Date: Mon, 20 Apr 2026 06:43:14 +0000 Subject: [PATCH 14/54] docs: fix validator lookup example to match IValidatorConfig::Validator struct Fields now match the on-chain Validator struct from validator_config.rs: publicKey, active, index, validatorAddress, inboundAddress, outboundAddress Co-authored-by: Amp Amp-Thread-ID: https://ampcode.com/threads/T-019da962-06e0-7083-806a-3d46ee63e7b1 --- src/pages/guide/node/validator-status.mdx | 47 ++++++----------------- 1 file changed, 11 insertions(+), 36 deletions(-) diff --git a/src/pages/guide/node/validator-status.mdx b/src/pages/guide/node/validator-status.mdx index a9258b89..9715fc93 100644 --- a/src/pages/guide/node/validator-status.mdx +++ b/src/pages/guide/node/validator-status.mdx @@ -28,48 +28,23 @@ Example output: ```json { - "current_epoch": 18, - "current_height": 5733680, - "last_boundary": 5732400, - "epoch_length": 1280, - "is_next_full_dkg": false, - "next_full_dkg_epoch": 25, - "validators": [ - { - "onchain_address": "0x1234567890abcdef1234567890abcdef12345678", - "public_key": "1a2b3c4d5e6f7a8b9c0d1e2f3a4b5c6d7e8f9a0b1c2d3e4f5a6b7c8d9e0f1a2b", - "inbound_address": "203.0.113.10:9000", - "outbound_address": "203.0.113.10:9001", - "active": true, - "is_player": true, - "is_dealer": true - } - ] + "publicKey": "0x1a2b3c4d5e6f7a8b9c0d1e2f3a4b5c6d7e8f9a0b1c2d3e4f5a6b7c8d9e0f1a2b", + "active": true, + "index": 5, + "validatorAddress": "0x1234567890abcdef1234567890abcdef12345678", + "inboundAddress": "203.0.113.10:9000", + "outboundAddress": "203.0.113.10:9001" } ``` -**Top-level fields:** - -| Field | Description | -|-------|-------------| -| `current_epoch` | The epoch at the time of the query | -| `current_height` | The block height at the time of the query | -| `last_boundary` | The boundary height from which the DKG outcome was read | -| `epoch_length` | The epoch length as set in the chain spec | -| `is_next_full_dkg` | Whether the next DKG ceremony is a full DKG (new polynomial) or a reshare | -| `next_full_dkg_epoch` | The epoch at which the next full DKG ceremony will be triggered | - -**Per-validator fields:** - | Field | Description | |-------|-------------| -| `onchain_address` | Validator control address | -| `public_key` | Ed25519 consensus public key (hex) | -| `inbound_address` | Inbound address (`IP:port`) for incoming connections | -| `outbound_address` | Outbound IP address for firewall whitelisting | +| `publicKey` | Ed25519 consensus public key (`bytes32`) | | `active` | Whether the validator is active in the current contract state | -| `is_player` | Whether the validator is a player (receiving shares) in the current epoch | -| `is_dealer` | Whether the validator is a dealer (distributing shares) in the current epoch | +| `index` | Index of the validator (constant under rotation) | +| `validatorAddress` | Validator control address | +| `inboundAddress` | Address where other validators connect to this validator (`IP:port`) | +| `outboundAddress` | Outbound IP address for firewall whitelisting | ## State transitions From ccc1b897594903d45533f947a2a091a541521fad Mon Sep 17 00:00:00 2001 From: Paradigm AI Date: Mon, 20 Apr 2026 06:48:27 +0000 Subject: [PATCH 15/54] docs: fix validator example output to match SingleValidatorOutput from 786c8ce Sourced from tempo_cmd.rs at commit 786c8ce. SingleValidatorOutput flattens ValidatorEntry fields (onchain_address, public_key, inbound_address, outbound_address, fee_recipient, index, active, is_dkg_dealer, is_dkg_player, in_committee) alongside current_epoch and current_height. Co-authored-by: Amp Amp-Thread-ID: https://ampcode.com/threads/T-019da962-06e0-7083-806a-3d46ee63e7b1 --- src/pages/guide/node/validator-status.mdx | 32 ++++++++++++++++------- 1 file changed, 22 insertions(+), 10 deletions(-) diff --git a/src/pages/guide/node/validator-status.mdx b/src/pages/guide/node/validator-status.mdx index 9715fc93..c3a76b62 100644 --- a/src/pages/guide/node/validator-status.mdx +++ b/src/pages/guide/node/validator-status.mdx @@ -28,23 +28,35 @@ Example output: ```json { - "publicKey": "0x1a2b3c4d5e6f7a8b9c0d1e2f3a4b5c6d7e8f9a0b1c2d3e4f5a6b7c8d9e0f1a2b", - "active": true, + "current_epoch": 18, + "current_height": 5733680, + "onchain_address": "0x1234567890abcdef1234567890abcdef12345678", + "public_key": "1a2b3c4d5e6f7a8b9c0d1e2f3a4b5c6d7e8f9a0b1c2d3e4f5a6b7c8d9e0f1a2b", + "inbound_address": "203.0.113.10:9000", + "outbound_address": "203.0.113.10:9001", + "fee_recipient": "0xabcdefabcdefabcdefabcdefabcdefabcdefabcd", "index": 5, - "validatorAddress": "0x1234567890abcdef1234567890abcdef12345678", - "inboundAddress": "203.0.113.10:9000", - "outboundAddress": "203.0.113.10:9001" + "active": true, + "is_dkg_dealer": true, + "is_dkg_player": true, + "in_committee": true } ``` | Field | Description | |-------|-------------| -| `publicKey` | Ed25519 consensus public key (`bytes32`) | -| `active` | Whether the validator is active in the current contract state | +| `current_epoch` | The epoch at the time of the query | +| `current_height` | The block height at the time of the query | +| `onchain_address` | Validator control address | +| `public_key` | Ed25519 consensus public key (hex) | +| `inbound_address` | Inbound address (`IP:port`) for incoming connections | +| `outbound_address` | Outbound IP address for firewall whitelisting | +| `fee_recipient` | Address that receives block proposal fees | | `index` | Index of the validator (constant under rotation) | -| `validatorAddress` | Validator control address | -| `inboundAddress` | Address where other validators connect to this validator (`IP:port`) | -| `outboundAddress` | Outbound IP address for firewall whitelisting | +| `active` | Whether the validator is active in the current contract state | +| `is_dkg_dealer` | Whether the validator is a dealer (distributing shares) in the current epoch | +| `is_dkg_player` | Whether the validator is a player (receiving shares) in the current epoch | +| `in_committee` | Whether the validator is in the committee for the current epoch | ## State transitions From 27981161e5bb49dd7d1144ba54086a503e9d28fb Mon Sep 17 00:00:00 2001 From: Paradigm AI Date: Mon, 20 Apr 2026 07:12:34 +0000 Subject: [PATCH 16/54] =?UTF-8?q?docs:=20clean=20up=20state=20transitions?= =?UTF-8?q?=20diagram=20=E2=80=94=20horizontal=20flow,=20epoch=20labels?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Amp Amp-Thread-ID: https://ampcode.com/threads/T-019da962-06e0-7083-806a-3d46ee63e7b1 --- src/pages/guide/node/validator-status.mdx | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/src/pages/guide/node/validator-status.mdx b/src/pages/guide/node/validator-status.mdx index c3a76b62..ec128db4 100644 --- a/src/pages/guide/node/validator-status.mdx +++ b/src/pages/guide/node/validator-status.mdx @@ -62,12 +62,12 @@ Example output: Every state transition happens on epoch boundaries. Currently on mainnet and testnet, the epoch length is around 3 hours. -```mermaid {style: {maxWidth: '480px', margin: '0 auto'}} -flowchart TD - A[Inactive] -->|epoch ends| B["Syncer
The validator starts syncing"] - B -->|+1 epoch| C["Syncer and Player"] - C -->|"+1 epoch, receives share"| D["Syncer
Player
Dealer"] - D -->|"once fully synced"| E["Voter ↔ Proposer"] +```mermaid {style: {maxWidth: '520px', margin: '0 auto'}} +flowchart LR + A["Inactive
Epoch E"] -->|"epoch boundary"| B["Syncer
Epoch E+1"] + B -->|"epoch boundary"| C["Player
Epoch E+2"] + C -->|"receives share"| D["Dealer
Epoch E+3"] + D -->|"fully synced"| E["Voter / Proposer"] click A "#not-a-participant-e" click B "#syncer-epoch-e1" From 2a771f5ab4c0644640bb6106a84502deb8c4e4bb Mon Sep 17 00:00:00 2001 From: Paradigm AI Date: Mon, 20 Apr 2026 07:19:26 +0000 Subject: [PATCH 17/54] =?UTF-8?q?docs:=20fix=20state=20transitions=20?= =?UTF-8?q?=E2=80=94=20syncer=20immediately=20on=20registration,=203=20sta?= =?UTF-8?q?ges?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Corrected flow: Syncer (E) → Player (E+1) → Dealer/Validator (E+2). Validator starts syncing immediately when added to on-chain contract. Co-authored-by: Amp Amp-Thread-ID: https://ampcode.com/threads/T-019da962-06e0-7083-806a-3d46ee63e7b1 --- src/pages/guide/node/validator-status.mdx | 34 ++++++++++------------- 1 file changed, 14 insertions(+), 20 deletions(-) diff --git a/src/pages/guide/node/validator-status.mdx b/src/pages/guide/node/validator-status.mdx index ec128db4..e724db1d 100644 --- a/src/pages/guide/node/validator-status.mdx +++ b/src/pages/guide/node/validator-status.mdx @@ -64,33 +64,27 @@ Every state transition happens on epoch boundaries. Currently on mainnet and tes ```mermaid {style: {maxWidth: '520px', margin: '0 auto'}} flowchart LR - A["Inactive
Epoch E"] -->|"epoch boundary"| B["Syncer
Epoch E+1"] - B -->|"epoch boundary"| C["Player
Epoch E+2"] - C -->|"receives share"| D["Dealer
Epoch E+3"] - D -->|"fully synced"| E["Voter / Proposer"] - - click A "#not-a-participant-e" - click B "#syncer-epoch-e1" - click C "#player-epoch-e2" - click D "#dealer-epoch-e3" - click E "#dealer-epoch-e3" + A["Syncer
Epoch E"] -->|"epoch boundary"| B["Player
Epoch E+1"] + B -->|"receives share"| C["Dealer / Validator
Epoch E+2"] + C -->|"fully synced"| D["Voter / Proposer"] + + click A "#syncer-epoch-e" + click B "#player-epoch-e1" + click C "#dealer--validator-epoch-e2" + click D "#dealer--validator-epoch-e2" ``` -#### Not a participant (E) +#### Syncer (epoch E) -Epoch E marks the addition of your validator to the on-chain validator configuration smart contract. Your validator isn't considered a peer by validators yet. This is because the validator hasn't been refreshed in the current epoch yet. It is normal that no height metrics progress during this period — your node has to be considered a syncer to receive blocks. +Once your validator is added to the on-chain contract, it is immediately registered on the p2p peer set and starts syncing blocks from the network. -#### Syncer (epoch E+1) +#### Player (epoch E+1) -Your validator is now considered a peer by validators. It's syncing with the network and will be considered a player in the next epoch. +Your validator is receiving consensus signing shares from dealers during the DKG ceremony. -#### Player (epoch E+2) +#### Dealer / Validator (epoch E+2) -Your validator is receiving consensus signing shares from dealers during the ceremony. - -#### Dealer (epoch E+3) - -Your validator is distributing consensus signing shares to other validators during the ceremony. Once your node is fully synced up, it will also be able to propose blocks and vote for other validators' proposed blocks. +Your validator is distributing consensus signing shares to other validators during the ceremony. Once your node is fully synced, it will also be able to propose blocks and vote on other validators' proposals. :::info The on-chain contract represents which validators should eventually be members of the committee, not which ones currently are. Every epoch, the Tempo network performs a distributed key generation ceremony, distributing keys to validators that should be committee members in the next epoch. So adding, deactivating, or rotating validator entries in epoch `E` can only be taken into account for the next DKG ceremony that runs during epoch `E+1`, and take effect in epoch `E+2`. From 8d30cd77bf90139e0942df3ed488a7c07d4e008b Mon Sep 17 00:00:00 2001 From: Paradigm AI Date: Mon, 20 Apr 2026 07:28:31 +0000 Subject: [PATCH 18/54] docs: remove misplaced data reset warning from validator status page Co-authored-by: Amp Amp-Thread-ID: https://ampcode.com/threads/T-019da962-06e0-7083-806a-3d46ee63e7b1 --- src/pages/guide/node/validator-status.mdx | 4 ---- 1 file changed, 4 deletions(-) diff --git a/src/pages/guide/node/validator-status.mdx b/src/pages/guide/node/validator-status.mdx index e724db1d..93e72be3 100644 --- a/src/pages/guide/node/validator-status.mdx +++ b/src/pages/guide/node/validator-status.mdx @@ -7,10 +7,6 @@ description: Understand validator state transitions, check your validator's part Your validator moves through different states after registration. Understanding these states helps you verify that your node is healthy and participating in consensus. -:::danger -**You cannot reset a validator's data and continue with the same identity.** Doing so risks inconsistent voting, which can cause irrecoverable network safety failures. If you need to reset your validator's data, you must rotate to a new validator identity — see [Signing key rotation](/guide/node/validator-keys#signing-key-rotation). -::: - ## Look up your validator You can query a single validator by its consensus public key (the ed25519 key generated during [initial setup](/guide/node/validator-setup#step-1-generate-a-signing-keypair) — see also [Managing validator keys](/guide/node/validator-keys#generating-a-signing-key)): From 1a76473c01fb618d6a10b9aa519105004e9bd45f Mon Sep 17 00:00:00 2001 From: Paradigm AI Date: Mon, 20 Apr 2026 07:35:00 +0000 Subject: [PATCH 19/54] docs: add entry/exit elements to state transitions diagram Co-authored-by: Amp Amp-Thread-ID: https://ampcode.com/threads/T-019da962-06e0-7083-806a-3d46ee63e7b1 --- src/pages/guide/node/validator-status.mdx | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/src/pages/guide/node/validator-status.mdx b/src/pages/guide/node/validator-status.mdx index 93e72be3..95beaaa6 100644 --- a/src/pages/guide/node/validator-status.mdx +++ b/src/pages/guide/node/validator-status.mdx @@ -58,11 +58,13 @@ Example output: Every state transition happens on epoch boundaries. Currently on mainnet and testnet, the epoch length is around 3 hours. -```mermaid {style: {maxWidth: '520px', margin: '0 auto'}} +```mermaid {style: {maxWidth: '600px', margin: '0 auto'}} flowchart LR - A["Syncer
Epoch E"] -->|"epoch boundary"| B["Player
Epoch E+1"] + Start(["Add / Rotate in"]) --> A["Syncer
Epoch E"] + A -->|"epoch boundary"| B["Player
Epoch E+1"] B -->|"receives share"| C["Dealer / Validator
Epoch E+2"] C -->|"fully synced"| D["Voter / Proposer"] + D --> End(["Deactivate / Rotate out"]) click A "#syncer-epoch-e" click B "#player-epoch-e1" From c65517e89fcbc156ec5d5ff5fac186614c39d3a4 Mon Sep 17 00:00:00 2001 From: Paradigm AI Date: Mon, 20 Apr 2026 07:42:55 +0000 Subject: [PATCH 20/54] docs: switch state transitions to block-beta diagram for cleaner layout Co-authored-by: Amp Amp-Thread-ID: https://ampcode.com/threads/T-019da962-06e0-7083-806a-3d46ee63e7b1 --- src/pages/guide/node/validator-status.mdx | 33 +++++++++++++++-------- 1 file changed, 22 insertions(+), 11 deletions(-) diff --git a/src/pages/guide/node/validator-status.mdx b/src/pages/guide/node/validator-status.mdx index 95beaaa6..1f62cb14 100644 --- a/src/pages/guide/node/validator-status.mdx +++ b/src/pages/guide/node/validator-status.mdx @@ -59,17 +59,28 @@ Example output: Every state transition happens on epoch boundaries. Currently on mainnet and testnet, the epoch length is around 3 hours. ```mermaid {style: {maxWidth: '600px', margin: '0 auto'}} -flowchart LR - Start(["Add / Rotate in"]) --> A["Syncer
Epoch E"] - A -->|"epoch boundary"| B["Player
Epoch E+1"] - B -->|"receives share"| C["Dealer / Validator
Epoch E+2"] - C -->|"fully synced"| D["Voter / Proposer"] - D --> End(["Deactivate / Rotate out"]) - - click A "#syncer-epoch-e" - click B "#player-epoch-e1" - click C "#dealer--validator-epoch-e2" - click D "#dealer--validator-epoch-e2" +block-beta + columns 3 + start(["Add / Rotate in"]):3 + space + A["Syncer — Epoch E"]:1 + space + space + B["Player — Epoch E+1"]:1 + space + space + C["Dealer / Validator — Epoch E+2"]:1 + space + space + D["Voter / Proposer"]:1 + space + end_(["Deactivate / Rotate out"]):3 + + start --> A + A --> B + B --> C + C --> D + D --> end_ ``` #### Syncer (epoch E) From 1bc3865a6dc4ada88dec03acae6dd5c029659fdc Mon Sep 17 00:00:00 2001 From: Paradigm AI Date: Mon, 20 Apr 2026 07:45:27 +0000 Subject: [PATCH 21/54] docs: use 'block' instead of 'block-beta' for mermaid diagram Co-authored-by: Amp Amp-Thread-ID: https://ampcode.com/threads/T-019da962-06e0-7083-806a-3d46ee63e7b1 --- src/pages/guide/node/validator-status.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/pages/guide/node/validator-status.mdx b/src/pages/guide/node/validator-status.mdx index 1f62cb14..6d9eae7a 100644 --- a/src/pages/guide/node/validator-status.mdx +++ b/src/pages/guide/node/validator-status.mdx @@ -59,7 +59,7 @@ Example output: Every state transition happens on epoch boundaries. Currently on mainnet and testnet, the epoch length is around 3 hours. ```mermaid {style: {maxWidth: '600px', margin: '0 auto'}} -block-beta +block columns 3 start(["Add / Rotate in"]):3 space From 44b667ff7fbb80fc0cab092fe26eb63c9380762f Mon Sep 17 00:00:00 2001 From: Paradigm AI Date: Mon, 20 Apr 2026 07:53:59 +0000 Subject: [PATCH 22/54] =?UTF-8?q?docs:=20fix=20broken=20diagram=20?= =?UTF-8?q?=E2=80=94=20use=20flowchart=20TD=20instead=20of=20unsupported?= =?UTF-8?q?=20block?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Amp Amp-Thread-ID: https://ampcode.com/threads/T-019da962-06e0-7083-806a-3d46ee63e7b1 --- src/pages/guide/node/validator-status.mdx | 33 +++++++---------------- 1 file changed, 10 insertions(+), 23 deletions(-) diff --git a/src/pages/guide/node/validator-status.mdx b/src/pages/guide/node/validator-status.mdx index 6d9eae7a..da025a38 100644 --- a/src/pages/guide/node/validator-status.mdx +++ b/src/pages/guide/node/validator-status.mdx @@ -58,29 +58,16 @@ Example output: Every state transition happens on epoch boundaries. Currently on mainnet and testnet, the epoch length is around 3 hours. -```mermaid {style: {maxWidth: '600px', margin: '0 auto'}} -block - columns 3 - start(["Add / Rotate in"]):3 - space - A["Syncer — Epoch E"]:1 - space - space - B["Player — Epoch E+1"]:1 - space - space - C["Dealer / Validator — Epoch E+2"]:1 - space - space - D["Voter / Proposer"]:1 - space - end_(["Deactivate / Rotate out"]):3 - - start --> A - A --> B - B --> C - C --> D - D --> end_ +```mermaid {style: {maxWidth: '320px', margin: '0 auto'}} +flowchart TD + Start(["Add / Rotate in"]) + A["Syncer — Epoch E"] + B["Player — Epoch E+1"] + C["Dealer / Validator — Epoch E+2"] + D["Voter / Proposer"] + End(["Deactivate / Rotate out"]) + + Start --> A --> B --> C --> D --> End ``` #### Syncer (epoch E) From b7bff0d77a13593e6a926b479b8ce35c42c07167 Mon Sep 17 00:00:00 2001 From: Paradigm AI Date: Mon, 20 Apr 2026 07:55:32 +0000 Subject: [PATCH 23/54] docs: revert to block-beta diagram Co-authored-by: Amp Amp-Thread-ID: https://ampcode.com/threads/T-019da962-06e0-7083-806a-3d46ee63e7b1 --- src/pages/guide/node/validator-status.mdx | 33 ++++++++++++++++------- 1 file changed, 23 insertions(+), 10 deletions(-) diff --git a/src/pages/guide/node/validator-status.mdx b/src/pages/guide/node/validator-status.mdx index da025a38..1f62cb14 100644 --- a/src/pages/guide/node/validator-status.mdx +++ b/src/pages/guide/node/validator-status.mdx @@ -58,16 +58,29 @@ Example output: Every state transition happens on epoch boundaries. Currently on mainnet and testnet, the epoch length is around 3 hours. -```mermaid {style: {maxWidth: '320px', margin: '0 auto'}} -flowchart TD - Start(["Add / Rotate in"]) - A["Syncer — Epoch E"] - B["Player — Epoch E+1"] - C["Dealer / Validator — Epoch E+2"] - D["Voter / Proposer"] - End(["Deactivate / Rotate out"]) - - Start --> A --> B --> C --> D --> End +```mermaid {style: {maxWidth: '600px', margin: '0 auto'}} +block-beta + columns 3 + start(["Add / Rotate in"]):3 + space + A["Syncer — Epoch E"]:1 + space + space + B["Player — Epoch E+1"]:1 + space + space + C["Dealer / Validator — Epoch E+2"]:1 + space + space + D["Voter / Proposer"]:1 + space + end_(["Deactivate / Rotate out"]):3 + + start --> A + A --> B + B --> C + C --> D + D --> end_ ``` #### Syncer (epoch E) From 895f5d24ff2aad8a542ae02d048a71f5b30736ab Mon Sep 17 00:00:00 2001 From: Paradigm AI Date: Mon, 20 Apr 2026 08:02:32 +0000 Subject: [PATCH 24/54] =?UTF-8?q?docs:=20replace=20block-beta=20with=20flo?= =?UTF-8?q?wchart=20TD=20=E2=80=94=20block-beta=20crashes=20in=20Vocs?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit block-beta mermaid diagrams cause a circular JSON serialization error in the docs site's React/Vocs rendering pipeline. All other diagrams in the codebase use flowchart or sequenceDiagram successfully. Co-authored-by: Amp Amp-Thread-ID: https://ampcode.com/threads/T-019da962-06e0-7083-806a-3d46ee63e7b1 --- src/pages/guide/node/validator-status.mdx | 30 ++++++----------------- 1 file changed, 7 insertions(+), 23 deletions(-) diff --git a/src/pages/guide/node/validator-status.mdx b/src/pages/guide/node/validator-status.mdx index 1f62cb14..394860c8 100644 --- a/src/pages/guide/node/validator-status.mdx +++ b/src/pages/guide/node/validator-status.mdx @@ -58,29 +58,13 @@ Example output: Every state transition happens on epoch boundaries. Currently on mainnet and testnet, the epoch length is around 3 hours. -```mermaid {style: {maxWidth: '600px', margin: '0 auto'}} -block-beta - columns 3 - start(["Add / Rotate in"]):3 - space - A["Syncer — Epoch E"]:1 - space - space - B["Player — Epoch E+1"]:1 - space - space - C["Dealer / Validator — Epoch E+2"]:1 - space - space - D["Voter / Proposer"]:1 - space - end_(["Deactivate / Rotate out"]):3 - - start --> A - A --> B - B --> C - C --> D - D --> end_ +```mermaid {style: {maxWidth: '360px', margin: '0 auto'}} +flowchart TD + Start(["Add / Rotate in"]) --> A["Syncer — Epoch E"] + A --> B["Player — Epoch E+1"] + B --> C["Dealer / Validator — Epoch E+2"] + C --> D["Voter / Proposer"] + D --> End(["Deactivate / Rotate out"]) ``` #### Syncer (epoch E) From 6b82f790814a1648b6f138ce8e56f421d32e0740 Mon Sep 17 00:00:00 2001 From: Emma Jamieson-Hoare Date: Tue, 21 Apr 2026 16:01:39 +0100 Subject: [PATCH 25/54] Apply suggestion from @emmajam --- src/pages/cli/node.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/pages/cli/node.mdx b/src/pages/cli/node.mdx index 0600ae69..d61992c4 100644 --- a/src/pages/cli/node.mdx +++ b/src/pages/cli/node.mdx @@ -37,7 +37,7 @@ Flags grouped by function: | `--consensus.datadir ` | Separate volume for consensus data | :::warning -`--consensus.fee-recipient` is deprecated as of `v1.5.2` and will be removed in an upcoming release. See [Update the fee recipient](/guide/node/validator-lifecycle#update-the-fee-recipient). +`--consensus.fee-recipient` is deprecated as of `v1.5.2` and will be removed in an upcoming release. See [updating the fee recipient](/guide/node/validator-lifecycle#update-the-fee-recipient). ::: ### Observability From 455032fa43b4bc691e9dc0a1f16842fa77115bd4 Mon Sep 17 00:00:00 2001 From: Emma Jamieson-Hoare Date: Tue, 21 Apr 2026 16:01:47 +0100 Subject: [PATCH 26/54] Apply suggestion from @emmajam --- src/pages/guide/node/network-upgrades.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/pages/guide/node/network-upgrades.mdx b/src/pages/guide/node/network-upgrades.mdx index 1be7a118..c5df136a 100644 --- a/src/pages/guide/node/network-upgrades.mdx +++ b/src/pages/guide/node/network-upgrades.mdx @@ -27,7 +27,7 @@ For detailed release notes and binaries, see the [Changelog](/changelog). | [v1.2.0](https://github.com/tempoxyz/tempo/releases/tag/v1.2.0) | Feb 13, 2026 | Mainnet only | Fixes validation bug rejecting transactions with gas limits above ~16.7M, blocking large contract deployments | Required | :::warning -`--consensus.fee-recipient` is deprecated as of `v1.5.2` and will be removed in an upcoming release. Validators should migrate fee-recipient management to [Update the fee recipient](/guide/node/validator-lifecycle#update-the-fee-recipient). +`--consensus.fee-recipient` is deprecated as of `v1.5.2` and will be removed in an upcoming release. Validators should migrate fee-recipient management to [update the fee recipient](/guide/node/validator-lifecycle#update-the-fee-recipient). ::: --- From b2c4e1eda344b45b7c761b0d29fa65d8bbec5600 Mon Sep 17 00:00:00 2001 From: Emma Jamieson-Hoare Date: Tue, 21 Apr 2026 16:05:26 +0100 Subject: [PATCH 27/54] Apply suggestion from @emmajam --- src/pages/guide/node/validator-troubleshooting.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/pages/guide/node/validator-troubleshooting.mdx b/src/pages/guide/node/validator-troubleshooting.mdx index 4726647b..0ef06c82 100644 --- a/src/pages/guide/node/validator-troubleshooting.mdx +++ b/src/pages/guide/node/validator-troubleshooting.mdx @@ -14,7 +14,7 @@ INFO handle_propose{epoch=18 view=387213 parent.view=387212 parent.digest=0x4388 ``` If you do not see logs like these: -- Check that your validator is part of the active set, and is both [a player and a dealer](/guide/node/validator-status#dealer-epoch-e3). +- Check that your validator is part of the active set, and is both [a player and a dealer](/guide/node/validator-status). - Check that your node is synced up to the [latest block height](https://explorer.tempo.xyz/). - Check that your outgoing IP address matches the one whitelisted on the validator smart contract: From b8eda8a8a2d4b6eff32cc73ce5821662e803932d Mon Sep 17 00:00:00 2001 From: Emma Jamieson-Hoare Date: Tue, 21 Apr 2026 16:17:18 +0100 Subject: [PATCH 28/54] docs: add network upgrade cadence guidelines Amp-Thread-ID: https://ampcode.com/threads/T-019db09b-38c8-7044-8f6c-767dfd4edb00 Co-authored-by: Amp --- src/pages/guide/node/network-upgrades.mdx | 32 +++++++++++++++++++++++ 1 file changed, 32 insertions(+) diff --git a/src/pages/guide/node/network-upgrades.mdx b/src/pages/guide/node/network-upgrades.mdx index c5df136a..54a5f6c2 100644 --- a/src/pages/guide/node/network-upgrades.mdx +++ b/src/pages/guide/node/network-upgrades.mdx @@ -11,6 +11,38 @@ Tempo uses scheduled network upgrades to introduce protocol changes. Each upgrad For detailed release notes and binaries, see the [Changelog](/changelog). +## Upgrade Cadence + +Tempo targets a **monthly network upgrade cadence**. Upgrades bundle protocol changes (TIPs) into named hardforks (T1, T2, T3, …) and follow a fixed rollout process: + +1. **Testnet first** — every upgrade activates on the Moderato testnet before mainnet. +2. **Mainnet follows** — mainnet activation is scheduled **approximately one week** after testnet, giving operators and integrators time to validate. +3. **Patch releases between upgrades** — security fixes and non-consensus improvements ship as patch releases (e.g. v1.5.1, v1.5.2) and do not require a hardfork. + +### How operators are notified + +- **Release announcements** are posted to the [Tempo GitHub releases](https://github.com/tempoxyz/tempo/releases) page with full changelogs. +- **Validator channels** — the Tempo team shares activation timestamps and migration checklists in dedicated operator channels ahead of each upgrade. +- **This page** is updated with dates, TIP references, and priority badges as soon as an upgrade is scheduled. + +### Upgrade priority levels + +| Badge | Meaning | +|-------|---------| +| Required | Consensus-breaking change — nodes **must** upgrade before the activation block or they will fork off the network. | +| Recommended | Non-consensus improvement (performance, hardening, bug fixes). Nodes will continue to function without upgrading, but the update is strongly encouraged. | +| RPC only | Only affects RPC-serving nodes (e.g. security patches for public endpoints). Validators without public RPC exposure can skip. | + +### Operator checklist for each upgrade + +1. Watch for the release announcement and review the changelog. +2. Upgrade your **testnet node first** and confirm it syncs past the activation block. +3. Verify your application or integration works against the upgraded testnet. +4. Upgrade your **mainnet node** before the mainnet activation timestamp. +5. Monitor logs after activation to confirm normal operation. + +--- + ## Node Operator Updates | Release | Date | Network | Description | Priority | From 52a94c97457d855af079babdefe97d06aef92f43 Mon Sep 17 00:00:00 2001 From: Emma Jamieson-Hoare Date: Tue, 21 Apr 2026 16:20:22 +0100 Subject: [PATCH 29/54] docs: add dedicated upgrade cadence page, revert network-upgrades changes Amp-Thread-ID: https://ampcode.com/threads/T-019db09b-38c8-7044-8f6c-767dfd4edb00 Co-authored-by: Amp --- src/pages/guide/node/network-upgrades.mdx | 32 ----------- src/pages/guide/node/upgrade-cadence.mdx | 67 +++++++++++++++++++++++ vocs.config.ts | 4 ++ 3 files changed, 71 insertions(+), 32 deletions(-) create mode 100644 src/pages/guide/node/upgrade-cadence.mdx diff --git a/src/pages/guide/node/network-upgrades.mdx b/src/pages/guide/node/network-upgrades.mdx index 54a5f6c2..c5df136a 100644 --- a/src/pages/guide/node/network-upgrades.mdx +++ b/src/pages/guide/node/network-upgrades.mdx @@ -11,38 +11,6 @@ Tempo uses scheduled network upgrades to introduce protocol changes. Each upgrad For detailed release notes and binaries, see the [Changelog](/changelog). -## Upgrade Cadence - -Tempo targets a **monthly network upgrade cadence**. Upgrades bundle protocol changes (TIPs) into named hardforks (T1, T2, T3, …) and follow a fixed rollout process: - -1. **Testnet first** — every upgrade activates on the Moderato testnet before mainnet. -2. **Mainnet follows** — mainnet activation is scheduled **approximately one week** after testnet, giving operators and integrators time to validate. -3. **Patch releases between upgrades** — security fixes and non-consensus improvements ship as patch releases (e.g. v1.5.1, v1.5.2) and do not require a hardfork. - -### How operators are notified - -- **Release announcements** are posted to the [Tempo GitHub releases](https://github.com/tempoxyz/tempo/releases) page with full changelogs. -- **Validator channels** — the Tempo team shares activation timestamps and migration checklists in dedicated operator channels ahead of each upgrade. -- **This page** is updated with dates, TIP references, and priority badges as soon as an upgrade is scheduled. - -### Upgrade priority levels - -| Badge | Meaning | -|-------|---------| -| Required | Consensus-breaking change — nodes **must** upgrade before the activation block or they will fork off the network. | -| Recommended | Non-consensus improvement (performance, hardening, bug fixes). Nodes will continue to function without upgrading, but the update is strongly encouraged. | -| RPC only | Only affects RPC-serving nodes (e.g. security patches for public endpoints). Validators without public RPC exposure can skip. | - -### Operator checklist for each upgrade - -1. Watch for the release announcement and review the changelog. -2. Upgrade your **testnet node first** and confirm it syncs past the activation block. -3. Verify your application or integration works against the upgraded testnet. -4. Upgrade your **mainnet node** before the mainnet activation timestamp. -5. Monitor logs after activation to confirm normal operation. - ---- - ## Node Operator Updates | Release | Date | Network | Description | Priority | diff --git a/src/pages/guide/node/upgrade-cadence.mdx b/src/pages/guide/node/upgrade-cadence.mdx new file mode 100644 index 00000000..9b9e2a68 --- /dev/null +++ b/src/pages/guide/node/upgrade-cadence.mdx @@ -0,0 +1,67 @@ +--- +title: Upgrade Cadence +description: How Tempo schedules and communicates network upgrades, including timelines, notification windows, and what to expect as a node operator. +--- + +import { Badge } from '../../../components/Badge' + +# Upgrade Cadence + +Tempo ships protocol upgrades on a **roughly monthly cadence**. Each upgrade bundles one or more protocol changes ([TIPs](/protocol/tips)) into a named hardfork (T1, T2, T3, …). + +## Rollout timeline + +Every upgrade follows the same two-stage rollout: + +| Stage | Lead time | Details | +|-------|-----------|---------| +| **Testnet** (Moderato) | Announced at least **3 days** before activation | The upgrade activates on the Moderato testnet first so operators and integrators can validate. | +| **Mainnet** (Presto) | Announced at least **7 days** before activation | After a successful testnet activation, the same release is scheduled for mainnet — typically one week later. | + +Operators should use the window between testnet and mainnet activation to upgrade their testnet node, verify it syncs past the activation block, and confirm their applications work against the new protocol rules. + +## Patch releases + +Between hardforks, Tempo publishes patch releases (e.g. v1.5.1, v1.5.2) for security fixes, performance improvements, and non-consensus bug fixes. Patch releases do not require a hardfork and can be adopted at your own pace, subject to the priority level. + +## Upgrade priority levels + +Each release on the [Network Upgrades and Releases](/guide/node/network-upgrades) page carries a priority badge: + +| Badge | Meaning | +|-------|---------| +| Required | Consensus-breaking change — nodes **must** upgrade before the activation block or they will fork off the network. | +| Recommended | Non-consensus improvement (performance, hardening, bug fixes). Nodes continue to function without upgrading, but the update is strongly encouraged. | +| RPC only | Only affects RPC-serving nodes (e.g. security patches for public endpoints). Validators without public RPC exposure can skip. | + +## How you will be notified + +- **GitHub releases** — every release is published to [tempoxyz/tempo releases](https://github.com/tempoxyz/tempo/releases) with a full changelog. +- **Operator channels** — the Tempo team shares activation timestamps and migration checklists in dedicated operator channels ahead of each upgrade. +- **Docs** — the [Network Upgrades and Releases](/guide/node/network-upgrades) page is updated with dates, TIP references, and priority badges as soon as an upgrade is scheduled. + +## Operator checklist + +:::::steps + +### Review the release + +Read the changelog and upgrade page to understand what is changing and whether any migration steps are required. + +### Upgrade your testnet node + +Update your Moderato node to the new release and confirm it syncs past the testnet activation block. + +### Validate your integration + +Run your application or test suite against the upgraded testnet to catch any breaking changes before mainnet. + +### Upgrade your mainnet node + +Update your mainnet node **before** the mainnet activation timestamp. + +### Monitor after activation + +Watch logs and metrics after the activation block to confirm normal operation. + +::::: diff --git a/vocs.config.ts b/vocs.config.ts index d92e1618..6f362d1d 100644 --- a/vocs.config.ts +++ b/vocs.config.ts @@ -840,6 +840,10 @@ export default defineConfig({ }, ], }, + { + text: 'Upgrade Cadence', + link: '/guide/node/upgrade-cadence', + }, { text: 'Network Upgrades and Releases', link: '/guide/node/network-upgrades', From 4488df7b2c554aaaab5a82fe056e890c31998922 Mon Sep 17 00:00:00 2001 From: Emma Jamieson-Hoare Date: Tue, 21 Apr 2026 16:24:09 +0100 Subject: [PATCH 30/54] Apply suggestion from @emmajam --- src/pages/guide/node/upgrade-cadence.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/pages/guide/node/upgrade-cadence.mdx b/src/pages/guide/node/upgrade-cadence.mdx index 9b9e2a68..2adf82d0 100644 --- a/src/pages/guide/node/upgrade-cadence.mdx +++ b/src/pages/guide/node/upgrade-cadence.mdx @@ -7,7 +7,7 @@ import { Badge } from '../../../components/Badge' # Upgrade Cadence -Tempo ships protocol upgrades on a **roughly monthly cadence**. Each upgrade bundles one or more protocol changes ([TIPs](/protocol/tips)) into a named hardfork (T1, T2, T3, …). +Tempo ships protocol upgrades on a **monthly cadence**. Each upgrade bundles one or more protocol changes ([TIPs](/protocol/tips)) into a named hardfork (T1, T2, T3, …). ## Rollout timeline From 662e4b558258f5ddef9cd119b10d47512ed47248 Mon Sep 17 00:00:00 2001 From: Emma Jamieson-Hoare Date: Tue, 21 Apr 2026 16:28:06 +0100 Subject: [PATCH 31/54] docs: add time synchronization guidance for node operators (INF-379) Amp-Thread-ID: https://ampcode.com/threads/T-019db09b-38c8-7044-8f6c-767dfd4edb00 Co-authored-by: Amp --- src/pages/guide/node/system-requirements.mdx | 33 ++++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/src/pages/guide/node/system-requirements.mdx b/src/pages/guide/node/system-requirements.mdx index b8a746b1..aacfe006 100644 --- a/src/pages/guide/node/system-requirements.mdx +++ b/src/pages/guide/node/system-requirements.mdx @@ -43,6 +43,39 @@ These dedicated servers meet or exceed the recommended specs for both RPC and va Cloud instances with network-attached storage (e.g., AWS EBS) do not provide sufficient I/O performance. Use dedicated servers or instances with local NVMe storage. ::: +## Time Synchronization + +Tempo validates that block timestamps are not in the future. If your system clock drifts even slightly, your node may reject valid blocks or produce blocks that other validators reject — leading to consensus errors and missed proposals. + +:::warning +`systemd-timesyncd` (the default on many minimal VMs) is **not sufficient**. Use `chrony` or `ntpd` for reliable sub-millisecond synchronization. +::: + +### Install and enable chrony (recommended) + +```bash +sudo apt install chrony +sudo systemctl enable --now chronyd +``` + +### Verify synchronization + +```bash +chronyc tracking +``` + +Check that **System time** offset is under a few milliseconds and **Leap status** is `Normal`. You can also verify with: + +```bash +timedatectl +``` + +Confirm `System clock synchronized: yes` and `NTP service: active`. + +### Cloud providers + +Most cloud providers (AWS, Hetzner, OVH) pre-configure NTP, but minimal VM images may ship without a proper NTP daemon. Always verify that `chrony` or `ntpd` is installed and running after provisioning a new machine. + ## Security measures We do not recommend using a cloud firewall / NAT gateway as it can introduce additional complexity and performance issues. Instead, we've implemented security measures in the networking layer of the node, including: From af4a40d22b6c090296000939fed0b6a5216bf910 Mon Sep 17 00:00:00 2001 From: Hamdi Allam Date: Tue, 21 Apr 2026 11:39:27 -0400 Subject: [PATCH 32/54] share recovery + syncer -> registration --- src/pages/guide/node/validator-keys.mdx | 2 +- src/pages/guide/node/validator-status.mdx | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/src/pages/guide/node/validator-keys.mdx b/src/pages/guide/node/validator-keys.mdx index 4ab9ef28..75b8e591 100644 --- a/src/pages/guide/node/validator-keys.mdx +++ b/src/pages/guide/node/validator-keys.mdx @@ -42,7 +42,7 @@ The ed25519 signing key can be rotated while preserving your validator index and ## Signing share recovery -If the signing share is lost — for example by deleting `/consensus` — the node will automatically recover its share from the network when it restarts. +If the signing share is lost — for example by deleting `/consensus` — the node will recover a new share in the following epochs from the network when it restarts. ## ValidatorConfig V2 precompile diff --git a/src/pages/guide/node/validator-status.mdx b/src/pages/guide/node/validator-status.mdx index 394860c8..902efca9 100644 --- a/src/pages/guide/node/validator-status.mdx +++ b/src/pages/guide/node/validator-status.mdx @@ -60,14 +60,14 @@ Every state transition happens on epoch boundaries. Currently on mainnet and tes ```mermaid {style: {maxWidth: '360px', margin: '0 auto'}} flowchart TD - Start(["Add / Rotate in"]) --> A["Syncer — Epoch E"] + Start(["Add / Rotate in"]) --> A["Registration — Epoch E"] A --> B["Player — Epoch E+1"] B --> C["Dealer / Validator — Epoch E+2"] C --> D["Voter / Proposer"] D --> End(["Deactivate / Rotate out"]) ``` -#### Syncer (epoch E) +#### Registration (epoch E) Once your validator is added to the on-chain contract, it is immediately registered on the p2p peer set and starts syncing blocks from the network. @@ -88,7 +88,7 @@ The on-chain contract represents which validators should eventually be members o Monitor these metrics to track your validator's state: ```bash -# Is your validator connected to other peers (i.e. is it considered a syncer)? +# Is your validator connected to other peers (i.e. is it registered)? # This should be >0 in at most 3 hours after your validator's addition. curl -s localhost:8002/metrics | grep consensus_network_spawner_connections From fb8f3da34514871cfdcdf96df8654a6af610c2f7 Mon Sep 17 00:00:00 2001 From: Emma Jamieson-Hoare Date: Tue, 21 Apr 2026 20:09:44 +0100 Subject: [PATCH 33/54] chore: address the comments --- src/pages/guide/node/validator-keys.mdx | 12 ++-- src/pages/guide/node/validator-lifecycle.mdx | 4 ++ src/pages/guide/node/validator-monitoring.mdx | 3 + .../guide/node/validator-troubleshooting.mdx | 55 +++++++++++++++++++ 4 files changed, 70 insertions(+), 4 deletions(-) diff --git a/src/pages/guide/node/validator-keys.mdx b/src/pages/guide/node/validator-keys.mdx index 4ab9ef28..8192f281 100644 --- a/src/pages/guide/node/validator-keys.mdx +++ b/src/pages/guide/node/validator-keys.mdx @@ -5,16 +5,20 @@ description: Generate, rotate, and recover validator signing keys and shares. # Managing validator keys -Tempo validators use two kinds of cryptographic keys. This page covers how to generate, rotate, and recover them. +Tempo validators use several keys and addresses. This page explains what each one does, how sensitive it is, and how to manage it. :::warning Never share your private signing key. Anyone with access to it can impersonate your validator. The Tempo team will never ask for your private key. Store keys securely and restrict file permissions. ::: -## Key hierarchy +## Key and address overview -* **Signing key** — an ED25519 keypair used to identify your validator amongst others. Can be rotated while preserving your validator index. -* **Signing share** — a share of a BLS12-381 keypair used to sign block notarizations and finalizations. Updated automatically every DKG ceremony (~3 hours). +| Key / Address | Type | What it does | Sensitivity | How to change | +|---|---|---|---|---| +| **Signing key** | Ed25519 keypair | Identifies your validator in the consensus protocol. Used for DKG participation, block proposals, and voting. | **Critical** — anyone with this key can impersonate your validator. | [Rotate validator identity](/guide/node/validator-lifecycle#rotate-validator-identity) | +| **Validator operator address** | Ethereum address (`0x…`) | The control address that authorizes on-chain operations: IP updates, key rotation, ownership transfer, and deactivation. | **High** — controls all validator configuration. | [Transfer validator ownership](/guide/node/validator-lifecycle#transfer-validator-ownership) | +| **Fee recipient** | Ethereum address (`0x…`) | Receives transaction fees from blocks your validator proposes. | **Low** — changing it only redirects future fee revenue, no security impact. | [Update fee recipient](/guide/node/validator-lifecycle#update-the-fee-recipient) | +| **Signing share** | BLS12-381 key share | A share of the committee's threshold signing key, used to sign block notarizations and finalizations. | **Managed automatically** — updated every DKG ceremony (~3 hours). Lost shares are recovered from the network on restart. | Automatic (see [recovery](#signing-share-recovery)) | ## Generating a signing key diff --git a/src/pages/guide/node/validator-lifecycle.mdx b/src/pages/guide/node/validator-lifecycle.mdx index 12ae895a..946348b3 100644 --- a/src/pages/guide/node/validator-lifecycle.mdx +++ b/src/pages/guide/node/validator-lifecycle.mdx @@ -29,6 +29,10 @@ The node will finish processing the current block before shutting down. Avoid us If you need to reset your validator's data, you must rotate to a new validator identity. This requires coordinating with the Tempo team to deactivate your old identity and register a new one. Don't hesitate to reach out — even with the Tempo team, this is a routine operation. +:::info +Self-service data resets are coming soon. Once available, you will be able to rotate to a new identity and reset your data without coordinating with the Tempo team. See [Rotate validator identity](#rotate-validator-identity). +::: + ## Rotate validator identity The ed25519 key can be changed while keeping your validator index stable. This is useful for key rotation or recovery without leaving and re-joining the committee. [Generate a new signing key](/guide/node/validator-keys#generating-a-signing-key) first. diff --git a/src/pages/guide/node/validator-monitoring.mdx b/src/pages/guide/node/validator-monitoring.mdx index 480129c4..25747e34 100644 --- a/src/pages/guide/node/validator-monitoring.mdx +++ b/src/pages/guide/node/validator-monitoring.mdx @@ -61,10 +61,13 @@ Both values should be nearly equal and increasing together. | --- | --- | --- | --- | | `consensus_engine_dkg_manager_ceremony_successes_total` | Number of successful DKG ceremonies | Critical when it hasn't increased in 12 hours | Your node has participated in a successful DKG ceremony | | `consensus_engine_dkg_manager_ceremony_failures_total` | Number of failed DKG ceremonies | Warning when it increases | Your node has failed to participate in a DKG ceremony | +| `consensus_engine_dkg_manager_how_often_dealer` | How many times your node has been a dealer (distributing shares) | Warning when it hasn't increased in 6 hours | Your node is distributing signing shares to other validators | +| `consensus_engine_dkg_manager_how_often_player` | How many times your node has been a player (receiving shares) | Warning when it hasn't increased in 6 hours | Your node is receiving signing shares from dealers | | `consensus_engine_marshal_finalized_height` | Latest finalized height your node is aware of | Warning when it hasn't increased in 1 hour, critical when it hasn't increased in 3 hours | Your node is aware of the latest finalized height | | `consensus_engine_marshal_processed_height` | Latest height your node has processed | Critical when it hasn't increased in an hour, warning when it's behind finalized height | Your node is processing blocks | | `consensus_network_spawner_connections` | Number of active peers on the consensus layer | Warning when it's below the expected validator count, critical when it's 0 | Your node is connected to the consensus layer | | `consensus_engine_epoch_manager_simplex_batcher_inbound_messages_total` | Number of inbound messages related to voting on the consensus layer | Warning when it's not increasing and your node is synced | Your node is receiving voting messages from the consensus layer | +| `consensus_application_parent_ahead_of_local_time` | Number of times the parent block timestamp was ahead of local time | Warning when it increases frequently | Your node's clock may be out of sync — check NTP configuration | | `reth_sync_checkpoint` | Current sync progress | Warning on when there are no changes in the `Finish` stage | Your node is syncing with the network | | `reth_payloads_resolved_block` | Number of built and resolved payloads | Warning when it hasn't increased in 12 hours | Your node has built and resolved blocks | diff --git a/src/pages/guide/node/validator-troubleshooting.mdx b/src/pages/guide/node/validator-troubleshooting.mdx index 0ef06c82..43bb9001 100644 --- a/src/pages/guide/node/validator-troubleshooting.mdx +++ b/src/pages/guide/node/validator-troubleshooting.mdx @@ -26,3 +26,58 @@ tempo consensus validators-info --rpc-url https://rpc.tempo.xyz tempo consensus validators-info --rpc-url https://rpc.testnet.tempo.xyz ``` :::: + +## My node is not connecting to peers + +If `consensus_network_spawner_connections` remains at `0` for more than 3 hours after your validator was added on-chain: + +- Verify your firewall allows inbound connections on the ingress port you registered. +- Verify your egress IP matches the one registered on-chain — check with `tempo consensus validator --rpc-url https://rpc.tempo.xyz`. +- If you have reset your validator's data without rotating to a new identity, your node may have been blocked due to double-signing. In that case, [reach out to the Tempo team](mailto:partners@tempo.xyz) to coordinate a new validator identity. + +## My node's DKG metrics are not increasing + +If `how_often_dealer` and `how_often_player` are not increasing after 6 hours: + +- Confirm your node is connected to peers (see above). +- Check that your validator has progressed past the [Syncer state](/guide/node/validator-status#state-transitions) — it takes at least one full epoch (~3 hours) after on-chain addition before your node participates in DKG. +- Check DKG failure count: if `consensus_engine_dkg_manager_ceremony_failures_total` is increasing, your node may be failing to complete ceremonies. Enable debug logging for more detail: + ```bash + RUST_LOG=info,tempo_commonware_node::dkg=debug + ``` + +## My node's clock appears out of sync + +If you see `parent_ahead_of_local_time` increasing in your metrics, your system clock is drifting behind the network. Tempo validators rely on accurate time for block proposal timestamps. + +- Ensure NTP is configured and running: + ```bash + timedatectl status # Linux + ``` +- Verify your NTP source is reachable and your clock offset is under 500ms. + +## I accidentally deleted my consensus data directory + +If you deleted `/consensus`, your signing share is lost but will be [automatically recovered](/guide/node/validator-keys#signing-share-recovery) from the network when the node restarts. Your node will re-join the committee after the next successful DKG ceremony. + +:::danger +Do **not** delete the entire data directory and attempt to re-sync with the same signing key. This risks double-signing and will require [rotating to a new identity](/guide/node/validator-lifecycle#resetting-your-validators-data). +::: + +## How long does it take for my validator to become active? + +After on-chain registration, your validator follows the [state transition timeline](/guide/node/validator-status#state-transitions): + +1. **Epoch E** (immediate) — registered on the p2p network, starts syncing. +2. **Epoch E+1** (~3 hours) — becomes a player, receives signing shares. +3. **Epoch E+2** (~6 hours) — becomes a dealer/validator, can propose and vote once synced. + +In most cases, your validator will be fully active within 6 hours. + +## How do I check which version I'm running? + +```bash +tempo --version +``` + +Compare with the latest release on the [network upgrades](/guide/node/network-upgrades) page to ensure you're on a supported version. From 4b73281feca56e6b3f070b83a132f9b1ba4805b4 Mon Sep 17 00:00:00 2001 From: Emma Jamieson-Hoare Date: Wed, 22 Apr 2026 10:24:49 +0100 Subject: [PATCH 34/54] chore: update docs --- src/pages/guide/node/installation.mdx | 2 +- src/pages/guide/node/validator.mdx | 51 +++++++++++++++++++++++---- vocs.config.ts | 17 +++++---- 3 files changed, 56 insertions(+), 14 deletions(-) diff --git a/src/pages/guide/node/installation.mdx b/src/pages/guide/node/installation.mdx index 6d2a16f9..42c078a8 100644 --- a/src/pages/guide/node/installation.mdx +++ b/src/pages/guide/node/installation.mdx @@ -46,7 +46,7 @@ docker logs tempo ## Verifying Releases -All release artifacts are cryptographically signed starting after **v1.1.1**. We recommend verifying signatures before running any binary. +All release artifacts are cryptographically signed. We recommend verifying signatures before running any binary. ### Binary Signatures (GPG) diff --git a/src/pages/guide/node/validator.mdx b/src/pages/guide/node/validator.mdx index a70fa81a..3a77f38e 100644 --- a/src/pages/guide/node/validator.mdx +++ b/src/pages/guide/node/validator.mdx @@ -2,6 +2,8 @@ description: Overview of running a Tempo validator node. --- +import { Cards, Card } from 'vocs' + # Running a validator node Validator nodes secure Tempo by validating blocks and participating in consensus. @@ -10,11 +12,46 @@ Validator nodes secure Tempo by validating blocks and participating in consensus The active validator set is currently permissioned. If you are interested in becoming a validator, please [get in touch](mailto:partners@tempo.xyz) with the Tempo team. See [Initial setup](/guide/node/validator-setup) for technical details. ::: -## Guides + + + + + + + + + + -- **[Initial setup](/guide/node/validator-setup)** — Register with the Tempo team, generate your signing key, download a snapshot, and start your validator for the first time. -- **[Checking validator status](/guide/node/validator-status)** — Understand state transitions, check participation via metrics, and query on-chain status. -- **[Controlling validator lifecycle](/guide/node/validator-lifecycle)** — Start, stop, register, rotate, deactivate, and transfer ownership of your validator. -- **[Managing validator keys](/guide/node/validator-keys)** — Key hierarchy, generation, rotation, and signing share recovery. -- **[Observing a validator](/guide/node/validator-monitoring)** — Monitor consensus and execution health, metrics glossary, Grafana dashboards, and log management. -- **[Troubleshooting and FAQ](/guide/node/validator-troubleshooting)** — Common issues and solutions. + + diff --git a/vocs.config.ts b/vocs.config.ts index 6f362d1d..75c1ea9f 100644 --- a/vocs.config.ts +++ b/vocs.config.ts @@ -831,7 +831,7 @@ export default defineConfig({ link: '/guide/node/validator-keys', }, { - text: 'Observing a validator', + text: 'Monitoring a validator', link: '/guide/node/validator-monitoring', }, { @@ -840,13 +840,18 @@ export default defineConfig({ }, ], }, - { - text: 'Upgrade Cadence', - link: '/guide/node/upgrade-cadence', - }, { text: 'Network Upgrades and Releases', - link: '/guide/node/network-upgrades', + items: [ + { + text: 'Upgrade Cadence', + link: '/guide/node/upgrade-cadence', + }, + { + text: 'Upgrades and Releases', + link: '/guide/node/network-upgrades', + }, + ], }, { text: 'Changelog', From 8064fefb475486f91d02db22699a24d75dd69377 Mon Sep 17 00:00:00 2001 From: Emma Jamieson-Hoare Date: Wed, 22 Apr 2026 10:28:41 +0100 Subject: [PATCH 35/54] chore: update docs --- src/pages/guide/node/validator-setup.mdx | 6 ++++- src/pages/guide/node/validator-status.mdx | 22 ++++++++++++------- .../guide/node/validator-troubleshooting.mdx | 4 ++++ 3 files changed, 23 insertions(+), 9 deletions(-) diff --git a/src/pages/guide/node/validator-setup.mdx b/src/pages/guide/node/validator-setup.mdx index ebef87a2..e4ac9e13 100644 --- a/src/pages/guide/node/validator-setup.mdx +++ b/src/pages/guide/node/validator-setup.mdx @@ -17,7 +17,7 @@ Registering a validator takes three steps: 1. **[Generate a signing keypair](#step-1-generate-a-signing-keypair)** — Create an ed25519 private key and derive the public key. 2. **[Create the add-validator signature](#step-2-create-the-add-validator-signature)** — Prove ownership of the signing key by producing a signature over your registration details. -3. **[Submit registration details](#step-3-submit-registration-details)** — Provide the Tempo team with all required values to add your validator on-chain. +3. **[Submit registration details](#step-3-submit-registration-details)** — Provide the required values to register your validator on-chain. ### Step 1: Generate a signing keypair @@ -68,6 +68,10 @@ tempo consensus create-add-validator-signature \ ### Step 3: Submit registration details +:::info +Operators can submit their own registration transaction on-chain. Before registering, please [reach out to the Tempo team](mailto:partners@tempo.xyz) to align on timelines and ensure we have awareness of your validator joining the set. +::: + Provide the following values along with the signature to the Tempo team: | Value | Format | Description | diff --git a/src/pages/guide/node/validator-status.mdx b/src/pages/guide/node/validator-status.mdx index 902efca9..8509819a 100644 --- a/src/pages/guide/node/validator-status.mdx +++ b/src/pages/guide/node/validator-status.mdx @@ -3,6 +3,8 @@ title: Checking validator status description: Understand validator state transitions, check your validator's participation status, and query on-chain status. --- +import { StaticMermaidDiagram } from '../../../components/StaticMermaidDiagram' + # Checking validator status Your validator moves through different states after registration. Understanding these states helps you verify that your node is healthy and participating in consensus. @@ -58,14 +60,18 @@ Example output: Every state transition happens on epoch boundaries. Currently on mainnet and testnet, the epoch length is around 3 hours. -```mermaid {style: {maxWidth: '360px', margin: '0 auto'}} -flowchart TD - Start(["Add / Rotate in"]) --> A["Registration — Epoch E"] - A --> B["Player — Epoch E+1"] - B --> C["Dealer / Validator — Epoch E+2"] - C --> D["Voter / Proposer"] - D --> End(["Deactivate / Rotate out"]) -``` + Registration: Add / Rotate in + Registration --> Player: Epoch boundary + Player --> Dealer: Epoch boundary + Dealer --> Active: Fully synced + Active --> [*]: Deactivate / Rotate out + + state "Registration — Epoch E" as Registration + state "Player — Epoch E+1\nReceiving signing shares" as Player + state "Dealer — Epoch E+2\nDistributing shares" as Dealer + state "Active — Voter & Proposer\nProposing and voting on blocks" as Active +`} /> #### Registration (epoch E) diff --git a/src/pages/guide/node/validator-troubleshooting.mdx b/src/pages/guide/node/validator-troubleshooting.mdx index 43bb9001..7888b272 100644 --- a/src/pages/guide/node/validator-troubleshooting.mdx +++ b/src/pages/guide/node/validator-troubleshooting.mdx @@ -74,6 +74,10 @@ After on-chain registration, your validator follows the [state transition timeli In most cases, your validator will be fully active within 6 hours. +## Can I register my validator without the Tempo team? + +Yes — validator registration is self-service. You generate your keys, create the registration signature, and submit the on-chain transaction yourself. However, please [contact the Tempo team](mailto:partners@tempo.xyz) before registering so we can align on timelines and have awareness of your validator joining the active set. + ## How do I check which version I'm running? ```bash From b473f898cc6055ab85333c60be2ba390c37daa8b Mon Sep 17 00:00:00 2001 From: Emma Jamieson-Hoare Date: Wed, 22 Apr 2026 10:33:10 +0100 Subject: [PATCH 36/54] chore: snapshots --- src/pages/guide/node/installation.mdx | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/src/pages/guide/node/installation.mdx b/src/pages/guide/node/installation.mdx index 42c078a8..1e240048 100644 --- a/src/pages/guide/node/installation.mdx +++ b/src/pages/guide/node/installation.mdx @@ -44,6 +44,24 @@ docker run -d --name tempo ghcr.io/tempoxyz/tempo: --version docker logs tempo ``` +## Snapshots + +Downloading a snapshot lets your node skip syncing from genesis and start participating much faster. This is recommended for both RPC nodes and validators. + +::::code-group +```bash [Minimal (validators)] +tempo download --chain mainnet --minimal +``` +```bash [Full (RPC / dApp backends)] +tempo download --chain mainnet +``` +```bash [Archive (indexers / RPC providers)] +tempo download --chain mainnet --archive +``` +:::: + +For testnet, replace `mainnet` with `moderato`. + ## Verifying Releases All release artifacts are cryptographically signed. We recommend verifying signatures before running any binary. From 991400a7b2dc71fca2679538d84d8248cbf5771e Mon Sep 17 00:00:00 2001 From: Emma Jamieson-Hoare Date: Wed, 22 Apr 2026 10:35:19 +0100 Subject: [PATCH 37/54] chore: security --- src/pages/guide/node/security.mdx | 52 ++++++++++++++++++++ src/pages/guide/node/system-requirements.mdx | 7 +-- vocs.config.ts | 4 ++ 3 files changed, 58 insertions(+), 5 deletions(-) create mode 100644 src/pages/guide/node/security.mdx diff --git a/src/pages/guide/node/security.mdx b/src/pages/guide/node/security.mdx new file mode 100644 index 00000000..2366c4ca --- /dev/null +++ b/src/pages/guide/node/security.mdx @@ -0,0 +1,52 @@ +--- +title: Node Security +description: Security best practices for Tempo node operators, covering key management, network configuration, release verification, and data integrity. +--- + +# Node Security + +This page covers security best practices for Tempo node operators. Following these recommendations helps protect your validator from impersonation, unauthorized access, and operational mistakes. + +## Key management + +Your signing key is the most sensitive asset on your validator. Anyone with access to it can impersonate your node in consensus. + +- **Restrict file permissions** — set `chmod 600` on key files so only the node process user can read them. +- **Never share your private key** — the Tempo team will never ask for it. +- **Rotate keys periodically** — use [key rotation](/guide/node/validator-lifecycle#rotate-validator-identity) to swap to a new ed25519 key without leaving the committee. +- **Separate the operator address** — the Ethereum address that controls on-chain operations (IP updates, rotation, ownership transfer) should be a dedicated address, not a general-purpose hot wallet. + +See [Managing validator keys](/guide/node/validator-keys) for the full key hierarchy and generation instructions. + +## Network configuration + +Tempo's networking layer includes built-in protections, making a cloud firewall or NAT gateway unnecessary in most setups. The node: + +- Only accepts consensus connections from validators that can prove their identity +- Only accepts connections from trusted IP addresses (active validators on-chain) +- Rate-limits connections and messages in both consensus and execution layers + +### Recommendations + +- **Expose only the required ports** — see [Ports](/guide/node/system-requirements#ports) for which ports need public access vs. internal-only. +- **Keep ingress addresses unique** — the on-chain contract enforces uniqueness, but verify your registered `IP:port` is correct after infrastructure changes. +- **Use a dedicated machine** — avoid running other internet-facing services on the same host as your validator. + +## Release verification + +Always verify release artifacts before running them. Tempo signs all binaries and Docker images — see [Verifying Releases](/guide/node/installation#verifying-releases) for GPG, Cosign, and SHA256 verification instructions. + +## Time synchronization + +An unsynchronized clock can cause your node to reject valid blocks or produce blocks that other validators reject. Use `chrony` or `ntpd` — not `systemd-timesyncd`. See [Time Synchronization](/guide/node/system-requirements#time-synchronization) for setup instructions. + +## Data integrity + +- **Never delete the data directory and re-sync with the same signing key** — this risks double-signing and will require [rotating to a new identity](/guide/node/validator-lifecycle#resetting-your-validators-data). Deleting only the `consensus` subdirectory is safe; the signing share will be [automatically recovered](/guide/node/validator-keys#signing-share-recovery). +- **Back up your signing key** — if the key file is lost and no backup exists, you will need to rotate to a new key and coordinate with the Tempo team. + +## Staying up to date + +- Subscribe to [Tempo GitHub releases](https://github.com/tempoxyz/tempo/releases) for security patches and upgrade announcements. +- Review the [Upgrade Cadence](/guide/node/upgrade-cadence) to understand notification timelines. +- Apply Required upgrades before the activation block — failing to do so will fork your node off the network. diff --git a/src/pages/guide/node/system-requirements.mdx b/src/pages/guide/node/system-requirements.mdx index aacfe006..990315c1 100644 --- a/src/pages/guide/node/system-requirements.mdx +++ b/src/pages/guide/node/system-requirements.mdx @@ -76,12 +76,9 @@ Confirm `System clock synchronized: yes` and `NTP service: active`. Most cloud providers (AWS, Hetzner, OVH) pre-configure NTP, but minimal VM images may ship without a proper NTP daemon. Always verify that `chrony` or `ntpd` is installed and running after provisioning a new machine. -## Security measures +## Security -We do not recommend using a cloud firewall / NAT gateway as it can introduce additional complexity and performance issues. Instead, we've implemented security measures in the networking layer of the node, including: -- only accepting connections from trusted IP addresses (active validators on-chain) -- ratelimiting connections and messages in consensus and execution to prevent abuse -- only accepting consensus connections from validators that can prove their identity +For network configuration, key management, release verification, and other security best practices, see the dedicated [Node Security](/guide/node/security) page. ## Network Tuning diff --git a/vocs.config.ts b/vocs.config.ts index 75c1ea9f..8fc19dc5 100644 --- a/vocs.config.ts +++ b/vocs.config.ts @@ -840,6 +840,10 @@ export default defineConfig({ }, ], }, + { + text: 'Node Security', + link: '/guide/node/security', + }, { text: 'Network Upgrades and Releases', items: [ From b3773eaa519f44cd1b97a02e6e4dc1511ca22302 Mon Sep 17 00:00:00 2001 From: Emma Jamieson-Hoare Date: Wed, 22 Apr 2026 10:42:53 +0100 Subject: [PATCH 38/54] chore: exiting validator --- src/pages/guide/node/validator-lifecycle.mdx | 14 ++++++++++++-- src/pages/guide/node/validator-status.mdx | 18 +++++++++++++++++- .../guide/node/validator-troubleshooting.mdx | 10 ++++++++++ 3 files changed, 39 insertions(+), 3 deletions(-) diff --git a/src/pages/guide/node/validator-lifecycle.mdx b/src/pages/guide/node/validator-lifecycle.mdx index 946348b3..33729b9e 100644 --- a/src/pages/guide/node/validator-lifecycle.mdx +++ b/src/pages/guide/node/validator-lifecycle.mdx @@ -154,7 +154,7 @@ The new address must not already be used by another active validator. ## Deactivate your validator -Deactivate your validator when you want to leave the active set. The validator is only removed from the committee once reaching the end of an epoch where the DKG was successful. +Deactivate your validator when you want to leave the active set. ::::code-group ```bash [Mainnet] @@ -169,4 +169,14 @@ tempo consensus deactivate-validator
\ ``` :::: -Keep your node running after deactivation. Use [validator lookup](/guide/node/validator-status#look-up-your-validator) to check when your validator is no longer listed — it is safe to shut down the node at that point. +### Exit timeline + +Deactivation is not instant — your validator is phased out over two epochs: + +| Epoch | State | What happens | +|-------|-------|-------------| +| **E** (deactivation) | Active | Transaction is submitted. Your validator is still fully active for the remainder of this epoch. | +| **E+1** | Exiting | Your validator is a **dealer** (distributing shares) but **no longer a player** (not receiving new shares). It is in the process of being removed. | +| **E+2** | Exited | Your validator is fully out of the committee (assuming no DKG failures in E+1). | + +Keep your node running until your validator shows `in_committee: false`. Use [validator lookup](/guide/node/validator-status#look-up-your-validator) to check — it is safe to shut down the node at that point. diff --git a/src/pages/guide/node/validator-status.mdx b/src/pages/guide/node/validator-status.mdx index 8509819a..5e122601 100644 --- a/src/pages/guide/node/validator-status.mdx +++ b/src/pages/guide/node/validator-status.mdx @@ -65,14 +65,20 @@ Every state transition happens on epoch boundaries. Currently on mainnet and tes Registration --> Player: Epoch boundary Player --> Dealer: Epoch boundary Dealer --> Active: Fully synced - Active --> [*]: Deactivate / Rotate out + Active --> Exiting: Deactivate / Rotate out + Exiting --> Exited: Epoch boundary + Exited --> [*] state "Registration — Epoch E" as Registration state "Player — Epoch E+1\nReceiving signing shares" as Player state "Dealer — Epoch E+2\nDistributing shares" as Dealer state "Active — Voter & Proposer\nProposing and voting on blocks" as Active + state "Exiting — Epoch E+1\nDealer only, no longer a player" as Exiting + state "Exited — Epoch E+2\nFully out of committee" as Exited `} /> +### Joining + #### Registration (epoch E) Once your validator is added to the on-chain contract, it is immediately registered on the p2p peer set and starts syncing blocks from the network. @@ -85,6 +91,16 @@ Your validator is receiving consensus signing shares from dealers during the DKG Your validator is distributing consensus signing shares to other validators during the ceremony. Once your node is fully synced, it will also be able to propose blocks and vote on other validators' proposals. +### Leaving + +#### Exiting (epoch E+1) + +After deactivation in epoch E, your validator is still a **dealer** (distributing shares) during epoch E+1, but is **no longer a player** (not receiving new shares). It is in the process of being removed from the committee. + +#### Exited (epoch E+2) + +Your validator is fully out of the committee, assuming no DKG failures occurred in E+1. It is safe to shut down the node once `in_committee` is `false`. + :::info The on-chain contract represents which validators should eventually be members of the committee, not which ones currently are. Every epoch, the Tempo network performs a distributed key generation ceremony, distributing keys to validators that should be committee members in the next epoch. So adding, deactivating, or rotating validator entries in epoch `E` can only be taken into account for the next DKG ceremony that runs during epoch `E+1`, and take effect in epoch `E+2`. ::: diff --git a/src/pages/guide/node/validator-troubleshooting.mdx b/src/pages/guide/node/validator-troubleshooting.mdx index 7888b272..657c92f8 100644 --- a/src/pages/guide/node/validator-troubleshooting.mdx +++ b/src/pages/guide/node/validator-troubleshooting.mdx @@ -74,6 +74,16 @@ After on-chain registration, your validator follows the [state transition timeli In most cases, your validator will be fully active within 6 hours. +## How long does it take for my validator to exit? + +After deactivation, your validator is phased out over two epochs: + +1. **Epoch E** — deactivation transaction submitted. Validator remains fully active. +2. **Epoch E+1** (~3 hours) — still a dealer but no longer a player. In the process of being removed. +3. **Epoch E+2** (~6 hours) — fully out of the committee (assuming no DKG failures). + +Keep your node running until `in_committee: false` — check with [validator lookup](/guide/node/validator-status#look-up-your-validator). + ## Can I register my validator without the Tempo team? Yes — validator registration is self-service. You generate your keys, create the registration signature, and submit the on-chain transaction yourself. However, please [contact the Tempo team](mailto:partners@tempo.xyz) before registering so we can align on timelines and have awareness of your validator joining the active set. From 6cd3f82d077e4bc73257b9ab4b4e61def65899c1 Mon Sep 17 00:00:00 2001 From: Emma Jamieson-Hoare Date: Wed, 22 Apr 2026 10:45:30 +0100 Subject: [PATCH 39/54] chore: exiting cmds --- src/pages/guide/node/validator-lifecycle.mdx | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/src/pages/guide/node/validator-lifecycle.mdx b/src/pages/guide/node/validator-lifecycle.mdx index 33729b9e..80b3a246 100644 --- a/src/pages/guide/node/validator-lifecycle.mdx +++ b/src/pages/guide/node/validator-lifecycle.mdx @@ -173,10 +173,10 @@ tempo consensus deactivate-validator
\ Deactivation is not instant — your validator is phased out over two epochs: -| Epoch | State | What happens | -|-------|-------|-------------| -| **E** (deactivation) | Active | Transaction is submitted. Your validator is still fully active for the remainder of this epoch. | -| **E+1** | Exiting | Your validator is a **dealer** (distributing shares) but **no longer a player** (not receiving new shares). It is in the process of being removed. | -| **E+2** | Exited | Your validator is fully out of the committee (assuming no DKG failures in E+1). | +| Epoch | State | `is_dealer` | `is_player` | `in_committee` | +|-------|-------|:-----------:|:-----------:|:--------------:| +| **E** (deactivation) | Active | `true` | `true` | `true` | +| **E+1** | Exiting | `true` | `false` | `true` | +| **E+2** | Exited | `false` | `false` | `false` | Keep your node running until your validator shows `in_committee: false`. Use [validator lookup](/guide/node/validator-status#look-up-your-validator) to check — it is safe to shut down the node at that point. From 20bee497dd48b2843c26162ac9d7ca67de056b94 Mon Sep 17 00:00:00 2001 From: Emma Jamieson-Hoare Date: Wed, 22 Apr 2026 10:51:19 +0100 Subject: [PATCH 40/54] chore: janis feedback --- src/pages/guide/node/validator-lifecycle.mdx | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/src/pages/guide/node/validator-lifecycle.mdx b/src/pages/guide/node/validator-lifecycle.mdx index 80b3a246..db04bb15 100644 --- a/src/pages/guide/node/validator-lifecycle.mdx +++ b/src/pages/guide/node/validator-lifecycle.mdx @@ -71,20 +71,18 @@ Rotation preserves your validator index and active validator count. The old entr After rotation, your validator goes through the [standard state transitions](/guide/node/validator-status#state-transitions) with the new identity. :::danger[Do not shut down the old validator immediately] -After rotation, the rotated-out validator is still a member of the current committee until the next successful DKG ceremony completes. Shutting it down early will degrade network liveness. +After rotation, the rotated-out validator is still a dealer in the current committee for at least one more epoch. Shutting it down early will degrade network liveness. See the [exit timeline](#exit-timeline) for the full epoch-by-epoch breakdown. -Keep the old validator running and use `validators-info` to monitor its status: +Keep the old validator running until it shows `in_committee: false`: ::::code-group ```bash [Mainnet] -tempo consensus validators-info --rpc-url https://rpc.tempo.xyz +tempo consensus validator --rpc-url https://rpc.tempo.xyz ``` ```bash [Testnet] -tempo consensus validators-info --rpc-url https://rpc.testnet.tempo.xyz +tempo consensus validator --rpc-url https://rpc.testnet.tempo.xyz ``` :::: - -Once the old validator shows `in_committee = false`, it is safe to shut down. ::: ## Update IP addresses @@ -156,6 +154,10 @@ The new address must not already be used by another active validator. Deactivate your validator when you want to leave the active set. +:::warning[Do not shut down your node immediately after deactivating] +Your validator remains a dealer in the committee for at least one more epoch after deactivation. Shutting it down early will degrade network liveness. See the [exit timeline](#exit-timeline) below. +::: + ::::code-group ```bash [Mainnet] tempo consensus deactivate-validator
\ From 73cddf8a24bdf745d5647705d472bebfa9c99058 Mon Sep 17 00:00:00 2001 From: Emma Jamieson-Hoare Date: Wed, 22 Apr 2026 11:04:33 +0100 Subject: [PATCH 41/54] chore: fix add validator --- src/pages/guide/node/validator-setup.mdx | 4 ++-- src/pages/guide/node/validator-troubleshooting.mdx | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/src/pages/guide/node/validator-setup.mdx b/src/pages/guide/node/validator-setup.mdx index e4ac9e13..9b02c6b8 100644 --- a/src/pages/guide/node/validator-setup.mdx +++ b/src/pages/guide/node/validator-setup.mdx @@ -17,7 +17,7 @@ Registering a validator takes three steps: 1. **[Generate a signing keypair](#step-1-generate-a-signing-keypair)** — Create an ed25519 private key and derive the public key. 2. **[Create the add-validator signature](#step-2-create-the-add-validator-signature)** — Prove ownership of the signing key by producing a signature over your registration details. -3. **[Submit registration details](#step-3-submit-registration-details)** — Provide the required values to register your validator on-chain. +3. **[Submit registration details](#step-3-submit-registration-details)** — Provide the Tempo team with all required values to add your validator on-chain. ### Step 1: Generate a signing keypair @@ -69,7 +69,7 @@ tempo consensus create-add-validator-signature \ ### Step 3: Submit registration details :::info -Operators can submit their own registration transaction on-chain. Before registering, please [reach out to the Tempo team](mailto:partners@tempo.xyz) to align on timelines and ensure we have awareness of your validator joining the set. +The active validator set is permissioned. Only the contract owner can add validators on-chain. Provide the values below to the Tempo team to complete registration. ::: Provide the following values along with the signature to the Tempo team: diff --git a/src/pages/guide/node/validator-troubleshooting.mdx b/src/pages/guide/node/validator-troubleshooting.mdx index 657c92f8..87e8e042 100644 --- a/src/pages/guide/node/validator-troubleshooting.mdx +++ b/src/pages/guide/node/validator-troubleshooting.mdx @@ -86,7 +86,7 @@ Keep your node running until `in_committee: false` — check with [validator loo ## Can I register my validator without the Tempo team? -Yes — validator registration is self-service. You generate your keys, create the registration signature, and submit the on-chain transaction yourself. However, please [contact the Tempo team](mailto:partners@tempo.xyz) before registering so we can align on timelines and have awareness of your validator joining the active set. +No — the active validator set is currently permissioned. Only the contract owner can add validators on-chain. You generate your keys and create the registration signature yourself, then provide them to the Tempo team for on-chain submission. [Contact the Tempo team](mailto:partners@tempo.xyz) to get started. ## How do I check which version I'm running? From ae897683fb35b7acf89804b81a0f6a3c84659f7b Mon Sep 17 00:00:00 2001 From: Emma Jamieson-Hoare Date: Wed, 22 Apr 2026 11:05:27 +0100 Subject: [PATCH 42/54] chore: CLEAN UP --- src/pages/guide/node/validator-setup.mdx | 4 ---- 1 file changed, 4 deletions(-) diff --git a/src/pages/guide/node/validator-setup.mdx b/src/pages/guide/node/validator-setup.mdx index 9b02c6b8..ebef87a2 100644 --- a/src/pages/guide/node/validator-setup.mdx +++ b/src/pages/guide/node/validator-setup.mdx @@ -68,10 +68,6 @@ tempo consensus create-add-validator-signature \ ### Step 3: Submit registration details -:::info -The active validator set is permissioned. Only the contract owner can add validators on-chain. Provide the values below to the Tempo team to complete registration. -::: - Provide the following values along with the signature to the Tempo team: | Value | Format | Description | From 4f5371310103e7391fde47d3e0418b663ebc8d0a Mon Sep 17 00:00:00 2001 From: Emma Jamieson-Hoare Date: Wed, 22 Apr 2026 11:09:43 +0100 Subject: [PATCH 43/54] chore: fix text --- src/pages/guide/node/validator-status.mdx | 14 +++++--------- 1 file changed, 5 insertions(+), 9 deletions(-) diff --git a/src/pages/guide/node/validator-status.mdx b/src/pages/guide/node/validator-status.mdx index 5e122601..72ddb3fd 100644 --- a/src/pages/guide/node/validator-status.mdx +++ b/src/pages/guide/node/validator-status.mdx @@ -77,27 +77,23 @@ Every state transition happens on epoch boundaries. Currently on mainnet and tes state "Exited — Epoch E+2\nFully out of committee" as Exited `} /> -### Joining - -#### Registration (epoch E) +### Registration (epoch E) Once your validator is added to the on-chain contract, it is immediately registered on the p2p peer set and starts syncing blocks from the network. -#### Player (epoch E+1) +### Player (epoch E+1) Your validator is receiving consensus signing shares from dealers during the DKG ceremony. -#### Dealer / Validator (epoch E+2) +### Dealer / Validator (epoch E+2) Your validator is distributing consensus signing shares to other validators during the ceremony. Once your node is fully synced, it will also be able to propose blocks and vote on other validators' proposals. -### Leaving - -#### Exiting (epoch E+1) +### Exiting (epoch E+1 after deactivation) After deactivation in epoch E, your validator is still a **dealer** (distributing shares) during epoch E+1, but is **no longer a player** (not receiving new shares). It is in the process of being removed from the committee. -#### Exited (epoch E+2) +### Exited (epoch E+2 after deactivation) Your validator is fully out of the committee, assuming no DKG failures occurred in E+1. It is safe to shut down the node once `in_committee` is `false`. From 8c6bafb97a53ec53e330f18d5890fac4cd48e9d2 Mon Sep 17 00:00:00 2001 From: Emma Jamieson-Hoare Date: Wed, 22 Apr 2026 11:12:28 +0100 Subject: [PATCH 44/54] chore: adjuist warning --- src/pages/guide/node/validator-lifecycle.mdx | 30 ++++++++++---------- 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/src/pages/guide/node/validator-lifecycle.mdx b/src/pages/guide/node/validator-lifecycle.mdx index db04bb15..81d6b175 100644 --- a/src/pages/guide/node/validator-lifecycle.mdx +++ b/src/pages/guide/node/validator-lifecycle.mdx @@ -37,6 +37,21 @@ Self-service data resets are coming soon. Once available, you will be able to ro The ed25519 key can be changed while keeping your validator index stable. This is useful for key rotation or recovery without leaving and re-joining the committee. [Generate a new signing key](/guide/node/validator-keys#generating-a-signing-key) first. +:::danger[Do not shut down the old validator] +Unlike Ethereum, you must **keep your old validator running** after rotation. The rotated-out validator is still a dealer in the committee for at least one more epoch. Shutting it down early will degrade network liveness. See the [exit timeline](#exit-timeline) for the full epoch-by-epoch breakdown. + +Keep the old validator running until it shows `in_committee: false`: + +::::code-group +```bash [Mainnet] +tempo consensus validator --rpc-url https://rpc.tempo.xyz +``` +```bash [Testnet] +tempo consensus validator --rpc-url https://rpc.testnet.tempo.xyz +``` +:::: +::: + The simplest way to rotate is using the `tempo` CLI, which handles signature creation and the on-chain transaction in one step: ::::code-group @@ -70,21 +85,6 @@ Rotation preserves your validator index and active validator count. The old entr After rotation, your validator goes through the [standard state transitions](/guide/node/validator-status#state-transitions) with the new identity. -:::danger[Do not shut down the old validator immediately] -After rotation, the rotated-out validator is still a dealer in the current committee for at least one more epoch. Shutting it down early will degrade network liveness. See the [exit timeline](#exit-timeline) for the full epoch-by-epoch breakdown. - -Keep the old validator running until it shows `in_committee: false`: - -::::code-group -```bash [Mainnet] -tempo consensus validator --rpc-url https://rpc.tempo.xyz -``` -```bash [Testnet] -tempo consensus validator --rpc-url https://rpc.testnet.tempo.xyz -``` -:::: -::: - ## Update IP addresses If your node's network endpoints change, update them on-chain. The change takes effect at the next finalized block. From 000bf40bf7b6ff9e2a8a41640fa1ce5e46138caf Mon Sep 17 00:00:00 2001 From: Emma Jamieson-Hoare Date: Wed, 22 Apr 2026 11:25:51 +0100 Subject: [PATCH 45/54] chore: fix onboarding --- .../guide/node/validator-troubleshooting.mdx | 21 ++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-) diff --git a/src/pages/guide/node/validator-troubleshooting.mdx b/src/pages/guide/node/validator-troubleshooting.mdx index 87e8e042..dc660338 100644 --- a/src/pages/guide/node/validator-troubleshooting.mdx +++ b/src/pages/guide/node/validator-troubleshooting.mdx @@ -86,7 +86,26 @@ Keep your node running until `in_committee: false` — check with [validator loo ## Can I register my validator without the Tempo team? -No — the active validator set is currently permissioned. Only the contract owner can add validators on-chain. You generate your keys and create the registration signature yourself, then provide them to the Tempo team for on-chain submission. [Contact the Tempo team](mailto:partners@tempo.xyz) to get started. +No — the active validator set is currently permissioned. Only the contract owner can add validators on-chain. To get started, [contact the Tempo team](mailto:partners@tempo.xyz). + +The onboarding process: + +1. **You** generate your signing key and registration signature ([Steps 1–2](/guide/node/validator-setup#initial-registration)). +2. **You** provide the required values to the Tempo team ([Step 3](/guide/node/validator-setup#step-3-submit-registration-details)). +3. **The Tempo team** adds your validator on-chain. +4. **You** download a snapshot and start your node ([Running the validator](/guide/node/validator-setup#running-the-validator)). + +## What can I do without the Tempo team? + +Once your validator is registered, most operations are self-service: + +- [Rotate your signing key](/guide/node/validator-lifecycle#rotate-validator-identity) +- [Update IP addresses](/guide/node/validator-lifecycle#update-ip-addresses) +- [Update your fee recipient](/guide/node/validator-lifecycle#update-the-fee-recipient) +- [Transfer validator ownership](/guide/node/validator-lifecycle#transfer-validator-ownership) +- [Deactivate your validator](/guide/node/validator-lifecycle#deactivate-your-validator) + +Only **initial registration** and **reactivation** require the Tempo team. ## How do I check which version I'm running? From 4fdfa1e245b13f047aa9b8803b0db7fce5ed70c8 Mon Sep 17 00:00:00 2001 From: Emma Jamieson-Hoare Date: Wed, 22 Apr 2026 11:47:33 +0100 Subject: [PATCH 46/54] chore: link snapshots.tempo.xyz --- src/pages/guide/node/installation.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/pages/guide/node/installation.mdx b/src/pages/guide/node/installation.mdx index 1e240048..bbe84e93 100644 --- a/src/pages/guide/node/installation.mdx +++ b/src/pages/guide/node/installation.mdx @@ -60,7 +60,7 @@ tempo download --chain mainnet --archive ``` :::: -For testnet, replace `mainnet` with `moderato`. +For testnet, replace `mainnet` with `moderato`. See [snapshots.tempoxyz.dev](https://snapshots.tempoxyz.dev/) for detailed component sizes and download options. ## Verifying Releases From 9108cfc6743a6da227612e7086fbf516480e2176 Mon Sep 17 00:00:00 2001 From: Jennifer Date: Wed, 22 Apr 2026 11:48:47 +0100 Subject: [PATCH 47/54] Update src/pages/guide/node/validator-monitoring.mdx --- src/pages/guide/node/validator-monitoring.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/pages/guide/node/validator-monitoring.mdx b/src/pages/guide/node/validator-monitoring.mdx index 25747e34..492595bc 100644 --- a/src/pages/guide/node/validator-monitoring.mdx +++ b/src/pages/guide/node/validator-monitoring.mdx @@ -1,5 +1,5 @@ --- -title: Observing a validator +title: Monitoring a validator description: Monitor validator health with metrics, Grafana dashboards, and log management. --- From 99323f461db04c73c51b0468e2202a181bf85e0d Mon Sep 17 00:00:00 2001 From: Jennifer Date: Wed, 22 Apr 2026 11:49:07 +0100 Subject: [PATCH 48/54] Update src/pages/guide/node/validator-setup.mdx --- src/pages/guide/node/validator-setup.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/pages/guide/node/validator-setup.mdx b/src/pages/guide/node/validator-setup.mdx index ebef87a2..2118808f 100644 --- a/src/pages/guide/node/validator-setup.mdx +++ b/src/pages/guide/node/validator-setup.mdx @@ -3,7 +3,7 @@ title: Initial setup description: Generate signing keys and run your Tempo validator node for the first time. --- -# Initial setup +# Validator onboarding :::info The active validator set is currently permissioned. If you are interested in becoming a validator, please [get in touch](mailto:partners@tempo.xyz) with the Tempo team. From ec5afb4fa3ef579749dad69b7edb2b4a6e12badc Mon Sep 17 00:00:00 2001 From: Jennifer Date: Wed, 22 Apr 2026 11:51:28 +0100 Subject: [PATCH 49/54] Update src/pages/guide/node/validator-setup.mdx --- src/pages/guide/node/validator-setup.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/pages/guide/node/validator-setup.mdx b/src/pages/guide/node/validator-setup.mdx index 2118808f..184adeb7 100644 --- a/src/pages/guide/node/validator-setup.mdx +++ b/src/pages/guide/node/validator-setup.mdx @@ -1,5 +1,5 @@ --- -title: Initial setup +title: Validator Onboarding description: Generate signing keys and run your Tempo validator node for the first time. --- From 0615cbbc0346add7cd5de8ffcd6fc61a85e3e4a2 Mon Sep 17 00:00:00 2001 From: Jennifer Date: Wed, 22 Apr 2026 11:51:36 +0100 Subject: [PATCH 50/54] Update vocs.config.ts --- vocs.config.ts | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/vocs.config.ts b/vocs.config.ts index 8fc19dc5..d091c8f7 100644 --- a/vocs.config.ts +++ b/vocs.config.ts @@ -847,6 +847,15 @@ export default defineConfig({ { text: 'Network Upgrades and Releases', items: [ + { + text: 'Upgrades and Releases', + link: '/guide/node/network-upgrades', + }, + { + text: 'Upgrade Cadence', + link: '/guide/node/upgrade-cadence', + }, + ], { text: 'Upgrade Cadence', link: '/guide/node/upgrade-cadence', From 8a5acf416055c7f00c8e70344a2e27d854748a4a Mon Sep 17 00:00:00 2001 From: Jennifer Date: Wed, 22 Apr 2026 11:51:45 +0100 Subject: [PATCH 51/54] Update src/pages/guide/node/validator-monitoring.mdx --- src/pages/guide/node/validator-monitoring.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/pages/guide/node/validator-monitoring.mdx b/src/pages/guide/node/validator-monitoring.mdx index 492595bc..b6b3b551 100644 --- a/src/pages/guide/node/validator-monitoring.mdx +++ b/src/pages/guide/node/validator-monitoring.mdx @@ -3,7 +3,7 @@ title: Monitoring a validator description: Monitor validator health with metrics, Grafana dashboards, and log management. --- -# Observing a validator +# Monitoring a validator This guide covers how to monitor your Tempo validator's health, diagnose issues, and manage logs. From ac48cf9d831a178c85cdd84dd5c905f1c661255e Mon Sep 17 00:00:00 2001 From: Emma Jamieson-Hoare Date: Wed, 22 Apr 2026 11:54:20 +0100 Subject: [PATCH 52/54] fix: remove duplicate sidebar entries in vocs.config.ts Amp-Thread-ID: https://ampcode.com/threads/T-019db09b-38c8-7044-8f6c-767dfd4edb00 Co-authored-by: Amp --- vocs.config.ts | 9 --------- 1 file changed, 9 deletions(-) diff --git a/vocs.config.ts b/vocs.config.ts index d091c8f7..8fc19dc5 100644 --- a/vocs.config.ts +++ b/vocs.config.ts @@ -847,15 +847,6 @@ export default defineConfig({ { text: 'Network Upgrades and Releases', items: [ - { - text: 'Upgrades and Releases', - link: '/guide/node/network-upgrades', - }, - { - text: 'Upgrade Cadence', - link: '/guide/node/upgrade-cadence', - }, - ], { text: 'Upgrade Cadence', link: '/guide/node/upgrade-cadence', From 8593143cdfc2e6011f5b5765c425a8f603aee285 Mon Sep 17 00:00:00 2001 From: Emma Jamieson-Hoare Date: Wed, 22 Apr 2026 11:58:28 +0100 Subject: [PATCH 53/54] chore: fix comments --- .../guide/node/validator-troubleshooting.mdx | 32 +++++++++++++++---- 1 file changed, 25 insertions(+), 7 deletions(-) diff --git a/src/pages/guide/node/validator-troubleshooting.mdx b/src/pages/guide/node/validator-troubleshooting.mdx index dc660338..d2736415 100644 --- a/src/pages/guide/node/validator-troubleshooting.mdx +++ b/src/pages/guide/node/validator-troubleshooting.mdx @@ -46,15 +46,33 @@ If `how_often_dealer` and `how_often_player` are not increasing after 6 hours: RUST_LOG=info,tempo_commonware_node::dkg=debug ``` -## My node's clock appears out of sync +## My node is rejecting blocks or missing proposals -If you see `parent_ahead_of_local_time` increasing in your metrics, your system clock is drifting behind the network. Tempo validators rely on accurate time for block proposal timestamps. +If your node logs block validation errors (e.g. "block timestamp is in the future") or you notice missed proposals, a potential root cause is **clock drift**. Check whether `parent_ahead_of_local_time` is increasing in your metrics — if so, your system clock is behind the network. -- Ensure NTP is configured and running: - ```bash - timedatectl status # Linux - ``` -- Verify your NTP source is reachable and your clock offset is under 500ms. +To fix: + +1. **Check your current sync status:** + ```bash + timedatectl status + ``` + Confirm `System clock synchronized: yes` and `NTP service: active`. + +2. **Install chrony** (if not already installed): + ```bash + sudo apt install chrony + sudo systemctl enable --now chronyd + ``` + +3. **Verify the offset is acceptable:** + ```bash + chronyc tracking + ``` + The **System time** offset should be under a few milliseconds. + +4. **Restart your node** after fixing the clock to clear any cached invalid block state. + +See [Time Synchronization](/guide/node/system-requirements#time-synchronization) for full setup details. ## I accidentally deleted my consensus data directory From a1f4c863a65ae5c3026ffcbc003df9474afd52b6 Mon Sep 17 00:00:00 2001 From: Jennifer Date: Wed, 22 Apr 2026 12:05:49 +0100 Subject: [PATCH 54/54] Apply suggestion from @jenpaff --- vocs.config.ts | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/vocs.config.ts b/vocs.config.ts index 8fc19dc5..cd065771 100644 --- a/vocs.config.ts +++ b/vocs.config.ts @@ -815,7 +815,7 @@ export default defineConfig({ link: '/guide/node/validator', }, { - text: 'Initial setup', + text: 'Validator Onboarding', link: '/guide/node/validator-setup', }, {