diff --git a/src/pages/cli/node.mdx b/src/pages/cli/node.mdx index 3d0e3b42..d61992c4 100644 --- a/src/pages/cli/node.mdx +++ b/src/pages/cli/node.mdx @@ -37,7 +37,7 @@ Flags grouped by function: | `--consensus.datadir ` | Separate volume for consensus data | :::warning -`--consensus.fee-recipient` is deprecated as of `v1.5.2` and will be removed in an upcoming release. See [Validator Config V2](/guide/node/validator-config-v2). +`--consensus.fee-recipient` is deprecated as of `v1.5.2` and will be removed in an upcoming release. See [updating the fee recipient](/guide/node/validator-lifecycle#update-the-fee-recipient). ::: ### Observability diff --git a/src/pages/guide/node/installation.mdx b/src/pages/guide/node/installation.mdx index 6d2a16f9..bbe84e93 100644 --- a/src/pages/guide/node/installation.mdx +++ b/src/pages/guide/node/installation.mdx @@ -44,9 +44,27 @@ docker run -d --name tempo ghcr.io/tempoxyz/tempo: --version docker logs tempo ``` +## Snapshots + +Downloading a snapshot lets your node skip syncing from genesis and start participating much faster. This is recommended for both RPC nodes and validators. + +::::code-group +```bash [Minimal (validators)] +tempo download --chain mainnet --minimal +``` +```bash [Full (RPC / dApp backends)] +tempo download --chain mainnet +``` +```bash [Archive (indexers / RPC providers)] +tempo download --chain mainnet --archive +``` +:::: + +For testnet, replace `mainnet` with `moderato`. See [snapshots.tempoxyz.dev](https://snapshots.tempoxyz.dev/) for detailed component sizes and download options. + ## Verifying Releases -All release artifacts are cryptographically signed starting after **v1.1.1**. We recommend verifying signatures before running any binary. +All release artifacts are cryptographically signed. We recommend verifying signatures before running any binary. ### Binary Signatures (GPG) diff --git a/src/pages/guide/node/network-upgrades.mdx b/src/pages/guide/node/network-upgrades.mdx index 0856818c..c5df136a 100644 --- a/src/pages/guide/node/network-upgrades.mdx +++ b/src/pages/guide/node/network-upgrades.mdx @@ -27,7 +27,7 @@ For detailed release notes and binaries, see the [Changelog](/changelog). | [v1.2.0](https://github.com/tempoxyz/tempo/releases/tag/v1.2.0) | Feb 13, 2026 | Mainnet only | Fixes validation bug rejecting transactions with gas limits above ~16.7M, blocking large contract deployments | Required | :::warning -`--consensus.fee-recipient` is deprecated as of `v1.5.2` and will be removed in an upcoming release. Validators should migrate fee-recipient management to [Validator Config V2](/guide/node/validator-config-v2). +`--consensus.fee-recipient` is deprecated as of `v1.5.2` and will be removed in an upcoming release. Validators should migrate fee-recipient management to [update the fee recipient](/guide/node/validator-lifecycle#update-the-fee-recipient). ::: --- diff --git a/src/pages/guide/node/security.mdx b/src/pages/guide/node/security.mdx new file mode 100644 index 00000000..2366c4ca --- /dev/null +++ b/src/pages/guide/node/security.mdx @@ -0,0 +1,52 @@ +--- +title: Node Security +description: Security best practices for Tempo node operators, covering key management, network configuration, release verification, and data integrity. +--- + +# Node Security + +This page covers security best practices for Tempo node operators. Following these recommendations helps protect your validator from impersonation, unauthorized access, and operational mistakes. + +## Key management + +Your signing key is the most sensitive asset on your validator. Anyone with access to it can impersonate your node in consensus. + +- **Restrict file permissions** — set `chmod 600` on key files so only the node process user can read them. +- **Never share your private key** — the Tempo team will never ask for it. +- **Rotate keys periodically** — use [key rotation](/guide/node/validator-lifecycle#rotate-validator-identity) to swap to a new ed25519 key without leaving the committee. +- **Separate the operator address** — the Ethereum address that controls on-chain operations (IP updates, rotation, ownership transfer) should be a dedicated address, not a general-purpose hot wallet. + +See [Managing validator keys](/guide/node/validator-keys) for the full key hierarchy and generation instructions. + +## Network configuration + +Tempo's networking layer includes built-in protections, making a cloud firewall or NAT gateway unnecessary in most setups. The node: + +- Only accepts consensus connections from validators that can prove their identity +- Only accepts connections from trusted IP addresses (active validators on-chain) +- Rate-limits connections and messages in both consensus and execution layers + +### Recommendations + +- **Expose only the required ports** — see [Ports](/guide/node/system-requirements#ports) for which ports need public access vs. internal-only. +- **Keep ingress addresses unique** — the on-chain contract enforces uniqueness, but verify your registered `IP:port` is correct after infrastructure changes. +- **Use a dedicated machine** — avoid running other internet-facing services on the same host as your validator. + +## Release verification + +Always verify release artifacts before running them. Tempo signs all binaries and Docker images — see [Verifying Releases](/guide/node/installation#verifying-releases) for GPG, Cosign, and SHA256 verification instructions. + +## Time synchronization + +An unsynchronized clock can cause your node to reject valid blocks or produce blocks that other validators reject. Use `chrony` or `ntpd` — not `systemd-timesyncd`. See [Time Synchronization](/guide/node/system-requirements#time-synchronization) for setup instructions. + +## Data integrity + +- **Never delete the data directory and re-sync with the same signing key** — this risks double-signing and will require [rotating to a new identity](/guide/node/validator-lifecycle#resetting-your-validators-data). Deleting only the `consensus` subdirectory is safe; the signing share will be [automatically recovered](/guide/node/validator-keys#signing-share-recovery). +- **Back up your signing key** — if the key file is lost and no backup exists, you will need to rotate to a new key and coordinate with the Tempo team. + +## Staying up to date + +- Subscribe to [Tempo GitHub releases](https://github.com/tempoxyz/tempo/releases) for security patches and upgrade announcements. +- Review the [Upgrade Cadence](/guide/node/upgrade-cadence) to understand notification timelines. +- Apply Required upgrades before the activation block — failing to do so will fork your node off the network. diff --git a/src/pages/guide/node/system-requirements.mdx b/src/pages/guide/node/system-requirements.mdx index b8a746b1..990315c1 100644 --- a/src/pages/guide/node/system-requirements.mdx +++ b/src/pages/guide/node/system-requirements.mdx @@ -43,12 +43,42 @@ These dedicated servers meet or exceed the recommended specs for both RPC and va Cloud instances with network-attached storage (e.g., AWS EBS) do not provide sufficient I/O performance. Use dedicated servers or instances with local NVMe storage. ::: -## Security measures +## Time Synchronization -We do not recommend using a cloud firewall / NAT gateway as it can introduce additional complexity and performance issues. Instead, we've implemented security measures in the networking layer of the node, including: -- only accepting connections from trusted IP addresses (active validators on-chain) -- ratelimiting connections and messages in consensus and execution to prevent abuse -- only accepting consensus connections from validators that can prove their identity +Tempo validates that block timestamps are not in the future. If your system clock drifts even slightly, your node may reject valid blocks or produce blocks that other validators reject — leading to consensus errors and missed proposals. + +:::warning +`systemd-timesyncd` (the default on many minimal VMs) is **not sufficient**. Use `chrony` or `ntpd` for reliable sub-millisecond synchronization. +::: + +### Install and enable chrony (recommended) + +```bash +sudo apt install chrony +sudo systemctl enable --now chronyd +``` + +### Verify synchronization + +```bash +chronyc tracking +``` + +Check that **System time** offset is under a few milliseconds and **Leap status** is `Normal`. You can also verify with: + +```bash +timedatectl +``` + +Confirm `System clock synchronized: yes` and `NTP service: active`. + +### Cloud providers + +Most cloud providers (AWS, Hetzner, OVH) pre-configure NTP, but minimal VM images may ship without a proper NTP daemon. Always verify that `chrony` or `ntpd` is installed and running after provisioning a new machine. + +## Security + +For network configuration, key management, release verification, and other security best practices, see the dedicated [Node Security](/guide/node/security) page. ## Network Tuning diff --git a/src/pages/guide/node/upgrade-cadence.mdx b/src/pages/guide/node/upgrade-cadence.mdx new file mode 100644 index 00000000..2adf82d0 --- /dev/null +++ b/src/pages/guide/node/upgrade-cadence.mdx @@ -0,0 +1,67 @@ +--- +title: Upgrade Cadence +description: How Tempo schedules and communicates network upgrades, including timelines, notification windows, and what to expect as a node operator. +--- + +import { Badge } from '../../../components/Badge' + +# Upgrade Cadence + +Tempo ships protocol upgrades on a **monthly cadence**. Each upgrade bundles one or more protocol changes ([TIPs](/protocol/tips)) into a named hardfork (T1, T2, T3, …). + +## Rollout timeline + +Every upgrade follows the same two-stage rollout: + +| Stage | Lead time | Details | +|-------|-----------|---------| +| **Testnet** (Moderato) | Announced at least **3 days** before activation | The upgrade activates on the Moderato testnet first so operators and integrators can validate. | +| **Mainnet** (Presto) | Announced at least **7 days** before activation | After a successful testnet activation, the same release is scheduled for mainnet — typically one week later. | + +Operators should use the window between testnet and mainnet activation to upgrade their testnet node, verify it syncs past the activation block, and confirm their applications work against the new protocol rules. + +## Patch releases + +Between hardforks, Tempo publishes patch releases (e.g. v1.5.1, v1.5.2) for security fixes, performance improvements, and non-consensus bug fixes. Patch releases do not require a hardfork and can be adopted at your own pace, subject to the priority level. + +## Upgrade priority levels + +Each release on the [Network Upgrades and Releases](/guide/node/network-upgrades) page carries a priority badge: + +| Badge | Meaning | +|-------|---------| +| Required | Consensus-breaking change — nodes **must** upgrade before the activation block or they will fork off the network. | +| Recommended | Non-consensus improvement (performance, hardening, bug fixes). Nodes continue to function without upgrading, but the update is strongly encouraged. | +| RPC only | Only affects RPC-serving nodes (e.g. security patches for public endpoints). Validators without public RPC exposure can skip. | + +## How you will be notified + +- **GitHub releases** — every release is published to [tempoxyz/tempo releases](https://github.com/tempoxyz/tempo/releases) with a full changelog. +- **Operator channels** — the Tempo team shares activation timestamps and migration checklists in dedicated operator channels ahead of each upgrade. +- **Docs** — the [Network Upgrades and Releases](/guide/node/network-upgrades) page is updated with dates, TIP references, and priority badges as soon as an upgrade is scheduled. + +## Operator checklist + +:::::steps + +### Review the release + +Read the changelog and upgrade page to understand what is changing and whether any migration steps are required. + +### Upgrade your testnet node + +Update your Moderato node to the new release and confirm it syncs past the testnet activation block. + +### Validate your integration + +Run your application or test suite against the upgraded testnet to catch any breaking changes before mainnet. + +### Upgrade your mainnet node + +Update your mainnet node **before** the mainnet activation timestamp. + +### Monitor after activation + +Watch logs and metrics after the activation block to confirm normal operation. + +::::: diff --git a/src/pages/guide/node/validator-config-v2.mdx b/src/pages/guide/node/validator-config-v2.mdx deleted file mode 100644 index 3e41d115..00000000 --- a/src/pages/guide/node/validator-config-v2.mdx +++ /dev/null @@ -1,215 +0,0 @@ ---- -title: ValidatorConfig V2 -description: Manage your validator with ValidatorConfig V2. Self-service key rotation, IP updates, and ownership transfer. ---- - -import { StaticMermaidDiagram } from '../../../components/StaticMermaidDiagram' - -# ValidatorConfig V2 - -ValidatorConfig V2 ([TIP-1017](/protocol/tips/tip-1017)) is the precompile that manages consensus participants. This config was [activated on mainnet](https://explore.mainnet.tempo.xyz/receipt/0x4716147e3c2bf5c8d014b8c27d6e2af0042d5a5f29bdead256d6f33038702d64) after the T2 hardfork. - -## Validator states - -Once added, a validator immediately becomes a player in the next epoch. - -|epoch ends| B["Player
Receives signing shares"] - B -->|"+1 epoch, receives share"| C["Player
Dealer"] - C -->|"once fully synced"| D["Voter ↔ Proposer"] -`} /> -## What changes for operators - -Validators can perform several operations themselves without coordinating with the Tempo team: - -- **Rotate your validator identity** — swap to a new ed25519 key without losing your committee slot -- **Update IP addresses** — change ingress and egress endpoints independently (V2 supports asymmetric ingress/egress IPs) -- **Transfer ownership** — rebind your validator entry to a new address - -All write operations require either the contract owner or the validator's own address. - -## Precompile address - -```solidity -address constant VALIDATOR_CONFIG_V2 = 0xCCCCCCCC00000000000000000000000000000001; -``` - -## Reading validator state - -### Query active validators - -Use `validators-info` to see the current state of all validators: - -```bash -tempo consensus validators-info --rpc-url https://rpc.tempo.xyz -``` - -This returns the committee of the current epoch (validators with `in_committee = true`) as well as validators that have been added but will only become active in a future epoch. Use this to verify your validator's status after registration or rotation. - -:::info -The on-chain contract represents which validators should eventually be members of the committee, not which ones currently are. Every epoch, the Tempo network performs a distributed key generation ceremony, distributing keys to validators that should be -committee members in the next epoch. So adding, deactivating, or rotating validators entries in epoch `E` can only be taken into account for the next DKG ceremony that runs during epoch `E+1`, and take effect in epoch `E+2`. -::: - -### Look up your validator - -You can query your validator by ethereum address, ed25519 public key, or index: - -```bash -tempo consensus validator
--rpc-url https://rpc.tempo.xyz -``` - -The returned `Validator` struct fields are: - -| Field | Type | Description | -|-------|------|-------------| -| `publicKey` | `bytes32` | Ed25519 communication public key | -| `validatorAddress` | `address` | Validator control address | -| `ingress` | `string` | Inbound address (`IP:port`) | -| `egress` | `string` | Outbound address (`IP`) | -| `index` | `uint64` | Index of the validator (constant under rotation)| -| `addedAtHeight` | `uint64` | Block height when added | -| `deactivatedAtHeight` | `uint64` | Block height when deactivated (`0` = active) | -| `feeRecipient` | `address` | Address that receives block proposal fees | - -## Operator guide - -### Initial registration - -To register a new validator, you must provide the following values to the Tempo team: - -| Value | Format | Description | -|-------|--------|-------------| -| **Validator address** | Ethereum address (`0x…`) | The control address for your validator. Used to authorize on-chain operations (IP updates, rotation, ownership transfer). | -| **Public key** | `0x`-prefixed 32-byte hex | Your ed25519 identity key. Generate one with `tempo consensus generate-private-key` (see [below](#generating-a-signing-key)). | -| **Ingress** | `IP:port` | The inbound address other validators use to reach your node. Must be unique across all active validators. | -| **Egress** | `IP` | The outbound IP address your node uses to connect to other validators. | -| **Fee recipient** | Ethereum address (`0x…`) | The address that receives transaction fees from blocks your validator proposes. If you are not prepared to accept fees, use `0x0000000000000000000000000000000000000000`. | -| **Signature** | `0x`-prefixed hex | An ed25519 signature proving you control the signing key. See below. | - -#### Generating a signing key - -:::warning -Never share your private signing key. Anyone with access to it can impersonate your validator. -::: - -Generate an ed25519 keypair: - -```bash -tempo consensus generate-private-key --output -``` - -Verify the public key: - -```bash -tempo consensus calculate-public-key --private-key -``` - -The public key should match the output of the `generate-private-key` command. - -#### Creating the add-validator signature - -The signature proves ownership of the ed25519 key being registered. Generate it with: - -```bash -tempo consensus create-add-validator-signature \ - --signing-key \ - --validator-address \ - --public-key \ - --ingress \ - --egress \ - --fee-recipient \ - --chain-id-from-rpc-url https://rpc.tempo.xyz -``` - -This outputs a hex-encoded signature. Provide this signature along with the values above to the Tempo team to complete registration. - -Once the Tempo team adds your validator on-chain, it will enter the active set in the [next epoch](#validator-states). - -### Update the Fee Recipient - -The fee recipient used by your validator when constructing block proposals is managed on-chain. This can be updated and takes effect on the next finalized block - -```bash -tempo consensus set-validator-fee-recipient
- --fee-recipient - --rpc-url https://rpc.tempo.xyz - --private-key -``` -### Update IP addresses - -If your node's network endpoints change, update them on-chain. The change takes effect at the next finalized block. - -```bash -tempo consensus set-validator-ip-address
- --ingress :NEW_PROT> - --egress - --rpc-url https://rpc.tempo.xyz - --private-key -``` - -:::warning -Ingress addresses must be unique across all active validators. The transaction will revert if another active validator already uses the same `IP:port`. -::: - -### Rotate validator identity - -The ed25519 key can be changed while keeping your validator index stable. This is useful for key rotation or recovery without leaving and re-joining the committee. [Generate a new signing key](#generating-a-signing-key) first. - -The simplest way to rotate is using the `tempo` CLI, which handles signature creation and the on-chain transaction in one step: - -```bash -tempo consensus rotate-validator \ - --validator-address \ - --public-key \ - --ingress \ - --egress \ - --signing-key \ - --private-key \ - --rpc-url https://rpc.tempo.xyz -``` - -If self-service rotation is not yet enabled for your validator, use `tempo consensus create-rotate-validator-signature` to generate the signature and provide it to the Tempo team. - -:::info -Rotation preserves your validator index and active validator count. The old entry is appended to history as deactivated, and the entry at your index is updated in place. You must use a different ingress address (changing the port is sufficient). -::: - -After rotation, your validator goes through the [standard lifecycle](#validator-states) with the new identity. - -:::danger[Do not shut down the old validator immediately] -After rotation, the rotated-out validator is still a member of the current committee until the next successful DKG ceremony completes. Shutting it down early will degrade network liveness. - -Keep the old validator running and use `validators-info` to monitor its status: - -```bash -tempo consensus validators-info --rpc-url https://rpc.tempo.xyz -``` - -Once the old validator shows `in_committee = false`, it is safe to shut down. -::: - -### Transfer validator ownership - -Rebind your validator entry to a new control address: - -```bash -tempo consensus transfer-validator-ownership
- --rpc-url https://rpc.tempo.xyz \ - --private-key - --new-private-key -``` - -The new address must not already be used by another active validator. - -### Deactivate your validator - -Deactivate your validator. Once deactivated, please keep your the node. The validator is only removed from the commitee once reaching the end of an epoch where the DKG was successful, thus the validator is no longer dealt any new shares. - -See the [validators-info](#query-active-validators) command which lists active validators. It is safe to shut down the node once the deactivated validator is no longer present in this list. - -```bash -tempo consensus deactivate-validator
- --rpc-url https://rpc.tempo.xyz \ - --private-key -``` diff --git a/src/pages/guide/node/validator-keys.mdx b/src/pages/guide/node/validator-keys.mdx new file mode 100644 index 00000000..41693643 --- /dev/null +++ b/src/pages/guide/node/validator-keys.mdx @@ -0,0 +1,59 @@ +--- +title: Managing validator keys +description: Generate, rotate, and recover validator signing keys and shares. +--- + +# Managing validator keys + +Tempo validators use several keys and addresses. This page explains what each one does, how sensitive it is, and how to manage it. + +:::warning +Never share your private signing key. Anyone with access to it can impersonate your validator. The Tempo team will never ask for your private key. Store keys securely and restrict file permissions. +::: + +## Key and address overview + +| Key / Address | Type | What it does | Sensitivity | How to change | +|---|---|---|---|---| +| **Signing key** | Ed25519 keypair | Identifies your validator in the consensus protocol. Used for DKG participation, block proposals, and voting. | **Critical** — anyone with this key can impersonate your validator. | [Rotate validator identity](/guide/node/validator-lifecycle#rotate-validator-identity) | +| **Validator operator address** | Ethereum address (`0x…`) | The control address that authorizes on-chain operations: IP updates, key rotation, ownership transfer, and deactivation. | **High** — controls all validator configuration. | [Transfer validator ownership](/guide/node/validator-lifecycle#transfer-validator-ownership) | +| **Fee recipient** | Ethereum address (`0x…`) | Receives transaction fees from blocks your validator proposes. | **Low** — changing it only redirects future fee revenue, no security impact. | [Update fee recipient](/guide/node/validator-lifecycle#update-the-fee-recipient) | +| **Signing share** | BLS12-381 key share | A share of the committee's threshold signing key, used to sign block notarizations and finalizations. | **Managed automatically** — updated every DKG ceremony (~3 hours). Lost shares are recovered from the network on restart. | Automatic (see [recovery](#signing-share-recovery)) | + +## Generating a signing key + +:::warning +Never share your private signing key. Anyone with access to it can impersonate your validator. The Tempo team will never ask for your private key. +::: + +Generate an ed25519 keypair: + +```bash +tempo consensus generate-private-key --output +``` + +Verify the public key: + +```bash +tempo consensus calculate-public-key --private-key +``` + +The public key should match the output of the `generate-private-key` command. + +## Signing key rotation + +The ed25519 signing key can be rotated while preserving your validator index and committee slot. See [Rotate validator identity](/guide/node/validator-lifecycle#rotate-validator-identity) for the full procedure. + +## Signing share recovery + +If the signing share is lost — for example by deleting `/consensus` — the node will recover a new share in the following epochs from the network when it restarts. + +## ValidatorConfig V2 precompile + +All key and identity operations are executed through the ValidatorConfig V2 precompile ([TIP-1017](/protocol/tips/tip-1017)): + +```solidity +address constant VALIDATOR_CONFIG_V2 = 0xCCCCCCCC00000000000000000000000000000001; +``` + +All write operations require either the contract owner or the validator's own address. diff --git a/src/pages/guide/node/validator-lifecycle.mdx b/src/pages/guide/node/validator-lifecycle.mdx new file mode 100644 index 00000000..81d6b175 --- /dev/null +++ b/src/pages/guide/node/validator-lifecycle.mdx @@ -0,0 +1,184 @@ +--- +title: Controlling validator lifecycle +description: Start, stop, register, rotate, deactivate, and transfer ownership of your Tempo validator. +--- + +# Controlling validator lifecycle + +This guide covers operational actions for managing a Tempo validator — from initial registration to deactivation. + +## Starting and stopping + +Use `SIGINT` (Ctrl+C) or `SIGTERM` to gracefully stop the node: + +```bash +# If running directly +kill -INT + +# If running via systemd +sudo systemctl stop tempo +``` + +The node will finish processing the current block before shutting down. Avoid using `SIGKILL` as it may corrupt the database. + +## Resetting your validator's data + +:::danger +**You cannot reset a validator's data and continue with the same identity.** Doing so risks inconsistent voting, which can cause irrecoverable network safety failures. +::: + +If you need to reset your validator's data, you must rotate to a new validator identity. This requires coordinating with the Tempo team to deactivate your old identity and register a new one. Don't hesitate to reach out — even with the Tempo team, this is a routine operation. + +:::info +Self-service data resets are coming soon. Once available, you will be able to rotate to a new identity and reset your data without coordinating with the Tempo team. See [Rotate validator identity](#rotate-validator-identity). +::: + +## Rotate validator identity + +The ed25519 key can be changed while keeping your validator index stable. This is useful for key rotation or recovery without leaving and re-joining the committee. [Generate a new signing key](/guide/node/validator-keys#generating-a-signing-key) first. + +:::danger[Do not shut down the old validator] +Unlike Ethereum, you must **keep your old validator running** after rotation. The rotated-out validator is still a dealer in the committee for at least one more epoch. Shutting it down early will degrade network liveness. See the [exit timeline](#exit-timeline) for the full epoch-by-epoch breakdown. + +Keep the old validator running until it shows `in_committee: false`: + +::::code-group +```bash [Mainnet] +tempo consensus validator --rpc-url https://rpc.tempo.xyz +``` +```bash [Testnet] +tempo consensus validator --rpc-url https://rpc.testnet.tempo.xyz +``` +:::: +::: + +The simplest way to rotate is using the `tempo` CLI, which handles signature creation and the on-chain transaction in one step: + +::::code-group +```bash [Mainnet] +tempo consensus rotate-validator \ + --validator-address \ + --public-key \ + --ingress \ + --egress \ + --signing-key \ + --private-key \ + --rpc-url https://rpc.tempo.xyz +``` +```bash [Testnet] +tempo consensus rotate-validator \ + --validator-address \ + --public-key \ + --ingress \ + --egress \ + --signing-key \ + --private-key \ + --rpc-url https://rpc.testnet.tempo.xyz +``` +:::: + +If self-service rotation is not yet enabled for your validator, use `tempo consensus create-rotate-validator-signature` to generate the signature and provide it to the Tempo team. + +:::info +Rotation preserves your validator index and active validator count. The old entry is appended to history as deactivated, and the entry at your index is updated in place. You must use a different ingress address (changing the port is sufficient). +::: + +After rotation, your validator goes through the [standard state transitions](/guide/node/validator-status#state-transitions) with the new identity. + +## Update IP addresses + +If your node's network endpoints change, update them on-chain. The change takes effect at the next finalized block. + +::::code-group +```bash [Mainnet] +tempo consensus set-validator-ip-address
\ + --ingress \ + --egress \ + --rpc-url https://rpc.tempo.xyz \ + --private-key +``` +```bash [Testnet] +tempo consensus set-validator-ip-address
\ + --ingress \ + --egress \ + --rpc-url https://rpc.testnet.tempo.xyz \ + --private-key +``` +:::: + +:::warning +Ingress addresses must be unique across all active validators. The transaction will revert if another active validator already uses the same `IP:port`. +::: + +## Update the fee recipient + +The fee recipient used by your validator when constructing block proposals is managed on-chain. This can be updated and takes effect on the next finalized block. + +::::code-group +```bash [Mainnet] +tempo consensus set-validator-fee-recipient
\ + --fee-recipient \ + --rpc-url https://rpc.tempo.xyz \ + --private-key +``` +```bash [Testnet] +tempo consensus set-validator-fee-recipient
\ + --fee-recipient \ + --rpc-url https://rpc.testnet.tempo.xyz \ + --private-key +``` +:::: + +## Transfer validator ownership + +Rebind your validator entry to a new control address: + +::::code-group +```bash [Mainnet] +tempo consensus transfer-validator-ownership
\ + --rpc-url https://rpc.tempo.xyz \ + --private-key \ + --new-private-key +``` +```bash [Testnet] +tempo consensus transfer-validator-ownership
\ + --rpc-url https://rpc.testnet.tempo.xyz \ + --private-key \ + --new-private-key +``` +:::: + +The new address must not already be used by another active validator. + +## Deactivate your validator + +Deactivate your validator when you want to leave the active set. + +:::warning[Do not shut down your node immediately after deactivating] +Your validator remains a dealer in the committee for at least one more epoch after deactivation. Shutting it down early will degrade network liveness. See the [exit timeline](#exit-timeline) below. +::: + +::::code-group +```bash [Mainnet] +tempo consensus deactivate-validator
\ + --rpc-url https://rpc.tempo.xyz \ + --private-key +``` +```bash [Testnet] +tempo consensus deactivate-validator
\ + --rpc-url https://rpc.testnet.tempo.xyz \ + --private-key +``` +:::: + +### Exit timeline + +Deactivation is not instant — your validator is phased out over two epochs: + +| Epoch | State | `is_dealer` | `is_player` | `in_committee` | +|-------|-------|:-----------:|:-----------:|:--------------:| +| **E** (deactivation) | Active | `true` | `true` | `true` | +| **E+1** | Exiting | `true` | `false` | `true` | +| **E+2** | Exited | `false` | `false` | `false` | + +Keep your node running until your validator shows `in_committee: false`. Use [validator lookup](/guide/node/validator-status#look-up-your-validator) to check — it is safe to shut down the node at that point. diff --git a/src/pages/guide/node/operate-validator.mdx b/src/pages/guide/node/validator-monitoring.mdx similarity index 57% rename from src/pages/guide/node/operate-validator.mdx rename to src/pages/guide/node/validator-monitoring.mdx index 0dc4bccf..b6b3b551 100644 --- a/src/pages/guide/node/operate-validator.mdx +++ b/src/pages/guide/node/validator-monitoring.mdx @@ -1,45 +1,18 @@ --- -title: Operate your validator node -description: Day-to-day operations for Tempo validators. Node lifecycle, monitoring, metrics, log management, and Grafana dashboards. +title: Monitoring a validator +description: Monitor validator health with metrics, Grafana dashboards, and log management. --- -# Operate your validator +# Monitoring a validator -This guide covers day-to-day operations for running a Tempo validator in production. +This guide covers how to monitor your Tempo validator's health, diagnose issues, and manage logs. -- **[Validator Configuration](/guide/node/validator-config-v2)** — self-service key rotation, IP updates, and ownership transfers +## Consensus states -## Checking your validator's state - -Monitor these metrics to track your validator's state: - -```bash -# Is your validator connected to other peers (i.e. is it considered a syncer)? This should be >0 in at most 3 hours after your validator's addition -curl -s localhost:8002/metrics | grep consensus_network_spawner_connections - -# How many times YOUR node has been a dealer (distributing shares) - this metric should be >0 over 6 hours -curl -s localhost:8002/metrics | grep consensus_engine_dkg_manager_how_often_dealer - -# How many times YOUR node has been a player (receiving shares) - this metric should be >0 over 6 hours -curl -s localhost:8002/metrics | grep consensus_engine_dkg_manager_how_often_player - -# Successful ceremonies (should increase every ~3 hours) -curl -s localhost:8002/metrics | grep consensus_engine_dkg_manager_ceremony_successes_total - -# Failed ceremonies (should stay at 0 or increase rarely) -curl -s localhost:8002/metrics | grep consensus_engine_dkg_manager_ceremony_failures_total -``` - -If `how_often_dealer` or `how_often_player` is increasing, your node is actively participating in DKG ceremonies. After your validator has been added to the network, you should alert on these metrics, as they indicate that your validator is actively participating in the network. - -If your validator is not connected to other peers, but at least 3 epochs have passed, please check that your node is properly configured - e.g. firewall settings are open to other peers. If you have reset your validator's state, -your validator might've been blocked due to double-signing a block. In that case, please reach out to the Tempo team. - -### Consensus - -#### Proposer +### Proposer Your validator is currently the leader and proposing blocks. Check proposal activity: + ```bash # Number of blocks your node has built and resolved curl -s localhost:9000/metrics | grep reth_payloads_resolved_block @@ -47,7 +20,7 @@ curl -s localhost:9000/metrics | grep reth_payloads_resolved_block If this counter is increasing, your validator is actively proposing blocks. -#### Voter +### Voter Your validator is voting on blocks proposed by others (notarization and finalization). This is the most common state. @@ -58,9 +31,9 @@ curl -s localhost:8002/metrics | grep consensus_engine_epoch_manager_simplex_bat This counter should increase steadily when your node is participating in consensus. -### Execution +## Execution states -#### Catching up +### Catching up Your node is syncing historical blocks. @@ -71,7 +44,7 @@ curl -s localhost:6060/metrics | grep reth_sync_checkpoint If `reth_sync_checkpoint` shows stages other than `Finish`, you're still syncing. -#### Up to sync +### Up to sync Your node is fully synced and processing new blocks in real-time. @@ -82,39 +55,53 @@ curl -s localhost:8002/metrics | grep -E "marshal_finalized_height|marshal_proce Both values should be nearly equal and increasing together. -## Node Lifecycle +## Metrics glossary + +| Metric Name | Description | When to alert? | Meaning | +| --- | --- | --- | --- | +| `consensus_engine_dkg_manager_ceremony_successes_total` | Number of successful DKG ceremonies | Critical when it hasn't increased in 12 hours | Your node has participated in a successful DKG ceremony | +| `consensus_engine_dkg_manager_ceremony_failures_total` | Number of failed DKG ceremonies | Warning when it increases | Your node has failed to participate in a DKG ceremony | +| `consensus_engine_dkg_manager_how_often_dealer` | How many times your node has been a dealer (distributing shares) | Warning when it hasn't increased in 6 hours | Your node is distributing signing shares to other validators | +| `consensus_engine_dkg_manager_how_often_player` | How many times your node has been a player (receiving shares) | Warning when it hasn't increased in 6 hours | Your node is receiving signing shares from dealers | +| `consensus_engine_marshal_finalized_height` | Latest finalized height your node is aware of | Warning when it hasn't increased in 1 hour, critical when it hasn't increased in 3 hours | Your node is aware of the latest finalized height | +| `consensus_engine_marshal_processed_height` | Latest height your node has processed | Critical when it hasn't increased in an hour, warning when it's behind finalized height | Your node is processing blocks | +| `consensus_network_spawner_connections` | Number of active peers on the consensus layer | Warning when it's below the expected validator count, critical when it's 0 | Your node is connected to the consensus layer | +| `consensus_engine_epoch_manager_simplex_batcher_inbound_messages_total` | Number of inbound messages related to voting on the consensus layer | Warning when it's not increasing and your node is synced | Your node is receiving voting messages from the consensus layer | +| `consensus_application_parent_ahead_of_local_time` | Number of times the parent block timestamp was ahead of local time | Warning when it increases frequently | Your node's clock may be out of sync — check NTP configuration | +| `reth_sync_checkpoint` | Current sync progress | Warning on when there are no changes in the `Finish` stage | Your node is syncing with the network | +| `reth_payloads_resolved_block` | Number of built and resolved payloads | Warning when it hasn't increased in 12 hours | Your node has built and resolved blocks | + +## Grafana dashboard -### Starting and Stopping +We provide a pre-built Grafana dashboard for monitoring your validator. It visualizes key metrics including node status, sync progress, voting activity, consensus latency, and execution performance. -Use `SIGINT` (Ctrl+C) or `SIGTERM` to gracefully stop the node: +### Importing the dashboard -```bash -# If running directly -kill -INT +1. Download the dashboard JSON from the [Tempo repository](https://github.com/tempoxyz/tempo/blob/main/contrib/grafana/dashboards/validator-health.json) -# If running via systemd -sudo systemctl stop tempo -``` +2. In Grafana, go to **Dashboards → Import** -The node will finish processing the current block before shutting down. Avoid using `SIGKILL` as it may corrupt the database. +3. Either paste the JSON content or upload the file -### Resetting your validator's data +4. Select your Prometheus and Loki datasources when prompted -:::danger -**You cannot reset a validator's data and continue with the same identity.** Doing so risks inconsistent voting, which can cause irrecoverable network safety failures. -::: +5. Click **Import** -If you need to reset your validator's data, you must rotate to a new validator identity. See key rotation under [Validator Configuration](/guide/node/validator-config-v2#rotate-validator-identity). +### Dashboard variables -## Key Management +The dashboard uses these template variables: -For signing key rotation, IP address updates, signing share recovery, and ownership transfers, see: +| Variable | Description | +| --- | --- | +| `datasource` | Prometheus datasource | +| `loki_ds` | Loki datasource (for log-based panels) | +| `network_name` | Filter by network/validator job name | -- **[Validator Configuration](/guide/node/validator-config-v2)** (post-T2) — self-service operations +Make sure your Prometheus is scraping metrics from your validator's metrics endpoint (default: `localhost:8002/metrics`). -## Log Management +## Log management -### Parsing Logs +### Parsing logs Tempo logs include ANSI escape codes for colors. To strip them for grep/awk: @@ -132,7 +119,7 @@ If you're using Loki, you can also use the `decolorize` filter to strip colors: {job="tempo-node"} | decolorize ``` -### Log Levels +### Log levels Control verbosity with `RUST_LOG`: @@ -146,46 +133,3 @@ RUST_LOG=info,tempo_commonware_node::consensus=debug # Quiet mode (warnings and errors only) RUST_LOG=warn ``` - -## Metrics glossary - -This is a table of the most important metrics to watch on your validator node: - -| Metric Name | Description | When to alert? | Meaning | -| --- | --- | --- | --- | -| `consensus_engine_dkg_manager_ceremony_successes_total` | Number of successful DKG ceremonies | Critical when it hasn't increased in 12 hours | Your node has participated in a successful DKG ceremony | -| `consensus_engine_dkg_manager_ceremony_failures_total` | Number of failed DKG ceremonies | Warning when it increases | Your node has failed to participate in a DKG ceremony | -| `consensus_engine_marshal_finalized_height` | Latest finalized height your node is aware of | Warning when it hasn't increased in 1 hour, critical when it hasn't increased in 3 hours | Your node is aware of the latest finalized height | -| `consensus_engine_marshal_processed_height` | Latest height your node has processed | Critical when it hasn't increased in an hour, warning when it's behind finalized height | Your node is processing blocks | -| `consensus_network_spawner_connections` | Number of active peers on the consensus layer | Warning when it's below the expected validator count, critical when it's 0 | Your node is connected to the consensus layer | -| `consensus_engine_epoch_manager_simplex_batcher_inbound_messages_total` | Number of inbound messages related to voting on the consensus layer | Warning when it's not increasing and your node is synced | Your node is receiving voting messages from the consensus layer | -| `reth_sync_checkpoint` | Current sync progress | Warning on when there are no changes in the `Finish` stage | Your node is syncing with the network | -| `reth_payloads_resolved_block` | Number of built and resolved payloads | Warning when it hasn't increased in 12 hours | Your node has built and resolved blocks | - -## Grafana Dashboard - -We provide a pre-built Grafana dashboard for monitoring your validator. It visualizes key metrics including node status, sync progress, voting activity, consensus latency, and execution performance. - -### Importing the dashboard - -1. Download the dashboard JSON from the [Tempo repository](https://github.com/tempoxyz/tempo/blob/main/contrib/grafana/dashboards/validator-health.json) - -2. In Grafana, go to **Dashboards → Import** - -3. Either paste the JSON content or upload the file - -4. Select your Prometheus and Loki datasources when prompted - -5. Click **Import** - -### Dashboard variables - -The dashboard uses these template variables: - -| Variable | Description | -| --- | --- | -| `datasource` | Prometheus datasource | -| `loki_ds` | Loki datasource (for log-based panels) | -| `network_name` | Filter by network/validator job name | - -Make sure your Prometheus is scraping metrics from your validator's metrics endpoint (default: `localhost:8002/metrics`). diff --git a/src/pages/guide/node/validator-setup.mdx b/src/pages/guide/node/validator-setup.mdx new file mode 100644 index 00000000..184adeb7 --- /dev/null +++ b/src/pages/guide/node/validator-setup.mdx @@ -0,0 +1,156 @@ +--- +title: Validator Onboarding +description: Generate signing keys and run your Tempo validator node for the first time. +--- + +# Validator onboarding + +:::info +The active validator set is currently permissioned. If you are interested in becoming a validator, please [get in touch](mailto:partners@tempo.xyz) with the Tempo team. +::: + +This guide walks through registering, generating your signing key, and starting your validator node for the first time. Before proceeding, make sure you have completed [system requirements](/guide/node/system-requirements) and [installation](/guide/node/installation). + +## Initial registration + +Registering a validator takes three steps: + +1. **[Generate a signing keypair](#step-1-generate-a-signing-keypair)** — Create an ed25519 private key and derive the public key. +2. **[Create the add-validator signature](#step-2-create-the-add-validator-signature)** — Prove ownership of the signing key by producing a signature over your registration details. +3. **[Submit registration details](#step-3-submit-registration-details)** — Provide the Tempo team with all required values to add your validator on-chain. + +### Step 1: Generate a signing keypair + +:::warning +Never share your private signing key. Anyone with access to it can impersonate your validator. The Tempo team will never ask for your private key. +::: + +Generate an ed25519 keypair: + +```bash +tempo consensus generate-private-key --output +``` + +Verify the public key: + +```bash +tempo consensus calculate-public-key --private-key +``` + +The public key should match the output of the `generate-private-key` command. + +### Step 2: Create the add-validator signature + +The signature proves ownership of the ed25519 key being registered. Generate it with: + +::::code-group +```bash [Mainnet] +tempo consensus create-add-validator-signature \ + --signing-key \ + --validator-address \ + --public-key \ + --ingress \ + --egress \ + --fee-recipient \ + --chain-id-from-rpc-url https://rpc.tempo.xyz +``` +```bash [Testnet] +tempo consensus create-add-validator-signature \ + --signing-key \ + --validator-address \ + --public-key \ + --ingress \ + --egress \ + --fee-recipient \ + --chain-id-from-rpc-url https://rpc.testnet.tempo.xyz +``` +:::: + +### Step 3: Submit registration details + +Provide the following values along with the signature to the Tempo team: + +| Value | Format | Description | +|-------|--------|-------------| +| **Validator address** | Ethereum address (`0x…`) | The control address for your validator. Used to authorize on-chain operations (IP updates, rotation, ownership transfer). | +| **Public key** | `0x`-prefixed 32-byte hex | Your ed25519 identity key (from [Step 1](#step-1-generate-a-signing-keypair)). | +| **Ingress** | `IP:port` | The inbound address other validators use to reach your node. Must be unique across all active validators. | +| **Egress** | `IP` | The outbound IP address your node uses to connect to other validators. | +| **Fee recipient** | Ethereum address (`0x…`) | The address that receives transaction fees from blocks your validator proposes. If you are not prepared to accept fees, use `0x0000000000000000000000000000000000000000`. | +| **Signature** | `0x`-prefixed hex | The ed25519 signature proving you control the signing key (from [Step 2](#step-2-create-the-add-validator-signature)). | + +Once the Tempo team adds your validator on-chain, it will enter the active set in the [next epoch](/guide/node/validator-status#state-transitions). + +## Running the validator + +The process for running a validator node is very similar to [running a full node](/guide/node/rpc). + +You should start by downloading the latest snapshot. Downloading the snapshot allows your validator to start participating in consensus much faster. + +::::code-group +```bash [Mainnet] +tempo download --chain mainnet +``` +```bash [Testnet] +tempo download --chain moderato +``` +:::: + +Once you've downloaded the snapshot and have been whitelisted on-chain, you can proceed to run the validator node: + +::::code-group +```bash [Mainnet] +tempo node --datadir \ + --chain mainnet \ + --consensus.signing-key \ + --telemetry-url +``` +```bash [Testnet] +tempo node --datadir \ + --chain moderato \ + --consensus.signing-key \ + --telemetry-url +``` +:::: +The notable difference between RPC nodes and validator nodes is the omission of the `--follow` argument and the addition of the `--consensus.signing-key` argument. + +Once your node is up, it may not start syncing immediately. This is because your node might not be part of the active set. In most cases, your validator will enter the active set in under 6 hours after the on-chain addition of the validator identity. + +### Optional flags + +| Flag | Description | +|------|-------------| +| `--telemetry-url ` | Unified metrics and logs export. See [Telemetry endpoint](#telemetry-endpoint) below for details. **We ask all validators to configure this so we can support troubleshooting.** | +| `--telemetry-metrics-interval ` | Interval for pushing metrics (default: `10s`). | +| `--consensus.datadir ` | Store consensus data on a separate volume (e.g., AWS EBS) while keeping execution state on high-performance local disks. Migrate by copying `/consensus` to the new location. | + +### Telemetry endpoint + +The `--telemetry-url` flag enables unified telemetry export. It pushes two types of data to a Tempo-operated metrics backend: + +- **Prometheus metrics** — execution-layer (reth) and consensus-layer metrics in Prometheus text format, pushed at a configurable interval (default every 10 seconds). +- **Structured logs** — operational logs exported via OTLP at `debug` level, covering consensus events, block processing, and sync progress. + +The URL must include credentials: `--telemetry-url https://user:pass@metrics.example.com` + +#### What is collected + +| Category | Examples | +|----------|---------| +| Execution metrics | Block processing times, transaction pool size, peer count, sync status, database stats | +| Consensus metrics | Epoch and view progress, DKG ceremony status, proposal and finalization counts | +| Operational logs | Consensus state transitions, block proposals, sync progress, error events | + +All consensus metrics are namespaced under a `consensus` prefix. + +#### What is **not** collected + +The telemetry endpoint does **not** collect any ambient information about the host machine. Specifically: + +- No hostname, IP address, or machine identifiers +- No operating system, CPU, memory, or disk information +- No information about other processes or services running on the machine +- No file paths or directory structures beyond the node's own data directory references in logs +- No network topology or firewall configuration + +The data is strictly limited to the node's own operational metrics and logs. diff --git a/src/pages/guide/node/validator-status.mdx b/src/pages/guide/node/validator-status.mdx new file mode 100644 index 00000000..72ddb3fd --- /dev/null +++ b/src/pages/guide/node/validator-status.mdx @@ -0,0 +1,132 @@ +--- +title: Checking validator status +description: Understand validator state transitions, check your validator's participation status, and query on-chain status. +--- + +import { StaticMermaidDiagram } from '../../../components/StaticMermaidDiagram' + +# Checking validator status + +Your validator moves through different states after registration. Understanding these states helps you verify that your node is healthy and participating in consensus. + +## Look up your validator + +You can query a single validator by its consensus public key (the ed25519 key generated during [initial setup](/guide/node/validator-setup#step-1-generate-a-signing-keypair) — see also [Managing validator keys](/guide/node/validator-keys#generating-a-signing-key)): + +::::code-group +```bash [Mainnet] +tempo consensus validator --rpc-url https://rpc.tempo.xyz +``` +```bash [Testnet] +tempo consensus validator --rpc-url https://rpc.testnet.tempo.xyz +``` +:::: + +Example output: + +```json +{ + "current_epoch": 18, + "current_height": 5733680, + "onchain_address": "0x1234567890abcdef1234567890abcdef12345678", + "public_key": "1a2b3c4d5e6f7a8b9c0d1e2f3a4b5c6d7e8f9a0b1c2d3e4f5a6b7c8d9e0f1a2b", + "inbound_address": "203.0.113.10:9000", + "outbound_address": "203.0.113.10:9001", + "fee_recipient": "0xabcdefabcdefabcdefabcdefabcdefabcdefabcd", + "index": 5, + "active": true, + "is_dkg_dealer": true, + "is_dkg_player": true, + "in_committee": true +} +``` + +| Field | Description | +|-------|-------------| +| `current_epoch` | The epoch at the time of the query | +| `current_height` | The block height at the time of the query | +| `onchain_address` | Validator control address | +| `public_key` | Ed25519 consensus public key (hex) | +| `inbound_address` | Inbound address (`IP:port`) for incoming connections | +| `outbound_address` | Outbound IP address for firewall whitelisting | +| `fee_recipient` | Address that receives block proposal fees | +| `index` | Index of the validator (constant under rotation) | +| `active` | Whether the validator is active in the current contract state | +| `is_dkg_dealer` | Whether the validator is a dealer (distributing shares) in the current epoch | +| `is_dkg_player` | Whether the validator is a player (receiving shares) in the current epoch | +| `in_committee` | Whether the validator is in the committee for the current epoch | + +## State transitions + +Every state transition happens on epoch boundaries. Currently on mainnet and testnet, the epoch length is around 3 hours. + + Registration: Add / Rotate in + Registration --> Player: Epoch boundary + Player --> Dealer: Epoch boundary + Dealer --> Active: Fully synced + Active --> Exiting: Deactivate / Rotate out + Exiting --> Exited: Epoch boundary + Exited --> [*] + + state "Registration — Epoch E" as Registration + state "Player — Epoch E+1\nReceiving signing shares" as Player + state "Dealer — Epoch E+2\nDistributing shares" as Dealer + state "Active — Voter & Proposer\nProposing and voting on blocks" as Active + state "Exiting — Epoch E+1\nDealer only, no longer a player" as Exiting + state "Exited — Epoch E+2\nFully out of committee" as Exited +`} /> + +### Registration (epoch E) + +Once your validator is added to the on-chain contract, it is immediately registered on the p2p peer set and starts syncing blocks from the network. + +### Player (epoch E+1) + +Your validator is receiving consensus signing shares from dealers during the DKG ceremony. + +### Dealer / Validator (epoch E+2) + +Your validator is distributing consensus signing shares to other validators during the ceremony. Once your node is fully synced, it will also be able to propose blocks and vote on other validators' proposals. + +### Exiting (epoch E+1 after deactivation) + +After deactivation in epoch E, your validator is still a **dealer** (distributing shares) during epoch E+1, but is **no longer a player** (not receiving new shares). It is in the process of being removed from the committee. + +### Exited (epoch E+2 after deactivation) + +Your validator is fully out of the committee, assuming no DKG failures occurred in E+1. It is safe to shut down the node once `in_committee` is `false`. + +:::info +The on-chain contract represents which validators should eventually be members of the committee, not which ones currently are. Every epoch, the Tempo network performs a distributed key generation ceremony, distributing keys to validators that should be committee members in the next epoch. So adding, deactivating, or rotating validator entries in epoch `E` can only be taken into account for the next DKG ceremony that runs during epoch `E+1`, and take effect in epoch `E+2`. +::: + +## Checking state via metrics + +Monitor these metrics to track your validator's state: + +```bash +# Is your validator connected to other peers (i.e. is it registered)? +# This should be >0 in at most 3 hours after your validator's addition. +curl -s localhost:8002/metrics | grep consensus_network_spawner_connections + +# How many times YOUR node has been a dealer (distributing shares) +# This metric should be >0 over 6 hours. +curl -s localhost:8002/metrics | grep consensus_engine_dkg_manager_how_often_dealer + +# How many times YOUR node has been a player (receiving shares) +# This metric should be >0 over 6 hours. +curl -s localhost:8002/metrics | grep consensus_engine_dkg_manager_how_often_player + +# Successful ceremonies (should increase every ~3 hours) +curl -s localhost:8002/metrics | grep consensus_engine_dkg_manager_ceremony_successes_total + +# Failed ceremonies (should stay at 0 or increase rarely) +curl -s localhost:8002/metrics | grep consensus_engine_dkg_manager_ceremony_failures_total +``` + +If `how_often_dealer` or `how_often_player` is increasing, your node is actively participating in DKG ceremonies. After your validator has been added to the network, you should alert on these metrics, as they indicate that your validator is actively participating in the network. + +If your validator is not connected to other peers, but at least 3 epochs have passed, check that your node is properly configured — e.g. firewall settings are open to other peers. If you have reset your validator's state, your validator might have been blocked due to double-signing a block. In that case, please reach out to the Tempo team — even with the Tempo team, resolving this requires coordinating a new validator identity. + + diff --git a/src/pages/guide/node/validator-troubleshooting.mdx b/src/pages/guide/node/validator-troubleshooting.mdx new file mode 100644 index 00000000..d2736415 --- /dev/null +++ b/src/pages/guide/node/validator-troubleshooting.mdx @@ -0,0 +1,134 @@ +--- +title: Troubleshooting and FAQ +description: Common issues and solutions for Tempo validator operators. +--- + +# Troubleshooting and FAQ + +## My node is not proposing blocks + +Once your node is proposing blocks, it will start emitting logs like these: + +``` +INFO handle_propose{epoch=18 view=387213 parent.view=387212 parent.digest=0x43885416b4a7ae7550c615ad4dc702045cd26540b78fa2bd75abdcbfbef02d9d}: tempo_commonware_node::consensus::application::actor: constructed proposal proposal.digest=0x6baa8fa813beea491cf598024d8b31cb2927e7ba1b92a29899a795dbafd682c6 proposal.height=5733680 +``` + +If you do not see logs like these: +- Check that your validator is part of the active set, and is both [a player and a dealer](/guide/node/validator-status). +- Check that your node is synced up to the [latest block height](https://explorer.tempo.xyz/). +- Check that your outgoing IP address matches the one whitelisted on the validator smart contract: + +::::code-group +```bash [Mainnet] +tempo consensus validators-info --rpc-url https://rpc.tempo.xyz +``` +```bash [Testnet] +tempo consensus validators-info --rpc-url https://rpc.testnet.tempo.xyz +``` +:::: + +## My node is not connecting to peers + +If `consensus_network_spawner_connections` remains at `0` for more than 3 hours after your validator was added on-chain: + +- Verify your firewall allows inbound connections on the ingress port you registered. +- Verify your egress IP matches the one registered on-chain — check with `tempo consensus validator --rpc-url https://rpc.tempo.xyz`. +- If you have reset your validator's data without rotating to a new identity, your node may have been blocked due to double-signing. In that case, [reach out to the Tempo team](mailto:partners@tempo.xyz) to coordinate a new validator identity. + +## My node's DKG metrics are not increasing + +If `how_often_dealer` and `how_often_player` are not increasing after 6 hours: + +- Confirm your node is connected to peers (see above). +- Check that your validator has progressed past the [Syncer state](/guide/node/validator-status#state-transitions) — it takes at least one full epoch (~3 hours) after on-chain addition before your node participates in DKG. +- Check DKG failure count: if `consensus_engine_dkg_manager_ceremony_failures_total` is increasing, your node may be failing to complete ceremonies. Enable debug logging for more detail: + ```bash + RUST_LOG=info,tempo_commonware_node::dkg=debug + ``` + +## My node is rejecting blocks or missing proposals + +If your node logs block validation errors (e.g. "block timestamp is in the future") or you notice missed proposals, a potential root cause is **clock drift**. Check whether `parent_ahead_of_local_time` is increasing in your metrics — if so, your system clock is behind the network. + +To fix: + +1. **Check your current sync status:** + ```bash + timedatectl status + ``` + Confirm `System clock synchronized: yes` and `NTP service: active`. + +2. **Install chrony** (if not already installed): + ```bash + sudo apt install chrony + sudo systemctl enable --now chronyd + ``` + +3. **Verify the offset is acceptable:** + ```bash + chronyc tracking + ``` + The **System time** offset should be under a few milliseconds. + +4. **Restart your node** after fixing the clock to clear any cached invalid block state. + +See [Time Synchronization](/guide/node/system-requirements#time-synchronization) for full setup details. + +## I accidentally deleted my consensus data directory + +If you deleted `/consensus`, your signing share is lost but will be [automatically recovered](/guide/node/validator-keys#signing-share-recovery) from the network when the node restarts. Your node will re-join the committee after the next successful DKG ceremony. + +:::danger +Do **not** delete the entire data directory and attempt to re-sync with the same signing key. This risks double-signing and will require [rotating to a new identity](/guide/node/validator-lifecycle#resetting-your-validators-data). +::: + +## How long does it take for my validator to become active? + +After on-chain registration, your validator follows the [state transition timeline](/guide/node/validator-status#state-transitions): + +1. **Epoch E** (immediate) — registered on the p2p network, starts syncing. +2. **Epoch E+1** (~3 hours) — becomes a player, receives signing shares. +3. **Epoch E+2** (~6 hours) — becomes a dealer/validator, can propose and vote once synced. + +In most cases, your validator will be fully active within 6 hours. + +## How long does it take for my validator to exit? + +After deactivation, your validator is phased out over two epochs: + +1. **Epoch E** — deactivation transaction submitted. Validator remains fully active. +2. **Epoch E+1** (~3 hours) — still a dealer but no longer a player. In the process of being removed. +3. **Epoch E+2** (~6 hours) — fully out of the committee (assuming no DKG failures). + +Keep your node running until `in_committee: false` — check with [validator lookup](/guide/node/validator-status#look-up-your-validator). + +## Can I register my validator without the Tempo team? + +No — the active validator set is currently permissioned. Only the contract owner can add validators on-chain. To get started, [contact the Tempo team](mailto:partners@tempo.xyz). + +The onboarding process: + +1. **You** generate your signing key and registration signature ([Steps 1–2](/guide/node/validator-setup#initial-registration)). +2. **You** provide the required values to the Tempo team ([Step 3](/guide/node/validator-setup#step-3-submit-registration-details)). +3. **The Tempo team** adds your validator on-chain. +4. **You** download a snapshot and start your node ([Running the validator](/guide/node/validator-setup#running-the-validator)). + +## What can I do without the Tempo team? + +Once your validator is registered, most operations are self-service: + +- [Rotate your signing key](/guide/node/validator-lifecycle#rotate-validator-identity) +- [Update IP addresses](/guide/node/validator-lifecycle#update-ip-addresses) +- [Update your fee recipient](/guide/node/validator-lifecycle#update-the-fee-recipient) +- [Transfer validator ownership](/guide/node/validator-lifecycle#transfer-validator-ownership) +- [Deactivate your validator](/guide/node/validator-lifecycle#deactivate-your-validator) + +Only **initial registration** and **reactivation** require the Tempo team. + +## How do I check which version I'm running? + +```bash +tempo --version +``` + +Compare with the latest release on the [network upgrades](/guide/node/network-upgrades) page to ensure you're on a supported version. diff --git a/src/pages/guide/node/validator.mdx b/src/pages/guide/node/validator.mdx index d0957772..3a77f38e 100644 --- a/src/pages/guide/node/validator.mdx +++ b/src/pages/guide/node/validator.mdx @@ -1,99 +1,57 @@ --- -description: Configure and run a Tempo validator node. Generate signing keys, participate in DKG ceremonies, and troubleshoot consensus issues. +description: Overview of running a Tempo validator node. --- +import { Cards, Card } from 'vocs' + # Running a validator node Validator nodes secure Tempo by validating blocks and participating in consensus. -## Deploying a validator - -The active validator set is currently permissioned. If you are interested in becoming a validator, please [get in touch](mailto:partners@tempo.xyz) with the Tempo team. See [Initial registration](/guide/node/validator-config-v2#initial-registration) for technical details. - -## Cryptographic key hierarchy - -All validators have two kinds of keys: -* signing key - this is an ED25519 keypair used to identify your validator amongst others -* signing share - this is a share of a BLS12-381 keypair used to sign block notarizations and finalizations - -The signing key is static. Pre-T2, it can only be rotated by removing and re-creating a validator on-chain with the Tempo team. Post-T2, [ValidatorConfig V2](/guide/node/validator-config-v2) enables self-service rotation via the `rotateValidator` precompile. The signing share is dynamic and is updated periodically on each DKG ceremony. Once updated, it will be saved to disk. Currently, the DKG ceremony is scheduled to run on-chain every ~3 hours. - -## Running the validator - -The process for running a validator node is very similar to [running a full node](/guide/node/rpc). - -You should start by downloading the latest snapshot using `tempo download --minimal`. Downloading the snapshot allows your validator to start participating in consensus much faster. -Once you've downloaded the snapshot and have been whitelisted on-chain, you can proceed to run the validator node as such: - -```bash -tempo node --datadir \ - --chain \ - --consensus.signing-key \ - --telemetry-url -``` - -:::warning -`--consensus.fee-recipient` is deprecated as of `v1.5.2` and will be removed in an upcoming release. Validators should migrate fee-recipient management to [Validator Config V2](/guide/node/validator-config-v2). +:::info +The active validator set is currently permissioned. If you are interested in becoming a validator, please [get in touch](mailto:partners@tempo.xyz) with the Tempo team. See [Initial setup](/guide/node/validator-setup) for technical details. ::: -### Optional flags - -| Flag | Description | -|------|-------------| -| `--telemetry-url ` | Unified metrics and logs export. See [Telemetry endpoint](#telemetry-endpoint) below for details. **We ask all validators to configure this so we can support troubleshooting.** | -| `--telemetry-metrics-interval ` | Interval for pushing metrics (default: `10s`). | -| `--consensus.datadir ` | Store consensus data on a separate volume (e.g., AWS EBS) while keeping execution state on high-performance local disks. Migrate by copying `/consensus` to the new location. | - -### Telemetry endpoint - -The `--telemetry-url` flag enables unified telemetry export. It pushes two types of data to a Tempo-operated metrics backend: - -- **Prometheus metrics** — execution-layer (reth) and consensus-layer metrics in Prometheus text format, pushed at a configurable interval (default every 10 seconds). -- **Structured logs** — operational logs exported via OTLP at `debug` level, covering consensus events, block processing, and sync progress. - -The URL must include credentials: `--telemetry-url https://user:pass@metrics.example.com` - -#### What is collected - -| Category | Examples | -|----------|---------| -| Execution metrics | Block processing times, transaction pool size, peer count, sync status, database stats | -| Consensus metrics | Epoch and view progress, DKG ceremony status, proposal and finalization counts | -| Operational logs | Consensus state transitions, block proposals, sync progress, error events | - -All consensus metrics are namespaced under a `consensus` prefix. - -#### What is **not** collected - -The telemetry endpoint does **not** collect any ambient information about the host machine. Specifically: - -- No hostname, IP address, or machine identifiers -- No operating system, CPU, memory, or disk information -- No information about other processes or services running on the machine -- No file paths or directory structures beyond the node's own data directory references in logs -- No network topology or firewall configuration - -The data is strictly limited to the node's own operational metrics and logs. - -The notable difference between RPC nodes and validator nodes is the omission of the `--follow` argument and the addition of the `--consensus.signing-key` argument. Validator fee recipients are moving to on-chain management via Validator Config V2, and the legacy `--consensus.fee-recipient` flag is deprecated. - -Once your node is up, it may not start syncing immediately. This is because your node might not be part of the active set. In most cases, your validator will enter the active set in under 6 hours after the on-chain addition of the validator identity. - -## Troubleshooting - -### Checking if your node is proposing blocks - -Once your node is proposing blocks, it will start emitting logs like these: - -``` -INFO handle_propose{epoch=18 view=387213 parent.view=387212 parent.digest=0x43885416b4a7ae7550c615ad4dc702045cd26540b78fa2bd75abdcbfbef02d9d}: tempo_commonware_node::consensus::application::actor: constructed proposal proposal.digest=0x6baa8fa813beea491cf598024d8b31cb2927e7ba1b92a29899a795dbafd682c6 proposal.height=5733680 -``` - -If you do not see logs like these: -- please check that your validator is part of the active set, and is both a player and a dealer. See [validator states](/guide/node/validator-config-v2#validator-states) -- please check that your node is synced up to the [latest block height](https://explorer.tempo.xyz/). -- please check that your outgoing IP address matches the one whitelisted on the validator smart contract: - -```bash -tempo consensus validators-info --rpc-url -``` + + + + + + + + + + + + + diff --git a/src/pages/protocol/upgrades/t2.mdx b/src/pages/protocol/upgrades/t2.mdx index 4b6ec8c8..aad74f71 100644 --- a/src/pages/protocol/upgrades/t2.mdx +++ b/src/pages/protocol/upgrades/t2.mdx @@ -44,7 +44,7 @@ T2 is Tempo's next network upgrade, building on T1. It introduces the following **Spec:** [TIP-1017](/protocol/tips/tip-1017) -**Operator guide:** [Validator Config V2](/guide/node/validator-config-v2) +**Operator guide:** [Controlling validator lifecycle](/guide/node/validator-lifecycle) **TLDR:** New precompile for managing consensus validators. Adds Ed25519 signature verification at registration, append-only history and stable validator indices. diff --git a/vocs.config.ts b/vocs.config.ts index c65bb9b5..cd065771 100644 --- a/vocs.config.ts +++ b/vocs.config.ts @@ -815,18 +815,47 @@ export default defineConfig({ link: '/guide/node/validator', }, { - text: 'Operation', - link: '/guide/node/operate-validator', + text: 'Validator Onboarding', + link: '/guide/node/validator-setup', }, { - text: 'ValidatorConfig V2', - link: '/guide/node/validator-config-v2', + text: 'Checking validator status', + link: '/guide/node/validator-status', + }, + { + text: 'Controlling validator lifecycle', + link: '/guide/node/validator-lifecycle', + }, + { + text: 'Managing validator keys', + link: '/guide/node/validator-keys', + }, + { + text: 'Monitoring a validator', + link: '/guide/node/validator-monitoring', + }, + { + text: 'Troubleshooting and FAQ', + link: '/guide/node/validator-troubleshooting', }, ], }, + { + text: 'Node Security', + link: '/guide/node/security', + }, { text: 'Network Upgrades and Releases', - link: '/guide/node/network-upgrades', + items: [ + { + text: 'Upgrade Cadence', + link: '/guide/node/upgrade-cadence', + }, + { + text: 'Upgrades and Releases', + link: '/guide/node/network-upgrades', + }, + ], }, { text: 'Changelog',