From 1873d8f654ae94cd3fb7b17113c58ae4668c52e8 Mon Sep 17 00:00:00 2001 From: Aditya Arora Date: Mon, 13 Oct 2025 22:22:59 +0530 Subject: [PATCH 1/7] chore(lazer) How Lazer works --- pages/lazer/how-lazer-works.mdx | 54 +++++++++++++++++++++++++++++++-- 1 file changed, 52 insertions(+), 2 deletions(-) diff --git a/pages/lazer/how-lazer-works.mdx b/pages/lazer/how-lazer-works.mdx index ea314c0e..ec39f4e3 100644 --- a/pages/lazer/how-lazer-works.mdx +++ b/pages/lazer/how-lazer-works.mdx @@ -1,5 +1,55 @@ # How Pyth Lazer works -Pyth Lazer is a permissioned service that provides ultra-low-latency price and market data to highly latency-sensitive users. +Pyth Lazer is a permissioned service that provides ultra-low-latency market data to consumers. +It aggregates data from multiple publishers and distributes it to consumers through a multi-tier architecture. -We are working on writing a detailed technical overview how Lazer works. Details will be added here as they are completed. +## System Components + +The architecture consists of four main services that work together to provide ultra-low-latency data to consumers. + +### INSERT DIAGRAM HERE + +### Publishers + +Publishers are the entities that provide market data to Lazer. They submit price updates via authenticated WebSocket connections. +Each publisher is configured with specific permissions defining which feeds they can update. + +### Relayers + +The Relayer is the ingestion layer that receives and validates all incoming updates from publishers. + +**Key responsibilities:** + +- **Authentication**: Validates publisher access tokens and optional Ed25519 signatures. +- **Rate limiting**: Enforces configurable limits of price updates per publisher. +- **Message forwarding**: Publishes validated updates to NATS or Kafka. +- **History Service**: Maintains a connection to the History Service to cache historical data for circuit breaker validation. + +The Relayer also maintains a connection to the History Service to cache historical data for circuit breaker validation. + +### Aggregator Component + +The Aggregator is a library component embedded within both the Router and History Service. +It consumes publisher updates from the message bus and computes aggregated price feeds. + +**Aggregation logic:** +The Aggregator computes median prices, confidence intervals (using interquartile range), and best bid/ask prices from multiple publisher inputs. +It filters out stale data and ensures minimum publisher requirements are met before publishing aggregated updates. + +### Router Service + +The Router is the real-time distribution layer that serves price feeds to consumers. + +**Key features:** + +- **WebSocket streaming**: Provides `/v1/stream` endpoint for real-time price updates +- **HTTP REST API**: Offers `/v1/latest_price` for on-demand price queries +- **Channel types**: Supports real-time and fixed-rate channels (1ms, 50ms, 200ms) +- **Multi-chain support**: Generates on-chain payloads for Solana, EVM, and other chains + +### Message Transport Layer + +The system supports dual message queue backends for flexibility. This allows the system to be deployed in a multi-datacenter environment. + +- **NATS Jetstream**: Provides pub/sub messaging with stream persistence. +- **Apache Kafka**: Can be used instead of NATS for message transport. From 85a5286f700de42779c53cffded75afb228cd4db Mon Sep 17 00:00:00 2001 From: Aditya Arora Date: Wed, 15 Oct 2025 11:01:59 +0530 Subject: [PATCH 2/7] chore(lazer) Add how lazer works-2 --- pages/lazer/how-lazer-works.mdx | 62 ++++++++++++++++++++++----------- 1 file changed, 42 insertions(+), 20 deletions(-) diff --git a/pages/lazer/how-lazer-works.mdx b/pages/lazer/how-lazer-works.mdx index ec39f4e3..2648f54b 100644 --- a/pages/lazer/how-lazer-works.mdx +++ b/pages/lazer/how-lazer-works.mdx @@ -1,44 +1,55 @@ +import { Callout } from "nextra/components"; + # How Pyth Lazer works Pyth Lazer is a permissioned service that provides ultra-low-latency market data to consumers. It aggregates data from multiple publishers and distributes it to consumers through a multi-tier architecture. -## System Components +## System Services -The architecture consists of four main services that work together to provide ultra-low-latency data to consumers. +The architecture consists of five main services that work together to provide ultra-low-latency data to consumers. ### INSERT DIAGRAM HERE ### Publishers -Publishers are the entities that provide market data to Lazer. They submit price updates via authenticated WebSocket connections. +Publishers are the entities that provide market data to Lazer. They submit updates via authenticated WebSocket connections. Each publisher is configured with specific permissions defining which feeds they can update. ### Relayers -The Relayer is the ingestion layer that receives and validates all incoming updates from publishers. +The Relayer service is the ingestion layer that receives and validates all incoming updates from publishers. **Key responsibilities:** - **Authentication**: Validates publisher access tokens and optional Ed25519 signatures. -- **Rate limiting**: Enforces configurable limits of price updates per publisher. -- **Message forwarding**: Publishes validated updates to NATS or Kafka. -- **History Service**: Maintains a connection to the History Service to cache historical data for circuit breaker validation. +- **Rate limiting**: Enforces configurable limits on publisher updates. +- **Message forwarding**: Publishes validated updates to an internal message queue (a NATS cluster). + +**Douro Labs** operates the relayer service for the Pyth Lazer network. + + + The Douro Labs-operated relayer follows a strict, deterministic processing model: + - **No price dropping**: All validated updates are forwarded to the message queue without dropping any prices. + - **FCFS processing**: Updates are processed on a first-come-first-served basis without prioritization. + - **Deterministic operation**: The relayer operates according to its configured logic and does not deviate from it. -The Relayer also maintains a connection to the History Service to cache historical data for circuit breaker validation. +This ensures reliable, predictable data flow from publishers to consumers. -### Aggregator Component + -The Aggregator is a library component embedded within both the Router and History Service. -It consumes publisher updates from the message bus and computes aggregated price feeds. +### Message Queue -**Aggregation logic:** -The Aggregator computes median prices, confidence intervals (using interquartile range), and best bid/ask prices from multiple publisher inputs. -It filters out stale data and ensures minimum publisher requirements are met before publishing aggregated updates. +The system uses NATS JetStream as the primary message queue for pub/sub messaging with stream persistence. +This allows the system to be deployed in a multi-datacenter environment and ensures reliable message delivery between services. -### Router Service +**Message ordering**: NATS JetStream guarantees message ordering within a single stream, ensuring that updates from publishers are processed in the order they were received by the Relayer. +This ordering guarantee is critical for maintaining consistent feed state across all aggregators. -The Router is the real-time distribution layer that serves price feeds to consumers. +### Routers + +The Router is the real-time distribution layer that serves data to consumers. +It embeds aggregation logic to compute median prices, confidence intervals (using interquartile range), and best bid/ask prices from multiple publisher inputs. **Key features:** @@ -47,9 +58,20 @@ The Router is the real-time distribution layer that serves price feeds to consum - **Channel types**: Supports real-time and fixed-rate channels (1ms, 50ms, 200ms) - **Multi-chain support**: Generates on-chain payloads for Solana, EVM, and other chains -### Message Transport Layer +#### Aggregation logic + +Each Router embeds an aggregator component that consumes publisher updates from NATS and computes aggregated data feeds. The aggregator: -The system supports dual message queue backends for flexibility. This allows the system to be deployed in a multi-datacenter environment. +- Computes median values resistant to outlier data from individual publishers. +- Calculates confidence intervals using interquartile range to measure data spread. +- Determines best bid/ask values filtered to ensure market consistency. +- Automatically removes stale publisher data based on configurable timeouts. + +### History Service + +The History Service provides persistence and historical data queries. + +**Key responsibilities:** -- **NATS Jetstream**: Provides pub/sub messaging with stream persistence. -- **Apache Kafka**: Can be used instead of NATS for message transport. +- Data persistence: Stores all publisher updates, aggregated data, and transactions in ClickHouse. +- Historical queries: Provides REST API for querying historical data. From 116c45d5dfa089608b1f9308302772e45bfd9c37 Mon Sep 17 00:00:00 2001 From: Aditya Arora Date: Thu, 16 Oct 2025 13:03:54 +0530 Subject: [PATCH 3/7] chore(lazer) How lazer works requested changes --- pages/lazer/how-lazer-works.mdx | 30 +++++++++++++----------------- 1 file changed, 13 insertions(+), 17 deletions(-) diff --git a/pages/lazer/how-lazer-works.mdx b/pages/lazer/how-lazer-works.mdx index 2648f54b..135e7198 100644 --- a/pages/lazer/how-lazer-works.mdx +++ b/pages/lazer/how-lazer-works.mdx @@ -1,5 +1,3 @@ -import { Callout } from "nextra/components"; - # How Pyth Lazer works Pyth Lazer is a permissioned service that provides ultra-low-latency market data to consumers. @@ -7,7 +5,8 @@ It aggregates data from multiple publishers and distributes it to consumers thro ## System Services -The architecture consists of five main services that work together to provide ultra-low-latency data to consumers. +The architecture consists of five main types of services that work together to provide ultra-low-latency data to consumers. +Each service has multiple instances running to ensure high availability and low latency. ### INSERT DIAGRAM HERE @@ -23,33 +22,30 @@ The Relayer service is the ingestion layer that receives and validates all incom **Key responsibilities:** - **Authentication**: Validates publisher access tokens and optional Ed25519 signatures. +- **Validation**: Performs sanity checks on incoming updates by examining feed IDs, timestamps, and values to ensure data integrity and proper formatting. - **Rate limiting**: Enforces configurable limits on publisher updates. -- **Message forwarding**: Publishes validated updates to an internal message queue (a NATS cluster). +- **Message forwarding**: Publishes validated updates to an internal message queue. -**Douro Labs** operates the relayer service for the Pyth Lazer network. +Douro Labs operates the relayer service for the Pyth Lazer network. It follows a strict, deterministic processing model: - - The Douro Labs-operated relayer follows a strict, deterministic processing model: - - **No price dropping**: All validated updates are forwarded to the message queue without dropping any prices. - - **FCFS processing**: Updates are processed on a first-come-first-served basis without prioritization. - - **Deterministic operation**: The relayer operates according to its configured logic and does not deviate from it. +- **No price dropping outside of circuit breakers**: All validated updates are forwarded to the message queue without dropping any prices. +- **FCFS processing**: Updates are processed on a first-come-first-served basis without prioritization. +- **Deterministic operation**: The relayer operates according to its configured logic and does not deviate from it. This ensures reliable, predictable data flow from publishers to consumers. - - ### Message Queue -The system uses NATS JetStream as the primary message queue for pub/sub messaging with stream persistence. +The system uses a distributed message queue for pub/sub messaging with stream persistence. This allows the system to be deployed in a multi-datacenter environment and ensures reliable message delivery between services. -**Message ordering**: NATS JetStream guarantees message ordering within a single stream, ensuring that updates from publishers are processed in the order they were received by the Relayer. +**Message ordering**: The message queue guarantees message ordering within a single stream, ensuring that updates from publishers are processed in the order they were received by the Relayer. This ordering guarantee is critical for maintaining consistent feed state across all aggregators. ### Routers The Router is the real-time distribution layer that serves data to consumers. -It embeds aggregation logic to compute median prices, confidence intervals (using interquartile range), and best bid/ask prices from multiple publisher inputs. +It embeds aggregation logic to compute median prices, confidence intervals (using interquartile range), and best bid/ask prices, funding rates, and more from multiple publisher inputs. **Key features:** @@ -60,7 +56,7 @@ It embeds aggregation logic to compute median prices, confidence intervals (usin #### Aggregation logic -Each Router embeds an aggregator component that consumes publisher updates from NATS and computes aggregated data feeds. The aggregator: +Each Router embeds an aggregator component that consumes publisher updates from the Message Queue and computes aggregated data feeds. The aggregator: - Computes median values resistant to outlier data from individual publishers. - Calculates confidence intervals using interquartile range to measure data spread. @@ -73,5 +69,5 @@ The History Service provides persistence and historical data queries. **Key responsibilities:** -- Data persistence: Stores all publisher updates, aggregated data, and transactions in ClickHouse. +- Data persistence: Stores all publisher updates, aggregated data, and transactions. - Historical queries: Provides REST API for querying historical data. From e1955d61f6fcc6146a9918c046d28410e954ca0d Mon Sep 17 00:00:00 2001 From: Aditya Arora Date: Thu, 16 Oct 2025 22:59:43 +0530 Subject: [PATCH 4/7] requested changes --- pages/lazer/how-lazer-works.mdx | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/pages/lazer/how-lazer-works.mdx b/pages/lazer/how-lazer-works.mdx index 135e7198..ef35b3e8 100644 --- a/pages/lazer/how-lazer-works.mdx +++ b/pages/lazer/how-lazer-works.mdx @@ -28,9 +28,8 @@ The Relayer service is the ingestion layer that receives and validates all incom Douro Labs operates the relayer service for the Pyth Lazer network. It follows a strict, deterministic processing model: -- **No price dropping outside of circuit breakers**: All validated updates are forwarded to the message queue without dropping any prices. +- **No price dropping outside of circuit breakers**: All validated updates are forwarded to the message queue without dropping any prices (except for when a circuit breaker is triggered). - **FCFS processing**: Updates are processed on a first-come-first-served basis without prioritization. -- **Deterministic operation**: The relayer operates according to its configured logic and does not deviate from it. This ensures reliable, predictable data flow from publishers to consumers. @@ -39,8 +38,9 @@ This ensures reliable, predictable data flow from publishers to consumers. The system uses a distributed message queue for pub/sub messaging with stream persistence. This allows the system to be deployed in a multi-datacenter environment and ensures reliable message delivery between services. -**Message ordering**: The message queue guarantees message ordering within a single stream, ensuring that updates from publishers are processed in the order they were received by the Relayer. -This ordering guarantee is critical for maintaining consistent feed state across all aggregators. +**Message ordering**: The message queue ensures reliable delivery and maintains the exact sequence of messages within each data stream. +This means every publisher update will be delivered at least once, and messages will be processed in the same order they arrived at the Relayer. +This sequential processing is essential for keeping all aggregators synchronized with the same feed state. ### Routers @@ -51,7 +51,7 @@ It embeds aggregation logic to compute median prices, confidence intervals (usin - **WebSocket streaming**: Provides `/v1/stream` endpoint for real-time price updates - **HTTP REST API**: Offers `/v1/latest_price` for on-demand price queries -- **Channel types**: Supports real-time and fixed-rate channels (1ms, 50ms, 200ms) +- **Channel types**: Supports real-time and fixed-rate channels (50ms, 200ms) - **Multi-chain support**: Generates on-chain payloads for Solana, EVM, and other chains #### Aggregation logic @@ -63,6 +63,9 @@ Each Router embeds an aggregator component that consumes publisher updates from - Determines best bid/ask values filtered to ensure market consistency. - Automatically removes stale publisher data based on configurable timeouts. +Lazer guarantees deterministic aggregation: all aggregators produce the exact same aggregated results by relying solely on the consistent stream of price updates from the Message Queue. +This ensures that every Router instance maintains identical feed state, providing consistent data to all consumers regardless of which Router they connect to. + ### History Service The History Service provides persistence and historical data queries. @@ -71,3 +74,4 @@ The History Service provides persistence and historical data queries. - Data persistence: Stores all publisher updates, aggregated data, and transactions. - Historical queries: Provides REST API for querying historical data. +- OHLC API: Provides Open, High, Low, Close (OHLC) data for charting applications through the history service. From 3ebeea1bf4ce38eab257112a4487202b2e9eef35 Mon Sep 17 00:00:00 2001 From: Aditya Arora Date: Mon, 20 Oct 2025 21:36:50 +0530 Subject: [PATCH 5/7] removed lazer arch diagram placeholder --- pages/lazer/how-lazer-works.mdx | 2 -- 1 file changed, 2 deletions(-) diff --git a/pages/lazer/how-lazer-works.mdx b/pages/lazer/how-lazer-works.mdx index ef35b3e8..a1e0d4a4 100644 --- a/pages/lazer/how-lazer-works.mdx +++ b/pages/lazer/how-lazer-works.mdx @@ -8,8 +8,6 @@ It aggregates data from multiple publishers and distributes it to consumers thro The architecture consists of five main types of services that work together to provide ultra-low-latency data to consumers. Each service has multiple instances running to ensure high availability and low latency. -### INSERT DIAGRAM HERE - ### Publishers Publishers are the entities that provide market data to Lazer. They submit updates via authenticated WebSocket connections. From ae48a38fa1fdf6ee39d29264169637184d2a8c57 Mon Sep 17 00:00:00 2001 From: Aditya Arora Date: Mon, 20 Oct 2025 21:37:36 +0530 Subject: [PATCH 6/7] added 1 ms --- pages/lazer/how-lazer-works.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pages/lazer/how-lazer-works.mdx b/pages/lazer/how-lazer-works.mdx index a1e0d4a4..ea34185e 100644 --- a/pages/lazer/how-lazer-works.mdx +++ b/pages/lazer/how-lazer-works.mdx @@ -49,7 +49,7 @@ It embeds aggregation logic to compute median prices, confidence intervals (usin - **WebSocket streaming**: Provides `/v1/stream` endpoint for real-time price updates - **HTTP REST API**: Offers `/v1/latest_price` for on-demand price queries -- **Channel types**: Supports real-time and fixed-rate channels (50ms, 200ms) +- **Channel types**: Supports real-time and fixed-rate channels (1ms, 50ms, 200ms) - **Multi-chain support**: Generates on-chain payloads for Solana, EVM, and other chains #### Aggregation logic From 2984901a201b4c6ca7cd43c8bfe0a71acb72556e Mon Sep 17 00:00:00 2001 From: Ali Behjati Date: Mon, 20 Oct 2025 18:40:01 +0200 Subject: [PATCH 7/7] Update pages/lazer/how-lazer-works.mdx --- pages/lazer/how-lazer-works.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pages/lazer/how-lazer-works.mdx b/pages/lazer/how-lazer-works.mdx index ea34185e..e0d12ace 100644 --- a/pages/lazer/how-lazer-works.mdx +++ b/pages/lazer/how-lazer-works.mdx @@ -49,7 +49,7 @@ It embeds aggregation logic to compute median prices, confidence intervals (usin - **WebSocket streaming**: Provides `/v1/stream` endpoint for real-time price updates - **HTTP REST API**: Offers `/v1/latest_price` for on-demand price queries -- **Channel types**: Supports real-time and fixed-rate channels (1ms, 50ms, 200ms) +- **Channel types**: Supports real-time and fixed-rate channels (50ms, 200ms, 1s) - **Multi-chain support**: Generates on-chain payloads for Solana, EVM, and other chains #### Aggregation logic