feat(flashblocks rpc): Full revamp of X Layer custom flashblocks RPC layer#176
feat(flashblocks rpc): Full revamp of X Layer custom flashblocks RPC layer#176
Conversation
🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…to state cache 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add a tertiary tx hash index (HashMap<TxHash, Arc<CachedTxInfo>>) to ConfirmCache, enabling O(1) lookups for transaction_by_hash, transaction_by_hash_with_meta, and receipt_by_hash from confirmed flashblocks that are ahead of the canonical chain. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add collect_cached_block_range_while helper that threads a predicate through both the provider prefix and cached suffix, enabling sealed_headers_while to serve confirmed flashblock headers instead of bypassing the cache entirely. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
… rpc crate Consolidates flashblocks cache provider trait implementations (block, header, factory, receipt, transaction) into a single eth.rs override module in the rpc crate. Wires FlashblockStateCache into the node's RPC registration. Simplifies the cache layer to only manage state. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…an up cache interfaces 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Consolidate the previously split flashblocks service setup and state cache creation into one cohesive `if let Some` block. This fixes a subtle bug where using `.map()` with `?` operators would produce `Option<Result<T, E>>` instead of the intended `Option<T>`, causing `?` to not propagate errors to the outer closure. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… access Restructure ConfirmCache to store ExecutedBlock directly instead of just BlockAndReceipts, enabling state provider overlay for flashblock state queries. - Add ConfirmedBlock struct wrapping ExecutedBlock + receipts - Update insert/remove signatures to accept ExecutedBlock - Make PendingSequence.tx_index pub for state access - Fix execution cache test assertions 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…cks state 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
f8a26bf to
5d96010
Compare
5d96010 to
6666f70
Compare
🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Consolidate early-return flush logic into a single return path, prefer newest raw cache entry via reverse iteration, fix typo in log message, and inline format args for clippy compliance. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
… build validation Thread payload_id from raw cache build args into PrefixExecutionMeta so the validator can verify that incremental builds belong to the same payload sequence. Also fix test assertions for rejected flashblocks and tighten prevalidate_incoming_sequence to detect payload_id mismatches. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Remove unused ConfirmCache methods (hash_for_number, remove_block_by_*, len/is_empty in non-test), move block_from_bar into test-only scope, drop unused type param on handle_execution_tasks, add Default impl for FlashblockStateCache, and use idiomatic is_none_or / contains_key. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Claude Review — Resolution StatusThe automated review identified 13 issues. Here is the resolution status for each: Resolved (All 13/13)
Additional fixes from cleanup session (commits
|
|
@claude review again for me. The handles logic have been added and the PR is completed and ready for review. Also ensure that all previously mentioned issues have been resolved. cc/ @JimmyShi22 |
|
Claude finished @sieniven's task in 5m 31s —— View job PR Review: feat(flashblocks rpc): Full revamp of X Layer custom flashblocks RPC layerTasks:
Verification: Previously Identified IssuesAll 13 previously flagged issues are resolved as described. The key confirmations:
New Issues Found in This ReviewCritical:
|
- Reorder insert() to remove old block entries before inserting new tx index entries, preventing silent tx_index corruption when old and new blocks share transaction hashes - Allow height replacement at full capacity by checking contains_key before rejecting inserts - Add test for overlapping tx hash scenario on duplicate height insert - Document PayloadId::default() usage in spawn_prewarm 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Second Claude Review — Resolution StatusThe second review (triggered after all 13 original issues were resolved) found 3 new issues. All are now resolved in
All 69 flashblocks tests pass (68 existing + 1 new). |
|
@claude review again for me |
|
Claude finished @sieniven's task in 4m 26s —— View job PR Review: feat(flashblocks rpc): Full revamp of X Layer custom flashblocks RPC layerTasks:
Verification: All Previously Identified IssuesAll issues from both prior reviews are correctly resolved. Spot-checked below.
New Issues FoundMinor:
|
🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Third Claude Review — Resolution StatusThe third review confirmed all 15 prior issues resolved and found 2 new minor items. Both addressed in
All 69 flashblocks tests pass. |
…for execution Reject incoming flashblocks when canonical height is still zero (not yet initialized) to prevent premature validation. Switch the execution task from spawn_critical_blocking_task to a dedicated OS thread via reth_tasks::spawn_os_thread since it runs a blocking loop. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Preserve canonical height continuity after a state cache flush instead of resetting confirm_height to zero, which could cause stale validation. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…tate Thread CanonicalInMemoryState into FlashblockStateCache so that get_overlay_data collects executed blocks from both the flashblocks cache and the engine's in-memory canonical chain, anchoring the state provider at the on-disk persistence boundary instead of the canon hash. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…nces to confirm cache Split the promotion logic in handle_pending_sequence to handle target_index completion and next-block arrival as separate cases. When target_index is reached for the current block, the complete sequence is promoted directly to the confirm cache and pending is cleared, preventing stale pending detection when the canonical block arrives. Also update validator prevalidation to use confirm_height instead of canon_height for flashblock height checks. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Move the cumulative gas offset application from merge_suffix_results into the per-receipt execution loop so that receipts sent to the incremental root builder already carry the correct cumulative_gas_used. Refactor FlashblockReceipt::add_cumulative_gas_offset to operate on a single receipt instead of a slice. Also improve logging consistency and add target_index to execution trace. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…vider For incremental builds, resolve the state provider from the pending sequence block hash instead of the parent hash so the overlay trie anchors at the latest executed prefix. Thread the parent SealedHeader into PendingSequence so the EVM env and state root remain consistent across incremental steps. Pass evm_env directly to execute_block to avoid cloning the full ExecutionEnv. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…ilds Incremental builds accumulate one revert entry per flashblock index via merge_transitions, but the engine persistence service expects a single revert per block. Flatten all revert transitions by keeping the earliest (parent-state) account info per address and merging storage slot reverts across transitions, preserving wipe_storage from later indices. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
… engine Add a debug mode (--xlayer.flashblocks-debug-state-comparison) that compares ExecutedBlock bundle states, reverts, and trie data between the flashblocks RPC cache and the engine canonical state on each new canonical block. Also add a flag to disable pre-warming so the engine computes payloads independently for accurate comparison. The heavy comparison runs on a blocking thread to avoid stalling the canonical stream handler. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Expose a unified lookup across pending and confirm caches for ExecutedBlock by block number, simplifying the debug comparison to a single call instead of inline pending sequence matching. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Drop DeferredTrieData comparison (hashed_state + trie_updates) from the debug state comparison since trie data is computed independently by the engine and not meaningful for execution output equivalence checks. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Summary
This PR revamps the entire flashblocks rpc layer design, similar to the previous Realtime-feature (RTA) cache capabilities on the since discontinued xlayer-erigon (refer to: https://github.com/okx/xlayer-erigon/tree/dev/zk/realtime).
Essentially, this revamp implements the full solution 2 mentioned in the upstream reth's issue discussion thread: paradigmxyz/reth#20491, which is a more elegant design to the complex op-reth's flashblock rpc layer that is tightly coupled to the underlying anchoring canonical chainstate.
Motivations
The best design for allowing minimal latencies is to allow a full state accumulation from incoming flashblocks that are ahead of the canonical chainstate, while allowing for a hybrid sync approach that:
CHANGESET_CACHE_RETENTION_BLOCKS), once canonical chainstate catches uplatesttags) on flashblocks rpc tries to read from both the flashblocks state cache and underlying canon providerNew flashblocks rpc design
1. Flashblock cache layer
FlashblockStateCachelayer which includes both the pending state and confirm state caches2. Revamp execution environment for flashblocks sequence validation
The flashblocks RPC layer revamp contains significant optimizations and alignment to the upstream reth's payload validator logic. The old logic was vastly over-complicated, and the revamped version now encapsulates the flashblocks sequence validation inside the new
FlashblockSequenceValidator. It aligns the sequence validation logic to the highly optimized upstream reth's engine payload validator.Ref: https://github.com/paradigmxyz/reth/blob/main/crates/engine/tree/src/tree/payload_validator.rs
Below are some of the key optimizations:
3. Flashblocks eth rpc extension
eth_sendRawTransactionSynceth_getRawTransactionByBlockHashAndIndexeth_getRawTransactionByBlockNumberAndIndex4. Remove sequence manager, revamp to RawCache
5. Add handlers for flashblocks state update logic
handle_flashblockshandler - handle receiving a raw flashblock, and updating the raw cachehandle_flashblocks_statehandler - handle spawning task execution for flashblocks sequence validation and updating the flashblocks state cachehandle_canonical_block- logic to handle canonical chainstate updates to flush flashblocks state cache layer, and also handle extreme cases like chain reorgs, etcSupported flashblocks eth APIs
Block apis
[✅]
eth_blockNumber[✅]
eth_getBlockByNumber[✅]
eth_getBlockByHash[✅]
eth_getBlockReceipts[✅]
eth_getBlockTransactionCountByNumber[✅]
eth_getBlockTransactionCountByHashTransaction apis
[✅]
eth_getTransactionByHash[✅]
eth_getRawTransactionByHash[✅]
eth_getTransactionReceipt[✅]
eth_getTransactionByBlockHashAndIndex[✅]
eth_getTransactionByBlockNumberAndIndex[✅]
eth_getRawTransactionByBlockHashAndIndex[✅]
eth_getRawTransactionByBlockNumberAndIndex[✅]
eth_sendRawTransactionSyncState apis
[✅]
eth_call[✅]
eth_estimateGas[✅]
eth_getBalance[✅]
eth_getTransactionCount[✅]
eth_getCode[✅]
eth_getStorageAtTODO:
Support eth_getLogs in the future
Design flaws on op-reth flashblocks rpc
On subsequent reth version upgrades (todo):
Todos: