feat: add substrate support

[mduthey]

Summary by CodeRabbit

• New Features

  • Added multi-chain support: Bitcoin, Ethereum, and Substrate sources alongside Cardano.
  • Introduced watch commands for Bitcoin, Ethereum, and Substrate chains.
  • Extended sink and filter options for multi-chain pipelines.

• Documentation

  • Added comprehensive v3 documentation covering sources, sinks, filters, usage, and installation.
  • Provided configuration guides and examples for all supported blockchains.

• Updates

  • Updated README to reflect v3 branch, multi-chain focus, and expanded CLI usage examples.

[coderabbitai]

Caution

Review failed

Failed to post review comments

Walkthrough

This PR introduces multi-chain support to Oura by adding Bitcoin, Ethereum, and Substrate sources alongside Cardano with feature-gated compilation. It refactors the core data model to wrap chain-specific records under a unified Record enum, introduces new Chain and ChainConfig types, and adds comprehensive v3 documentation including guides, advanced features, and installation instructions. New watch subcommands enable CLI interaction with each supported chain.

Changes

Cohort / File(s)	Summary
Manifest & Configuration Cargo.toml	Adds three new top-level features (eth, btc, substrate), introduces optional dependencies (bitcoind-async-client, alloy with provider-ws, subxt, corepc-types), wires them into respective features.
README	Replaces V2/V1 warning with v3 branch note, adds multi-chain focus to production guidance, expands CLI examples for Cardano, Bitcoin, Ethereum, Substrate, adds Bitcoin to Features/Sources list.
V3 Installation Docs docs/v3/installation/_meta.yaml, binary_release.mdx, docker.mdx, from_source.mdx, kubernetes.mdx	Adds metadata and documentation for multiple installation methods (binary releases, Docker, Kubernetes sidecar/standalone, from source).
V3 Usage Docs docs/v3/usage/_meta.yaml, watch.mdx, daemon.mdx, dump.mdx, library.mdx	Documents daemon mode, watch CLI with multi-chain support (Cardano, Bitcoin, Ethereum, Substrate), dump mode, and library usage.
V3 Advanced Docs docs/v3/advanced/_meta.yaml, custom_network.mdx, finalize_options.mdx, intersect_options.mdx, pipeline_metrics.mdx, retry_policy.mdx, stateful_cursor.mdx	Documents advanced daemon features: custom network configuration, finalize/intersect options, pipeline metrics, retry policies, stateful cursor persistence.
V3 Filters Docs docs/v3/filters/_meta.yaml, legacy_v1.mdx, parse_cbor.mdx, rollback_buffer.mdx, select.mdx, split_block.mdx, wasm.mdx	Documents filter stages: LegacyV1, ParseCbor, RollbackBuffer, Select with predicates, SplitBlock, WebAssembly.
V3 Sources Docs docs/v3/sources/_meta.yaml, n2c.mdx, n2n.mdx, btc.mdx, eth.mdx, substrate.mdx, hydra.mdx, mithril.mdx, s3.mdx, utxorpc.mdx	Documents blockchain data sources: Cardano (N2C, N2N, Hydra, Mithril, UTXORPC), Bitcoin RPC, Ethereum RPC, Substrate RPC, AWS S3.
V3 Sinks Docs docs/v3/sinks/_meta.yaml, stdout.mdx, terminal.mdx, kafka.mdx, redis.mdx, elasticsearch.mdx, file_rotate.mdx, webhook.mdx, aws_lambda.mdx, aws_s3.mdx, aws_sqs.mdx, gcp_cloudfunction.mdx, gcp_pubsub.mdx, rabbitmq.mdx	Documents output sinks: Stdout/Terminal, Kafka, Redis Streams, Elasticsearch, file rotation, webhooks, AWS (Lambda, S3, SQS), GCP (Cloud Functions, Pub/Sub), RabbitMQ.
V3 Reference Docs docs/v3/reference/_meta.yaml, data_dictionary.mdx	Documents event data structures and fields for different filter modes (ParseCbor, Legacy V1).
V3 Guides docs/v3/guides/_meta.yaml, cardano_2_kafka.mdx	Provides guide for streaming Cardano to Kafka.
V3 Index docs/v3/index.mdx	Introduces Oura tool overview, core concepts, and usage scenarios.
Example Configs examples/btc/daemon.toml, examples/eth/daemon.toml, examples/substrate/daemon.toml	Provides example daemon configurations for Bitcoin RPC, Ethereum RPC, and Substrate RPC sources with FileRotate sinks.
Framework Core: Cardano Module src/framework/cardano/mod.rs	Adds ChainConfig enum (Mainnet, Testnet, PreProd, Preview, Custom), Record enum wrapping chain-specific data (CborBlock, CborTx, ParsedBlock, ParsedTx, OuraV1Event), and JSON serialization support.
Framework Core: Bitcoin Module src/framework/bitcoin/mod.rs	Adds Record enum (ParsedBlock, RawBlock) with JSON serialization via From<Record> for JsonValue.
Framework Core: Ethereum Module src/framework/ethereum/mod.rs	Adds Record enum (ParsedBlock, RawBlock) with JSON serialization via From<Record> for JsonValue.
Framework Core: Substrate Module src/framework/substrate/mod.rs	Adds data structures (BlockHeader, SignatureInfo, CallInfo, ExtrinsicEvent, Extrinsic, ParsedBlock) and Record enum (ParsedBlock, RawBlock) with JSON serialization.
Framework Integration src/framework/mod.rs	Introduces Chain enum (Cardano, Bitcoin, Ethereum, Substrate), refactors Record enum to wrap chain-specific variants, adds Context struct with chain/intersect/finalize/breadcrumbs, updates JSON serialization for new variants.
Sources: New Implementations src/sources/btc.rs, src/sources/eth.rs, src/sources/substrate.rs	Implements Bitcoin RPC, Ethereum (WebSocket), and Substrate RPC data sources with Worker/Stage/Config for gasket framework.
Sources: Integration src/sources/mod.rs	Adds feature-gated modules, extends Config enum with BitcoinRpc/EthereumRpc/SubstrateRpc, extends Bootstrapper to support new sources.
Sources: Cardano Updates src/sources/n2c.rs, src/sources/n2n.rs, src/sources/hydra.rs, src/sources/mithril.rs, src/sources/s3.rs, src/sources/u5c.rs	Wraps Cardano records in Record::Cardano(cardano::Record::...) variant, updates chain config extraction to handle new Chain enum.
Filters: Legacy V1 Module Updates src/filters/legacy_v1/...	Updates import paths from crate::framework::legacy_v1 to crate::framework::cardano::legacy_v1, wraps records in Record::Cardano(...) variant.
Filters: Core Logic Updates src/filters/parse_cbor.rs, src/filters/split_block.rs, src/filters/select/eval/mod.rs, src/filters/into_json.rs, src/filters/wasm_plugin.rs	Wraps Cardano records in Record::Cardano(...) variant, adds selective JSON conversion logic.
Sinks: Updates src/sinks/assert/*, src/sinks/elasticsearch.rs, src/sinks/terminal/format.rs, src/sinks/aws_s3.rs	Updates record handling to work with Record::Cardano(...) variant, adds feature-gated handlers for Ethereum/Bitcoin/Substrate records, updates chain config extraction.
Watch CLI: New Modules src/watch/mod.rs, src/watch/cardano.rs, src/watch/btc.rs, src/watch/eth.rs, src/watch/substrate.rs	Introduces subcommand-based watch for each chain with dedicated Args/run functions.
Watch CLI: Main Refactor src/bin/oura/watch.rs	Refactors to dispatch-based design with WatchCommand enum (Cardano, Bitcoin, Ethereum, Substrate), removes direct Bearer/Chain enums, delegates to feature-gated watcher modules.
Dump CLI src/bin/oura/dump.rs	Replaces Chain with DumpChain type for magic option, updates chain construction.
Daemon Config src/daemon/mod.rs	Changes ConfigRoot.chain field type from Option<ChainConfig> to Option<Chain>.
Library Export src/lib.rs	Adds public watch module.

Sequence Diagram(s)

sequenceDiagram
    participant CLI as Watch CLI
    participant Dispatch as Command Dispatcher
    participant Source as Chain Source
    participant Filter as Filters
    participant Sink as Terminal Sink

    CLI->>Dispatch: args: WatchCommand::Bitcoin(btc::Args)
    Dispatch->>Source: initialize BitcoinRpc source
    Source->>Source: connect to RPC node
    Source->>Source: poll for new blocks
    Source->>Filter: emit ChainEvent::Apply<br/>(point, Record::Cardano(...))
    Filter->>Filter: process record
    Filter->>Sink: forward event
    Sink->>Sink: format & display
    Sink-->>CLI: stdout/terminal output

Loading

sequenceDiagram
    participant Pipeline as Pipeline
    participant RecordOld as Old Record Enum
    participant RecordNew as New Record Enum
    participant CardanoMod as Cardano Module

    Pipeline->>RecordOld: Record::CborBlock(bytes)
    Note over RecordOld: Legacy flat variants
    Pipeline->>RecordNew: Record::Cardano(cardano::Record::CborBlock(bytes))
    Note over RecordNew: Feature-gated chain-specific wrappers
    RecordNew->>CardanoMod: nested variant path
    CardanoMod->>CardanoMod: serialize to JSON

Loading

Estimated code review effort

🎯 5 (Critical) | ⏱️ ~120 minutes

Rationale: This is a substantial, high-density refactor introducing multi-chain support across the entire codebase. Key complexity drivers include:

• Large scope: 50+ new files (mostly documentation) + significant framework/source/filter/sink refactoring
• High logic density: new Chain/Record enums with feature-gated variants and nested wrapping logic
• Heterogeneous changes: new data sources, framework restructuring, watch CLI dispatch refactor, extensive import path updates
• Many affected files: framework, sources (6+), filters (5+), sinks (5+), watch modules (5+)
• Feature-gate interactions: need to verify conditional compilation correctness across eth/btc/substrate flags
• Public API surface changes: Chain, Record, Context, WatchCommand structures and watch module organization

Possibly related PRs

• feat: add ethereum support #912: Adds Ethereum support with alloy dependency, eth framework/sources, and watch subcommand refactoring—directly related at code level.
• refactor: scaffold multi-chain pipeline #910: Implements the multi-chain refactor with Cardano::Record wrappers, Chain/ChainConfig types, feature-gated modules—core related changes.

Suggested reviewers

• scarmuega
• paulobressan

Poem

🐰 Hop, hop! A multi-chain feast,
Bitcoin, Eth, and Substrate released,
Records now wrapped in chain-specific grace,
Docs blooming bright across the place,
Oura's journey spans the blockchain space! 🌍✨

Pre-merge checks and finishing touches

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 10.91% which is insufficient. The required threshold is 80.00%.	You can run @coderabbitai generate docstrings to improve docstring coverage.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title Check	✅ Passed	The title "feat: add substrate support" accurately refers to a real and significant part of the changeset—Substrate source support is indeed being added. However, the PR also introduces equally substantial changes: Bitcoin source support (btc.rs, documentation, watch command), Ethereum source support (eth.rs, documentation, watch command), and a fundamental architectural refactor to support multiple blockchains. The title captures only one aspect of the multi-chain support being implemented rather than the broader scope of the changes. While the title is factually correct and not misleading about what it describes, it falls short of conveying the complete picture of the main change.

✨ Finishing touches

• 📝 Generate docstrings

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}