Narwhal++ (narwhal-cpp)

A high-performance C++20 port of the Narwhal and Tusk DAG-based mempool and consensus engine, featuring modular support for Shoal++ and Mysticeti

This project is designed as a modular, high-fidelity implementation of state-of-the-art consensus protocols, focusing on low latency and horizontal scalability.

🎯 Project Goal

The primary goal of this project is to evaluate the performance of the Narwhal++ mempool implementation in C++20 when integrated with various state-of-the-art BFT consensus engines.

By providing a high-fidelity C++ port, this repository enables a direct performance comparison between:

• Tusk: The baseline DAG-based consensus.
• Shoal++: Performance-optimized selection with reputation.
• Mysticeti: The latest advancement in low-latency uncertified DAG consensus.

This evaluation focuses on identifying bottlenecks, measuring throughput (certificates/sec), and analyzing latency characteristics across different consensus strategies within a unified C++20 framework.

🏗️ Architecture Overview

       [ Clients ]
            │
            ▼
┌───────────────────────┐      ┌─────────────────────────┐
│     Narwhal Mempool   │      │    Consensus Engines    │
│  (Data Availability)  │      │  (Total Ordering)       │
├───────────────────────┤      ├─────────────────────────┤
│ ┌───┐   ┌───┐   ┌───┐ │      │  ┌───────────────┐      │
│ │ B │──▶│ B │──▶│ B │ │      │  │    Mysticeti  │      │
│ └───┘   └───┘   └───┘ │      │  └───────────────┘      │
│   ▲       ▲       ▲   │      │          ▲              │
│   │       │       │   ◀──────┼──────────┘              │
│ ┌───┐   ┌───┐   ┌───┐ │      │  ┌───────────────┐      │
│ │ C │──▶│ C │──▶│ C │ │      │  │    Shoal++    │      │
│ └───┘   └───┘   └───┘ │      │  └───────────────┘      │
│   ▲       ▲       ▲   │      │          ▲              │
│   │       │       │   ◀──────┼──────────┘              │
│ ┌───┐   ┌───┐   ┌───┐ │      │  ┌───────────────┐      │
│ │ G │──▶│ G │──▶│ G │ │      │  │      Tusk     │      │
│ └───┘   └───┘   └───┘ │      │  └───────────────┘      │
└───────────────────────┘      └─────────────────────────┘
     Shared Mempool DAG             Modular Pluggable
      (C++20 Buffers)                Logic Layers

🚀 Key Features

• Modular Consensus Engines: Switch seamlessly between:
  • Tusk: High-throughput DAG-based consensus.
  • Shoal++: Advanced anchor selection with reputation systems.
  • Mysticeti: Ultra-low latency "3-hop" consensus path.
• DAG-based Architecture: Decouples data availability from transaction ordering.
• High-Performance C++20: Utilizing modern C++ features for maximum efficiency.
• Windows-Friendly: Integrated PowerShell scripts for local cluster simulation and benchmarking.
• Pluggable Backend: Support for both production-ready dependencies (RocksDB, Sodium, Boost.Asio) and internal mocks for rapid testing/CI.

🛠 Tech Stack

• Core: C++20
• Build System: CMake 3.20+
• Asynchronous I/O: Boost.Asio
• Storage: RocksDB
• Cryptography: Libsodium (Ed25519)
• Networking: TLS 1.3 / Noise Protocol

🏁 Getting Started

Prerequisites

• A modern C++ compiler (MSVC 2022+ or GCC 11+)
• CMake 3.20 or higher

Build

# Create build directory and configure
cmake -B build -S .

# Build in Release mode
cmake --build build --config Release

Run a Local Cluster

Launch a 4-node local cluster with the Mysticeti engine:

./run_cluster.ps1 -Engine mysticeti -Load

Possible engines: tusk, shoal++, mysticeti.

📊 Benchmark Results (Local Mock Mode)

These benchmarks were performed on a local 4-node cluster using internal mocks for networking and storage to measure the maximum theoretical throughput of the consensus engines.

Engine	Throughput (Cert/sec)	Latency (ms)	Key Mechanism
Tusk	~19,800	~450	DAG-link based commit
Shoal++	~22,600	~320	Adaptive Anchor selection + Reputation
Mysticeti	~2,150 (Leaders)	~120	3-hop Fast Path (Fastest Finality)

Note: Tusk and Shoal++ report throughput of all committed certificates in the DAG, while Mysticeti throughput focuses on individual leader-round commitments for ultra-low latency.

⚠️ Current State & Verification

Implementation Status

Currently Implemented (Real):

• ✅ Core DAG consensus logic for all three engines
• ✅ Certificate creation, validation, and ordering
• ✅ Multi-threaded consensus execution
• ✅ Modular engine architecture

Currently Mocked (For Testing):

• 🔶 Network layer (TLS 1.3 connections simulated)
• 🔶 Cryptographic signatures (Ed25519 placeholder)
• 🔶 Persistent storage (In-memory maps instead of RocksDB)

Important: This is a research and evaluation platform, not a production-ready system. The mocks enable rapid prototyping and benchmarking of consensus algorithms without the overhead of full network I/O.

Verification Strategy

Given the critical nature of BFT consensus correctness, we are planning a multi-layered verification approach:

1. Formal Verification (Planned):

   • TLA+ specifications for consensus invariants
   • Property-based testing with RapidCheck (C++ equivalent to Rust's proptest)
   • Model checking for liveness and safety properties

2. Byzantine Fault Simulation (Phase 4):

   • Adversarial node behavior injection
   • Network partition and delay simulation
   • Equivocation detection testing

3. Observability (Phase 1):

   • Prometheus/Grafana metrics integration
   • OpenTelemetry distributed tracing
   • Consensus state visualization

🗺 Roadmap

• Core DAG Logic & Tusk Implementation
• Shoal++ & Mysticeti engines
• Phase 0: Formal Verification Foundation
  • TLA+ specifications for consensus safety/liveness
  • Property-based testing suite (RapidCheck)
  • Invariant checking in CI/CD
• Phase 1: Infrastructure Professionalization
  • Asynchronous Multi-threaded Networking (Boost.Asio)
  • Real Ed25519 signatures with batch verification
  • Prometheus metrics & Grafana dashboards
  • OpenTelemetry distributed tracing
• Phase 2: Advanced BFT Liveness
  • Pacemaker & Timeout Management
  • View-change protocol
  • Hot-reloading configuration
• Phase 3: Scalable Data Availability (Separate Worker-Layer)
• Phase 4: Hardening (Byzantine Simulation & Fuzzing)

📄 License

This project is licensed under the MIT License.