blake3-bao

Pure JavaScript BLAKE3 cryptographic hash and Bao verified streaming implementation with Iroh compatibility and optional WASM SIMD acceleration.

1,158 tests passing (228 BLAKE3 + 930 Bao)

Features

• Pure JavaScript - No native dependencies, works everywhere
• BLAKE3 Hashing - Full implementation with keyed hashing and key derivation
• Bao Verified Streaming - Encode, decode, and slice with Merkle tree verification
• Iroh Chunk Groups - 16x smaller outboard size for efficient verified streaming
• WASM SIMD - Optional 4-way parallel chunk processing for maximum performance
• Streaming API - Memory-efficient processing for large files
• Self-contained - Single files with no external dependencies

Installation

npm

npm install blake3-bao

From Source

git clone https://github.com/lamb356/blake3-optimized.git
cd blake3-optimized

# Run all tests
npm test

# Or run individually:
node test.js                    # BLAKE3 tests (228)
node test-bao-vectors.js        # Bao official vectors (574)
node test-bao-primitives.js     # Bao primitives (41)
node test-bao-encode.js         # Bao encoding (38)
node test-bao-decode.js         # Bao decoding (33)
node test-bao-slice.js          # Bao slicing (24)
node test-bao-streaming.js      # Bao streaming (39)
node test-bao-iroh.js           # Bao Iroh chunk groups (54)
node test-bao-partial.js        # Bao resumable downloads (55)
node test-bao-sequence.js       # Bao hash sequences (72)

# Run benchmarks
node benchmark.js               # BLAKE3 benchmarks
node benchmark-bao.js           # Bao benchmarks

# Build browser bundles
npm run build

Browser Usage

Include the minified bundle via script tag:

<!-- Full bundle (BLAKE3 + Bao) - 37.8 KB -->
<script src="https://unpkg.com/blake3-bao/dist/blake3-bao.min.js"></script>

<!-- Or individual bundles -->
<script src="https://unpkg.com/blake3-bao/dist/blake3.min.js"></script>  <!-- 18.9 KB -->
<script src="https://unpkg.com/blake3-bao/dist/bao.min.js"></script>     <!-- 36.9 KB -->

// Full bundle - access via blake3Bao global
const hash = blake3Bao.blake3.hash('hello');
const hex = blake3Bao.blake3.toHex(hash);
const { encoded } = blake3Bao.bao.baoEncode(data);

// Individual bundles - direct global access
const hash = blake3.hash('hello');
const { encoded } = bao.baoEncode(data);

TypeScript Support

Full TypeScript definitions are included:

import {
  hash, hashHex, Hasher, createHasher,
  baoEncode, baoDecode, PartialBao, HashSequence
} from 'blake3-bao';
import type { BaoEncodeResult, PartialBaoState } from 'blake3-bao/bao';

// Hashing with type inference
const digest: Uint8Array = hash('hello world');
const hex: string = hashHex('hello world');

// Streaming hasher
const hasher: Hasher = createHasher();
hasher.update('hello ').update('world');
const result: Uint8Array = hasher.finalize();

// Bao encoding with typed result
const data = new Uint8Array(2048);
const { encoded, hash: rootHash }: BaoEncodeResult = baoEncode(data);

// Resumable downloads
const partial = new PartialBao(rootHash, data.length);
partial.addChunkGroupTrusted(0, data.slice(0, 16384));
const progress: number = partial.getProgress();

// Hash sequences for collections
const seq = new HashSequence();
seq.addHash(digest).addHash(rootHash);
const collectionHash: Uint8Array = seq.finalize();

Quick Start

// CommonJS
const { hash, hashHex, baoEncode, baoDecode, baoSlice, baoDecodeSlice } = require('blake3-bao');

// Or import individual modules
const blake3 = require('blake3-bao/blake3');
const bao = require('blake3-bao/bao');

// ESM
import { hash, hashHex, baoEncode, baoDecode } from 'blake3-bao';
import blake3 from 'blake3-bao/blake3';
import bao from 'blake3-bao/bao';

// Simple hashing
const digest = blake3.hash('hello world');
console.log(blake3.hashHex('hello world'));
// d74981efa70a0c880b8d8c1985d075dbcbf679b99a5f9914e5aaf96b831a9e24

// Bao encode for verified streaming
const data = new Uint8Array(10000);
const { encoded, hash: rootHash } = bao.baoEncode(data);

// Decode with verification
const decoded = bao.baoDecode(encoded, rootHash);

// Extract and verify a slice (random access)
const slice = bao.baoSlice(encoded, 5000, 1000);
const sliceData = bao.baoDecodeSlice(slice, rootHash, 5000, 1000);

---

BLAKE3 API

hash(input, outputLen = 32)

Hash data and return bytes.

const digest = blake3.hash('hello world');           // Uint8Array(32)
const digest64 = blake3.hash('hello world', 64);     // Uint8Array(64) - XOF
const digest = blake3.hash(new Uint8Array([1,2,3])); // Binary input

hashHex(input, outputLen = 32)

Hash data and return hex string.

const hex = blake3.hashHex('hello world');
// "d74981efa70a0c880b8d8c1985d075dbcbf679b99a5f9914e5aaf96b831a9e24"

hashKeyed(key, input, outputLen = 32)

Keyed hash (MAC) with 32-byte key.

const key = new Uint8Array(32).fill(0x42);
const mac = blake3.hashKeyed(key, 'message');

deriveKey(context, keyMaterial, outputLen = 32)

Derive a key from context string and key material.

const derivedKey = blake3.deriveKey('myapp v1 encryption', masterKey);

Hasher - Streaming API

For processing large data incrementally:

const hasher = new blake3.Hasher();
hasher.update('chunk 1');
hasher.update('chunk 2');
hasher.update(new Uint8Array([1, 2, 3]));
const digest = hasher.finalize();      // Uint8Array(32)
const hex = hasher.finalizeHex();      // hex string

SIMD Acceleration

// Initialize WASM SIMD (optional, for best performance)
await blake3.initSimd();
console.log('SIMD enabled:', blake3.isSimdEnabled());

// All hash functions automatically use SIMD when available

---

Bao API

Bao provides verified streaming - the ability to verify portions of a file without downloading the whole thing.

baoEncode(data, outboard = false)

Encode data with Bao tree structure.

const data = new Uint8Array(10000);

// Combined mode: tree + data interleaved
const { encoded, hash } = bao.baoEncode(data);

// Outboard mode: tree only (data stored separately)
const { encoded: tree, hash } = bao.baoEncode(data, true);

baoDecode(encoded, rootHash, outboardData = null)

Decode and verify Bao-encoded data.

// Combined mode
const decoded = bao.baoDecode(encoded, hash);

// Outboard mode
const decoded = bao.baoDecode(tree, hash, originalData);

baoSlice(encoded, start, len, outboardData = null)

Extract a minimal slice for verifying a byte range.

// Extract bytes 5000-6000 from a file
const slice = bao.baoSlice(encoded, 5000, 1000);

// Slice is much smaller than full encoding
console.log(`Full: ${encoded.length}, Slice: ${slice.length}`);

baoDecodeSlice(slice, rootHash, start, len)

Decode and verify a slice.

const slice = bao.baoSlice(encoded, 5000, 1000);
const data = bao.baoDecodeSlice(slice, hash, 5000, 1000);
// data contains verified bytes 5000-6000

BaoEncoder - Streaming Encoder

const encoder = new bao.BaoEncoder(outboard = false);

// Feed data incrementally
encoder.write(chunk1);
encoder.write(chunk2);
encoder.write(chunk3);

// Get final encoding
const { encoded, hash } = encoder.finalize();

BaoDecoder - Streaming Decoder

const decoder = new bao.BaoDecoder(rootHash, contentLen, isOutboard = false);

// For outboard mode
decoder.setOutboardData(originalData);

// Feed encoded data incrementally
decoder.write(encodedChunk1);
decoder.write(encodedChunk2);

// Read verified data as it becomes available
const verifiedData = decoder.read();

// Check completion
if (decoder.isComplete()) {
  const allData = decoder.finalize();
}

---

Iroh Chunk Groups API

Iroh uses chunk groups (16 chunks = 16 KiB) to reduce outboard size by ~16x compared to standard Bao. The root hash is identical, making it compatible with standard Bao verification.

baoEncodeIroh(data, outboard = false, chunkGroupLog = 4)

Encode data with Iroh-compatible chunk groups.

const data = new Uint8Array(1024 * 1024);  // 1 MB

// Combined mode (same as standard Bao)
const { encoded, hash } = bao.baoEncodeIroh(data, false);

// Outboard mode (16x smaller than standard!)
const { encoded: outboard, hash } = bao.baoEncodeIroh(data, true);

// Standard Bao outboard: ~64 KB
// Iroh outboard: ~4 KB (16x smaller!)

baoVerifyIroh(outboard, rootHash, data, chunkGroupLog = 4)

Verify data against Iroh outboard encoding.

const data = new Uint8Array(1024 * 1024);
const { encoded: outboard, hash } = bao.baoEncodeIroh(data, true);

// Verify later
const isValid = bao.baoVerifyIroh(outboard, hash, data);
console.log('Data is valid:', isValid);  // true

baoDecodeIroh(outboard, rootHash, data, chunkGroupLog = 4)

Verify and return data (throws on failure).

const data = new Uint8Array(1024 * 1024);
const { encoded: outboard, hash } = bao.baoEncodeIroh(data, true);

try {
  const verified = bao.baoDecodeIroh(outboard, hash, data);
  // verified === data (same reference on success)
} catch (e) {
  console.error('Verification failed:', e.message);
}

Helper Functions

// Count chunk groups for a given content length
const groups = bao.countChunkGroups(1024 * 1024);  // 64 groups for 1 MB

// Calculate outboard size
const size = bao.irohOutboardSize(1024 * 1024);  // ~4 KB for 1 MB

// Compute chunk group chaining value
const cv = bao.chunkGroupCV(groupData, startChunkIndex, isRoot);

Outboard Size Comparison

Content Size	Standard Bao	Iroh Outboard	Reduction
100 KB	6.2 KB	0.4 KB	93.8%
1 MB	63.9 KB	3.9 KB	93.8%
10 MB	639.9 KB	39.9 KB	93.8%
100 MB	6.4 MB	400 KB	93.8%

---

PartialBao - Resumable Downloads

The PartialBao class enables resumable downloads and multi-source fetching by tracking which chunk groups have been downloaded and verified.

Basic Usage

const bao = require('./bao.js');

// Get hash and size from metadata/header
const rootHash = /* 32-byte hash */;
const contentLen = 1024 * 1024;  // 1 MB

// Create partial download tracker
const partial = new bao.PartialBao(rootHash, contentLen);

console.log('Total groups:', partial.numGroups);  // 64 for 1 MB
console.log('Progress:', partial.getProgress());  // 0%

// Add chunk groups as they're downloaded
partial.addChunkGroupTrusted(0, groupData0);
partial.addChunkGroupTrusted(5, groupData5);

console.log('Progress:', partial.getProgress());  // ~3%
console.log('Missing:', partial.getMissingGroups().length);

// When complete, finalize to get the data
if (partial.isComplete()) {
  const data = partial.finalize();
}

Multi-Source Download

// Track what we need
const partial = new bao.PartialBao(rootHash, contentLen);
const needed = partial.getMissingRanges();
// [{start: 0, end: 64}] - need all 64 groups

// Download from source A (has groups 0-31)
for (let i = 0; i < 32; i++) {
  const data = await fetchFromSourceA(i);
  partial.addChunkGroupTrusted(i, data);
}

// Check what's still missing
const stillNeeded = partial.getMissingRanges();
// [{start: 32, end: 64}] - need groups 32-63

// Download remaining from source B
for (const range of stillNeeded) {
  for (let i = range.start; i < range.end; i++) {
    const data = await fetchFromSourceB(i);
    partial.addChunkGroupTrusted(i, data);
  }
}

const complete = partial.finalize();

Resumable Download with Persistence

// Start download
const partial = new bao.PartialBao(rootHash, contentLen);

// Download some groups...
partial.addChunkGroupTrusted(0, group0);
partial.addChunkGroupTrusted(1, group1);

// Save state before closing
const state = partial.exportState();
fs.writeFileSync('download.state', JSON.stringify(state));

// --- Later, resume ---

// Restore state
const savedState = JSON.parse(fs.readFileSync('download.state'));
const resumed = bao.PartialBao.importState(savedState);

console.log('Already have:', resumed.receivedGroups);
console.log('Still need:', resumed.getMissingGroups());

// Continue downloading...

With Merkle Proofs

// Server creates proofs for each chunk group
const complete = new bao.PartialBao(rootHash, contentLen);
// ... add all groups ...

const proof = complete.createProof(groupIndex);

// Client verifies proof before accepting
const client = new bao.PartialBao(rootHash, contentLen);
client.addChunkGroup(groupIndex, groupData, proof);  // Throws if invalid

PartialBao API Reference

Method	Description
new PartialBao(rootHash, contentLen, chunkGroupLog?)	Create tracker
numGroups	Total number of chunk groups
receivedGroups	Number of groups received
isComplete()	Check if all groups received
getProgress()	Get completion percentage (0-100)
hasGroup(index)	Check if specific group is present
getGroupSize(index)	Get expected size of a group
addChunkGroup(index, data, proof)	Add with Merkle proof verification
addChunkGroupTrusted(index, data)	Add without proof (trusted source)
getGroupData(index)	Get data for a group (or null)
getMissingRanges()	Get [{start, end}, ...] of missing groups
getPresentRanges()	Get [{start, end}, ...] of present groups
getMissingGroups()	Get array of missing group indices
getPresentGroups()	Get array of present group indices
getBitfield()	Get bitfield as Uint8Array
setBitfield(bf)	Set bitfield (for loading state)
finalize(verify?)	Assemble and return complete data
exportState()	Export state for serialization
PartialBao.importState(state)	Import serialized state
createProof(index)	Create Merkle proof for a group

Bitfield Helpers

// Create a bitfield for tracking N items
const bf = bao.createBitfield(100);  // 100 bits = 13 bytes

// Set/get/clear individual bits
bao.setBit(bf, 5);
bao.getBit(bf, 5);   // true
bao.clearBit(bf, 5);
bao.getBit(bf, 5);   // false

// Count set bits
bao.countSetBits(bf, 100);  // Number of bits set

---

HashSequence - Blob Collections

Hash sequences are ordered lists of blob hashes representing collections like directories or datasets. The sequence itself has a hash, allowing the entire collection to be verified with one hash.

Basic Usage

const bao = require('./bao.js');
const blake3 = require('./blake3.js');

// Create a collection of files
const seq = new bao.HashSequence();

// Add hashes of individual files
seq.addHash(blake3.hash(file1Data));
seq.addHash(blake3.hash(file2Data));
seq.addHash(blake3.hash(file3Data));

// Get the collection hash (verifies entire collection)
const collectionHash = seq.finalize();
console.log('Collection hash:', seq.finalizeHex());

// Check collection contents
console.log('Files in collection:', seq.length);
console.log('Has file1:', seq.hasHash(file1Hash));
console.log('File1 index:', seq.indexOf(file1Hash));

Serialization

// Serialize for storage/transmission
const bytes = seq.toBytes();  // 4-byte count + concatenated hashes

// Deserialize
const restored = bao.HashSequence.fromBytes(bytes);

// JSON serialization
const json = seq.toJSON();
const fromJson = bao.HashSequence.fromJSON(json);

// Hex string input
const seq2 = bao.HashSequence.fromHex([
  '0123456789abcdef...', // 64-char hex strings
  'fedcba9876543210...'
]);

Sequence Format (Matching Iroh)

Field	Size	Description
Count	4 bytes	Little-endian number of hashes
Hashes	N × 32 bytes	Concatenated 32-byte hashes

Total size: 4 + (count × 32) bytes

HashSequence API Reference

Method	Description
new HashSequence([hashes])	Create sequence, optionally with initial hashes
addHash(hash)	Add 32-byte hash, returns this
length	Number of hashes in sequence
getHash(index)	Get hash at index (copy)
getHashHex(index)	Get hash at index as hex string
hasHash(hash)	Check if hash exists in sequence
indexOf(hash)	Find index of hash (-1 if not found)
[Symbol.iterator]	Iterate over hashes (for...of)
toArray()	Get all hashes as array
finalize()	Get BLAKE3 hash of the sequence
finalizeHex()	Get sequence hash as hex string
toBytes()	Serialize to bytes
fromBytes(bytes)	Deserialize from bytes (static)
from(hashes)	Create from hash array (static)
fromHex(hexStrings)	Create from hex strings (static)
toJSON()	Export to JSON-serializable object
fromJSON(json)	Create from JSON object (static)
clear()	Remove all hashes
removeAt(index)	Remove and return hash at index
insertAt(index, hash)	Insert hash at index
slice(start, end?)	Create new sequence with slice
concat(other)	Create new sequence combining both
equals(other)	Check if sequences are equal

---

Examples

Verifying File Integrity

const fs = require('fs');
const bao = require('./bao.js');

// Encode a file
const fileData = fs.readFileSync('largefile.bin');
const { encoded, hash } = bao.baoEncode(fileData);

// Save encoding and hash
fs.writeFileSync('largefile.bao', encoded);
fs.writeFileSync('largefile.hash', hash);

// Later: verify and decode
const savedEncoded = fs.readFileSync('largefile.bao');
const savedHash = fs.readFileSync('largefile.hash');
const verified = bao.baoDecode(savedEncoded, savedHash);

Random Access Verification

// Only download and verify specific byte range
const slice = bao.baoSlice(encoded, 1000000, 4096);  // 4KB at offset 1MB
const data = bao.baoDecodeSlice(slice, hash, 1000000, 4096);

// Slice is ~2KB regardless of file size (only tree nodes + 1-2 chunks)

Streaming Large Files

const encoder = new bao.BaoEncoder();

// Process file in chunks
const stream = fs.createReadStream('hugefile.bin', { highWaterMark: 64 * 1024 });
for await (const chunk of stream) {
  encoder.write(chunk);
}

const { encoded, hash } = encoder.finalize();

Keyed Hashing for Authentication

const blake3 = require('./blake3.js');

// Create MAC
const key = crypto.randomBytes(32);
const mac = blake3.hashKeyed(key, message);

// Verify MAC
const expectedMac = blake3.hashKeyed(key, receivedMessage);
if (mac.every((b, i) => b === expectedMac[i])) {
  console.log('Message authenticated');
}

---

Performance

v1.1.0 Optimizations

Version 1.1.0 brings significant performance improvements:

• Buffer pooling: Reusable Uint32Array buffers reduce GC pressure
• Single allocation: baoEncode pre-calculates exact output size
• Streaming encoder: BaoEncoder with O(log n) memory in outboard mode
• WASM acceleration: Optional WASM modules for large files

BLAKE3 Throughput

Input Size	Scalar	With SIMD
1 KB	~400 MB/s	~800+ MB/s
16 KB	~450 MB/s	~1200+ MB/s
1 MB	~450 MB/s	~1500+ MB/s

Bao Throughput (v1.1.0)

Operation	Throughput	vs v1.0.0
Encode (combined)	~185 MB/s	83% faster
Encode (outboard)	~198 MB/s	29% faster
BaoEncoder streaming	~150-168 MB/s	O(log n) memory
chunkCV primitive	~203 MB/s	-
Decode	~173 MB/s	-

Slice Efficiency

File Size	1KB Slice Size	Reduction
100 KB	1.45 KB	98.6%
1 MB	1.63 KB	99.8%
10 MB	1.88 KB	100.0%

Encoding Overhead

• Combined mode: ~6.25% (tree nodes interleaved with data)
• Outboard mode: ~6.25% of input size (tree only)

---

Files

File	Description
blake3.js	BLAKE3 implementation (self-contained)
bao.js	Bao verified streaming implementation
test.js	BLAKE3 test suite (228 tests)
test-bao-*.js	Bao test suites (930 tests total)
test-vectors.json	Official Bao test vectors
benchmark.js	BLAKE3 performance benchmarks
benchmark-bao.js	Bao performance benchmarks
compress4x.wat	WASM SIMD source
compress4x.wasm	Compiled WASM binary

---

Test Coverage

Component	Tests
BLAKE3 core	35
BLAKE3 keyed	35
BLAKE3 derive_key	35
BLAKE3 XOF	123
Bao primitives	41
Bao encoding	38
Bao decoding	33
Bao slicing	24
Bao streaming	39
Bao Iroh chunk groups	54
Bao resumable downloads	55
Bao hash sequences	72
Bao official vectors	574
Total	1,158

---

Platform Support

BLAKE3

• Node.js 14+
• All modern browsers
• Deno, Bun

WASM SIMD

• macOS: M1/M2/M3/M4 chips
• Linux: Modern x64 with Node.js 16+
• Windows: Node.js with SIMD support

Bao

• Same as BLAKE3 (pure JavaScript)

---

References

• BLAKE3 Specification
• Bao Specification
• Iroh Bao Format - Chunk group optimization for smaller outboard
• Fleek Network Case Study

License

MIT