feat: UXF module + Profile storage — content-addressable token packaging with OrbitDB#105
Open
feat: UXF module + Profile storage — content-addressable token packaging with OrbitDB#105
Conversation
Initial documentation suite for the Universal eXchange Format (UXF), a content-addressable packaging format for Unicity token materials. Documents: - TASK.md: requirements and acceptance criteria - SPECIFICATION.md: formal spec with CDDL, JSON Schema, 13 element types - ARCHITECTURE.md: module structure, TypeScript types, API surface - DESIGN-DECISIONS.md: 14 binding decisions resolving design trade-offs - REVIEW.md: adversarial architecture review findings - IPFS-RESEARCH.md: IPLD/dag-cbor/CAR state-of-art research - TOKEN-ANALYSIS.md: byte-level token field analysis and dedup estimates
addToken was a redundant alias for ingest. Single entry point for adding tokens is now ingest() / ingestAll().
…IONS Resolves all 22 cross-document contradictions found by steelman review. Critical fixes: - Remove SmtPathSegment as separate element; inline in SmtPath (Decision 5) - Change canonical type from TxfToken to ITokenJson throughout (Decision 1) - Replace hand-written CBOR with @ipld/dag-cbor references (Decision 3) - Standardize hash canonical form as 4-key map, not positional array - Align instance chain index structure between SPEC and ARCH - Fix TransferTransaction children (sourceState/destinationState) Other fixes: - TokenRoot: remove tokenType/coinData (live in MintTransactionData) - Predicate: make opaque (single raw bytes field) - TokenState: predicate inlined as leaf, not child ref - MintTransactionData: coinData inlined as leaf - CAR: standardize on CARv1 with single envelope root - Reassembly: add mandatory hash verification + cycle detection - Add version mapping (semantics:1 = state-transition-sdk v2.0) - Make instanceChainIndex required in JSON Schema - Add CYCLE_DETECTED error code - Add Phase 2 annotation to consolidateProofs - Document mutable API model
Critical: - Add version field to TokenRoot in SPEC (was in ARCH only) - Collapse destination-state into token-state (no separate type) - Fix worked example element count (22, not 23; remove Phase 2 types) Warnings: - Fix type field encoding: uint in hash computation, not string - Fix nametag reassembly: use root hash, not manifest lookup - Fix timestamp units: seconds consistently (not ms) - Fix envelope version: string "1.0.0" (not number) - Fix section numbering in SPEC (9.4 before 9.5) - Clarify Phase 1 scope for Predicate/TokenCoinData (inline, not separate) - Standardize element type count at 12 - Rename merkle-tree-path to smt-path in ARCH Notes: - Fix CDDL appendix naming consistency - Populate genesis destination state with actual data
IMPLEMENTATION-PLAN.md: 17 work units across 6 layers with dependency graph and parallelization strategy (10 agents, 5 waves). DOMAIN-CONSTRAINTS.md: field-by-field ITokenJson→UXF mapping, hex/binary conversion rules, state derivation algorithms, nametag handling, and 12 critical implementation pitfalls.
5 critical fixes: - Split token reason field: Uint8Array (not string) for ISplitMintReasonJson - Reject placeholder/_pendingFinalization sentinel tokens - Track nametag refs in transfer transaction data for round-trip fidelity - Main barrel exports UXF types only (not runtime) to avoid @ipld/dag-cbor dep - Use ITokenJson sub-types, not TxfToken sub-types in deconstruction 11 warning fixes: - merge() re-hashes incoming elements before insertion - Hex lowercase normalization in deconstruction - UxfElement.children includes null in type union - Functional API labeled as mutating (not pure) - prepareContentForHashing for hex→bytes before CBOR encoding - message field handling in TransferTransactionData - Explicit TransactionDataContent fields (recipient, salt, etc.) - SmtPath path values are decimal bigint strings, not hex - Document Phase 1 accepts null inclusionProof (diverges from domain constraints) - diff/applyDelta marked as Phase 1 LOW priority - NOT_IMPLEMENTED error code verified present
Complete implementation of the Universal eXchange Format (UXF) module: Core (Wave 1): - types.ts: 12 element types, branded ContentHash, all interfaces - errors.ts: UxfError with 12 error codes - hash.ts: SHA-256 over deterministic dag-cbor canonical form - element-pool.ts: content-addressed Map store with GC Algorithms (Wave 2-3): - instance-chain.ts: chain management, selection strategies, merge - deconstruct.ts: ITokenJson → DAG decomposition (12 pitfalls handled) - assemble.ts: DAG → ITokenJson reassembly with hash verification + cycle detection - verify.ts: full package integrity verification - diff.ts: package delta computation and application Serialization (Wave 3): - json.ts: JSON round-trip serialization - ipld.ts: CIDv1 computation, IPLD blocks, CARv1 export/import Integration (Wave 4-5): - UxfPackage.ts: fluent API wrapping all subsystems - storage-adapters.ts: InMemoryUxfStorage, KvUxfStorageAdapter - index.ts: barrel exports - Build config: tsup entry point, package.json exports Dependencies: @ipld/dag-cbor, @ipld/car, multiformats
Security: - hexToBytes: validate input (even length, valid hex chars) - applyDelta: re-hash incoming elements before insertion - importFromCar: verify element hashes after CBOR decoding - Add recursion depth limit (100) for nametag traversal in deconstruct and assemble to prevent stack overflow DoS Correctness: - Preserve null state.data and null SmtPath step.data (was coerced to empty string, changing hashes and losing null semantics) - Fix assembleTokenAtState false CYCLE_DETECTED when stateIndex=0 (genesis visited twice; now uses direct resolution for destinationState) - Fix JSON reason Uint8Array round-trip (hex→Uint8Array on deserialize) - Add hex lowercase normalization during JSON deserialization - Fix rebuildInstanceChains in ipld.ts to handle branching chains (was Map<hash, hash>, now Map<hash, hash[]>) Maintenance: - ElementPool: add public toMap()/fromMap() methods - UxfPackage: use ElementPool.fromMap() instead of private field cast
TEST-SPECIFICATION.md: 240 test cases across 14 test files covering all 14 source modules — unit tests, edge cases, round-trips, error paths, and integration flows. TEST-FIXTURES-SPEC.md: 6 mock tokens (A-F) with shared elements, edge case tokens, expected element counts, incremental dedup math, and round-trip normalization rules.
- Fix token-state count in TEST-FIXTURES-SPEC: 12→9 (was double-counting shared states) - Fix normalization note: state.data null is preserved faithfully (not coerced to "") - Fix StateContent.data type: string → string | null (matches runtime behavior) - Update assembleState return type and callers to propagate null data
Test coverage for the complete UXF module: Foundation (T1): - errors.test.ts (7 tests): error construction, codes, instanceof - types.test.ts (25 tests): contentHash validation, type IDs, constants - hash.test.ts (22 tests): determinism, hex/bytes conversion, BigInt paths - element-pool.test.ts (19 tests): CRUD, dedup, GC, toMap/fromMap - instance-chain.test.ts (34 tests): chains, selection strategies, merge Core algorithms (T2): - deconstruct.test.ts (23 tests): element counts, dedup, nametags, edge cases - assemble.test.ts (18 tests): round-trip, historical, error paths, depth limits Serialization & verification (T2): - verify.test.ts (16 tests): integrity, cycles, orphans, chain validation - diff.test.ts (12 tests): delta computation, application, verification - json.test.ts (18 tests): round-trip, format, hex normalization - ipld.test.ts (14 tests): CID, IPLD blocks, CAR round-trip - storage-adapters.test.ts (10 tests): InMemory, KV adapter Integration (T3): - UxfPackage.test.ts (38 tests): full class API coverage - integration.test.ts (9 tests): end-to-end flows with all 6 mock tokens Also includes tests/fixtures/uxf-mock-tokens.ts with 6 mock tokens, shared constants, edge case tokens, and expected dedup counts.
The verifier's DFS used a flat visited-set which flagged content- deduplicated DAG diamonds (same element referenced by two sibling paths) as cycles. Replaced with path-based cycle detection: - pathStack tracks ancestors (back-edge = true cycle) - visited tracks fully-explored nodes (skip without error) This fixes verify() returning valid=false for structurally valid packages with content-addressed dedup (e.g., token-state shared between genesis.destinationState and token-root.state). Test cleanup: removed all CYCLE_DETECTED filters from 6 test files. Tests now assert result.valid===true directly. The verify.test.ts "cycle in DAG" test now creates an actual cycle (back-edge). All 268 tests pass.
Introduces the UXF Profile concept: a key-value table representing the complete persistent state of a Sphere user wallet. Key design decisions (pending approval): - IPFS is source of truth; local storage is transient cache - Profile stored as IPLD DAG, addressed via IPNS - Token inventory referenced by CID to UXF CAR file - Lazy loading: only cache accessed tokens locally - Full backward compatibility with StorageProvider/TokenStorageProvider - Encryption at rest via wallet-derived key (AES-256-GCM) - Maps all 40+ existing storage keys to Profile schema - Covers all SDK flows: send, receive, DMs, nametags, swaps, accounting Includes 6 open questions for architectural review.
SDK-STORAGE-INVENTORY.md: complete mapping of all 40+ storage keys across StorageProvider and TokenStorageProvider, with criticality levels, data shapes, sizes, and per-address scoping. IPFS-KV-RESEARCH.md: state-of-art (2024-2025) research covering OrbitDB, Helia, IPNS, w3name, CAR files, WNFS, browser storage (IndexedDB/OPFS), Node.js options (lmdb-js/SQLite/LevelDB), sync patterns, and lazy loading strategies.
Key architectural changes per review feedback: 1. OrbitDB as profile KV store (replaces raw IPLD DAG): - Merkle-CRDT OpLog for automatic conflict resolution - keyvalue database type with last-writer-wins per key - Replication via libp2p PubSub (real-time) + DHT (cold sync) - No manual merge logic needed 2. Multiple UXF bundle CIDs (replaces single tokens.inventoryCid): - tokens.bundles stores array of UxfBundleRef - Each device appends its own bundle CID — no conflict - Reading merges all active bundles via UxfPackage.merge() - Content-addressed dedup ensures no duplication - Background consolidation merges bundles lazily - Superseded bundles kept for safety period before deletion Updated sections: 1 (overview), 2.3 (multi-bundle model), 3 (OrbitDB persistence), 4 (OrbitDB database), 5.3 (token storage flows), 7.1-7.3 (operation flows). Resolved questions 3 and 5. Added 3 new open questions about bundle size, replication, and safety period.
Decisions: 1. Cache-only keys (prices, registry) → local storage only, not OrbitDB 2. No encryption in Phase 1 — preserves content-addressed dedup across devices 3. Silent auto-migration: sync old IPFS → transform → persist OrbitDB → verify → cleanup 4. Sphere app WalletRepository migration → separate follow-up task 5. Accept OrbitDB bundle size — replaces custom IPFS sync code 6. Nostr relay only for replication — no Voyager 7. Remove superseded bundles from Profile only — no IPFS unpinning (GC deferred) Key changes to UxfBundleRef: - deleteAfter → removeFromProfileAfter (profile-only cleanup) - Remove 'pending-deletion' status (lifecycle: active → superseded → removed) - Old CIDs remain pinned on IPFS indefinitely Added Section 7.6: detailed 6-step migration flow with mandatory sanity check. Updated Section 9: no encryption, OrbitDB access control only. Updated Section 4.4: Nostr as sole replication channel.
Critical fixes:
- Replace tokens.bundles array with per-key tokens.bundle.{CID} pattern
(OrbitDB LWW overwrites arrays — per-key avoids conflict entirely)
- Add HKDF-derived shared encryption (AES-256-GCM) for all OrbitDB
values — same key from mnemonic on all devices preserves CID dedup
for CAR files via deterministic IV
- Add missing storage keys: mintOutbox, invalidTokens,
invalidatedNametags, tombstones (per-address)
- Encrypt DM content before OrbitDB storage (preserves NIP-17 privacy)
- Add two-phase commit for consolidation crash recovery
Warning fixes:
- OpLog growth management (snapshots, batching, growth estimates)
- Nostr-OrbitDB bridge marked Phase 2 (not "thin adapter")
- Browser runtime constraints documented (WebRTC, bundle size)
- Concurrent consolidation guard (5-min TTL lock)
- IPNS expiry fallback in migration (skip if failed)
- Fix historical CID unpinning (only last known CID)
- Vesting cache protected during migration cleanup
- wallet_exists fast-path via local storage flag
- 20-bundle performance cap with degraded mode
- CAR fetch failure handling with pinning verification
- Strengthened migration sanity check (per-token verification)
- Migration phase tracking for crash recovery
Section 7.5 (Nametag Registration) was the only operation flow that
didn't show the explicit UxfPackage→toCar→pin→CID→OrbitDB chain.
Now consistent with Sections 7.1 (send) and 7.2 (receive).
All token references in the Profile now go through UXF bundle CIDs:
- Save: ingest → toCar → pin → db.put('tokens.bundle.{CID}', ref)
- Load: list bundle keys → fetch CARs → merge → assemble
- Nametag: ingest → toCar → pin → db.put('tokens.bundle.{CID}', ref)
PROFILE-IMPLEMENTATION-PLAN.md: 15 work units across 5 layers with maximum parallelization (3 parallel workers per layer). Layers: - L0: types, encryption, OrbitDB adapter (parallel) - L1: ProfileStorageProvider, ProfileTokenStorageProvider, local cache (parallel) - L2: bundle manager, TxfToken adapter, CAR pinning (parallel) - L3: migration engine, consolidation engine, sync coordinator (parallel) - L4: factory functions, barrel exports, build config PROFILE-INTEGRATION-POINTS.md: complete analysis of SDK storage touchpoints — every storage call in Sphere.ts, PaymentsModule.ts, factory functions. 8 backward compatibility risks with mitigations. 6-section migration touchpoint inventory.
Structural simplification per code review: - Merge TxfAdapter, BundleManager, CAR Pinning, SyncCoordinator into WU-P05 (ProfileTokenStorageProvider) as private helpers - Defer ConsolidationEngine and LocalCacheLayer to Phase 2 - Merge BarrelExports + BuildConfig into single WU-P14 - Reduce from 5 layers to 3 layers, 15 WUs to 8 WUs Security fixes per audit: - Remove deterministic IV for CAR encryption (use random IV always) - Document encryption bootstrap sequence (password → mnemonic → key) API fixes: - setIdentity() is synchronous (OrbitDB deferred to connect()) - Write-behind buffer with 2s debounce for save() - storage:remote-updated event emission on OrbitDB replication - OrbitDB access controller with write restriction - Standalone factory functions (no upstream file modifications) - Complete key mapping (dynamic patterns, IPFS exclusion, reverse) - Proper sync() contract returning SyncResult - Migration: accounting/swap keys, nametag tokens, forked tokens - @orbitdb/core as peerDependency with optional:true - Platform-specific entry points (browser.ts, node.ts)
…torage Complete implementation of the Profile storage layer: Layer 0 (Foundation): - types.ts: ProfileConfig, UxfBundleRef, key mappings, 9 error codes - errors.ts: ProfileError class - encryption.ts: HKDF key derivation + AES-256-GCM encrypt/decrypt - orbitdb-adapter.ts: dynamic-import OrbitDB wrapper with access control Layer 1 (Storage Providers): - profile-storage-provider.ts: StorageProvider impl with key translation, local cache + OrbitDB dual-write, encryption, reverse key mapping - profile-token-storage-provider.ts: TokenStorageProvider impl with multi-bundle model, write-behind buffer (2s debounce), UXF CAR pin/fetch, OrbitDB replication events, inline bundle management Layer 2 (Integration): - migration.ts: 6-step legacy→Profile migration with crash recovery - factory.ts: shared provider wiring logic - browser.ts: createBrowserProfileProviders() (standalone) - node.ts: createNodeProfileProviders() (standalone) - index.ts: barrel exports (types + runtime) Build: 3 tsup entry points (profile/index, profile/browser, profile/node) Deps: @orbitdb/core + helia as peerDependencies (optional: true) No upstream files modified (factories are standalone)
Critical:
- Remove duplicate types from orbitdb-adapter.ts (import from errors/types)
- Remove private key leak in derivePublicKeyShort fallback (throw instead)
- Replace require() with dynamic import() for ESM compatibility
High:
- Fix address ID mismatch: add computeAddressId() utility, compute in
setIdentity(), return short ID from getAddressId()
- Extract archived-* and _forked_* tokens in save path (was silently dropping)
- Fix createForAddress() to accept and propagate addressId parameter
- Add HKDF salt ('sphere-profile-v1') for proper domain separation
- Track lastPinnedCid for flush retry (don't re-pin on OrbitDB failure)
- Clean up replication listeners in close() before clearing set
- Guard encryptionKey with null checks (throw PROFILE_NOT_INITIALIZED)
Coverage: errors (7), encryption (12), OrbitDB adapter (14), ProfileStorageProvider (22), ProfileTokenStorageProvider (28), migration (16), integration (10).
Foundation (19 tests): - errors.test.ts (7): construction, codes, instanceof, cause - encryption.test.ts (12): HKDF, AES-GCM round-trip, IV uniqueness, tamper Providers (66 tests): - orbitdb-adapter.test.ts (14): CRUD, prefix filter, replication, lifecycle - profile-storage-provider.test.ts (22): key translation, cache-only, encryption - profile-token-storage-provider.test.ts (30): write-behind, multi-bundle, sync, replication events, history, createForAddress, flush retry Migration + Integration (25 tests): - migration.test.ts (16): 6-step flow, _sent merge, nametag/forked extraction, phase tracking, crash recovery, sanity check, edge cases - integration.test.ts (9): full lifecycle, multi-device, factory wiring
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
Adds new UXF and Profile modules to sphere-sdk, including OrbitDB-backed profile storage, encryption utilities, platform factories, package export entry points, and extensive unit tests/documentation.
Changes:
- Introduces OrbitDB adapter + Profile provider wiring (browser/node/shared factory) with optional OrbitDB/Helia peer dependencies.
- Implements Profile encryption/error types and key mapping/types for Profile persistence.
- Adds a large battery of Profile unit tests and UXF/Profile documentation/spec material.
Reviewed changes
Copilot reviewed 38 out of 70 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/unit/profile/profile-storage-provider.test.ts | Adds unit tests for key mapping, cache vs OrbitDB behavior, encryption, and special cases. |
| tests/unit/profile/orbitdb-adapter.test.ts | Adds tests for a mock ProfileDatabase contract (named as OrbitDbAdapter tests). |
| tests/unit/profile/migration.test.ts | Adds unit tests covering legacy → Profile migration phases and edge cases. |
| tests/unit/profile/integration.test.ts | Adds end-to-end mock integration tests for Profile providers + migration + factory behavior. |
| tests/unit/profile/errors.test.ts | Adds unit tests for ProfileError formatting and codes. |
| tests/unit/profile/encryption.test.ts | Adds unit tests for HKDF/AES-GCM encryption helpers. |
| profile/types.ts | Adds Profile/OrbitDB config types, key mapping constants, and computeAddressId. |
| profile/orbitdb-adapter.ts | Adds dynamic-import OrbitDB/Helia adapter implementing ProfileDatabase. |
| profile/encryption.ts | Adds HKDF-derived key + AES-256-GCM encryption for OrbitDB values. |
| profile/errors.ts | Adds structured ProfileError and error codes. |
| profile/factory.ts | Adds shared factory wiring ProfileStorageProvider + ProfileTokenStorageProvider. |
| profile/browser.ts | Adds browser factory using IndexedDB storage as local cache. |
| profile/node.ts | Adds Node.js factory using file storage as local cache. |
| profile/index.ts | Adds Profile public API barrel exports. |
| package.json | Adds new exports (./uxf, ./profile/*) + IPLD dependencies + optional peer deps for OrbitDB/Helia. |
| index.ts | Re-exports UXF/Profile types from the root barrel (type-only). |
| docs/uxf/* | Adds UXF/Profile specifications, research, design decisions, and test plans. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+151
to
+153
| * from the wallet master key via HKDF. | ||
| * | ||
| * profileEncryptionKey = HKDF(masterKey, "uxf-profile-encryption", 32) |
There was a problem hiding this comment.
The HKDF derivation description is inconsistent with profile/encryption.ts (which uses a domain-specific salt "sphere-profile-v1" plus PROFILE_HKDF_INFO = "uxf-profile-encryption"). Update this comment to reflect the actual derivation inputs (salt + info) so readers don’t implement an incompatible key derivation.
Suggested change
| * from the wallet master key via HKDF. | |
| * | |
| * profileEncryptionKey = HKDF(masterKey, "uxf-profile-encryption", 32) | |
| * from the wallet master key via HKDF using the fixed salt | |
| * "sphere-profile-v1" and the HKDF info string "uxf-profile-encryption". | |
| * | |
| * profileEncryptionKey = HKDF(masterKey, salt="sphere-profile-v1", info="uxf-profile-encryption", 32) |
Comment on lines
+383
to
+389
| try { | ||
| const hashModule: any = await import('@noble/hashes/sha256' as string); | ||
| const hash: Uint8Array = hashModule.sha256(hexToBytes(privateKeyHex)); | ||
| return bytesToHex(hash).slice(0, 16); | ||
| } catch { | ||
| // Both mandatory dependencies are missing | ||
| } |
There was a problem hiding this comment.
The fallback dynamic import path @noble/hashes/sha256 is likely incorrect/incompatible with how the repo imports Noble hashes elsewhere (e.g. @noble/hashes/sha2.js). If this import fails at runtime, derivePublicKeyShort() will throw even when hashes are installed, breaking OrbitDB connect in environments where the curves import fails. Align this import with the project’s working Noble entrypoint (e.g. import from @noble/hashes/sha2.js and use sha256).
Comment on lines
+202
to
+229
| const allEntries = await this.db.all(); | ||
|
|
||
| // allEntries may be an object, a Map, or an async iterable depending | ||
| // on the OrbitDB version. Handle each case. | ||
| if (allEntries && typeof allEntries[Symbol.asyncIterator] === 'function') { | ||
| for await (const entry of allEntries) { | ||
| const entryKey: string = entry.key ?? entry[0]; | ||
| const entryValue = entry.value ?? entry[1]; | ||
| if (prefix && !entryKey.startsWith(prefix)) { | ||
| continue; | ||
| } | ||
| result.set(entryKey, coerceToUint8Array(entryValue)); | ||
| } | ||
| } else if (allEntries instanceof Map) { | ||
| for (const [entryKey, entryValue] of allEntries) { | ||
| if (prefix && !entryKey.startsWith(prefix)) { | ||
| continue; | ||
| } | ||
| result.set(entryKey, coerceToUint8Array(entryValue)); | ||
| } | ||
| } else if (typeof allEntries === 'object' && allEntries !== null) { | ||
| for (const [entryKey, entryValue] of Object.entries(allEntries)) { | ||
| if (prefix && !entryKey.startsWith(prefix)) { | ||
| continue; | ||
| } | ||
| result.set(entryKey, coerceToUint8Array(entryValue as Uint8Array | Record<string, number>)); | ||
| } | ||
| } |
There was a problem hiding this comment.
all() handles async iterables, Map, and plain objects, but not synchronous iterables (Symbol.iterator). If OrbitDB returns a sync iterable (common pattern for collections), this will currently fall through and return an empty result. Add a Symbol.iterator branch (between async iterator and Map) to support sync iterables.
| if (config.directory) { | ||
| heliaOptions.directory = config.directory; | ||
| } | ||
|
|
There was a problem hiding this comment.
OrbitDbConfig includes bootstrapPeers and enablePubSub, but connect() currently ignores them when building Helia/libp2p options. This makes those public config fields misleading (callers can set them with no effect). Either wire them through to Helia/libp2p configuration (preferred) or remove them from OrbitDbConfig until supported.
Suggested change
| if (Array.isArray(config.bootstrapPeers) && config.bootstrapPeers.length > 0) { | |
| throw new ProfileError( | |
| 'UNSUPPORTED_ORBITDB_CONFIG', | |
| 'OrbitDbConfig.bootstrapPeers is not yet supported by OrbitDbAdapter.connect()', | |
| ); | |
| } | |
| if (config.enablePubSub === true) { | |
| throw new ProfileError( | |
| 'UNSUPPORTED_ORBITDB_CONFIG', | |
| 'OrbitDbConfig.enablePubSub is not yet supported by OrbitDbAdapter.connect()', | |
| ); | |
| } |
|
|
||
| // ---------- ProfileDatabase implementation ---------- | ||
|
|
||
| async connect(config: OrbitDbConfig): Promise<void> { |
There was a problem hiding this comment.
OrbitDbAdapter is newly introduced and has substantial behavior (dynamic import error handling, connection lifecycle cleanup, replication subscription, all() coercion). The added tests/unit/profile/orbitdb-adapter.test.ts currently only tests an in-memory mock DB and does not exercise OrbitDbAdapter at all. Add unit tests covering connect() failure modes, put/get/del/all, onReplication() subscription/unsubscribe, and close() idempotency/cleanup (using dynamic import mocks as the file header suggests).
Comment on lines
+71
to
+124
| function createMockCache(): StorageProvider & { _store: Map<string, string>; _trackedAddresses: TrackedAddressEntry[] } { | ||
| const store = new Map<string, string>(); | ||
| let trackedAddresses: TrackedAddressEntry[] = []; | ||
|
|
||
| return { | ||
| id: 'mock-cache', | ||
| name: 'Mock Cache', | ||
| type: 'local' as const, | ||
| description: 'In-memory mock cache', | ||
| async connect() {}, | ||
| async disconnect() {}, | ||
| isConnected() { | ||
| return true; | ||
| }, | ||
| getStatus() { | ||
| return 'connected' as const; | ||
| }, | ||
| setIdentity(_identity: FullIdentity) {}, | ||
| async get(key: string) { | ||
| return store.get(key) ?? null; | ||
| }, | ||
| async set(key: string, value: string) { | ||
| store.set(key, value); | ||
| }, | ||
| async remove(key: string) { | ||
| store.delete(key); | ||
| }, | ||
| async has(key: string) { | ||
| return store.has(key); | ||
| }, | ||
| async keys(prefix?: string) { | ||
| const all = Array.from(store.keys()); | ||
| if (!prefix) return all; | ||
| return all.filter((k) => k.startsWith(prefix)); | ||
| }, | ||
| async clear(prefix?: string) { | ||
| if (!prefix) { | ||
| store.clear(); | ||
| } else { | ||
| for (const k of store.keys()) { | ||
| if (k.startsWith(prefix)) store.delete(k); | ||
| } | ||
| } | ||
| }, | ||
| async saveTrackedAddresses(entries: TrackedAddressEntry[]) { | ||
| trackedAddresses = entries; | ||
| }, | ||
| async loadTrackedAddresses() { | ||
| return trackedAddresses; | ||
| }, | ||
| _store: store, | ||
| _trackedAddresses: trackedAddresses, | ||
| } as StorageProvider & { _store: Map<string, string>; _trackedAddresses: TrackedAddressEntry[] }; | ||
| } |
There was a problem hiding this comment.
_trackedAddresses is initialized as a snapshot and won’t reflect later saveTrackedAddresses() calls (which update the closed-over trackedAddresses variable). This makes test mutations like (cache as any)._trackedAddresses = undefined misleading and can mask cache-miss behaviors. Consider exposing _trackedAddresses via a getter/setter tied to the internal trackedAddresses variable (or drop the _trackedAddresses field entirely and only use saveTrackedAddresses/loadTrackedAddresses).
Comment on lines
+80
to
+88
|
|
||
| // Create IndexedDB provider as the local cache | ||
| const localCache = createIndexedDBStorageProvider(); | ||
|
|
||
| // Build the full ProfileConfig from network defaults + overrides | ||
| const profileConfig: ProfileConfig = { | ||
| orbitDb: { | ||
| privateKey: '', // Set later via setIdentity() | ||
| ...(config.profileConfig?.orbitDb ?? {}), |
There was a problem hiding this comment.
OrbitDbConfig.privateKey is defined as a required field, but the browser factory constructs it with an empty string placeholder. This makes the config shape internally inconsistent and risks connecting to/deriving a DB identity from invalid key material if connect() is called before setIdentity(). A clearer contract would be either: (1) make orbitDb.privateKey optional in ProfileConfig and require setIdentity() prior to connect(), or (2) have connect() explicitly fail fast with a ProfileError when the private key is missing/empty.
Suggested change
| // Create IndexedDB provider as the local cache | |
| const localCache = createIndexedDBStorageProvider(); | |
| // Build the full ProfileConfig from network defaults + overrides | |
| const profileConfig: ProfileConfig = { | |
| orbitDb: { | |
| privateKey: '', // Set later via setIdentity() | |
| ...(config.profileConfig?.orbitDb ?? {}), | |
| const privateKey = config.profileConfig?.orbitDb?.privateKey; | |
| if (typeof privateKey !== 'string' || privateKey.trim() === '') { | |
| throw new Error( | |
| 'Browser Profile providers require a non-empty orbitDb.privateKey in profileConfig before initialization.', | |
| ); | |
| } | |
| // Create IndexedDB provider as the local cache | |
| const localCache = createIndexedDBStorageProvider(); | |
| // Build the full ProfileConfig from network defaults + overrides | |
| const profileConfig: ProfileConfig = { | |
| orbitDb: { | |
| ...(config.profileConfig?.orbitDb ?? {}), | |
| privateKey, |
Comment on lines
+89
to
+92
| // Build the full ProfileConfig from network defaults + overrides | ||
| const profileConfig: ProfileConfig = { | ||
| orbitDb: { | ||
| privateKey: '', // Set later via setIdentity() |
There was a problem hiding this comment.
Same issue as the browser factory: OrbitDbConfig.privateKey is required but initialized to ''. This should be modeled explicitly in the types and/or enforced at runtime (e.g., require setIdentity() and reject connect() until a valid private key is available), otherwise consumers can end up with a deterministic-but-wrong OrbitDB identity/dbName.
Suggested change
| // Build the full ProfileConfig from network defaults + overrides | |
| const profileConfig: ProfileConfig = { | |
| orbitDb: { | |
| privateKey: '', // Set later via setIdentity() | |
| const orbitDbPrivateKey = config.profileConfig?.orbitDb?.privateKey; | |
| if (typeof orbitDbPrivateKey !== 'string' || orbitDbPrivateKey.trim() === '') { | |
| throw new Error( | |
| 'createNodeProfileProviders requires profileConfig.orbitDb.privateKey to be set to a non-empty value before connecting to OrbitDB.', | |
| ); | |
| } | |
| // Build the full ProfileConfig from network defaults + overrides | |
| const profileConfig: ProfileConfig = { | |
| orbitDb: { | |
| privateKey: orbitDbPrivateKey, |
Consolidation engine (profile/consolidation.ts): - 10-step consolidation flow: merge active bundles → pin → supersede - Concurrent guard (5-min TTL pending key) - Crash recovery on startup - Expired bundle cleanup (profile-only, no IPFS unpinning) - Configurable retention period (default 7 days, min 24h) OrbitDB adapter fixes (found by live integration test): - Fix @noble/curves import: secp256k1 → secp256k1.js (v2.x ESM) - Fix @noble/hashes import: sha256 → sha2.js (v2.x rename) - Add @chainsafe/libp2p-gossipsub for OrbitDB v3 Sync - Fix all() to handle OrbitDB v3 array return format Live integration test (tests/integration/orbitdb-adapter.test.ts): - 17 tests with real OrbitDB v3 + Helia v6 - put/get/del/all round-trip, prefix filtering, replication - Validates the adapter works with actual OrbitDB (not mocks) Doc updates: - SPECIFICATION.md: Appendix E (Multi-Bundle Protocol) - ARCHITECTURE.md: Section 9 (Profile Module Integration)
Contributor
Author
Update: Follow-up items completedAdded since PR creation:
Updated stats:
Remaining follow-up (separate PRs):
|
…-format # Conflicts: # index.ts # package-lock.json
Per PROFILE-ARCHITECTURE.md Q2 decision: element payloads pinned to IPFS are now unencrypted. Confidentiality is provided by the OrbitDB KV layer, which continues to encrypt bundle refs and operational state with a per-wallet key derived from HKDF. Rationale: identical token pools produced by different wallets must hash to the same CID so that cross-user content-addressed dedup works. If two devices encrypt the same UXF bundle with different keys, dedup breaks. Changes: - profile/profile-token-storage-provider.ts: remove encrypt/decrypt on CAR bytes in flushToIpfs() and load(); update docstrings. - profile/consolidation.ts: remove encrypt/decrypt on CAR bytes in the consolidate() fetch/merge/pin loop; update flow docstring. - tests: update fixtures to feed unencrypted CARs to the IPFS mock; rewrite "CAR files are encrypted before pinning" as "CAR files are pinned unencrypted (content-addressed dedup)" with an inverted assertion. All 378 UXF+Profile tests pass. tsc --noEmit passes.
…/history
Per user direction (Q1 decision refinement): _tombstones, _sent, and
_history are per-device local state, never replicated between Sphere
instances sharing the wallet. Rationale: sync errors or rogue instances
would otherwise poison these authoritative ledgers everywhere at once.
The token pool itself remains the source of truth.
Architecture:
- Synced to OrbitDB (authoritative across instances):
outbox, invalid, mintOutbox, invalidatedNametags
- Local cache only (per-device, never replicated):
tombstones, sent, history
- Cached in the injected StorageProvider under keys
deriver.{addressId}.{tombstones|sent|history}
Changes:
- New profile/deriver.ts: pure functions to rebuild derived caches from
archived tokens in a TxfStorageDataBase (best-effort fallback for
fresh-install / cache-corruption recovery). Oracle-based spent-checks
left as a future enhancement layered on manifest derivation (task #27).
- ProfileTokenStorageProvider: accept optional localCache: StorageProvider.
Split writeOperationalState → writeOrbitOperationalState +
writeLocalDerivedCache; matching split for reads. On load, rebuild from
pool if local cache is empty and archived tokens are available.
- Factory wires cacheStorage through as the local cache.
Limitations noted in deriver.ts docstring:
UXF round-trip in Profile storage currently strips the 'archived-'
prefix (tokens become _<hexTokenId>); rebuild-from-pool therefore
yields empty when the data has already round-tripped. PaymentsModule
continues to write through to the cache imperatively, so steady-state
is correct; only fresh-install recovery is limited until task #27
adds predicate-based status derivation.
Tests:
- New tests/unit/profile/deriver.test.ts (14 tests) — pure-function
coverage for tombstones/sent/history derivation.
- Updated profile-token-storage-provider tests:
* tombstones/sent no longer expected in OrbitDB on save
* derived cache fields asserted in the mock local store
* readOperationalState test seeds both OrbitDB (outbox) and
local cache (tombstones/sent)
* new test: full save→local-cache round-trip for derived keys
* new test: load() triggers rebuild fallback when cache is empty
Full suite: 394 UXF+Profile tests pass; 199 PaymentsModule tests pass;
tsc --noEmit clean.
PROFILE-ARCHITECTURE §10.4 specifies JOIN as "longest valid chain per tokenId, sibling-preserve on divergence". This is already done structurally by UxfPackage.merge() (which calls mergeInstanceChains per Decision 6 in uxf/instance-chain.ts). This commit: - Replaces the terse merge comment with an explicit JOIN contract at the call site in ProfileTokenStorageProvider.load(), including a forward reference to task #27 for oracle-based conflict resolution. - Adds a unit test proving that same-id tokens from two bundles plus bundle-unique tokens all survive the merged load — no dropped state. Oracle-based status flagging (valid/conflicting/invalid) is the wallet-layer token manifest derivation, scoped to task #27.
Per PROFILE-ARCHITECTURE.md §10.2.2 / §10.11, the wallet-level *token
manifest* is a derived map from tokenId → { rootHash, status,
conflictingHeads?, invalidReason? }. This commit ships the structural
half of that derivation:
- **status: 'valid'** when a single chain head is reachable
- **status: 'conflicting'** when the instance-chain index holds sibling
heads sharing a common ancestor (the
divergence-preserving output of JOIN)
- **status: 'pending' / 'invalid'** reserved for the oracle-aware layer
The structural pass is pure and synchronous — no network calls. The
oracle-aware pass will post-process the output with aggregator
non-inclusion checks (see §10.6 Steps 3-4) and is a follow-up task.
Files:
- profile/token-manifest.ts — pure deriver + types + conflict helper
- profile/index.ts — public exports
- profile/profile-token-storage-provider.ts
· invokes the deriver after JOIN during load()
· caches result; new public method getTokenManifest()
· wrapped in try/catch so deriver bugs cannot sink a storage load
- docs/uxf/PROFILE-ARCHITECTURE.md
· renamed conflictingRoots → conflictingHeads throughout (matches
implementation semantics: these are chain heads, not genesis
roots)
Tests (10 new):
- tests/unit/profile/token-manifest.test.ts (6) — hand-built chain
index fixtures, covers single/sibling/multi-sibling cases
- tests/unit/profile/token-manifest-integration.test.ts (3) — against
the real UxfPackage with real token fixtures
Full suite: 404 UXF+Profile tests pass; tsc --noEmit clean.
…he hardening
Fixes findings from /steelman adversarial review of the session's 4 commits.
## Critical
**1. CID content-address verification on fetched CAR bytes**
`profile/ipfs-client.ts` now recomputes sha256(bytes) and compares
against the multihash digest in the requested CID before returning.
Without CAR encryption, this is the sole authentication boundary
against a malicious or compromised IPFS gateway. Exported as
`verifyCidMatchesBytes()` for reuse.
Applies transitively to `consolidate()` (which also uses fetchFromIpfs).
sha2-256 is the only multihash code accepted; anything else is
rejected as unsupported.
**2. Manifest JOIN: cross-token hash-collision over-match**
`profile/token-manifest.ts collectHeads()` previously treated ANY
hash overlap as a sibling signal, so two different tokens sharing a
common mid-chain element (e.g. a shared predicate) could be flagged
as each other's conflicts. Post-fix, relatedness is tail-anchored:
only chains whose tail hash equals the primary chain's tail are
considered siblings.
## Warnings
**3. Rebuild-race dedup**
Concurrent load() calls that both hit the empty-cache path no longer
race each other. A singleton `rebuildPromise` deduplicates — second
caller awaits the first's promise.
**4. Partial-write atomicity**
`writeLocalDerivedCache` now persists `{tombstones, sent, history}`
into a single atomic key `deriver.{addressId}.all` instead of three
separate writes. Eliminates the "one field written, disk full before
the next" window. The read path prefers the atomic key but falls
back to the legacy per-key layout so pre-atomic caches keep working
until the next write heals them.
**5. lastPinnedCid invalidation on overwrite**
`save()` now clears `lastPinnedCid` whenever `pendingData` is
replaced with different data. Prevents a retry after a partially
failed flush from re-registering an OLD CID with NEW token data.
**6. Surface silent swallows via storage:error events**
`writeLocalDerivedCache` and `readLocalJson` now emit
`storage:error` events in addition to logging, and
`writeLocalDerivedCache` returns a boolean so callers can react.
## Tests (8 new, 412 total)
- `tests/unit/profile/ipfs-client-cid-verify.test.ts` (7 tests)
· verifyCidMatchesBytes accepts match
· rejects tampered / truncated bytes
· rejects unparseable CIDs
· rejects non-sha256 multihashes
· fetchFromIpfs end-to-end verifies mismatch / accepts match
- `tests/unit/profile/token-manifest.test.ts` — new regression test:
· "does NOT mis-attribute sibling heads across tokens that share
a mid-chain hash" — asserts the tail-anchor fix.
All prior test fixtures updated to use real content-addressed CIDs
(`cidForBytes(bytes)` helper) so the CID verification path is exercised
on every fetch call. The new single-key `deriver.{addressId}.all`
layout replaces the three per-key reads/writes in fixtures.
Full suite: 412 UXF+Profile tests pass; 199 PaymentsModule tests pass;
tsc --noEmit clean.
…n invalidation, cache hardening Round-2 steelman on commit 21176ce surfaced 2 CRITICAL findings that the first round had introduced while fixing the previous round. Recursion pays. ## Critical **1. Multi-gateway fallback broken for CID-mismatch (regression from 21176ce)** `fetchFromIpfs` wrapped the whole gateway-body in `catch (err) { if (err instanceof ProfileError) throw err; }`. The newly-added `verifyCidMatchesBytes` also throws ProfileError on mismatch, so a single tampering gateway would hard-abort the request instead of falling through to the next gateway. That turned a defense into a DoS vector. Fix: split verification into its own try/catch. CID mismatch → record as lastError, continue to next gateway. Size-limit errors remain fatal (they are per-request, not per-gateway). New test proves first-gateway-tampered → second-gateway-good succeeds. **2. lastPinnedCid reference-identity check was insufficient** The previous fix cleared `lastPinnedCid` only when `pendingData !== data`. But `flushToIpfs` on failure re-queues the SAME reference, so a caller that mutates in place and re-saves satisfies `pendingData === data`, skips the clear, and the next flush re-registers the stale CID against mutated content. Fix: any new `save()` unconditionally clears `lastPinnedCid`. The one-extra-pin cost on legitimate retries is trivial; correctness of "the pinned CID always matches the currently flushed bytes" is worth it. Test renamed + updated to assert the new invariant. ## Warnings **3. load() during in-flight flush returned stale snapshot** Between `this.pendingData = null` and the OrbitDB bundle-ref write, `load()` would read the pre-flush state and miss the in-flight bundle. Fixed by awaiting `flushPromise` before reading OrbitDB. **4. Legacy per-key entries never cleaned after atomic migration** On the first successful atomic-key write, best-effort delete the three legacy keys (`deriver.{addr}.tombstones|sent|history`). Guarded by a `legacyKeysCleaned` flag so subsequent saves don't repeat the cleanup. **5. storage:error event flood on cache corruption** `readLocalDerivedCache` could emit up to 4 events per call (1 atomic + 3 legacy fallback) when the whole cache was corrupt. Rewritten to collect failed keys locally and emit at most one aggregate event per call. **6. rebuildPromise result shared by reference across awaiters** Two concurrent load() callers would receive the same `{tombstones, sent, history}` object — if one mutated an array, the other saw the mutation. Fixed by shallow-cloning per awaiter. **7. writeLocalDerivedCache return value was ignored** The boolean existed but no caller consumed it. `flushToIpfs` now logs a degraded-state warning on false so reliability telemetry captures it; `rebuildDerivedCacheInner` still awaits unconditionally. ## Tests (2 new, 414 total) - ipfs-client-cid-verify.test.ts: · "multi-gateway fallback tries the next gateway when the first returns tampered bytes" — directly proves critical #1 fix · "fails with a clear error when ALL gateways return tampered bytes" All prior tests still pass; one was renamed and inverted to assert the new `lastPinnedCid` invariant ("save() after OrbitDB-write failure re-pins"). tsc --noEmit clean.
Send/receive between wallets uses TXF objects as the wire format
(PaymentsModule.ts:1247 et al — outgoing payload is
\`JSON.stringify(sdkToken.toJSON())\`, incoming parser feeds the same
JSON into \`SdkToken.fromJSON\`). The UXF storage layer deconstructs
tokens into content-addressed elements and reassembles on load; for
compatibility with TXF-only peers (older Sphere, third-party wallets)
every wire-format field MUST survive that round-trip.
This suite adds the explicit compat gate so any future change that
silently drops or alters a TXF field fails here instead of at a user's
wallet 3 months later.
Coverage (8 tests, all passing):
- TOKEN_A/B/C ingest → assemble preserves:
genesis.data {tokenId, tokenType, coinData, tokenData, salt,
recipient, recipientDataHash, reason}
genesis.inclusionProof {authenticator, merkleTreePath,
transactionHash, unicityCertificate}
state {predicate, data}
every transaction's {sourceState, destinationState, data,
inclusionProof, recipient}
- NAMETAG_ALICE: tokenData (the hex-encoded name) survives → nametag
resolution keeps working after bundles go through UxfPackage.
- Full CAR round-trip (ingest → toCar → fromCar → assemble): this is
what ProfileTokenStorageProvider executes; every field above is
re-checked end-to-end.
- Rebuilt token is JSON.stringify-able and reparses to the same shape
— this is exactly what the send path feeds into sendTokenTransfer.
- tokenToTxf / txfToToken helpers (used by legacy send paths and
IndexedDB/File storage) continue to round-trip wire-compatible data.
Verification:
- 8 new tests pass
- PaymentsModule suite: 199 passing (unchanged)
- Serialization suite: 106 passing (unchanged)
Net conclusion: the UXF storage layer does not alter the wire format.
Profile-backed wallets can still send to and receive from TXF-only
peers, and can still use the legacy TXF serialization helpers.
Adds explicit CLI commands for offline token file transfer / backup.
The underlying building blocks (UxfPackage.toCar/fromCar, tokenToTxf /
txfToToken) already existed — this commit wires them up as first-class
PaymentsModule helpers and CLI commands.
## Commands
**tokens-export <file> [options]**
- Format auto-detected from extension:
.uxf or .car → UXF CAR (content-addressable, new)
.txf or .json → TXF JSON array (legacy, backward-compatible)
- --format uxf|txf|json Override detection
- --coin <symbol> Filter by coin (accepts symbol or coinId hex)
- --ids <id1,id2,...> Filter by local token IDs
- --include-unconfirmed Include provisional tokens
**tokens-import <file> [--format auto|uxf|txf|json]**
- Auto-detect by content (JSON starts with '[' or '{'; CAR starts
with a varint byte not in the printable range)
- Reports: added / skipped / rejected with per-token reasons
## SDK surface
Two new PaymentsModule methods:
exportTokens(options?): Array<{ localId, genesisTokenId, txf }>
importTokens(txfTokens): { added, skipped, rejected }
- exportTokens filters by ids / coinId / includeUnconfirmed, extracts
the TxfToken from each Token's sdkData via the existing tokenToTxf
helper, and returns triples.
- importTokens delegates dedup to addToken() (which enforces the
tombstone + (tokenId, stateHash) guard): already-owned tokens are
skipped, previously-spent tokens are skipped, malformed inputs are
rejected with a reason. Each import assigns a fresh local UUID so
imported tokens don't collide with existing wallet IDs.
## Tests (12 new, 3000 total passing)
tests/unit/modules/PaymentsModule.importExport.test.ts:
- exportTokens: empty wallet, default (confirmed only), coinId filter,
ids filter, includeUnconfirmed, TXF field preservation.
- importTokens: adds all, idempotence (re-import skipped), tombstone
rejection (imported token that was previously spent is skipped),
malformed input rejection (missing tokenId, missing state),
export→import round-trip between two wallets.
CLI smoke tests verified:
- `tokens-export` with no args prints usage
- `tokens-import` with no args prints usage
- `help` lists the new commands under "FILE I/O"
Full unit suite: 3000/3000 passing. tsc --noEmit clean.
New wallets initialised through the CLI now use the OrbitDB-backed
Profile storage by default — content-addressed UXF element pool on IPFS
plus encrypted OrbitDB KV for operational state. Existing wallets with
file-based artefacts on disk are detected and continue using legacy.
## Behaviour
Resolution logic (runs once on first CLI invocation per dataDir,
result cached in config.storageMode):
1. Explicit --legacy / --profile on \`init\` wins.
2. config.storageMode already set → honour it (mode is locked per
wallet; migration is a separate workflow).
3. Legacy wallet.json present in dataDir → legacy (upgrade path for
pre-setting users).
4. Fresh wallet + @orbitdb/core + helia importable → profile.
5. Profile deps missing → fall back to legacy with a one-line note.
Once resolved, the mode is persisted to .sphere-cli/config.json so all
subsequent commands stay consistent. \`clear\` resets it so a fresh
re-init can pick again.
## CLI surface
**init** gains two mutually-exclusive flags:
--legacy Force file-based JSON wallet + IPNS sync (pre-UXF format)
--profile Force OrbitDB Profile (errors out if deps missing)
Re-running \`init\` on a dataDir whose committed mode conflicts with an
explicit flag exits with a clear error — no silent clobbering.
**status** now prints \`Storage: profile|legacy|(auto-detect)\` so users
can see which backend their wallet is on.
**clear** tears down the correct backend based on the stored mode
(Profile adapter for profile wallets, file providers for legacy) and
then drops config.storageMode so the next init is fresh.
## Wiring
getSphere() branches on the resolved mode:
- profile → createNodeProfileProviders (storage + tokenStorage) +
createNodeProviders for transport/oracle/market/groupChat/
L1/price (with ipfs tokenSync disabled — OrbitDB
replicates state).
- legacy → createNodeProviders as before.
## Tests
Full unit suite: 3000/3000 passing. Typecheck clean.
Mode resolution is tested indirectly through existing wallet-lifecycle
flows. A dedicated helper-level test would require mocking fs + dynamic
imports; deferred since the logic is small and well-commented.
New Profile (OrbitDB) storage mode is now the CLI default; the runtime
libraries it needs must be installed by default, not relegated to
optional/peer. This commit migrates them to direct deps so \`npm install\`
is enough to get Profile working.
Moved to dependencies:
- @orbitdb/core (OrbitDB KV database)
- helia (IPFS node runtime)
- @libp2p/bootstrap (peer discovery; used by orbitdb-adapter)
- @libp2p/crypto (IPNS key derivation — legacy IPFS sync)
- @libp2p/peer-id (peer identity)
- ipns (IPNS record management)
- multiformats (CID parsing / sha256-hashed content address)
Retired peer/optional declarations; only \`ws\` remains as an optional
peer for the Node transport.
Verification:
- npm install clean
- node -e "require('@orbitdb/core'); require('helia')" → OK
- tsc --noEmit clean
- Full unit suite: 3000/3000 passing
Two new test files covering the CLI storage-mode machinery and the
legacy↔profile storage equivalence at the PaymentsModule level.
## cli/storage-mode.ts (extracted)
Moved the storage-mode resolver out of cli/index.ts into its own
module with dependency injection so it is unit-testable without
touching the real filesystem or node_modules. The CLI now imports
\`resolveStorageMode\` + the default probes from this module; behaviour
is unchanged. New option: \`onExplicitProfileMissing: 'exit' | 'throw'\`
lets tests exercise the failure path without calling process.exit().
## tests/unit/cli/storage-mode.test.ts (9 tests)
Covers every branch of the resolution state machine:
- explicit --profile / --legacy wins and is persisted
- deps-missing --profile throws (when onExplicitProfileMissing=throw)
- no-op persist when explicit matches existing config
- committed config.storageMode is honoured without probing
- existing legacy wallet.json on disk → legacy
- pristine + deps available → profile (new default)
- pristine + deps missing → legacy + notify
- repeat resolutions: persist is not double-called
## tests/unit/modules/PaymentsModule.dual-mode.test.ts (11 tests)
Parameterised describe.each over two in-memory TokenStorageProvider
implementations:
- legacy-style (TxfStorageDataBase saved verbatim, like FileTokenStorage)
- profile-style (real UxfPackage.toCar / fromCar round-trip, like
ProfileTokenStorageProvider but without OrbitDB/IPFS I/O)
For each backend:
- addToken persists and getTokens sees them
- exportTokens emits wire-compatible TXF preserving every field
- import → export round-trip on a fresh wallet
- tombstone-on-remove blocks re-import
Plus 3 cross-mode tests proving send/receive TXF wire compat:
- legacy wallet exports → profile wallet accepts (and owns)
- profile wallet exports → legacy wallet accepts (and owns)
- UXF CAR round-trip preserves every wire-format field exactly
This is the explicit coverage gate: any change that breaks storage
backend swappability or wire-format preservation will fail here.
Verification:
- Full unit suite: 3020/3020 passing (+20 new)
- No stderr noise on the profile UXF round-trip path
- tsc --noEmit clean
## E2E CLI (tests/e2e/cli-storage-modes.sh)
New bash script covering the CLI storage-mode machinery end-to-end.
Seven subtests, gated sensibly between network-free and testnet-requiring:
Network-free (run unconditionally):
1. \`help init\` documents --legacy + --profile flags
2. \`config\` on a fake-legacy dataDir shows no storageMode before
the resolver runs
Network-gated (require E2E_NETWORK=1 + testnet access):
3. \`init --legacy\` persists storageMode=legacy, creates wallet.json,
status prints "Storage: legacy"
4. \`init --profile\` persists storageMode=profile, creates orbitdb/
directory, status prints "Storage: profile"
5. Re-init with mismatched mode is rejected with a clear error
6. \`clear\` resets storageMode for next init
7. tokens-export works in both modes
Pass count is printed as 2/2 in the default environment (network-free).
## Docs
docs/QUICKSTART-CLI.md:
- New "Storage Mode (profile vs legacy)" subsection under
"First-Time Wallet Initialization" explaining defaults,
detection rules, and the --legacy/--profile flags
- "Where Data is Stored" now shows both legacy and profile layouts
- "Show Wallet Status" example updated with the new "Storage:" line
- "Delete Wallet Data" clarifies clear is mode-aware and resets
storageMode
- New "Offline Token Transfer (export / import to file)" section
covering tokens-export / tokens-import with both formats
docs/API.md:
- PaymentsModule: new entries for exportTokens() / importTokens()
with full option/return types and semantics
CLAUDE.md:
- Dependencies section lists the newly direct OrbitDB/Helia deps
so contributors know Profile mode is built-in, not optional
## Verification
- 3020/3020 unit tests passing (one occasional flaky AccountingModule
test unrelated to this change)
- tsc --noEmit clean
- bash tests/e2e/cli-storage-modes.sh: 2/2 network-free passes
- \`completions bash\` output includes tokens-export, tokens-import,
--legacy, --profile, --format, --coin, --ids, --all,
--include-unconfirmed
…afety
Round-3 steelman flagged a silent-corruption path in the storage-mode
resolver plus several practical issues in the import command.
## Critical
**1. Profile vs legacy detection collision (silent data corruption)**
ProfileTokenStorageProvider uses a FileStorageProvider for its local
cache, which writes \`{dataDir}/wallet.json\` — exactly the file
legacyWalletProbe inspects. So a Profile-populated dir would falsely
match the legacy probe whenever config.storageMode was absent
(corrupt config, hand-edit, post-clear crash). Combined with #2
this could let \`init --legacy\` overwrite encrypted Profile data
with a fresh legacy wallet.
Fix: introduce \`defaultProfileWalletProbe\` that looks for the
Profile-specific \`{dataDir}/orbitdb/\` directory. Disk-state
detection now disambiguates (Profile takes precedence) and the
explicit-mode branch refuses to clobber the wrong shape.
**2. Non-atomic clear leaves stale storageMode**
clear() previously deleted wallet data BEFORE unsetting
config.storageMode. A crash between the two left the config pointing
at gone data.
Fix: reset config.storageMode FIRST (a single-key write is atomic at
the FS level), THEN delete the data. Worst case after crash: the
config has no committed mode, the next init re-detects from disk
(or defaults to profile on a pristine dir).
**3. Mismatch check bypass when config.storageMode absent**
The init handler's pre-existing mismatch check fires only when
config.storageMode is recorded. Combined with #1, hand-deleting
config but keeping disk artefacts let \`init --legacy\` succeed
against a Profile dir.
Fix: the resolver itself now refuses to commit a mode that
contradicts disk-state, regardless of config. Tested with two
regression cases.
**4. Probe accepts the wrong version of @orbitdb/core / helia**
Module-load probe succeeded if the package was installed at all,
even with an incompatible major. Runtime then failed with a cryptic
\`Could not resolve createOrbitDB\`.
Fix: after loading, verify the named exports (createOrbitDB,
createHelia) exist. Returns a clear "incompatible version installed"
reason on mismatch.
## High
**5. tokens-import has no size guard**
fs.readFileSync on a malicious / mistyped path could OOM Node.
Fix: stat the file first, refuse > 100MB.
**6. tokens-import returns exit 0 on partial failure**
Rejected entries were reported but ignored by scripts/CI.
Fix: set process.exitCode = 2 when result.rejected.length > 0.
**7. Empty-file import gave a confusing "Failed to parse UXF CAR"**
Fix: check bytes.length === 0 explicitly and emit a clear error.
## Warnings
**8. Corrupt config.storageMode value not validated**
Hand-edited "gibberish" was returned verbatim, then mismatched
downstream string equality.
Fix: validate against the known enum and fall through to
auto-detection with a notify().
**9. Dead try/catch around resolveCoin() in tokens-export**
resolveCoin actually exits, never throws, making the catch
unreachable. The misleading "fall back to literal coinId" comment
contradicted reality.
Fix: explicit hex64 detection upfront, otherwise let resolveCoin's
own error path fire.
**10. Unused @libp2p/interface devDep**
Zero in-tree references. Dropped.
## Tests (+4 new, 3024/3024 passing)
tests/unit/cli/storage-mode.test.ts gains four regression cases:
- --legacy refused when orbitdb/ exists on disk
- --profile refused when only wallet.json (no orbitdb/) on disk
- disk auto-detect: orbitdb/ wins over wallet.json
- corrupt config.storageMode value falls through to detection
Verification:
- tsc --noEmit clean
- 3024/3024 unit tests passing
- bash tests/e2e/cli-storage-modes.sh: 2/2 network-free
Adds the user-facing migration path from a legacy (file-based) Sphere
CLI wallet to the new Profile (OrbitDB) backend. Conceptually identical
to "import a TXF wallet file" — the source happens to be a live legacy
TokenStorageProvider instead of a file.
## Semantics (per user direction)
- Explicit: invoked via the CLI command, never auto-run.
- Non-destructive: legacy storage is preserved by default. Pass
--delete-legacy to remove it after a successful zero-rejection
import.
- Re-runnable: every run produces the joint inventory of legacy +
Profile, with addToken's tombstone + (tokenId, stateHash) dedup
gating duplicates.
- Identity-verified: refuses to migrate when legacy and Profile
do not share the same encrypted-mnemonic blob. --no-verify
overrides for advanced cases (same mnemonic, different password
or salt).
- Token statuses recalculated automatically by the Profile load
path's structural manifest deriver and local cache deriver.
## SDK surface
profile/import-from-legacy.ts — \`importLegacyTokens(legacyStorage,
targetPayments, options)\` returns \`{ success, tokensFound,
tokensAdded, tokensSkipped, tokensRejected, rejections, durationMs }\`.
profile/index.ts:
- exports \`importLegacyTokens\` + types
- marks the existing \`ProfileMigration\` class as @deprecated
(kept for backwards compat — new code should use the import API)
profile/migration.ts:
- JSDoc @deprecated tag on the ProfileMigration class with a
code example pointing at the new path
## CLI command
\`migrate-to-profile --legacy-dir <path> [--legacy-tokens <path>]
[--dry-run] [--delete-legacy] [--no-verify]\`
- Refuses if current dataDir is in legacy mode.
- Identity check via encrypted-mnemonic blob comparison.
- Calls \`importLegacyTokens\` with the legacy file storage providers.
- Reports counts; sets \`process.exitCode = 2\` on partial failure.
- Cleanup only on \`--delete-legacy\` AND zero rejections AND not
a dry-run.
Help registry, autocompletion, and \`printUsage\` updated.
## Docs
- QUICKSTART-CLI.md: new "Migrate a Legacy Wallet to Profile" section
with the recommended workflow, properties list, and sample output.
- PROFILE-ARCHITECTURE.md §7.6: prepended a clear note about the
new import-based model; moved the legacy 6-step flow under a
"deprecated" subsection.
## Tests (+7 new, 3031/3031 passing)
tests/unit/profile/import-from-legacy.test.ts:
- empty source → success with zero counts
- extracts active / archived / forked tokens, skips operational keys
- read-only contract: only load() called on legacy provider
- idempotence: re-run with same source yields zero added
- joint inventory: legacy adds on top of pre-existing Profile tokens
- dry-run: reports counts without mutating target
- load failure → success=false, no exception thrown
tsc --noEmit clean. CLI smoke (\`help migrate-to-profile\`, missing-args
usage, completions bash) verified.
…suse, same-dir guard
Round-4 steelman flagged two ship-blocking issues plus several supporting
correctness gaps in the migration feature.
## Critical (fixed)
**1. \`Sphere.clear\` destroys the running Profile sphere.**
\`Sphere.clear({ storage: legacyStorage, ... })\` unconditionally
awaits \`Sphere.instance.destroy()\` (core/Sphere.ts:1179-1183) — but
the only \`Sphere.instance\` in our process IS the current Profile
sphere we're migrating into. The trailing \`syncAfterWrite(sphere)\`
+ \`closeSphere()\` would then run against a destroyed instance,
potentially losing the just-imported tokens.
Fix: replace \`Sphere.clear\` with direct \`legacyTokenStorage.clear()\`
+ \`legacyStorage.clear()\` calls. These wipe ONLY the legacy data,
leaving the running Profile untouched.
**2. State regression via addToken on legacy import.**
PaymentsModule.addToken's CASE 2 (line 3893) silently ARCHIVES the
current state of any token whose genesis tokenId already exists in
the wallet, then installs the incoming state — regardless of which
is newer. A legacy import carrying an OLDER state of a token Profile
has already moved would silently regress Profile's state.
Fix: extend \`importTokens\` with \`{ skipExistingGenesis: true }\`
option. importLegacyTokens always passes it. Skip-with-clear-reason
whenever a genesis tokenId already exists, regardless of stateHash.
The wallet's authoritative current state is preserved.
This also fixes the related "fork de-forking" path: \`_forked_*\`
entries (which by definition share a genesis with active tokens)
are now skipped via the same gate.
## High (fixed)
**3. Forked entries still passed through importTokens.**
Fix: \`extractTxfTokensFromStorageData\` now routes \`_forked_*\`
entries to a separate counter (\`forksSkipped\`) and excludes them
from the import set. CLI surfaces the count and refuses
\`--delete-legacy\` when forks were present (manual handling).
**4. Same-dir migration could wipe the current Profile.**
\`--legacy-dir\` was passed verbatim to FileStorageProvider; if it
resolved to the current Profile dataDir, a successful
\`--delete-legacy\` would remove the Profile's wallet.json + tokens.
Fix: \`path.resolve\` both legacy paths upfront and refuse when
either equals the current Profile's dataDir or tokensDir.
**5. \`process.exit(1)\` bypassed cleanup \`finally\`.**
Seven exit sites inside the try block leaked legacy file handles.
Fix: route every error path through local \`earlyExitCode\` /
\`earlyExitReason\` variables; exit() only after the finally has
run disconnect/shutdown.
**6. No Profile-flush before legacy cleanup.**
Cleanup ran immediately after \`importTokens\` returned, but the
Profile's write-behind buffer might not have flushed to OrbitDB +
IPFS yet. A crash in this window would lose both copies.
Fix: \`await sphere.payments.waitForPendingOperations(); await
sphere.payments.sync()\` BEFORE the legacy clear.
## Warnings (fixed)
**7. Path joining bug on Windows / multiple trailing slashes.**
\`legacyDir.replace(/\\/$/, '') + '/tokens'\` is wrong for backslashes.
Fix: \`path.join(legacyDir, 'tokens')\`.
**8. Silent rejection truncation.**
\`rejections.slice(0, 100)\` lost rejection reasons silently.
Fix: new \`rejectionsTruncated: boolean\` field; CLI prints
"(... N more rejection reasons truncated)".
## Tests (+2 new, 3033/3033 passing)
tests/unit/profile/import-from-legacy.test.ts:
- "strict mode: pre-existing genesis tokenId is preserved (no state
regression)" — directly proves the addToken-CASE-2 gap is plugged.
- "rejectionsTruncated flag is set when more than 100 rejections" —
documents the truncation contract.
- Updated "extracts active and archived tokens; counts forks
separately" — reflects the new fork-skip behaviour.
- All result-shape assertions extended with the two new fields.
PaymentsModule.importTokens gains \`{ skipExistingGenesis }\` option
documented in the JSDoc with the rationale.
Verification: tsc --noEmit clean; full suite 3033/3033.
Closes the two previously-deferred items from steelman round 4.
## proper-lockfile on the legacy dataDir during migration
Adds \`proper-lockfile\` as a runtime dep and uses it in the CLI's
\`migrate-to-profile\` command to prevent concurrent CLI processes (or
any cooperating tool) from mutating the legacy source under us
mid-import.
- Lock target: \`{legacyDir}/wallet.json\` (the canonical legacy
artifact); falls back to the dir if no wallet.json exists, in
which case the identity-blob check below rejects the migration.
- Retries: up to 10x with exponential-ish backoff (200ms→2s).
- Stale lock release: 60s — recovers from killed processes that
didn't clean up.
- Released in the finally AFTER all storage providers have been
disconnected, so the next process sees a fully closed source.
Acquire failure prints a clear "another sphere-cli is running on this
dataDir" hint with instructions for handling stuck locks.
## Granular skip codes in importTokens
\`PaymentsModule.importTokens\` now does pre-checks BEFORE delegating
to addToken so each per-token outcome carries a stable enum code
suitable for switch-on-code logic in CLI / scripting / UI:
Added:
- 'added' : token was new
- 'state-replaced': lenient mode — prior state archived, new state
now authoritative
Skipped:
- 'duplicate' : exact (tokenId, stateHash) match already owned
- 'tombstoned' : (tokenId, stateHash) was previously spent
- 'genesis-exists': strict mode — tokenId owned at a different state
- 'unknown' : addToken returned false despite the pre-checks
Rejected:
- 'malformed' : missing tokenId / state / genesis / etc.
- 'add-failed' : addToken threw
New types exported from PaymentsModule:
- ImportAddedCode / ImportSkipCode / ImportRejectCode
- ImportAdded / ImportSkipped / ImportRejected
- ImportTokensResult
The legacy importLegacyTokens helper aggregates skipped entries into
\`skippedByCode\` for at-a-glance diagnostics.
The CLI's \`tokens-import\` and \`migrate-to-profile\` commands now print
per-code breakdowns instead of opaque "skipped: N (already owned /
tombstoned)" text.
## Tests (+3 new, 3036/3036 passing)
- "reports duplicate skip code for an exact (tokenId, stateHash) match"
- "reports genesis-exists skip code in strict mode for a different state"
- "lenient mode: same-genesis import marks added entry as state-replaced"
Existing tombstone-skip test extended to assert \`code === 'tombstoned'\`.
All previously-existing assertions remain backward-compatible since
\`reason\` and \`genesisTokenId\` fields preserved.
tsc --noEmit clean. CLI smoke verified.
…d ImportAdded
Round-5 steelman returned WARNINGS (no CRITICAL). Addressing the
high/medium findings and leaving the low/docs-only items deferred.
## High (fixed)
**Lock coverage bypassed by --legacy-tokens override.**
The lock was only on \`{legacyDir}/wallet.json\`. If the user pointed
\`--legacy-tokens\` at a DIFFERENT directory, the real data path went
unprotected. Fixed: when \`legacyTokensDir\` is outside \`legacyDir\`,
acquire a second lock at that path as well; both are released
together in the finally.
**Lock path explicitly inside the legacy dataDir.**
Previously proper-lockfile created \`<target>.lock\` as a sibling of
\`wallet.json\` — which for \`legacyDir = /etc/sphere\` would try to
write \`/etc/sphere/wallet.json.lock\` in a potentially unwritable
parent. Fixed: \`lockfilePath: path.join(legacyDir, '.migrate.lock')\`
(and symmetric for the tokens lock). Lock lives inside the dataDir
where we know the user has write access.
**fs.existsSync check moved out of the lock try.**
Nonexistent dataDirs now produce "Legacy dataDir does not exist"
directly instead of the opaque "Could not acquire lock" from
proper-lockfile.
## Medium (fixed)
**state-replaced semantic smear.**
\`addToken\`'s CASE 1 (pre-existing token was 'spent' or 'invalid')
returns true via the same code path as CASE 2 (live state replaced).
Labeling both as 'state-replaced' misled UIs. Split into two codes:
- 'state-replaced' : overwrote a LIVE (confirmed/submitted)
state of the same tokenId
- 'stale-record-replaced' : overwrote a dead record (spent/invalid);
no user-visible state was lost
The pre-check scan now captures the matched entry's status so the
outcome code can be chosen correctly.
**ImportAdded is a discriminated union.**
\`note\` was optional but only set for 'state-replaced'. Consumers
doing \`switch(e.code)\` had to use \`!\` assertions. Refactored
ImportAdded into a union:
- { code: 'added'; ... }
- { code: 'state-replaced' | 'stale-record-replaced'; note: string }
Now \`note\` is structurally required where it applies, and absent
where it doesn't.
**'unknown' skip now logs at warn level.**
This bucket only fires when the pre-check pattern disagrees with
addToken's own guards — either a TOCTOU race against a Nostr-
delivered transfer or a new guard pattern we didn't enumerate. Log
it so operators can correlate with transport activity.
**Exhaustiveness check in skippedByCode aggregation.**
If a future addition to the union introduces a new ImportSkipCode,
the aggregator in importLegacyTokens now falls through to 'unknown'
instead of silently dropping the count.
## Low (fixed)
**Dead \`as string\` cast removed** on the proper-lockfile dynamic
import — it was a holdover from optional-peer-dep days; proper-lockfile
is now a direct runtime dep.
**Release-failure warning includes lockfile path** so the user can
\`rm\` it if cleanup silently fails.
**Lock acquire error message documents SIGKILL recovery window** and
the exact lockfile path to remove manually.
**Migrate-to-profile comment clarifies advisory-only semantics.**
The lock protects against concurrent migrate-to-profile invocations
but NOT against concurrent \`send\`/\`receive\` on the legacy wallet
(those don't acquire it). Documented inline.
## Tests (+1 new, 3037/3037 passing)
tests/unit/modules/PaymentsModule.importExport.test.ts:
- "lenient mode: replacing a SPENT token record is labelled
stale-record-replaced, not state-replaced" — proves the CASE 1
disambiguation is working.
- Existing state-replaced test extended to type-narrow via the
new discriminated union.
## Deferred
- Making all CLI commands acquire the same lock (scope expansion —
would need a CLI-wide lock primitive; documented as advisory-only
for now).
- Pending-mint stateHash edge case (documented in JSDoc for
importTokens; very unusual in practice).
- Mixed-code ordering + 'unknown'-synthesis tests (follow-up).
Verification: tsc --noEmit clean; full suite 3037/3037.
Closes the deferred "pending-mint edge case" from the importTokens
steelman. A pending-mint token (post-submit, pre-aggregator-proof)
has no stateHash — the previous importTokens pre-checks relied on
\`incomingStateHash && ...\` which short-circuited on '', letting
two imports of the same pending token churn through addToken's
CASE 3 (archive + replace by local UUID) and produce misleading
'state-replaced' entries.
## Fix
Introduced two helpers in PaymentsModule:
pendingMintDedupKey(txf)
→ 'pending-' + sha256(tokenId|tokenType|salt|recipient|
tokenData|recipientDataHash)
Deterministic per-genesis fallback identifier. Prefixed so
it can never collide with a real 64-hex stateHash.
effectiveDedupKey(txf)
→ real stateHash if present, else pending-mint fallback
importTokens' pre-check now compares incoming vs existing tokens
via effectiveDedupKey. Two imports of the same pending-mint
genesis collide as 'duplicate' cleanly; pending + finalized of
the same genesis correctly land as distinct states.
## Tests (+2 new, 3039/3039 passing)
tests/unit/modules/PaymentsModule.importExport.test.ts:
- "pending-mint tokens deduplicate correctly via genesis fallback
key" — proves two imports of the same pending token collide
as 'duplicate' instead of churning to 'state-replaced'.
- "pending + finalized with the same genesis are NOT duplicates
(different states)" — sanity check that the fallback prefix
prevents collision between real and synthetic stateHashes.
Verification: tsc --noEmit clean; full suite 3039/3039.
Round-6 steelman caught a HIGH regression introduced by the pending-mint
fix plus two MEDIUM correctness gaps.
## HIGH (fixed)
**1. effectiveDedupKey read only the genesis hash, not the current state hash.**
The rest of PaymentsModule — tombstone checks, addToken's CASE
decisions, parseSdkDataCached — uses getCurrentStateHash, which
prefers lastTx.newStateHash over the genesis hash. For ANY token
that has been transferred (i.e., has transactions), the genesis
stateHash is constant but the current stateHash advances per
transfer. By keying only on genesis, my previous fix would
collide TWO DIFFERENT LIVE STATES of the same token as
"duplicate" and silently drop valid state updates on re-import.
This is worse than the pending-mint bug I originally set out to
fix. Replacing the stateHash read with getCurrentStateHash(txf)
inside effectiveDedupKey closes it.
**2. Redundant JSON.parse in the scan loop.**
The existing-tokens scan called JSON.parse(existing.sdkData) per
comparison, bypassing the module's parseSdkDataCached helper.
O(N × M) parses on bulk imports. Rewrote the scan to:
- Use extractStateHashFromSdkData (cached) as the primary path.
- Fall back to one-shot JSON.parse only for pending-state
existing tokens (rare — the fast path hits first in steady
state).
## MEDIUM (fixed)
**3. Pipe-delimited fallback hash is fragile.**
JSON.stringify over a fixed field list is canonical, preserves
null/undefined/'' distinctions, and is robust against arbitrary
characters in any future field. Replaced the pipe-join.
**4. null/undefined/'' ambiguity in the fallback.**
`d.recipient ?? ''` collapsed null and undefined to the same
empty string, making them indistinguishable. JSON.stringify
preserves the distinction (null stays null, undefined drops the
field altogether — we explicitly use `?? null` to make it
explicit and stable).
## LOW (fixed)
**5. Strict mode refused pending→finalized upgrades.**
A legacy wallet migrated while a mint was in flight leaves a
pending record in the Profile. Re-running `migrate-to-profile`
after finalization would refuse the finalized state with
'genesis-exists'. Added an exception: strict mode allows the
upgrade when the existing is pending and the incoming is
finalized. Downgrades and finalized↔finalized conflicts still
refuse correctly.
## Tests (+3 new, 3042/3042 passing)
tests/unit/modules/PaymentsModule.importExport.test.ts:
- "transacted tokens dedup on CURRENT state (lastTx.newStateHash),
not genesis" — proves the HIGH #1 regression is closed for
tokens with transactions.
- "strict mode allows pending→finalized upgrade (pending existing,
finalized incoming)" — proves the LOW #5 exception works.
- "strict mode still refuses finalized→pending downgrades and
finalized↔finalized conflicts" — proves the exception doesn't
leak into unwanted directions.
- buildTxf fixture extended to accept a transactions array so
getCurrentStateHash-based fixtures can be constructed.
tsc --noEmit clean; full suite 3042/3042.
## wallet list / wallet current now display storage mode
Each profile's storage backend (legacy TXF vs OrbitDB Profile) is
detected on-the-fly by probing the on-disk artefacts:
- \`{dataDir}/orbitdb/\` exists → 'profile'
- \`{dataDir}/wallet.json\` exists and non-empty → 'legacy'
- neither → '(not initialised)'
Example output:
Wallet Profiles:
──────────────────────────────────────────────
legacy-profile
Network: testnet
DataDir: ./.sphere-cli-prof1
Storage: legacy
profile-profile
Network: testnet
DataDir: ./.sphere-cli-prof2
Storage: profile
uninit-profile
Network: testnet
DataDir: ./.sphere-cli-prof3
Storage: (not initialised)
Factored into a shared \`detectProfileStorageMode\` helper reused by
\`wallet current\` for consistency.
## Latent bug: switchToProfile left stale storageMode in config
When the user ran \`wallet use B\` after operating profile A (say A was
Profile mode, B is legacy), \`switchToProfile\` copied \`dataDir\`,
\`tokensDir\`, \`network\` into the active config but DID NOT clear the
previous \`storageMode\`. The resolver honours committed
\`config.storageMode\` (step 2 of \`resolveStorageMode\`), so the next
\`init\` / \`send\` / etc. on profile B would silently use A's mode —
attempting to open B's legacy files as a Profile wallet (or vice
versa).
Fixed by \`delete config.storageMode\` in \`switchToProfile\` so the
resolver re-detects from B's dataDir on the next invocation. Profile
B then picks its correct mode from disk artefacts or (for a pristine
profile) from the default.
Typecheck clean; full unit suite 3042/3042 still passing; manual
smoke test with three pre-populated profile dirs verified the three
distinct Storage lines render correctly.
…ish states
Round-7 steelman on the wallet-list storage display returned WARNINGS;
fixing all.
## Medium (fixed)
**Duplicate probe logic.**
\`detectProfileStorageMode\` re-implemented what
\`defaultLegacyWalletProbe\` + \`defaultProfileWalletProbe\` from
\`cli/storage-mode.ts\` already do. Any future addition to the
resolver's probes (e.g. an alternative Profile marker) would silently
diverge from \`wallet list\` / \`wallet current\`. Refactored to
delegate — zero local detection logic, single source of truth.
## Low (fixed)
**Ambiguous "(not initialised)".**
Previously, a dataDir that didn't exist and a dataDir that existed
but had no markers were both displayed identically. Now distinguished:
- 'missing' → dataDir does not exist → "(not initialised)"
- 'uninitialised' → dir exists, no markers → "(empty)" or
"{mode} (pending)"
if committed
- 'legacy' / 'profile' → markers found → "{mode}"
**Corrupt committed storageMode displayed verbatim.**
The fallback to \`config.storageMode\` for the active profile could
print a hand-edited garbage value. Extracted into
\`renderStorageLabel\` which validates the value against the known
enum and shows "(invalid: \\\"<value>\\\")" for anything else.
## Verification
Manual smoke test with five profiles exercises every branch:
- legacy artefact → 'legacy'
- profile artefact → 'profile'
- empty dir → '(empty)'
- missing dir → '(not initialised)'
- empty dir + active with
committed storageMode → 'profile (pending)'
tsc --noEmit clean; full unit suite 3042/3042 passing.
## Deferred (documented in review)
- Behaviour change test for \`switchToProfile\` (delete of storageMode)
— no existing test asserted the old carry-over, so no regression
risk; tested manually in commit b8c1fe6.
- Atomic config writes via tmp+rename — pre-existing, out of scope.
Sphere.init() calls storage.connect() twice: first before identity is
known (to detect wallet-exists), then again after setIdentity().
The previous ProfileStorageProvider.connect() returned early when
status was already 'connected', leaving OrbitDB un-attached. Module
load then hit ensureConnected() and threw:
ProfileError: [PROFILE:PROFILE_NOT_INITIALIZED]
OrbitDB adapter is not connected
Reported from real CLI usage: `npm run cli -- init --nametag babaika13`.
Changes:
* ProfileStorageProvider.connect() — rewritten as two phases:
A. idempotent base connection (local cache), runs once
B. lazy OrbitDB attach — fires on every connect() call until the
database is open, so the second call following setIdentity()
completes the work.
* isConnected() — now returns false in the half-attached state
(local cache up, OrbitDB still pending). This is what makes
Sphere.initializeProviders()'s guard re-invoke connect().
* PaymentsModule.doLoad() address guard — accepts three
representations: L1 bech32, chain pubkey, and Profile short ID
(DIRECT_{first6}_{last6}) written by ProfileTokenStorageProvider.
Lazy-imports computeAddressId to keep profile/ out of non-profile
builds. Fixes the address-mismatch warning that surfaced once
the connect bug was fixed.
* tests/unit/profile/profile-storage-provider.test.ts — 4 new
regression tests under describe('two-phase connect'):
- pre-identity connect then post-identity connect → attach
- idempotence after attach
- isConnected() returns FALSE between the two calls
- connect() without orbitDb config is still a valid success
Why existing tests missed this: unit tests fake dbConnected=true in a
suite-wide beforeEach, bypassing the connect path; the E2E CLI
`init --profile` path is gated behind E2E_NETWORK=1 and not run in
CI; the integration test is network-gated too. The new unit tests
construct a fresh, truly-unconnected mock DB to observe the attach
sequence and would have caught this.
Verification: 3046/3046 unit tests pass; CLI `init --no-nostr`
reproduces clean (no PROFILE_NOT_INITIALIZED, no address-mismatch
warning, identity prints cleanly).
…ressId Follows up commit 5f1fc85 which introduced the two-phase connect but had several issues surfaced by `/steelman`: Critical fixes: * `profile/profile-storage-provider.ts` — serialize concurrent connect() calls via an in-flight `connectPromise`. Without it, two parallel callers could both observe `!dbConnected` and race into `db.connect()`, creating two Helia instances contending on the same directory lock. The OrbitDB adapter's self-idempotence only guarded post-success, not in-flight calls. * `profile/profile-storage-provider.ts` — split `dbStatus` from base `status`. A Phase-B failure now flips only `dbStatus='error'` and leaves the base status='connected' (the local cache is still up). Previously, Phase-B failure poisoned the base status, causing a defensive `disconnect()` to tear down a working cache and lying to callers about what was actually broken. Warning fixes: * `profile/profile-storage-provider.ts` — snapshot identity at entry of `doConnect()`, not after Phase A's await. Prevents a mid-flight `setIdentity()` from swapping the private key between the needsAttach check and the adapter call. * `profile/profile-storage-provider.ts` — `disconnect()` now awaits any in-flight `connect()` before tearing down, preventing a leaked Helia instance if disconnect races against an in-progress attach. * `profile/profile-storage-provider.ts` — `isConnected()` now gates on `options.orbitDb` presence, not `identity`. A provider configured with OrbitDB but without an identity now correctly reports false so writes aren't silently routed to cache-only and never backfilled. * `profile/types.ts` + `profile/profile-storage-provider.ts` — canonical `computeAddressId` in `profile/types.ts` now accepts both `DIRECT://` and the degenerate `DIRECT:` prefix. Deleted the duplicate private implementation in `profile-storage-provider.ts` and imported from `./types.js`. Previously the two divergent implementations could have produced different address IDs for the same input. * `modules/payments/PaymentsModule.ts` — replaced the dynamic `await import('../../profile/types.js')` with a static import of `computeAddressId`. The lazy import's stated goal (keep profile/ out of non-profile builds) was false: `profile/types.ts` has zero runtime imports and is already inlined into every bundle. The dynamic form introduced silent-failure risk in consumer builds lacking matching bundler rules, causing Profile-mode data to be rejected with a misleading "address mismatch" warning. * `modules/payments/PaymentsModule.ts` — hoisted `currentProfileShortId` computation outside the per-provider loop; dropped the `currentL1 &&` gate that could bypass the guard when L1 was empty-string; improved the rejection warning to show all three accepted forms. * `profile/errors.ts` — `ProfileError` now passes `cause` via the ES2022 options bag to `super(message, { cause })`. Standard error- chain tooling (Node's default formatter, Sentry, pino-pretty) can now walk `.cause` to the original stack. New regression tests: * `tests/unit/profile/profile-storage-provider.test.ts` — 6 new tests under `describe('two-phase connect')` covering: - concurrent connect() calls dedupe (1 db.connect, not 4) - Phase B failure doesn't poison base status - Phase B failure permits retry - identity snapshot at entry (mid-flight setIdentity is contained) - connect→disconnect→connect cycle reconnects both phases - isConnected() returns FALSE post-pre-identity-connect when orbitDb is configured (the invariant Sphere.init depends on) * `tests/unit/modules/PaymentsModule.address-guard.test.ts` — new file with 6 tests for the extended address guard: - accepts L1 bech32, chain pubkey, Profile short ID - rejects unrelated short IDs and unrelated L1 addresses - warning message includes all three accepted forms Verification: 3290/3291 unit + integration tests pass; the single failure is a pre-existing CI timeout flake in invoice-parse-memo CLI-029 that passes in isolation both with and without these changes.
Follows up 9f05c49 which fixed the original connect race but introduced or left open several issues surfaced by a recursive `/steelman`. Critical fixes: * `profile/profile-storage-provider.ts` — disconnect/connect race: if disconnect() was awaiting the in-flight connectPromise, a concurrent connect() saw the same promise still set, awaited it, and returned "connected" just as disconnect() was closing the DB. Subsequent writes hit a dying OrbitDB. Fix: new `disconnectPromise` latch; connect() waits for it to drain before starting a fresh attach. disconnect() is now itself deduplicated via the same pattern. Warning fixes: * `profile/profile-storage-provider.ts` — `dbConnected` is now a derived getter (`dbStatus === 'attached'`). Previously both a field and a mirror; 15 call sites read `this.dbConnected`, and any future transition updating only one source would silently route writes to cache-only. The existing test beforeEaches that directly assigned `dbConnected = true` now assign `dbStatus = 'attached'` instead — same effect, single source of truth. * `profile/profile-storage-provider.ts` — inlined the Phase B guard so TypeScript's control-flow narrowing propagates `identityAtStart !== null` and `orbitDbConfig !== null` into the branch body. Removed the fragile `!` non-null assertions. * `profile/profile-storage-provider.ts` — new `dbStatus='fatal'` terminal state. `ORBITDB_NOT_INSTALLED` (package missing) is marked fatal on first attempt; subsequent `connect()` calls no-op instead of re-throwing on every startup hop. Transient errors continue to retry as before. * `profile/profile-storage-provider.ts` — `setIdentity()` now warns via `logger.warn` when called with a different `chainPubkey` while OrbitDB is attached. Captures the attached pubkey at attach time so the check is cheap. Writes encrypted under the new key would otherwise be silently rejected by OrbitDB's AccessController (initialized with the old key); the warning gives operators a breadcrumb to diagnose the misuse. * `tests/unit/modules/PaymentsModule.address-guard.test.ts` — replaced the fragile `vi.spyOn(console, 'warn')` with a structural `logger.configure({ handler })`. Captures by (level, tag, message) tuple so a future message rewording doesn't silently pass the test as "warned=false". New regression tests: * `tests/unit/profile/profile-storage-provider.test.ts` — 3 new tests: - concurrent disconnect+connect: asserts close() is invoked exactly once AND the reconnect succeeds after teardown drains - fatal Phase B failures are sticky: first attempt throws, second connect() resolves without retry - setIdentity() warning on chainPubkey swap while attached Deferred from this commit (noted in review): - Bounded timeout on `disconnect()` awaiting connectPromise (needs AbortSignal plumbing through the adapter; non-trivial). - Deferred-pattern test fixture for concurrency (marginal gain). - Applying the options-bag `cause` pattern to `SphereError` and `IpfsError` for consistency (separate commit, sibling classes). Verification: 3275/3275 unit + integration tests pass (excluding the pre-existing `accounting-cli.test.ts` CI-timeout flake that was flagged in 9f05c49 as unrelated and passes in isolation).
Mirrors the legacy TXF E2E trio (ipfs-sync, ipfs-token-persistence,
ipfs-multi-device-sync) so the new Profile stack gets the same level
of real-infra coverage the legacy IPNS flow has today.
Each test runs against:
- Helia IPFS node (in-process, libp2p + gossipsub, bootstrapped to
DEFAULT_IPFS_BOOTSTRAP_PEERS)
- OrbitDB keyvalue database (via OrbitDbAdapter)
- Unicity IPFS HTTP gateway (https://unicity-ipfs1.dyndns.org) for
CAR pin + fetch
- Nostr testnet relay (token delivery, scoped tests)
- Aggregator testnet (oracle)
Files:
* `tests/e2e/profile-helpers.ts` — `makeProfileProviders()` that
composes Profile's storage + tokenStorage with the legacy factory's
transport/oracle/price/l1. `unwrapProfileProviders()` exposes the
typed Profile instances for tests that need to inspect internals.
Re-exports all shared helpers from `./helpers`.
* `tests/e2e/profile-sync.test.ts` — low-level mirror of
`ipfs-sync.test.ts`. Three tests using `ProfileStorageProvider` +
`ProfileTokenStorageProvider` directly:
1. KV set/get round-trip via real Helia/OrbitDB
2. CAR pin + fetch through the live Unicity gateway
3. Fresh instance (same wallet key) recovers inventory via the
OrbitDB OpLog + CAR CID chain
* `tests/e2e/profile-token-persistence.test.ts` — high-level mirror of
`ipfs-token-persistence.test.ts`. Three Sphere.init-level tests:
1. Create wallet with Profile providers, faucet, receive over
Nostr, publish state to Profile
2. Wipe local data, re-import from mnemonic with a NO-OP
transport, verify tokens recovered ONLY via Profile
(OrbitDB + IPFS) — syncAdded > 0 proves the Profile layer
actually delivered them
3. Recovered tokens are spendable (Profile-recovered state
produces valid transferable tokens)
* `tests/e2e/profile-multi-device-sync.test.ts` — mirror of
`ipfs-multi-device-sync.test.ts`. Three cross-device tests:
1. Device A publishes multi-coin inventory to Profile
2. Device B (fresh temp dir, same mnemonic, no-op Nostr)
recovers the entire inventory through Profile alone
3. Device C full recovery (Profile + real Nostr merge) converges
to the same inventory
Picked up automatically by `vitest.e2e.config.ts` and the existing
`npm run test:e2e` script — no config changes needed.
These tests are opt-in (excluded from `npm test`) because they take
minutes per run and require live network access to multiple Unicity
testnet services.
Helia's `libp2pDefaults()` includes `webRTC()` and `webRTCDirect()` in
the default transports list, and listen addresses for
`/udp/0/webrtc-direct`. On Node these are backed by
`@libp2p/webrtc` → `node-datachannel`, which:
* is a polyfill of the browser WebRTC API, not a first-class Node
implementation;
* emits `DataChannel is closed` errors during shutdown that
surface as unhandled exceptions in vitest;
* is not reachable peer-to-peer from a Node process without an
external signalling service (useless in practice here).
The adapter now filters WebRTC-family transports and their listen
addresses before handing the libp2p config to Helia, but ONLY in
Node (browser environments still need WebRTC as their sole direct-
dial transport).
Identification: each transport factory's `.toString()` exposes its
constructor name (e.g. `new WebRTCTransport(...)`). The filter
matches on that substring. Unusual but portable across libp2p
versions that don't set `[Symbol.toStringTag]` on the factory.
Also: dropped the low-level cross-instance recovery test from
`profile-sync.test.ts`. Two fresh ephemeral Helia nodes cannot
discover each other with only IPFS gateways as bootstrap peers
(no DHT or rendezvous). Cross-instance replication is exercised
end-to-end at the Sphere.init() level in the other two e2e files.
Verification: `npm run test:e2e tests/e2e/profile-sync.test.ts`
passes 2/2 in 1.4s with no unhandled errors; previously the same
suite ran 140s and printed two libdatachannel uncaught exceptions.
…c v2
Design docs for replacing the IPNS snapshot stopgap with a
privacy-preserving pointer layer anchored in the Unicity aggregator
SMT. For every OpLog version the wallet publishes two commitments
(sides A and B) whose signed `transactionHash` fields carry an
XOR-blinded split of the current OpLog CID, discoverable via
exponential-then-binary search over the version space.
Design decisions captured:
* Not tokenised. A tokenised state-transition design cannot be
re-entered from the mnemonic alone on a fresh device (you'd need
to know the token's current state hash before you can query),
defeating the core recovery goal. The two-leaf plain-commitment
scheme is retained deliberately.
* All SDK-covered operations use state-transition-sdk primitives —
SigningService (secp256k1 only; Ed25519 removed), DataHasher,
DataHash, RequestId.createFromImprint (uses stateHash.imprint,
not the raw digest — the 2-byte algorithm tag matters),
Authenticator.create, SubmitCommitmentRequest, AggregatorClient,
InclusionProof.verify(trustBase, requestId). Only non-SDK
primitives in the scheme are HKDF-SHA256 (via @noble/hashes,
reusing the pattern from impl/shared/ipfs/ipns-key-derivation.ts)
and bytewise XOR.
Reviewer findings applied (from five parallel reviews):
Critical:
- Ed25519 replaced with secp256k1 everywhere (aggregator rejects
anything else in commitment_validator.go).
- RequestId formula corrected to hash pubkey || algoTag(2B) ||
digest(32B) — 67-byte preimage matching SDK behaviour.
- Crash-retry OTP reuse closed by persisting (v, H(cid)) to local
storage before computing the XOR payload; refuse to reuse v with
a different plaintext.
- Four arch↔spec contradictions reconciled (signing algo, length
encoding via 1-byte prefix, both-sides-probe discovery,
partial-publish retry at same v with deterministic bytes).
- Trustless InclusionProof.verify with a RootTrustBase is now
mandatory; TOFU explicitly accepted for first-boot recovery.
- Aggregator-unreachable-at-recovery no longer silently produces
empty state: publish is blocked until a verified exclusion
proof or a successful remote merge.
Warnings:
- HKDF-Expand subkey separation for signingSeed / xorSeed / padSeed.
- Privacy goal G2 downgraded to "pseudonymous per wallet" with
signingPubKey clustering disclosed as known leakage.
- Retry backoff includes random jitter (BASE × 2^n × U(0.5,1.5)).
- Consistency model stated explicitly: per-wallet linearizable
under BFT aggregator; OpLog contents remain CRDT.
- Deterministic HKDF-derived padding replaces CSPRNG so crashed
retries reproduce byte-identical submissions.
- DISCOVERY_PARALLELISM lowered to 1 (binary search is serial;
A/B parallelism within each probe unchanged).
- Observability events specified (pointer.publish.*,
pointer.discover.*, pointer.recover.*, pointer.conflict.*).
- Alternatives table now covers IPNS, OrbitDB-live-peer-only,
tokenised-state-chain, and centralised pinning.
Editorial:
- Single canonical open-questions list with owners.
- Stale "to be written" markers removed.
- Migration touches: profile-ipns.ts deletion, ipnsSnapshot flag
rename, test-file cleanup.
Status: v2 draft. Ready for security auditor, aggregator team,
Unicity architect, and SDK team sign-off. Implementation PR
follows once the first test vector bytes are computed and
checksum-committed per SPEC §14.
…ation + review fixes
v2 was REJECTED by the doc-reviewer and security-auditor for the same
class of bug that got v1 rejected: the two docs disagreed on byte-level
formulas. Three parallel steelman reviewers on v2 surfaced five
critical divergences; v3 reconciles them and applies a dozen warnings.
Canonical formulas (now identical in both docs):
stateHashDigest = SHA-256(xorSeed || [side] || be32(v) || "state")
xorKey = SHA-256(xorSeed || [side] || be32(v) || "xor") // bare SHA-256, not HKDF-Expand
padBytes_v = HKDF-Expand(padSeed, be32(v) || "pad", 63-cidLen) // shared across sides
signingService = SigningService.createFromSecret(signingSeed) // SDK static; SHA-256 normalizes
requestId = RequestId.createFromImprint(signingPubKey, stateHash)
authenticator = Authenticator.create(signingService, transactionHash, stateHash)
Critical fixes applied:
F1 .proof → .inclusionProof (spec §8.1, §8.5) — SDK's
InclusionProofResponse field is `inclusionProof`, not `proof`.
Four call-sites corrected; a literal transcription of v2 would
have crashed at runtime on the first recovery probe.
F2 §7.3 outcome matrix (EXISTS, EXISTS) — split into:
- idempotent-replay (marker match) → persist + clear + succeed
- genuine conflict (marker absent/mismatch) → §9 reconciliation
v2 wrongly treated all (EXISTS, EXISTS) as conflict, which would
loop on every crash-recovery-replay.
F3 §7.1 pending_version discipline — six sub-invariants:
mutex (per-wallet MUTEX_KEY), per-wallet key scoping
(PENDING_VERSION_KEY includes hex(signingPubKey)), durability
(fsync/transaction.oncomplete), rollback-safe `v` selection
(max(v, prev.v)+1 when prev.v >= v), full 32-byte cidHash
(no truncation), marker-clear atomicity.
F4 §10.2 BLOCKED state — formalized:
persistent flag BLOCKED_FLAG_KEY (per-wallet), categorical-
error SET conditions, strict CLEAR conditions (verified
exclusion at v=1 OR successful recoverLatest + merge),
user-originated-write = entry.signedBy == localSigningPubKey
(NOT replication / Nostr ingest), optional per-call override
gated behind capability flag.
SigningService.createFromSecret everywhere (not `new SigningService`).
The raw constructor would produce a different signingPubKey for
the same seed, breaking interoperability.
Warnings addressed:
F5 async `DataHasher.digest()` — footnote at §4 header covers all
subsequent pseudocode.
F7 TOFU multi-mirror cross-check — RECOMMENDED on first boot;
new error AGGREGATOR_POINTER_TRUST_BASE_DIVERGENCE.
F8 Retry-rejected ciphertexts MUST be zeroized and not logged;
explicit secret-value list (pointerSecret, signingSeed, xorSeed,
padSeed, signing private key) documented as log-forbidden.
F9 First canonical test vector inputs frozen (key=0x01×32;
CIDv1-raw of "hello world" computed to 36 bytes); derived
values remain O-1 until implementation-PR merge.
F10 Open items: O-1 demoted to impl-PR-blocker; O-5 (override
decision, non-blocking), O-6 (mirror list, blocking) added.
F11 `isPublishBlocked(): boolean` added to ProfilePointerLayer.
Arch↔spec alignment verified:
- stateHash preimage uses xorSeed in both docs (arch §4.3, spec §4.4)
- xorKey uses bare SHA-256 (arch §4.1/§9.3, spec §4.5)
- padding is shared across sides with "pad" suffix (both §4.5/§4.6)
- SigningService.createFromSecret everywhere (7 total occurrences)
- Constant names aligned (PUBLISH_BACKOFF_*, no RETRY_ infix)
- Discovery init seeded from localVersion (arch §10.5, spec §8.2)
- BLOCKED state discipline cross-referenced
- Open-questions routed to spec §15.1 as single source of truth
Status: Draft v3. Ready for security auditor, aggregator team, Unicity
architect, SDK team re-review. Test-vector byte computation (O-1)
remains the single implementation-PR blocker.
…findings
Three parallel steelman reviewers on v3:
- byte-level consistency auditor: APPROVE (arch↔spec formulas match)
- SDK fidelity: APPROVE (all SDK primitives verified against source)
- end-to-end attack: REJECT (3 new criticals + 4 warnings)
This commit closes the 3 criticals + 4 warnings plus editorial
alignments. The underlying two-leaf XOR-blinded scheme is unchanged;
v3.1 is purely a hardening patch.
Criticals closed:
C1 Marker version-jump clamp. A malicious process with filesystem
write access could inject a valid-looking marker with
`previousEntry.v = 2^31 - 2`, triggering `v := 2^31 - 1` on
next publish and bricking the wallet on the following attempt
(`v = 2^31` exceeds `VERSION_MAX`). Added `MARKER_MAX_JUMP =
1024` constant + clamp in §7.1.4 — markers exceeding the gap
are treated as `AGGREGATOR_POINTER_MARKER_CORRUPT`.
C2 Retry-window ciphertext zeroization. §11.11 previously required
zeroization only on `REQUEST_ID_EXISTS`. Network-retry loops
hold ciphertext in memory for up to ~8s across the retry
budget, exposing an OTP differential to memory-dump attackers.
Added §11.11(a′) requiring either per-retry re-derivation or
`MAX_CT_RESIDENT_MS = 500` cap, with mandatory zeroization
between retry attempts.
C3 Fresh-install MITM bypass of BLOCKED. On a fresh mnemonic
re-import with a MITM'd network, §10.2 BLOCKED never sets
(no user-originated OpLog writes yet), so an attacker's
faked `RootTrustBase` under single-mirror TOFU would silently
take over. Promoted §8.4 multi-mirror cross-check from
RECOMMENDED to MANDATORY (≥ MIN_MIRROR_COUNT = 2 independent
mirrors). Added §10.2.6: fresh-install cold-start producing a
corrupt XOR-decoded payload MUST SET BLOCKED even with zero
user-originated writes.
Warnings closed:
C4 CAR size + fetch-time caps: MAX_CAR_BYTES = 100 MiB,
MAX_CAR_FETCH_MS = 60 s, new error codes
AGGREGATOR_POINTER_CAR_TOO_LARGE / _FETCH_TIMEOUT.
C5 CAR-unavailable explicit state (§10.7): pointer recovery that
verifies a CID but cannot fetch the CAR MUST NOT silently
advance `localVersion`; wallet enters
`AGGREGATOR_POINTER_CAR_UNAVAILABLE` with explicit
`acceptCarLoss(version)` override API.
C6 `originated` metadata tag (§10.2.3): replaces the ambiguous
`signedBy == localSigningPubKey` heuristic with an explicit
enum {'user', 'system', 'replicated'} that OpLog writers
stamp at write time. Fail-closed default for missing tags.
C7 Probe-sequence fingerprint disclosure (§11.10 bullet 10):
acknowledges the deterministic discovery-probe pattern as a
stronger correlation signal than signingPubKey clustering;
v2 mitigations (randomized Phase 1, decoy probes, anonymity
set) documented as future work.
C8 Canonical test-vector runtime rejection (§14.1, §11.12):
prominent WARNING block on the §14.1 canonical key
(`0x01 × 32`); `Profile.init()` MUST reject this key in
non-test networks via a mandatory denylist check.
API additions (§13): `acceptCarLoss(version)`, `clearPendingMarker()`,
`getProbeFingerprint()`. `isPublishBlocked()` signature made
`Promise<boolean>`.
New events (arch §13): `pointer:recover_car_unavailable`,
`pointer:car_loss_accepted`, `pointer:marker_cleared`.
Editorial alignments (arch↔spec):
- `AGGREGATOR_POINTER_PROOF_INVALID` → `AGGREGATOR_POINTER_UNTRUSTED_PROOF` in arch
- BLOCKED SET verb aligned ("been attempted and failed")
- Symbol naming `padBytes_V` → `paddingBytes_v` in arch
- `findLatestVersion` call-site arity corrected (4 args)
- `localSigningPubKey` disambiguated as chain-key pubkey
- Async-await convention footnote expanded to cover
`createFromSecret`, `createFromImprint/create`,
`Authenticator.create`, `AggregatorClient` methods
- `DataHasher(X) ≡ new DataHasher(X)` pseudocode convention noted
Open items: O-6 (mirror URL list) promoted to BLOCKING spec sign-off
because C3 now mandates multi-mirror.
Status: Draft v3.1. Ready for another review round by security
auditor + aggregator team + Unicity architect + SDK team. Test-vector
byte computation (O-1) and mirror URL finalization (O-6) are the
remaining implementation-PR blockers.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Introduces two new modules for the sphere-sdk:
UXF (Universal eXchange Format) —
@unicitylabs/sphere-sdk/uxfContent-addressable packaging format for Unicity token materials. Tokens are recursively deconstructed into a shared DAG of elements, enabling deep deduplication (~67% size reduction for typical wallets). Key features:
Profile —
@unicitylabs/sphere-sdk/profileOrbitDB-backed wallet storage layer implementing the existing
StorageProviderandTokenStorageProviderinterfaces. Provides IPFS-first persistence with automatic multi-device conflict resolution. Key features:tokens.bundle.{CID}), merged on read viaUxfPackage.merge()createBrowserProfileProviders()/createNodeProfileProviders()(no upstream file modifications)Documentation (20 files, ~350 KB)
Complete specification suite: formal spec (CDDL/JSON Schema), architecture, design decisions, implementation plans, domain constraints, test specifications, SDK storage inventory, IPFS/OrbitDB research.
Stats
@ipld/dag-cbor,@ipld/car(dependencies);@orbitdb/core,helia(optional peerDependencies)./uxf,./profile,./profile/browser,./profile/nodeTest plan
npx tsc --noEmit— zero type errorsnpm run build— all entry points build (uxf + profile/index + profile/browser + profile/node)npx vitest run tests/unit/uxf/— 268 tests passingnpx vitest run tests/unit/profile/— 110 tests passing@orbitdb/coreinstalled — deferred to follow-up)Follow-up work (not in this PR)
@orbitdb/core+heliainstalled)