[EPIC] The Rust API for CocoIndex v1.0's engine

[bashandbone]

With Upstream's Move to v1.0.0, Recoco will need a complete Rust API

Recoco is building the first and only pure-Rust API for the incremental data processing engine that powers CocoIndex. Upstream has committed fully to Python for their v1 user-facing API, Recoco must provide a native Rust experience: typed operations, proc macros, explicit context management, and zero Python dependencies.

If you're looking for a Rust API for CocoIndex, this is it.

Upstream context

The CocoIndex community has expressed clear demand for a Rust API:

• [FEATURE] Rust API cocoindex-io/cocoindex#1372 — [FEATURE] Rust API requested by @gkgoat1. A founder responded positively but no work has been planned or milestoned.
• [FEATURE] Ergonomic Rust SDK cocoindex-io/cocoindex#1667 — [FEATURE] Ergonomic Rust SDK proposed by @tomz-alt with a detailed, well-received design. A team member provided constructive feedback that improved the proposal. The issue remains open with no milestone or assignee. (While clearly not aligning with the planned direction for v1, which moves all integration to Python)

Upstream's v1 milestone is exclusively Python-focused. Neither Rust API issue is milestoned. The v1 branch has rebuilt all API and integration layers in Python, with the Rust engine serving only as an internal runtime.

Recoco's v1.0.0 branch has full parity with upstream's Rust -- sharing the same high-performance Rust engine (LMDB-backed state, component memoization, target reconciliation, incremental processing). But with upstream's divergence, we need to plan for a proper all-Rust API while keeping in sync and benefiting from upstream's engine development. I would have preferred more alignment, but it is what it is -- their motivations and needs don't align with mine. It makes sense for them, but doesn't give me the foundation I need for Thread

This issue will serve as the tracking and planning issue for Recoco's v1 design and API.

What we're building

Goals:

• Continue to match upstream's engine in recoco-core, optimizing for Rust where it makes sense, and contributing improvements upstream where directions/needs align. (I've admittedly been bad about this; I need to do better)
• Continue our obsessiveness about feature gating and performance optimization.
• Build the Rust API for upstream's engine. This means to the extent practical extending idioms and API surfaces in upstream's python to Rust. There will necessarily be large divergences -- Rust is a very different language and Rustaceans expect different things from their APIs.
• Aim to implement parity with upstream's integrations in Rust (what were functions/sources/targets in pre-v1). Granularly feature gated without compromise. I'm hoping we pick up some community interest to help in this department.

#[recoco::function(memo)]
async fn process_file(ctx: &Ctx, file: FileEntry, table: &TableTarget) -> Result<()> {
    let text = file.read_text().await?;
    let chunks = RecursiveSplitter::default().split(&text, SplitOptions::default());
    let embedder = ctx.use_resource(&EMBEDDER);
    for chunk in &chunks {
        let embedding = embedder.embed(&chunk.text).await?;
        table.declare_row(/* ... */)?;
    }
    Ok(())
}

Programming model: Persistent-state-driven — transformations are plain Rust async fn functions. The engine handles incrementality, memoization, lineage, and fault tolerance transparently. No DSL, no graph builder, no string-based dispatch.

Key features:

• #[recoco::function] proc macro with memo, batching, and code-hash-based cache invalidation (our blake3 implementation vice upstream's blake2)
• Environment + App for LMDB-backed persistent state
• Ctx with typed ContextKey<T> for explicit resource management
• mount_each() for parallel, incremental component processing
• Sources as iterators, functions as direct method calls, targets as declarative mounts
• Feature-gated everything — only compile what you use

Design ancestry

Our API design draws directly from @tomz-alt's proposal in cocoindex-io/cocoindex#1667, refined with the feedback from upstream maintainers and adapted for Recoco's pure-Rust context. We diverge where Rust idioms demand it (e.g., ContextKey<T> over type-erased lookup, Environment/App separation, richer target abstractions).

@tomz-alt — if you're interested in contributing to a Rust implementation of the ideas you proposed, we'd welcome you. Your design work and understanding of the engine are exactly what this project needs. You mentioned having a partial implementation; we'd love to build on that foundation.

Current state

Component	Status	Location
Engine (LMDB, components, memoization)	Done	v1.0.0 branch, crates/core/
Operations (sources, functions, targets)	Done (old arch)	main branch, crates/recoco-core/src/ops/
Concrete RecocoProfile	Not started	—
Ctx / ContextKey / Environment / App API	Not started	—
#[recoco::function] proc macro	Not started	—
Operation port (old arch → new API)	Not started	—

The engine is ready. The operations exist but are wired for the pre-v1 FlowBuilder architecture. This epic tracks bridging them with a Rust-native API layer.

Implementation plan

Phase 1: Foundation — Concrete RecocoProfile, Ctx, ContextKey<T>, Environment, App
Phase 2: Proc macros — recoco-macros crate, #[recoco::function(memo, batching)]
Phase 3: Port operations — Sources, functions, targets as standalone Rust types
Phase 4: Transient mode — One-shot execution without LMDB persistence
Phase 5: Query handlers — Search endpoints, graph DB mappings, reusable transforms

Areas requiring deep design

Each of these will get their own issue before implementation begins:

• RecocoProfile concrete types — 8 associated types that define the entire API surface
• Proc macro design — #[recoco::function] code generation, error quality, compile-fail tests
• Ctx lifetime and ownership — Thread safety, child contexts, reference ergonomics
• Target state reconciliation — Serialization format, row identity, diffing, backend mapping
• Incremental source change detection — Engine integration, CDC, file watching
• Crate structure — Workspace organization, feature flag design, public API paths
• Error handling — thiserror types for public API, propagation through the engine
• Operation porting strategy — Priority order, standard patterns, config migration
• Docs - Update v1 docs to reflect the new surface and API.

Related links

Upstream Rust API demand:

• [FEATURE] Rust API cocoindex-io/cocoindex#1372 — [FEATURE] Rust API
• [FEATURE] Ergonomic Rust SDK cocoindex-io/cocoindex#1667 — [FEATURE] Ergonomic Rust SDK (design proposal)

Upstream v1 direction (Python-only):

• v1 milestone — all Python-focused issues
• v1 branch — Python API, PyO3 bindings

Recoco design documents:

• docs/v1-way-ahead.md — Current analysis and recommendations

[github-actions]

🤖 Hi @bashandbone, I've received your request, and I'm working on it now! You can track my progress in the logs for more details.

[github-actions]

Claude finished @bashandbone's task —— View job

---

EPIC Issue #132 Analysis: The Rust API for CocoIndex v1.0's Engine

I've completed my analysis of this EPIC issue. Here's my comprehensive review and recommendations.

---

Executive Summary

This is a well-structured, ambitious, and necessary EPIC that addresses a critical architectural transition for Recoco. The issue correctly identifies that upstream CocoIndex's v1.0 commitment to Python-only APIs creates both an opportunity and an obligation for Recoco to provide a first-class Rust API.

Key Strengths:

• Clear articulation of the upstream divergence and Recoco's positioning
• Thoughtful API design that respects Rust idioms while preserving engine semantics
• Comprehensive scoping with explicit "areas requiring deep design"
• Strong documentation and planning foundation in docs/v1-way-ahead.md
• Realistic phasing with clear dependencies

Primary Concerns:

• Significant implementation complexity with many interdependent design decisions
• Risk of "analysis paralysis" given the depth of planning required

---

Context and Current State

Repository Structure (Current)

Based on my analysis:

✅ What exists:

• The v1.0.0 branch with LMDB-backed engine (crates/core/src/engine/)
• 4-crate workspace: recoco, recoco-core, recoco-utils, recoco-splitters
• Pre-v1 architecture with FlowBuilder, JSON-based configs, string-based operation dispatch
• Comprehensive operation library: 5 sources, 6 functions, 4 targets (all feature-gated)
• 81 Rust source files in recoco-core
• Strong examples and documentation

❌ What's missing (as stated in issue):

• EngineProfile / RecocoProfile infrastructure
• Ctx / ContextKey / Environment / App API layer
• #[recoco::function] proc macros

Related Issues

Found 11 closed v1.0.0 issues, all [upstream-sync] tasks tracking engine parity:

• Issue [upstream-sync] [v1-parity]Tracking: Reach parity with upstream cocoindex v1 branch #29: Main v1 parity tracking issue (CLOSED)
• Issues [upstream-sync] Batch LMDB write transactions automatically for improved throughput #35-60, [upstream-sync] Expose progress watching API for App/execution engine (upstream PR #1767) #99-100, [upstream-sync] Expose progress watching API for App/execution engine (upstream PR #1767) #118: Specific engine features synced from upstream

This EPIC (#132) is the first issue focused on the Rust API layer above the engine.

---

Design Assessment

Proposed Programming Model

The API design is conceptually sound and appropriately Rust-idiomatic:

// Proposed API (from issue)
#[recoco::function(memo)]
async fn process_file(ctx: &Ctx, file: FileEntry, table: &TableTarget) -> Result<()> {
    let text = file.read_text().await?;
    let chunks = RecursiveSplitter::default().split(&text, SplitOptions::default());
    let embedder = ctx.use_resource(&EMBEDDER);
    for chunk in &chunks {
        let embedding = embedder.embed(&chunk.text).await?;
        table.declare_row(/* ... */)?;
    }
    Ok(())
}

Strengths:

• Plain Rust functions - No DSL, no graph builder, natural async/await
• Explicit context management - ContextKey<T> over type-erased lookup improves discoverability
• Declarative targets - Engine-managed reconciliation is more powerful than manual writes
• Optional proc macros - Memoization via #[recoco::function(memo)] but manual ComponentProcessor impl always available

Comparison to current API:

// Current pre-v1 API (from examples/transient.rs)
let split_slice = builder.transform(
    "SplitBySeparators".to_string(),  // String-based dispatch
    json!({ "separators_regex": [" "], "keep_separator": null }).as_object().unwrap().clone(),  // JSON config
    vec![(input_slice, Some("text".to_string()))],
    None,
    "splitter".to_string(),
).await?;

The proposed API is a significant ergonomic improvement - typed operations, builder patterns for config, direct method calls.

Identified Design Challenges

The issue correctly identifies 9 deep design areas. I'd prioritize them as follows:

Critical Path (must resolve first):

1. RecocoProfile concrete types (Section 5.1) - Defines the entire API surface
2. Ctx lifetime and ownership (5.4) - Core abstraction, affects everything
3. Crate structure (5.7) - Early decision constrains public API paths

Implementation Phase:
4. Proc macro design (5.2) - Complex but well-scoped, can prototype independently
5. Target state reconciliation (5.5) - Engine integration, needs RecocoProfile first
6. Operation porting strategy (5.3) - Tactical, follows from foundation

Post-MVP:
7. Incremental source change detection (5.6) - Advanced feature
8. Error handling strategy (5.8) - Can iterate; anyhow acceptable initially
9. Crate structure details (utils absorption, separate target crates) - Optimize later

---

Implementation Feasibility

Phase Assessment

Phase 1: Foundation - High Risk, High Value

• Requires resolving the 3 critical-path design areas
• Estimated: ~1 week of focused design + implementation

Phase 2: Proc Macros - Medium Risk

• Well-understood Rust patterns; good examples exist (tokio, axum)
• Separate recoco-macros crate with own test harness
• Estimated: 2-3 weeks

Phase 3: Port Operations - Medium Risk, High Effort

• 15 operations to port (5 sources, 6 functions, 4 targets)
• Each needs: standalone types, typed config, new tests
• Can parallelize; good for community contribution
• Estimated: 3-4 weeks (or faster with contributors)

Phase 4 & 5 - Lower Priority

• Transient mode may be easier than expected if operations are plain Rust types
• Query handlers are advanced features, not MVP-blocking

Risk Factors

1. Design interdependencies - RecocoProfile, Ctx, and proc macros have circular dependencies. Wrong early decisions are expensive to fix.

2. Community capacity - Issue mentions @tomz-alt's upstream proposal and invites collaboration. Has there been interest? Solo implementation is feasible but slower.

---

Recommendations

1. Break Down Into Actionable Sub-Issues

Create individual issues for critical-path items:

• Issue: "Design RecocoProfile associated types for Rust API" (references EPIC [EPIC] The Rust API for CocoIndex v1.0's engine #132)
• Issue: "Design Ctx lifetime and ownership model" (references EPIC [EPIC] The Rust API for CocoIndex v1.0's engine #132)
• Issue: "Decide final crate structure for v1.0.0" (references EPIC [EPIC] The Rust API for CocoIndex v1.0's engine #132)

Each should be tagged with needs-design, v1.0.0, and reference this EPIC.

2. Create Design Spike Prototype

Before committing to full implementation:

• Create a spike/v1-api branch
• Implement minimal viable RecocoProfile, Ctx, Environment, App
• Port one simple operation (e.g., SplitBySeparators) to new API
• Write example showing end-to-end usage
• Document learnings and API refinements

Goal: Validate API ergonomics and identify hidden complexity before scaling to 15 operations.

3. Prioritize Community Engagement

The issue invites @tomz-alt but lacks outreach:

• Open a GitHub Discussion: "Designing Recoco's v1 Rust API - Request for Feedback"
• Share in upstream CocoIndex community (if appropriate)
• Tag relevant Rust communities (r/rust, This Week in Rust) once design stabilizes

Operation porting (Phase 3) is ideal for first-time contributors - create "good first issue" labels when ready.

4. Update Documentation Proactively

The issue mentions docs need updates (Section "Implementation Plan"):

• Create docs/v1-api-design.md capturing design decisions as they're made
• Update site/ docs incrementally, not as final phase
• Consider writing "v1 Migration Guide" comparing old FlowBuilder → new API

---

Suggested Labels

Based on repository labels, I recommend:

• ✅ v1.0.0 (already applied)
• ✅ needs-design (already applied)
• ✅ enhancement (already applied)
• ✅ help wanted (already applied - good for community)
• ✅ documentation (already applied - docs updates are significant)
• ✅ upstream-sync (already applied - tracking upstream divergence)

All appropriate labels are already applied. Well done!

---

Related Issues & Links

Existing Recoco Issues

• Issue [upstream-sync] [v1-parity]Tracking: Reach parity with upstream cocoindex v1 branch #29 (CLOSED): "[v1-parity] Tracking: Reach parity with upstream cocoindex v1 branch"
  → This EPIC is the natural successor focusing on API layer

Upstream References (from issue)

• [FEATURE] Rust API cocoindex-io/cocoindex#1372: [FEATURE] Rust API - community request
• [FEATURE] Ergonomic Rust SDK cocoindex-io/cocoindex#1667: [FEATURE] Ergonomic Rust SDK - @tomz-alt's detailed design proposal
• cocoindex-io/cocoindex v1 milestone: Python-focused, no Rust API work

Similar Rust Projects (for inspiration)

• Polars (polars-plan, polars-lazy) - similar dataflow + proc macro approach
• Ratatui (formerly tui-rs) - good example of pure-Rust fork establishing own identity
• Tokio - excellent proc macro patterns (#[tokio::main], #[tokio::test])

---

Difficulty & Impact Estimate

Difficulty: High (8/10)

• Requires deep Rust expertise (proc macros, lifetimes, LMDB integration)
• Many interdependent design decisions
• No direct precedent (first pure-Rust API for this engine)
• Estimated full implementation: 3-4 months (1 person) or 6-8 weeks (small team)

Impact: Critical (10/10)

• Defines Recoco's identity as "the Rust API for CocoIndex engine"
• Enables all future Rust users - this is the foundation everything builds on
• Differentiates from upstream - becomes Recoco's primary value proposition
• Unblocks Thread (mentioned in issue) and other downstream Knitli projects

Recommendation: Proceed, but with structured design phase first

This is the right work at the right time. The planning is thoughtful and comprehensive. However, jumping straight to implementation risks expensive rework. I recommend:

1. Week 1-2: Resolve critical-path design decisions, document in docs/v1-api-design.md
2. Week 3-5: Implement Phase 1 (Foundation)
3. Week 6-7: Implement Phase 2 (Proc macros)
4. Week 8-11: Implement Phase 3 (Port operations) - invite community
5. Week 12: Polish, docs, examples, announce

If community contributors join, timeline compresses significantly.

---

Final Thoughts

This is excellent technical planning. The issue demonstrates deep understanding of:

• Upstream's architectural direction and Recoco's positioning
• Rust idioms and ecosystem expectations
• The engine's actual implementation (despite missing branch)
• Realistic complexity and scope

My only concerns are:

1. Avoiding over-planning - the 9 "areas requiring deep design" are valid, but don't let design become a blocker. Ship MVP, iterate.
2. Community engagement - this is a lot of work for one person; broadcast the vision and invite collaboration early

I recommend proceeding with this EPIC. It's the right direction for Recoco and positions the project uniquely in the Rust ecosystem. No other pure-Rust API exists for this class of incremental dataflow engine.

---

Next Actions:

1. Create 3 design issues for critical-path items
2. Start design spike in spike/v1-api branch
3. Open GitHub Discussion for community feedback
   | Branch: claude/issue-132-20260329-1421

---

Editor's note: Claude's original post had a lot of focus on its inability to find our v1.0.0 branch, which was an environment/permissions issue. I edited to focus on everything else.