Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
108 changes: 99 additions & 9 deletions docs/plans/2026-01-22-tywrapped-ecosystem-vision.md
Original file line number Diff line number Diff line change
Expand Up @@ -110,12 +110,26 @@ Examples:
| Tywrap fix for same numpy | `@tywrapped/numpy@1.26.0-tywrap.1` |
| New numpy release | `@tywrapped/numpy@1.26.1` (resets suffix) |

**Republish policy**: Every tywrap core release triggers regeneration and republish of all active packages with compound version. Users on any upstream version benefit from improvements.
**Republish policy**: Every tywrap core release can trigger regeneration and republish of all active packages with a compound version so users on supported upstream versions benefit from improvements.

Operational scalability (suggested defaults):
- Canary wave: 5-10% of packages first; require green CI and no runtime regressions before expanding.
- Publish gating: require unit + integration smoke tests; fail-closed on flaky/timeout tests.
- Batching: cap concurrent publishes (e.g., 3-5) to avoid CI bottlenecks and npm rate limits.
- Rate limits: retry with exponential backoff (e.g., 5 retries, 30s → 5m); pause the queue on repeated 429s.
- Rollback: move dist-tags back to the last known-good build; open an incident issue and stop the wave.

**Version support breadth**: Separate indicator from quality tiers. Packages aim to match upstream's official support matrix (e.g., if numpy supports 1.24+, so does `@tywrapped/numpy`).

**Breaking changes**: Mirror upstream directly. No buffering, no parallel support periods. The wrapper is a transparent pass-through - breaking changes in numpy mean breaking changes in `@tywrapped/numpy`. Document this clearly so users understand they're accepting upstream's upgrade path.

### Enterprise stability options

- Optional channels: `stable`/`lts` dist-tags for org-blessed "slow lane" packages when demand justifies it; explicitly time-box support windows.
- Migration guides: require a short migration note for any breaking change (template + checklist) before publish.
- Pinning guidance (enterprise): prefer exact versions + lockfiles for production; treat `^` upgrades as opt-in.
- Trade-off: strict mirroring maximizes transparency and reduces maintainer load; stability channels increase operational overhead but improve adoption.

---

## Template Repository Structure
Expand Down Expand Up @@ -168,6 +182,12 @@ Releases triggered by **either**:

Library maintainer must approve before publish - automation does the heavy lifting, humans ensure quality.

Fallbacks when maintainers are unavailable (suggested defaults):
- Minimum 2 maintainers per package (primary + backup).
- Patch releases: auto-approve after 72 hours with no response (logged), unless a maintainer vetoes.
- Security fixes: emergency override by technical committee (2-of-3) with mandatory postmortem.
- All overrides require a public audit trail (issue + PR link + rationale).

---

## Package Health Indicators
Expand All @@ -188,9 +208,51 @@ Two separate badge systems for orthogonal concerns:
| Level | Indicator | Meaning |
|-------|-----------|---------|
| Basic | `latest` | Only current upstream version |
| Standard | `latest+1` | Latest + previous major |
| Standard | `latest-1` | Latest + previous major |
| Full | `upstream` | Matches upstream's support matrix |
| Extended | `upstream+LTS` | Upstream support + extended LTS |
| Extended | `upstream-lts` | Upstream support + extended LTS |

#### Implementation notes

Recommended approach: single package name + npm dist-tags.

- One package per library (e.g., `@tywrapped/numpy`) publishes multiple tested wrapper versions (mirroring upstream + `-tywrap.N` patches).
- Use dist-tags to make supported tracks easy to install:
- `latest` → newest supported upstream release line
- `latest-1` → previous major (or previous supported line)
- `upstream` → "full matrix" track (only if you actually publish/test it)
- `upstream-lts` → extended support line (explicitly time-boxed)
- Wrapper repo `package.json` stays straightforward: name is stable, version mirrors upstream with optional `-tywrap.N`, and `publishConfig.access` is public.

Publish/tag flow (sketch):

```bash
npm publish --tag latest
npm dist-tag add @tywrapped/numpy@1.26.0-tywrap.3 latest
npm dist-tag add @tywrapped/numpy@1.25.4-tywrap.2 latest-1
```

CI matrix outline (sketch):

```yaml
strategy:
matrix:
python: ["3.10", "3.11", "3.12"]
upstream: ["numpy==1.24.*", "numpy==1.25.*", "numpy==1.26.*"]
```

Publish script logic (sketch):
1. Install target upstream version(s) into an isolated env (per matrix entry).
2. Generate wrappers (tywrap) and run the test tier for that package.
3. Compute package version (`<upstream>-tywrap.N`) and publish.
4. Apply/update dist-tags for the supported tracks.

User-facing installs:
- Default: `npm i @tywrapped/numpy` (uses `latest`)
- Specific track: `npm i @tywrapped/numpy@latest-1`
- Exact pin: `npm i @tywrapped/numpy@1.26.0-tywrap.3`

Alternative (if tags become unwieldy): separate packages per track (e.g., `@tywrapped/numpy-1.24`), at the cost of more maintenance and user confusion.

### Display Format

Expand Down Expand Up @@ -220,18 +282,19 @@ Quality: 🥇 Gold | Support: upstream (1.24+) | Runtimes: node, browser
- Browser environment → Pyodide runtime
- Node.js environment → Subprocess runtime (safest, no native deps)

**Override for power users:**
**Override for power users (current tywrap API):**

```typescript
import { setRuntime, InProcessBridge } from '@tywrapped/numpy';
setRuntime(new InProcessBridge()); // Use node-calls-python for speed
import { setRuntimeBridge, NodeBridge } from 'tywrap';

setRuntimeBridge(new NodeBridge()); // Explicit Node.js subprocess runtime
```

### Runtime Packages

Scope policy:
- `@tywrapped/*` are public library wrappers (e.g., `@tywrapped/numpy`, exports `setRuntime` and `InProcessBridge`)
- `@tywrap/*` are runtime and infrastructure packages (e.g., `@tywrap/runtime-subprocess`, `@tywrap/runtime-inprocess`, `@tywrap/runtime-pyodide`)
- `@tywrapped/*` are public library wrappers (e.g., `@tywrapped/numpy`, exports `setRuntime` and wrapper APIs)
- `@tywrap/*` are runtime and infrastructure packages (e.g., `@tywrap/runtime-subprocess`, `@tywrap/runtime-inprocess`, `@tywrap/runtime-pyodide`) and export runtime implementations like `InProcessBridge`

- `@tywrap/runtime-subprocess` - Bundled by default, no native dependencies
- `@tywrap/runtime-inprocess` - Optional, requires node-calls-python native addon
Expand All @@ -247,6 +310,15 @@ Packages declare supported runtimes in metadata:

Visible in README badge, npm package metadata, and package documentation.

#### Validation & enforcement (template repo)

- Automated validation: add a `browser-compat` CI job that runs bundler builds (esbuild/rollup/webpack) and Playwright smoke tests for packages that declare `runtimes: ["browser"]` (or `["node", "browser"]`).
- Enforcement at package boundary:
- Build-time: fail CI if declared runtimes don't have matching artifacts/entry points.
- Runtime: add a guard in Node-only entry points that throws a clear error when imported in the browser.
- Publish-time checks: lint `package.json` `runtimes`, generate README badges, and fail publish if declared runtime artifacts are missing.
- Pyodide handling: for browser-enabled packages, lazy-load Pyodide/WASM and keep heavy Python deps optional; include a headless-browser smoke test that imports the package and initializes the runtime.

---

## Initial Launch Libraries
Expand Down Expand Up @@ -286,10 +358,18 @@ Criteria for selection:
- 15% - Bounties for new packages, test improvements, documentation
- 5% - Reserve for legal/unexpected expenses

#### Governance and Fund Allocation

- Decision body: technical committee + maintainers (define this in a governance charter).
- Maintainer stipends (60%): split using a simple, auditable formula (e.g., base stipend per maintained package + a variable component tied to monthly activity); publish the formula.
- Bounties (15%): bounties are proposed via issues, approved by the committee within a fixed SLA (e.g., 7 days), and paid on merge + release of the deliverable.
- Reserve (5%): only tapped for legal/incidents; require explicit approval (e.g., 2-of-3 committee vote) and a public rationale.
- Transparency: publish quarterly reports + a public ledger of disbursements; define a dispute-resolution path (mediation → committee decision → community escalation).

### Trademark: Register Early

- File trademark application before public launch
- Estimated cost: $250-400 per class
- Estimated cost (fees-only): ~$250-400 per class (USPTO filing fees); budget ~$1,200-3,000+ per class including search + attorney filing/prosecution and possible office actions
- Protects brand from squatting
- Establishes legitimacy for enterprise adoption
- Required classes: software development tools, software distribution
Expand Down Expand Up @@ -332,6 +412,16 @@ interface Transport extends Disposable {

`BridgeProtocol` is already transport-agnostic - it accepts any `Transport` implementation.

#### Large-data handling notes

The current `Transport` interface is string-based, which is great for simplicity but has practical limits for multi-GB tensors/DataFrames.

High-level strategies (future work):
- Binary channel: add optional `sendBinary(...)` support (or extend `send(...)` to accept `ArrayBuffer`) and use `Transferable` objects where available.
- Out-of-band transfer: for large payloads, send references (file paths, HTTP URLs, object-store URIs) instead of inlining data.
- Streaming/chunking: add `streamStart`/`streamChunk`/`streamEnd` semantics with backpressure and `AbortSignal`.
- Capability negotiation: have transports advertise features (binary/streaming) via a meta call (e.g., `getBridgeInfo()` capabilities).

### Adding node-calls-python Support

**Recommended approach: JSON passthrough**
Expand Down