feat: enable arbitrary json in the ingest apis

[callicles]

Note

Enables IngestApi/Stream to accept arbitrary extra fields (index signatures), pass them through to streaming functions, and document/test the behavior.

• Core (Rust)
  • Add allow_extra_fields to DataModel/PartialIngestApi and propagate into ingress routes.
  • Update JSON parsing to use DataModelVisitor::new_with_extra_fields when allowed; DataModelVisitor now optionally passes unknown keys through; nested/NamtTuples still strict.
• TypeScript Compiler/SDK
  • toColumns supports allowIndexSignatures option; compiler plugin permits index signatures for IngestApi, Stream, and IngestPipeline (when no table) and injects allowExtraFields.
  • SDK (TypedBase, Stream, IngestApi, IngestPipeline) constructors accept/propagate allowExtraFields; infra JSON includes allowExtraFields.
• Templates/Tests
  • Add TS test models and transform to capture extra fields into properties JSON; add UserEventOutput table schema.
  • E2E test validates extra fields flow from ingest → stream → transform → JSON column.
• Docs
  • Add "Accepting Arbitrary Fields" sections (TS index signatures, Python extra fields) to Ingest API docs (v1 and v2).
• Misc
  • Minor lockfile metadata adjustments.

^{Written by Cursor Bugbot for commit 7634f0c. This will update automatically on new commits. Configure here.}

[linear]

ENG-1617 Support index signatures in IngestApi data models

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Preview	Comments	Updated (UTC)
docs-v2	Ready	Preview	Comment	Dec 2, 2025 4:56am
framework-docs	Ready	Preview	Comment	Dec 2, 2025 4:56am

[Copilot]

Pull request overview

This PR enables IngestApi and Stream resources to accept payloads with arbitrary additional fields beyond their defined schema using TypeScript index signatures and Python's extra='allow' configuration. The API validates known fields while gracefully accepting extra fields without validation errors.

Key Changes:

• Modified the TypeScript compiler to allow index signatures for IngestApi and Stream types while extracting only named properties as columns
• Added comprehensive test coverage including templates, transforms, and E2E tests
• Updated documentation for both TypeScript and Python implementations

Reviewed changes

Copilot reviewed 8 out of 9 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
packages/ts-moose-lib/src/dataModels/typeConvert.ts	Added ToColumnsOptions interface with allowIndexSignatures flag to conditionally permit index signatures
packages/ts-moose-lib/src/dmv2/dataModelMetadata.ts	Enabled allowIndexSignatures for IngestApi and Stream resource types
templates/typescript-tests/src/ingest/indexSignatureTests.ts	Created test types and resources demonstrating index signature usage with IngestApi
templates/typescript-tests/src/ingest/transforms.ts	Added transform mapping UserEventInput to UserEventOutput with logging
apps/framework-cli-e2e/test/utils/schema-definitions.ts	Added expected schema for UserEventOutput table
apps/framework-cli-e2e/test/templates.test.ts	Added E2E test validating ingestion and transform of payloads with extra fields
apps/framework-docs/src/pages/moose/apis/ingest-api.mdx	Added "Accepting Arbitrary Fields" documentation section
apps/framework-docs-v2/content/moosestack/apis/ingest-api.mdx	Added "Accepting Arbitrary Fields" documentation section

Files not reviewed (1)

• pnpm-lock.yaml: Language not supported

[phiSgr]

Unknown fields are skipped in rust.

moosestack/apps/framework-cli/src/utilities/validate_passthrough.rs

Line 1055 in 9bb2167

map.next_value::<serde::de::IgnoredAny>()?;