|
| 1 | +# M033: ORM Expressiveness & Schema Extras |
| 2 | + |
| 3 | +**Gathered:** 2026-03-24 |
| 4 | +**Status:** Ready for planning |
| 5 | + |
| 6 | +## Project Description |
| 7 | + |
| 8 | +M033 is the follow-on to M032. Its job is to strengthen Mesh's data layer against the real recurring pressure still visible in `mesher/`, not to reopen solved Mesh-language folklore or redesign the app. |
| 9 | + |
| 10 | +The intended shape is now clearer than the draft: |
| 11 | +- a **broader neutral core** for honestly reusable query/update/insert/select expression work |
| 12 | +- that broader core should take the form of a **new expression DSL**, not just a few narrow helper methods |
| 13 | +- **explicit Postgres extras** for behavior that is genuinely PG-specific |
| 14 | +- a clean path for **SQLite extras later**, after this extension shape is proven |
| 15 | + |
| 16 | +The user still does **not** want fake portability, does **not** want a PG-only trap, and does **not** want a purity chase. The target is pragmatic raw SQL / DDL reduction with a short justified keep-list, while keeping `mesher/` behaviorally stable and using it as the proof surface. |
| 17 | + |
| 18 | +## Why This Milestone |
| 19 | + |
| 20 | +M032 already did the cleanup pass that this work depends on: it retired stale workaround folklore, separated real Mesh/tooling blockers from real data-layer pressure, and left a short honest keep-list anchored in live Mesher files. |
| 21 | + |
| 22 | +That means the remaining friction is no longer “Mesh cannot do this at the language level.” The remaining friction is that `Repo`, `Query`, and `Migration` still cannot honestly express several recurring shapes that `mesher/` already uses today: expression-heavy updates, JSONB-heavy read/write paths, parameterized select expressions, recurring subquery patterns, and partition lifecycle DDL. |
| 23 | + |
| 24 | +This milestone needs to happen now because the repo already has a truthful pressure map. Leaving those boundaries as permanent app-local raw SQL / DDL after M032 would amount to knowingly preserving the next wave of dogfood pain instead of turning it into platform capability. |
| 25 | + |
| 26 | +## User-Visible Outcome |
| 27 | + |
| 28 | +### When this milestone is complete, the user can: |
| 29 | + |
| 30 | +- rewrite a meaningful share of `mesher/storage/queries.mpl` and `mesher/storage/writer.mpl` from raw SQL to stronger Mesh data-layer surfaces without changing Mesher behavior |
| 31 | +- manage Mesher's partitioned-events schema through honest migration / PG helper surfaces instead of the current `PARTITION BY` and partition-maintenance raw DDL keep-sites |
| 32 | + |
| 33 | +### Entry point / environment |
| 34 | + |
| 35 | +- Entry point: `mesher/` data layer, targeted compiler/runtime tests, and live Mesher flows against the rewritten storage paths |
| 36 | +- Environment: local dev with a live Postgres database |
| 37 | +- Live dependencies involved: PostgreSQL now; SQLite is a design constraint for later extras, not a live dependency for M033 closeout |
| 38 | + |
| 39 | +## Completion Class |
| 40 | + |
| 41 | +- Contract complete means: targeted compiler/runtime tests and milestone artifacts prove that the new neutral expression DSL covers the recurring honestly reusable cases, the PG extras cover the recurring PG-only families, and the retained raw SQL / DDL keep-list is short and justified |
| 42 | +- Integration complete means: live Postgres-backed Mesher query, write, search, alert, and migration paths still work after the rewrites, with a meaningful share of the current `ORM boundary` and `PARTITION BY` keep-sites retired |
| 43 | +- Operational complete means: real Postgres migration and partition lifecycle operations (create/list/drop) work against a live database and real catalogs, not just SQL builder snapshots; SQLite implementation is not required in this milestone |
| 44 | + |
| 45 | +## Final Integrated Acceptance |
| 46 | + |
| 47 | +To call this milestone complete, we must prove: |
| 48 | + |
| 49 | +- a live Postgres-backed Mesher path can ingest and store events, upsert/query the affected issue and event data, and preserve the same user-visible behavior while the underlying storage code uses the new neutral expression DSL and explicit PG extras instead of today's recurring raw SQL |
| 50 | +- a live Postgres-backed Mesher path can exercise the JSONB-heavy, search-heavy, and alert-rule write/read flows that currently sit on `ORM boundary` comments, with targeted runtime/e2e tests covering the new reusable query/update/insert/select surfaces underneath |
| 51 | +- partition lifecycle work cannot be treated as done through mocks or SQL-string tests alone; the milestone must run create/list/drop partition behavior against a real Postgres database because catalog behavior and partition DDL are part of the truth surface |
| 52 | + |
| 53 | +## Risks and Unknowns |
| 54 | + |
| 55 | +- The new expression DSL could drift into a giant abstract ORM detached from real `mesher/` pressure — that would violate the dogfood-first bar and likely make the API worse rather than better |
| 56 | +- A broader neutral core can easily blur the line between honest reusable behavior and PG-only behavior — if that line is wrong, the milestone recreates the fake-portability problem it was supposed to fix |
| 57 | +- PG lifecycle helpers for partitions can easily hardcode the wrong abstraction boundary — if they leak into the neutral core, later SQLite extras may require backing out the design |
| 58 | +- Aggressive rewrites in `mesher/` could accidentally change behavior, data shape, or operational signals — M033 is supposed to improve the platform underneath the app, not smuggle in product redesign |
| 59 | +- A “retire everything” mentality would turn the milestone into a purity chase — the success bar remains a short justified keep-list, not zero raw SQL or zero raw DDL |
| 60 | + |
| 61 | +## Existing Codebase / Prior Art |
| 62 | + |
| 63 | +- `mesher/storage/queries.mpl` — the concentrated `ORM boundary` map for computed `ON CONFLICT` updates, function-valued insert/update expressions, parameterized select expressions, multi-subquery reads, full-text search, JSONB-heavy read/write paths, and partition cleanup helpers |
| 64 | +- `mesher/storage/writer.mpl` — the insert-side JSONB extraction boundary and the storage-local dogfood surface that should benefit from stronger insert expressions |
| 65 | +- `mesher/migrations/20260216120000_create_initial_schema.mpl` — the current migration-time `PARTITION BY` keep-site plus existing PG-specific extension/index prior art |
| 66 | +- `compiler/mesh-rt/src/db/query.rs` — the current query surface already supports joins, raw where/select, and a limited subquery path; this is the starting point for the broader neutral expression DSL |
| 67 | +- `compiler/mesh-rt/src/db/repo.rs` — the current insert/update/delete/upsert surfaces show the literal-field-map bias and the raw escape hatches M033 is meant to reduce |
| 68 | +- `compiler/mesh-rt/src/db/migration.rs` — the current neutral migration baseline plus raw execute escape hatch; the obvious home for honest PG lifecycle helpers |
| 69 | +- `.gsd/milestones/M032/M032-SUMMARY.md` — the authoritative handoff separating supported-now Mesh behavior from the real M033 data-layer boundary families |
| 70 | + |
| 71 | +> See `.gsd/DECISIONS.md` for all architectural and pattern decisions — it is an append-only register; read it during planning, append to it during execution. |
| 72 | +
|
| 73 | +## Relevant Requirements |
| 74 | + |
| 75 | +- R036 — advances the neutral-core-plus-explicit-extras contract by deciding that M033 should build a broader neutral core through a new expression DSL while keeping PG-only behavior explicit |
| 76 | +- R037 — advances the PG-specific query/migration requirement by explicitly targeting JSONB-heavy paths, expression-heavy updates, full-text search, pgcrypto-adjacent helpers, and partition lifecycle work |
| 77 | +- R038 — advances the pragmatic cleanup bar by aiming to retire a meaningful share of the current raw SQL / DDL sites while keeping a short justified keep-list |
| 78 | +- R039 — advances migration and schema capability by requiring real partition lifecycle coverage instead of stopping at the current `PARTITION BY` raw DDL note |
| 79 | +- R040 — advances the SQLite-path constraint by requiring M033's design to leave a clean extension path instead of baking PG assumptions into the wrong layer |
| 80 | + |
| 81 | +## Scope |
| 82 | + |
| 83 | +### In Scope |
| 84 | + |
| 85 | +- build a broader neutral core for honestly reusable expression-heavy query/update/insert/select work, even if that requires a new expression DSL rather than only small helper additions |
| 86 | +- cover the must-win neutral-core families now: expression-heavy updates/inserts, parameterized select expressions, and honest reusable subquery forms that retire recurring Mesher raw SQL |
| 87 | +- cover JSONB-heavy data paths used in `mesher/` where the reusable part belongs in the broader neutral core and the PG-only part belongs in explicit extras |
| 88 | +- add explicit PG extras for the real recurring PG-only families: full-text search, pgcrypto-adjacent helpers, partition lifecycle helpers, and other genuinely PG-specific behavior |
| 89 | +- add PG lifecycle helpers for partition work all the way through the real create/list/drop path, while keeping truly dynamic escape hatches explicit when necessary |
| 90 | +- clean up the small sharp gaps that are cheap and honest to solve during this wave, such as `now()`-driven updates and other narrow recurring storage edges |
| 91 | +- rewrite a meaningful share of the current `mesher/storage/queries.mpl` and `mesher/storage/writer.mpl` raw SQL sites onto the stronger surfaces while keeping Mesher behavior stable |
| 92 | +- keep docs dogfood-first: public docs/examples should stay truthful and not drift, but M033 is not primarily a docs milestone |
| 93 | + |
| 94 | +### Out of Scope / Non-Goals |
| 95 | + |
| 96 | +- fake portability that hides PG-only behavior inside a misleading neutral API |
| 97 | +- a PG-only trap that makes future SQLite extras awkward or forces a later redesign |
| 98 | +- raw-SQL purity or near-zero raw DDL as the goal |
| 99 | +- product redesign in `mesher/` |
| 100 | +- a giant abstract ORM disconnected from the actual recurring Mesher pressure map |
| 101 | +- full SQLite-specific extras in this milestone |
| 102 | +- a broad public-docs push beyond light truth-sync needed to keep the new surfaces legible |
| 103 | + |
| 104 | +## Technical Constraints |
| 105 | + |
| 106 | +- The broader neutral core is allowed to be ambitious, but it still has to be justified by recurring real Mesher pressure rather than abstraction for its own sake |
| 107 | +- Vendor-specific behavior must stay explicit where the capability is not honestly portable, especially around JSONB, full-text search, pgcrypto, partition lifecycle, and catalog behavior |
| 108 | +- `mesher/` should remain behaviorally stable from the product point of view while the platform underneath it improves |
| 109 | +- M033 must leave a clean SQLite extension path even though SQLite extras themselves are deferred |
| 110 | +- Acceptance has to include live Postgres proof, not compile-only proof, SQL-string snapshots alone, or comment-level cleanup alone |
| 111 | +- Truly dynamic or rare catalog behavior may remain an explicit escape hatch if a first-class surface would be dishonest or too specific |
| 112 | + |
| 113 | +## Integration Points |
| 114 | + |
| 115 | +- `mesher/` — primary proof surface and the main consumer of the stronger data-layer surfaces |
| 116 | +- `compiler/mesh-rt/src/db/query.rs` — neutral query/expression surface to expand |
| 117 | +- `compiler/mesh-rt/src/db/repo.rs` — neutral insert/update/delete/upsert surface to expand or reshape around the new expression DSL |
| 118 | +- `compiler/mesh-rt/src/db/migration.rs` — migration baseline plus explicit PG lifecycle helper surface |
| 119 | +- PostgreSQL — required live integration target for JSONB, `ON CONFLICT`, `now()`, full-text search, pgcrypto, partitioned tables, and catalog-backed partition lifecycle behavior |
| 120 | +- Mesh compiler/runtime tests — required lower-level proof surface for the new neutral DSL and explicit PG helpers |
| 121 | +- SQLite — not a live M033 target, but the boundary decisions here must preserve a credible later explicit-extras path |
| 122 | + |
| 123 | +## Open Questions |
| 124 | + |
| 125 | +- How far the first neutral expression DSL pass should go on the hardest multi-derived-table / multi-scalar-subquery read shapes in `mesher/storage/queries.mpl` — current thinking: cover the recurring shapes that retire multiple real Mesher sites, but stop before inventing a fake universal SQL AST |
| 126 | +- Whether any truly dynamic partition/catalog maintenance beyond the common create/list/drop path should stay as explicit raw SQL after PG lifecycle helpers land — current thinking: the helpers should own the common Mesher path, and rare catalog-specific operations can remain raw if the abstraction starts lying |
0 commit comments