feat: implicit argument derivation and structural record synthesis#5903
feat: implicit argument derivation and structural record synthesis#5903
Conversation
When no direct implicit match exists, the compiler now derives implicit arguments from functions that themselves have implicit parameters. For example, if `Array.compare<T>` has an implicit `(T, T) -> Order`, the compiler can automatically derive a `([Nat], [Nat]) -> Order` by instantiating `T := Nat` and recursively resolving `Nat.compare`. This works transitively (e.g. `[[Nat]]` derives through two levels) and with both polymorphic and monomorphic candidates. Resolution depth is bounded by `--implicit-derivation-depth` (default: 5). Error messages distinguish missing vs ambiguous inner implicits and report which candidate was tried and which inner implicit failed. Made-with: Cursor
… ambiguous candidates
…rameters and enhance error reporting for ambiguous implicit arguments - Removed the `inst` parameter from `synthesize_derived_wrapper` as it was not utilized. - Improved error messages for ambiguous implicit arguments, updating error codes and notes to provide clearer context on the candidates involved. - Updated test cases to reflect changes in error reporting for ambiguous implicit derivations.
|
Cool, any ideas how handle record and variants, where you don't have a function predefined? |
I think that records and variants need an explicit compare function to say
Then we could have helper functions defined in core (or compiler-derived) and use like: type Foo = {
name : Text;
id : Nat;
subs : List.List<Bar>;
};
module FooCompareByNameAndId {
public func compare(a : Foo, b : Foo) : Order.Order {
a.compareBy(b, func (r) { (r.name, r.id) })
// `compareBy` should be defined in `Order.mo`
}
};So that a record/variant comparison could be defined via tuple comparison. |
- Updated the implicit derivation process to clarify the search order for candidates, prioritizing direct local values over derived ones. - Improved documentation in the language manual and implicit parameters guide to reflect changes in implicit derivation and resolution order. - Fixed minor typos and inconsistencies in comments and documentation across multiple files. - Adjusted test cases to ensure clarity and accuracy in expected outcomes for implicit derivation scenarios.
- Introduced a new function `display_typ_oneline` for concise type display. - Updated `render_derivation_tree` to improve handling of error messages, including clearer differentiation between failed and attempted implicit derivations. - Adjusted test cases to reflect changes in error reporting format, ensuring consistency in the output for ambiguous and not found cases.
- Removed the unused `~tldr` parameter from `render_derivation_tree` for clarity. - Simplified the handling of inner errors by always including the detailed type display. - Adjusted indentation logic for better readability in error reporting.
…e types Detect recursion during implicit derivation by tracking in-progress goals in a shared mutable list. When a cycle is found, return a VarE referencing the already-in-progress function. After resolution succeeds, wrap the result in a BlockE with mutually recursive LetD bindings so all generated $derived_implicit_N functions can reference each other. Made-with: Cursor
…riant helpers Demonstrates how Order.compareBy, Variant3, and Variant4 reduce compare boilerplate for records and variants. Uses mutable fields, optional List<Text> payloads, and two alternative sort orderings. Made-with: Cursor
Made-with: Cursor
- Use `is_matching_typ` in `matching_vals` and `matching_fields` instead of inline `T.sub` (eliminates dead helper) - Refactor `synthesize_derived_wrapper` from mutable refs to `List.fold_left`, making data flow explicit - Document `Local/Returns` restriction on derivation candidates - Update `render_derivation_tree` comment (kept pending full review) - Fix trailing whitespace - Add subtyping edge-case test: `Int.compare` satisfies inner `(Nat, Nat) -> Order` implicit via function contravariance - Add PR number (#5903) to changelog - Accept pre-existing tc-human indentation drift Made-with: Cursor
- Add comment explaining why `assert false` in `resolve_hole` is safe (depends on no-backtracking invariant) - Fix changelog to say "functions (possibly polymorphic)" instead of "polymorphic functions" since derivation also works for monomorphic functions with implicit parameters Made-with: Cursor
…docs
typing.ml:
- Remove dead render_derivation_tree function (was marked TODO, never called)
- Fix try_derive_structural: return Committed(Error(DepthLimited)) instead of
Empty when the depth limit is hit, so the error message includes the depth
hint ("increase with --implicit-derivation-depth") rather than falling through
to a generic "Cannot determine implicit" message
- Replace assert(eligible_ids=[]) with a comment explaining the invariant
docs:
- 16-language-manual.md: add Structural derivation paragraph to the implicit
resolution section (parameter name __record convention, reserved names)
- 11-implicit-parameters.md: add tiers 7-9 to resolution order, add Structural
derivation subsection with __record convention and example, add performance
note, fix four typos, clarify the parameter-name signal wording
- Changelog.md: fix wording ("without writing a serialiser for each record type")
tests:
- test/fail/implicit-derivation-structural-depth.mo: new fail test verifying
the depth-limit error message for structural derivation (nested records)
- test/run/implicit-derivation-structural.mo: new end-to-end runtime test
(no SKIP comp) exercising structural synthesis through full Wasm compilation;
verifies field ordering and three different field types (Text, Nat, Bool)
Made-with: Cursor
…able field test
- Add `record_fields <> [] &&` to the DepthLimited guard in try_derive_structural,
matching try_commit's `h.holes <> [] &&` guard. An empty record {} should
synthesise trivially at any depth, not return DepthLimited.
- Add mutable-field case to the structural runtime test, verifying that T.as_immut
correctly strips var wrappers before per-field implicit resolution.
Made-with: Cursor
The test_src filter in nix/tests.nix only included core-stub/ and base-stub/ but not json-stub/. The three structural derivation fail tests use --package json ../json-stub/src, which didn't exist in the Nix sandbox on CI, causing all three tests to fail. Made-with: Cursor
The human-format note indentation differs between the local macOS build and the Linux CI build. Revert the six .tc-human.ok files modified in the review-findings commit back to the original 17-space indentation that matches the CI moc output. Made-with: Cursor
…ents - disambiguate_resolutions: return computed Pareto frontier on ambiguity instead of the original candidate list (affected error messages only) - Changelog: mention __tuple structural derivation alongside __record - Language manual: clarify direct-library-fields tier precedes derivation - 11-implicit-parameters.md: fix typos (comma splice, "the call site", stray space, missing semicolon), fix broken contextual-dot link, normalize to American spelling (synthesize, serialization) - fail/implicit-derivation-json.mo: update comments, re-accept .ok files Made-with: Cursor
| |----------------|-----------------------|------------------------------------------|------------------------------------------------| | ||
| | `__record` | `[(Text, E)] -> R` | `Rec -> R` or `(Rec, Rec) -> R` | Record: one or two records, arity from hole | | ||
| | `__tuple` | `[E] -> R` | `(A, B, ...) -> R` or `((A,B,...), (A,B,...)) -> R` (≥ 2 elements) | Tuple: one implicit per element | | ||
| | `__variant` | — | — | Reserved for future extension | |
There was a problem hiding this comment.
| `__variant` | — (Nat, Text, () -> R) | — `Var -> R` or `(Var, Var) -> R` | Reserved for future extension |
Might work for variants.... (Nat is the ordinal of the tag, which, for records, can be inferred from the array index)
Maybe arrays should just be delayed (or use an iterator if ok with forcing the ordering)
[(Text, () -> E)] -> R
There was a problem hiding this comment.
That's a really good idea to be go with lazy values () -> R, it'd give more flexibility to package authors (and they are the ones who write these __record/tuple/variant functions).
Better than Iter because they can choose which one to evaluate and when, yet still simple enough!
Just for the record: I'd implement variants in another PR. But I'm curious - Why would anyone need the Nat ordinal of the tag? Aren't we sorting tags in a variant anyway so that would make it tricky to preserve, right? Or am I misunderstanding or missing sth?
There was a problem hiding this comment.
For efficient compare I would compare the ordinals before comparing the contents when ordinals match. Hash would work too but maybe more arbitrary and implementation specific than the ordinal of the label in the variants (which should be know statically, right?).
Real-world nested types (e.g. records containing records containing tuples) easily exceed depth 5. A limit of 100 is generous enough for any practical nesting while still guaranteeing termination. Made-with: Cursor
Structural combiners now receive lazy thunks instead of eagerly evaluated values: __record : [(Text, () -> E)] -> R __tuple : [() -> E] -> R The compiler wraps each per-field/element implicit call in a thunk, giving the combiner full control over evaluation order. Combiners that can short-circuit (compare, eq) now genuinely skip remaining fields; combiners that need everything (toJson, describe) call each thunk eagerly. Also consolidates AST synthesis helpers into a SynthesizeWrapper module, eliminating duplication across derived_wrapper, record_wrapper, and tuple_wrapper. Made-with: Cursor
Reflect the new structural combiner signatures ([(Text, () -> T)] -> R and [() -> T] -> R) and the depth limit bump to 100. Made-with: Cursor
Apply @@, @~, @?, @! shorthand operators for AST node construction, let open Lib.Option.Syntax for monadic Option chaining, and share SynthesizeWrapper.var for synthesized variable references. Adapted from #5921. Made-with: Cursor
- Add Lib.Result.Syntax with let* for Result.bind - Replace structural_kind enum + dom with polymorphic variant `Record of T.field list | `Tuple of T.typ list carrying pre-extracted data, eliminating redundant T.promote + assert false re-extraction - Extract is_matching_structural_combiner to share combiner-matching logic between module-field and val candidate sources - Unify try_derive and try_derive_structural via try_derive_with, centralizing depth-limit checks and disambiguate/commit flow Made-with: Cursor
…vation Made-with: Cursor
Allow the compiler to derive implicit arguments from functions that themselves have implicit parameters (e.g. `compare` for `[Nat]` from `Array.compare<Nat>` + `Nat.compare`). Works transitively and is depth-limited via `--implicit-derivation-depth`. Structural derivation (`__record`/`__tuple` combiners) will follow in a separate PR. Made-with: Cursor
Allow the compiler to derive implicit arguments from functions that themselves have implicit parameters (e.g. `compare` for `[Nat]` from `Array.compare<Nat>` + `Nat.compare`). Works transitively and is depth-limited via `--implicit-derivation-depth`. Structural derivation (`__record`/`__tuple` combiners) will follow in a separate PR. Made-with: Cursor
Allow the compiler to derive implicit arguments from functions that themselves have implicit parameters (e.g. `compare` for `[Nat]` from `Array.compare<Nat>` + `Nat.compare`). Works transitively and is depth-limited via `--implicit-derivation-depth`. Structural derivation (`__record`/`__tuple` combiners) will follow in a separate PR. Made-with: Cursor
| | _ -> None in | ||
| match T.promote candidate_typ with | ||
| | T.Func (T.Local, T.Returns, [], [T.Named ("__record", inner_typ)], [ret_typ]) -> | ||
| (match T.promote inner_typ with |
There was a problem hiding this comment.
dubious promote on negative occurrence?
| with_thunk_elem `Record thunk_typ ret_typ | ||
| | _ -> None) | ||
| | T.Func (T.Local, T.Returns, [], [T.Named ("__tuple", inner_typ)], [ret_typ]) -> | ||
| (match T.promote inner_typ with |
There was a problem hiding this comment.
Ditto: dubious promote on negative occurrence?
Summary
Extends the Motoko compiler's implicit argument resolution with two new mechanisms:
1. Implicit derivation from functions with inner implicits
The compiler can now resolve implicit arguments by composing them from functions (possibly polymorphic) that themselves have implicit parameters. For example, an implicit
comparefor[Nat]is automatically derived fromArray.compare<Nat>+Nat.compare, eliminating boilerplate wrapper modules. Works transitively (e.g.,[[Nat]]). Depth-limited with--implicit-derivation-depth(default: 5).Before:
After:
2. Structural derivation for records and tuples
A function whose sole explicit parameter is named
__record(typed[(Text, T)] -> R) or__tuple(typed[T] -> R) acts as a structural combiner. When the compiler needs an implicit for a record or tuple type, it automatically decomposes the type, resolves a per-field/per-element implicit (using the same search label), and synthesizes a wrapper. Both unary (X -> R) and binary ((X, X) -> R) hole types are supported.Example — generic JSON serialization for any record:
The
Jsonpackage defines a single_toJson(__record : [(Text, Json)]) : Jsoncombiner. That's all — the compiler handles every record type automatically, as long as each field type has a_toJsoninstance.Resolution order
Direct matches are always preferred over derived ones. Within each tier, the most specific candidate (by subtyping) wins:
--implicit-package)--implicit-package)__record/__tuplecombiner)--implicit-package)Test plan
Run tests
test/run/implicit-derivation.mo— basic derivation, monomorphic, polymorphic, multiple implicits, priority ordering, subtypingtest/run/implicit-derivation-transitive.mo—[[Nat]]through two derivation levelstest/run/implicit-derivation-recursive.mo— single recursion, mutual recursion, recursive treestest/run/implicit-derivation-record-variant.mo— real-world record/variant comparison withcompareBytest/run/implicit-derivation-core.mo— integration withmo:corelibrary (Array,Nat,Int,Text)test/run/implicit-derivation-implicit-package.mo— derivation from unimported library modulestest/run/implicit-derivation-json.mo— structural derivation for JSON serialization (records, tuples, maps, lists)test/run/implicit-derivation-structural.mo— basic structural record derivation (unary)test/run/implicit-derivation-structural-compare.mo— binary structural derivation for record/tuple comparisontest/run/implicit-derivation-structural-dispatch.mo—__recordvs__tupledispatch and binary/unary disambiguationFail tests (14 files)
test/fail/implicit-derivation.mo— missing inner implicit, basic derivation errorstest/fail/implicit-derivation-ambiguous.mo— ambiguous derived candidates at the head leveltest/fail/implicit-derivation-ambiguous-deep.mo— ambiguous inner implicit during derivationtest/fail/implicit-derivation-bimatch.mo— bi-matching limits during type instantiationtest/fail/implicit-derivation-deep.mo— derivation chain errors at multiple depthstest/fail/implicit-derivation-depth.mo—--implicit-derivation-depthlimit hittest/fail/implicit-derivation-no-backtracking.mo— no backtracking across derivation candidatestest/fail/implicit-derivation-json.mo— real-world JSON errors: missing leaves, expanded Map types, nested recordstest/fail/implicit-derivation-structural-missing.mo— record with a field type that has no instancetest/fail/implicit-derivation-structural-ambiguous.mo— two structural combiners in scopetest/fail/implicit-derivation-structural-hint.mo— structural combiner in an unimported library (import hint)test/fail/implicit-derivation-structural-depth.mo— structural derivation depth limittest/fail/implicit-derivation-structural-record2-missing.mo— binary record derivation with missing field instancetest/fail/implicit-derivation-structural-tuple-missing.mo— tuple derivation with missing element instanceTest package
test/json-stub/— minimal JSON package used by the structural derivation tests; demonstrates the__recordconvention and companion per-type modules (IntJson,TextJson,ArrayJson,ListJson,MapJson,Tuple2Json,Tuple3Json)