Skip to content

feat: implicit argument derivation and structural record synthesis#5903

Open
Kamirus wants to merge 55 commits intomasterfrom
kamil/implicit-derivation
Open

feat: implicit argument derivation and structural record synthesis#5903
Kamirus wants to merge 55 commits intomasterfrom
kamil/implicit-derivation

Conversation

@Kamirus
Copy link
Copy Markdown
Contributor

@Kamirus Kamirus commented Mar 11, 2026

Summary

Extends the Motoko compiler's implicit argument resolution with two new mechanisms:

1. Implicit derivation from functions with inner implicits

The compiler can now resolve implicit arguments by composing them from functions (possibly polymorphic) that themselves have implicit parameters. For example, an implicit compare for [Nat] is automatically derived from Array.compare<Nat> + Nat.compare, eliminating boilerplate wrapper modules. Works transitively (e.g., [[Nat]]). Depth-limited with --implicit-derivation-depth (default: 5).

Before:

module MyArray {
  public func compare(a : [Nat], b : [Nat]) : Order { Array.compare(a, b) };
};
let m = Map.empty<[Nat], Text>();
m.add([1, 2, 3], "abc"); // uses MyArray.compare

After:

let m = Map.empty<[Nat], Text>();
m.add([1, 2, 3], "abc"); // just works — derived from Array.compare<Nat> + Nat.compare

2. Structural derivation for records and tuples

A function whose sole explicit parameter is named __record (typed [(Text, T)] -> R) or __tuple (typed [T] -> R) acts as a structural combiner. When the compiler needs an implicit for a record or tuple type, it automatically decomposes the type, resolves a per-field/per-element implicit (using the same search label), and synthesizes a wrapper. Both unary (X -> R) and binary ((X, X) -> R) hole types are supported.

Example — generic JSON serialization for any record:

import Json "mo:json/Json";
import IntJson "mo:json/IntJson";
import TextJson "mo:json/TextJson";

type Person = { name : Text; age : Int };

let p : Person = { name = "Alice"; age = 30 };
let json = p.toJson();
// #obj([("name", #text "Alice"), ("age", #number 30)])

The Json package defines a single _toJson(__record : [(Text, Json)]) : Json combiner. That's all — the compiler handles every record type automatically, as long as each field type has a _toJson instance.

Resolution order

Direct matches are always preferred over derived ones. Within each tier, the most specific candidate (by subtyping) wins:

  1. Direct local values
  2. Direct module fields
  3. Direct library fields (requires --implicit-package)
  4. Derived from local values
  5. Derived from module fields
  6. Derived from library fields (requires --implicit-package)
  7. Structural from local values (__record / __tuple combiner)
  8. Structural from module fields
  9. Structural from library fields (requires --implicit-package)

Test plan

Run tests

  • test/run/implicit-derivation.mo — basic derivation, monomorphic, polymorphic, multiple implicits, priority ordering, subtyping
  • test/run/implicit-derivation-transitive.mo[[Nat]] through two derivation levels
  • test/run/implicit-derivation-recursive.mo — single recursion, mutual recursion, recursive trees
  • test/run/implicit-derivation-record-variant.mo — real-world record/variant comparison with compareBy
  • test/run/implicit-derivation-core.mo — integration with mo:core library (Array, Nat, Int, Text)
  • test/run/implicit-derivation-implicit-package.mo — derivation from unimported library modules
  • test/run/implicit-derivation-json.mo — structural derivation for JSON serialization (records, tuples, maps, lists)
  • test/run/implicit-derivation-structural.mo — basic structural record derivation (unary)
  • test/run/implicit-derivation-structural-compare.mo — binary structural derivation for record/tuple comparison
  • test/run/implicit-derivation-structural-dispatch.mo__record vs __tuple dispatch and binary/unary disambiguation

Fail tests (14 files)

  • test/fail/implicit-derivation.mo — missing inner implicit, basic derivation errors
  • test/fail/implicit-derivation-ambiguous.mo — ambiguous derived candidates at the head level
  • test/fail/implicit-derivation-ambiguous-deep.mo — ambiguous inner implicit during derivation
  • test/fail/implicit-derivation-bimatch.mo — bi-matching limits during type instantiation
  • test/fail/implicit-derivation-deep.mo — derivation chain errors at multiple depths
  • test/fail/implicit-derivation-depth.mo--implicit-derivation-depth limit hit
  • test/fail/implicit-derivation-no-backtracking.mo — no backtracking across derivation candidates
  • test/fail/implicit-derivation-json.mo — real-world JSON errors: missing leaves, expanded Map types, nested records
  • test/fail/implicit-derivation-structural-missing.mo — record with a field type that has no instance
  • test/fail/implicit-derivation-structural-ambiguous.mo — two structural combiners in scope
  • test/fail/implicit-derivation-structural-hint.mo — structural combiner in an unimported library (import hint)
  • test/fail/implicit-derivation-structural-depth.mo — structural derivation depth limit
  • test/fail/implicit-derivation-structural-record2-missing.mo — binary record derivation with missing field instance
  • test/fail/implicit-derivation-structural-tuple-missing.mo — tuple derivation with missing element instance

Test package

  • test/json-stub/ — minimal JSON package used by the structural derivation tests; demonstrates the __record convention and companion per-type modules (IntJson, TextJson, ArrayJson, ListJson, MapJson, Tuple2Json, Tuple3Json)

When no direct implicit match exists, the compiler now derives implicit
arguments from functions that themselves have implicit parameters. For
example, if `Array.compare<T>` has an implicit `(T, T) -> Order`, the
compiler can automatically derive a `([Nat], [Nat]) -> Order` by
instantiating `T := Nat` and recursively resolving `Nat.compare`.

This works transitively (e.g. `[[Nat]]` derives through two levels)
and with both polymorphic and monomorphic candidates. Resolution depth
is bounded by `--implicit-derivation-depth` (default: 5).

Error messages distinguish missing vs ambiguous inner implicits and
report which candidate was tried and which inner implicit failed.

Made-with: Cursor
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Mar 11, 2026

Comparing from c8cf79a to 01de41c:
In terms of gas, no changes are observed in 5 tests.
In terms of size, no changes are observed in 5 tests.

Kamirus added 5 commits March 11, 2026 14:01
…rameters and enhance error reporting for ambiguous implicit arguments

- Removed the `inst` parameter from `synthesize_derived_wrapper` as it was not utilized.
- Improved error messages for ambiguous implicit arguments, updating error codes and notes to provide clearer context on the candidates involved.
- Updated test cases to reflect changes in error reporting for ambiguous implicit derivations.
@crusso
Copy link
Copy Markdown
Contributor

crusso commented Mar 11, 2026

Cool, any ideas how handle record and variants, where you don't have a function predefined?

@Kamirus
Copy link
Copy Markdown
Contributor Author

Kamirus commented Mar 11, 2026

Cool, any ideas how handle record and variants, where you don't have a function predefined?

I think that records and variants need an explicit compare function to say

  1. Which fields take part in the comparison
  2. In which order should the fields be compared

Then we could have helper functions defined in core (or compiler-derived) and use like:

type Foo = {
  name : Text;
  id : Nat;
  subs : List.List<Bar>;
};
module FooCompareByNameAndId {
  public func compare(a : Foo, b : Foo) : Order.Order {
    a.compareBy(b, func (r) { (r.name, r.id) })
    // `compareBy` should be defined in `Order.mo`
  }
};

So that a record/variant comparison could be defined via tuple comparison.
And to support tuple comparison we could either rely on helpers in mo:core in the Tuples module (that would work for tuples up to size N).
Or have the compiler derive those.

Kamirus and others added 21 commits March 12, 2026 09:01
- Updated the implicit derivation process to clarify the search order for candidates, prioritizing direct local values over derived ones.
- Improved documentation in the language manual and implicit parameters guide to reflect changes in implicit derivation and resolution order.
- Fixed minor typos and inconsistencies in comments and documentation across multiple files.
- Adjusted test cases to ensure clarity and accuracy in expected outcomes for implicit derivation scenarios.
- Introduced a new function `display_typ_oneline` for concise type display.
- Updated `render_derivation_tree` to improve handling of error messages, including clearer differentiation between failed and attempted implicit derivations.
- Adjusted test cases to reflect changes in error reporting format, ensuring consistency in the output for ambiguous and not found cases.
- Removed the unused `~tldr` parameter from `render_derivation_tree` for clarity.
- Simplified the handling of inner errors by always including the detailed type display.
- Adjusted indentation logic for better readability in error reporting.
…e types

Detect recursion during implicit derivation by tracking in-progress
goals in a shared mutable list. When a cycle is found, return a VarE
referencing the already-in-progress function. After resolution succeeds,
wrap the result in a BlockE with mutually recursive LetD bindings so
all generated $derived_implicit_N functions can reference each other.

Made-with: Cursor
…riant helpers

Demonstrates how Order.compareBy, Variant3, and Variant4 reduce
compare boilerplate for records and variants. Uses mutable fields,
optional List<Text> payloads, and two alternative sort orderings.

Made-with: Cursor
- Use `is_matching_typ` in `matching_vals` and `matching_fields`
  instead of inline `T.sub` (eliminates dead helper)
- Refactor `synthesize_derived_wrapper` from mutable refs to
  `List.fold_left`, making data flow explicit
- Document `Local/Returns` restriction on derivation candidates
- Update `render_derivation_tree` comment (kept pending full review)
- Fix trailing whitespace
- Add subtyping edge-case test: `Int.compare` satisfies inner
  `(Nat, Nat) -> Order` implicit via function contravariance
- Add PR number (#5903) to changelog
- Accept pre-existing tc-human indentation drift

Made-with: Cursor
- Add comment explaining why `assert false` in `resolve_hole` is
  safe (depends on no-backtracking invariant)
- Fix changelog to say "functions (possibly polymorphic)" instead of
  "polymorphic functions" since derivation also works for monomorphic
  functions with implicit parameters

Made-with: Cursor
Kamirus and others added 10 commits March 26, 2026 11:06
…docs

typing.ml:
- Remove dead render_derivation_tree function (was marked TODO, never called)
- Fix try_derive_structural: return Committed(Error(DepthLimited)) instead of
  Empty when the depth limit is hit, so the error message includes the depth
  hint ("increase with --implicit-derivation-depth") rather than falling through
  to a generic "Cannot determine implicit" message
- Replace assert(eligible_ids=[]) with a comment explaining the invariant

docs:
- 16-language-manual.md: add Structural derivation paragraph to the implicit
  resolution section (parameter name __record convention, reserved names)
- 11-implicit-parameters.md: add tiers 7-9 to resolution order, add Structural
  derivation subsection with __record convention and example, add performance
  note, fix four typos, clarify the parameter-name signal wording
- Changelog.md: fix wording ("without writing a serialiser for each record type")

tests:
- test/fail/implicit-derivation-structural-depth.mo: new fail test verifying
  the depth-limit error message for structural derivation (nested records)
- test/run/implicit-derivation-structural.mo: new end-to-end runtime test
  (no SKIP comp) exercising structural synthesis through full Wasm compilation;
  verifies field ordering and three different field types (Text, Nat, Bool)

Made-with: Cursor
…able field test

- Add `record_fields <> [] &&` to the DepthLimited guard in try_derive_structural,
  matching try_commit's `h.holes <> [] &&` guard. An empty record {} should
  synthesise trivially at any depth, not return DepthLimited.
- Add mutable-field case to the structural runtime test, verifying that T.as_immut
  correctly strips var wrappers before per-field implicit resolution.

Made-with: Cursor
The test_src filter in nix/tests.nix only included core-stub/ and base-stub/
but not json-stub/. The three structural derivation fail tests use
--package json ../json-stub/src, which didn't exist in the Nix sandbox on CI,
causing all three tests to fail.

Made-with: Cursor
The human-format note indentation differs between the local macOS build and
the Linux CI build. Revert the six .tc-human.ok files modified in the
review-findings commit back to the original 17-space indentation that matches
the CI moc output.

Made-with: Cursor
…ents

- disambiguate_resolutions: return computed Pareto frontier on ambiguity
  instead of the original candidate list (affected error messages only)
- Changelog: mention __tuple structural derivation alongside __record
- Language manual: clarify direct-library-fields tier precedes derivation
- 11-implicit-parameters.md: fix typos (comma splice, "the call site",
  stray space, missing semicolon), fix broken contextual-dot link,
  normalize to American spelling (synthesize, serialization)
- fail/implicit-derivation-json.mo: update comments, re-accept .ok files

Made-with: Cursor
|----------------|-----------------------|------------------------------------------|------------------------------------------------|
| `__record` | `[(Text, E)] -> R` | `Rec -> R` or `(Rec, Rec) -> R` | Record: one or two records, arity from hole |
| `__tuple` | `[E] -> R` | `(A, B, ...) -> R` or `((A,B,...), (A,B,...)) -> R` (≥ 2 elements) | Tuple: one implicit per element |
| `__variant` | — | — | Reserved for future extension |
Copy link
Copy Markdown
Contributor

@crusso crusso Mar 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

| `__variant`    | —  (Nat, Text, () -> R)   | — `Var -> R`  or `(Var, Var) -> R`                                    | Reserved for future extension                  |

Might work for variants.... (Nat is the ordinal of the tag, which, for records, can be inferred from the array index)

Maybe arrays should just be delayed (or use an iterator if ok with forcing the ordering)
[(Text, () -> E)] -> R

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a really good idea to be go with lazy values () -> R, it'd give more flexibility to package authors (and they are the ones who write these __record/tuple/variant functions).
Better than Iter because they can choose which one to evaluate and when, yet still simple enough!

Just for the record: I'd implement variants in another PR. But I'm curious - Why would anyone need the Nat ordinal of the tag? Aren't we sorting tags in a variant anyway so that would make it tricky to preserve, right? Or am I misunderstanding or missing sth?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For efficient compare I would compare the ordinals before comparing the contents when ordinals match. Hash would work too but maybe more arbitrary and implementation specific than the ordinal of the label in the variants (which should be know statically, right?).

Kamirus added 12 commits March 29, 2026 19:37
Real-world nested types (e.g. records containing records containing
tuples) easily exceed depth 5. A limit of 100 is generous enough for
any practical nesting while still guaranteeing termination.

Made-with: Cursor
Structural combiners now receive lazy thunks instead of eagerly
evaluated values:

  __record : [(Text, () -> E)] -> R
  __tuple  : [() -> E]         -> R

The compiler wraps each per-field/element implicit call in a thunk,
giving the combiner full control over evaluation order.  Combiners
that can short-circuit (compare, eq) now genuinely skip remaining
fields; combiners that need everything (toJson, describe) call each
thunk eagerly.

Also consolidates AST synthesis helpers into a SynthesizeWrapper
module, eliminating duplication across derived_wrapper,
record_wrapper, and tuple_wrapper.

Made-with: Cursor
Reflect the new structural combiner signatures ([(Text, () -> T)] -> R
and [() -> T] -> R) and the depth limit bump to 100.

Made-with: Cursor
Apply @@, @~, @?, @! shorthand operators for AST node construction,
let open Lib.Option.Syntax for monadic Option chaining, and share
SynthesizeWrapper.var for synthesized variable references.

Adapted from #5921.

Made-with: Cursor
- Add Lib.Result.Syntax with let* for Result.bind
- Replace structural_kind enum + dom with polymorphic variant
  `Record of T.field list | `Tuple of T.typ list carrying pre-extracted
  data, eliminating redundant T.promote + assert false re-extraction
- Extract is_matching_structural_combiner to share combiner-matching
  logic between module-field and val candidate sources
- Unify try_derive and try_derive_structural via try_derive_with,
  centralizing depth-limit checks and disambiguate/commit flow

Made-with: Cursor
Kamirus added a commit that referenced this pull request Apr 1, 2026
Allow the compiler to derive implicit arguments from functions that
themselves have implicit parameters (e.g. `compare` for `[Nat]` from
`Array.compare<Nat>` + `Nat.compare`).

Works transitively and is depth-limited via `--implicit-derivation-depth`.

Structural derivation (`__record`/`__tuple` combiners) will follow in a
separate PR.

Made-with: Cursor
Kamirus added a commit that referenced this pull request Apr 1, 2026
Allow the compiler to derive implicit arguments from functions that
themselves have implicit parameters (e.g. `compare` for `[Nat]` from
`Array.compare<Nat>` + `Nat.compare`).

Works transitively and is depth-limited via `--implicit-derivation-depth`.

Structural derivation (`__record`/`__tuple` combiners) will follow in a
separate PR.

Made-with: Cursor
Kamirus added a commit that referenced this pull request Apr 1, 2026
Allow the compiler to derive implicit arguments from functions that
themselves have implicit parameters (e.g. `compare` for `[Nat]` from
`Array.compare<Nat>` + `Nat.compare`).

Works transitively and is depth-limited via `--implicit-derivation-depth`.

Structural derivation (`__record`/`__tuple` combiners) will follow in a
separate PR.

Made-with: Cursor
Comment thread src/mo_frontend/typing.ml
| _ -> None in
match T.promote candidate_typ with
| T.Func (T.Local, T.Returns, [], [T.Named ("__record", inner_typ)], [ret_typ]) ->
(match T.promote inner_typ with
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

dubious promote on negative occurrence?

Comment thread src/mo_frontend/typing.ml
with_thunk_elem `Record thunk_typ ret_typ
| _ -> None)
| T.Func (T.Local, T.Returns, [], [T.Named ("__tuple", inner_typ)], [ret_typ]) ->
(match T.promote inner_typ with
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ditto: dubious promote on negative occurrence?

@crusso
Copy link
Copy Markdown
Contributor

crusso commented Apr 27, 2026

Is there any way to present this PR as a diff to part 1 #5966? I tried to open a PR with this branch and the branch of #5966 as a base but it didn't reduce the diff for me.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants