feat: Add ABI Generation, including leo-abi-types and leo-abi crates#29064
feat: Add ABI Generation, including leo-abi-types and leo-abi crates#29064mohammadfawaz merged 12 commits intomasterfrom
leo-abi-types and leo-abi crates#29064Conversation
7034738 to
ec0366e
Compare
|
Ahhh I just realised we're not handling generic types (types generic over some constant, like fixed-size arrays) correctly in ABI generation. Currently we try to generate the ABI after type-checking, but these generic types have not yet been monomorphised. I might try doing ABI generation after @mohammadfawaz any thoughts on this? I'm guessing we'll want to expose the monomorphised types rather than the generic ones, as we mostly care about types that appear in the program interface ( Edit: CI error seems unrelated, but I don't seem to be able to rerun the test myself. |
There was a problem hiding this comment.
Thank you Mitch 🚀 ! This is well designed and overall looks like the right approach. Just made a few comments.
I would also be nice a "big" test that exercises most if not all parts of JSON ABI generation, all types, as many corner cases as possible, etc. E.g. transitions with 1 output v.s. transitions with multiple outputs.
|
Regarding generics... I suggest we ignore them in this PR but we do need to address them.
so we need a way to describe const generic structs in the ABI (at definition site) as well as a way to specify instances of them. Which means |
|
Thanks for the review!
Thanks for the example. It sounds like we want:
We might still be able to do the whole ABI generation following Failing this, we might need to separate the ABI gen into two passes, where we first do a pass after type-checking, then traverse again after ConstProp to fill the generic parameters in transitions. |
Yeah that is indeed a challenging problem. We want the generate the ABI after all consts have been folded (including const parameters, loop iterators, etc.) but we also want const generic struct refs to be kept in their generic form (not monomorphized). I suppose generating the ABI in phases is okay given your design. You could do this in a follow up PR if you like. |
I might do this, and leave in the current approach of always including the monomorphised form of the generic types for now. I'm still addressing the rest of the feedback, but after I'll open up a dedicated issue for handling generics in ABI and the multi-phased approach for follow-up 👌 |
Introduce `leo-abi-types` and `leo-abi` crates for describing and
generating program ABIs. The ABI captures the public interface of a
Leo program including transitions, mappings, structs, and records,
enabling downstream tooling to interact with deployed programs.
Key changes:
- `leo-abi-types`: Serde-serializable ABI type definitions
- `leo-abi`: AST-to-ABI conversion after monomorphisation
- Compiler returns `Compiled { programs, abi }` from compilation
- CLI writes `build/abi.json` alongside compiled bytecode
- Move `CompiledPrograms`/`Bytecode` from leo-ast to leo-passes
3b03be7 to
2dde4f5
Compare
Add StorageVariable and StorageType to the ABI, enabling downstream tooling to understand program storage declarations. Doc comments explain how storage is lowered to mappings in bytecode: - `storage x: T` → `mapping x__: bool => T` (value at key `false`) - `storage vec: Vector<T>` → two mappings for elements and length
Replace single `name: String` with `path: Vec<String>` in StructRef and RecordRef to support types defined in modules. Path segments represent the full qualified path (e.g., `["utils", "Vector3"]` for `utils::Vector3`).
Add storage variable declarations to exercise the new ABI storage support: simple primitive, struct type, and vector type. Also updates all tests with `REWRITE_EXPECTATIONS` to include recent path, storage variables and is_async changes.
- Change Struct.name and Record.name to Struct.path and Record.path - Update generate() to accept full ast::Program - Iterate over ast.modules to collect module composites - Add is_record check for module composites - Add test with module struct (utils::Vector3) Module structs now appear in the ABI structs array with their full path (e.g., ["utils", "Vector3"] for utils::Vector3).
Add `prune_non_interface_types()` to filter the generated ABI, removing structs and records that don't appear in transitions, mappings, or storage variables. The pruning: - Collects type paths from transition inputs/outputs, mapping keys/values, and storage variable types - Resolves transitive dependencies (e.g., Rectangle containing Point) - Retains only types referenced directly or transitively by the interface Add test case with an `Unused` struct to verify pruning behavior.
|
OK, I think most of this feedback has now been addressed.
I'll open a dedicated issue now to track the generics-related ABI generation work discussed above. EDIT: opened: EDIT2: @mohammadfawaz I just recalled our slack conversation about generating separate, dedicated ABIs for leo and aleo program stubs in our build output. I'm looking into addressing this now as a part of this PR. |
Restructure `Compiled` to include ABIs for both primary and imported
programs:
- Add `CompiledProgram` struct with name, bytecode, and ABI fields.
- Change `Compiled` to use `primary: CompiledProgram` and
`imports: Vec<CompiledProgram>`.
- Update `generate_abi()` to return ABIs for all stubs, handling both
`FromLeo` and `FromAleo` variants.
- Write `{program}.abi.json` files for each import to `build/imports/`.
Output structure:
build/
├── main.aleo
├── abi.json
└── imports/
├── child.aleo
└── child.abi.json ← NEW
|
OK I've just updated the PR to include ABI generation for imports under
This partly addresses #29074, though doesn't yet expose a CLI interface for generating |
|
I've opened a start on some ABI docs + lowering spec here: |
mohammadfawaz
left a comment
There was a problem hiding this comment.
Fantastic work 🚀 . This looks solid, thanks!
I see you're tracking the rest of the tasks in open issues, which I just added a sub-issues to #28892.
Regen expectations Regen expectations Add warn_and_confirm feat(leo-fmt): add formatter crate scaffolding (#29083) Foundation for `leo fmt` command with Output struct, format_source() and check_formatted() API, plus test harness with initial fixtures. feat(parser): Add `leo-parser-rowan` crate and initial lexer implementation (#29080) * feat(parser): WIP Add rowan-based parser crate with feature flag Introduce `leo-parser-rowan` crate as the foundation for a new rowan-based lossless parser with IDE-grade error recovery. - Create `compiler/parser-rowan/` with minimal SyntaxKind enum (ERROR, EOF, WHITESPACE, COMMENT, ROOT) and stub entry points - Implement rowan::Language trait for LeoLanguage type - Add `rowan = "0.16"` to workspace dependencies - Gate new parser behind `rowan` feature flag in leo-parser - Re-export leo_parser_rowan when feature is enabled The stub functions (parse_expression, parse_statement, parse_module, parse_main) return todo!() and will be implemented incrementally. * feat(parser-rowan): add SyntaxKind enum and logos-based lexer Add complete flattened SyntaxKind enum with ~170 token and node variants. Include helper methods (is_trivia, is_keyword, is_literal, etc.) and safe Language trait implementation using const lookup table. Port lexer from parser-lossless to produce rowan-compatible tokens. All trivia (whitespace, comments) preserved as explicit tokens for lossless parsing. Add comprehensive lexer tests using expect-test. feat: Add ABI Generation, including `leo-abi-types` and `leo-abi` crates (#29064) * feat(abi): add ABI generation to Leo compiler Introduce `leo-abi-types` and `leo-abi` crates for describing and generating program ABIs. The ABI captures the public interface of a Leo program including transitions, mappings, structs, and records, enabling downstream tooling to interact with deployed programs. Key changes: - `leo-abi-types`: Serde-serializable ABI type definitions - `leo-abi`: AST-to-ABI conversion after monomorphisation - Compiler returns `Compiled { programs, abi }` from compilation - CLI writes `build/abi.json` alongside compiled bytecode - Move `CompiledPrograms`/`Bytecode` from leo-ast to leo-passes * feat(abi): add is_async field to Transition * feat(abi): add storage variable support Add StorageVariable and StorageType to the ABI, enabling downstream tooling to understand program storage declarations. Doc comments explain how storage is lowered to mappings in bytecode: - `storage x: T` → `mapping x__: bool => T` (value at key `false`) - `storage vec: Vector<T>` → two mappings for elements and length * feat(abi): use Path type for struct and record references Replace single `name: String` with `path: Vec<String>` in StructRef and RecordRef to support types defined in modules. Path segments represent the full qualified path (e.g., `["utils", "Vector3"]` for `utils::Vector3`). * test(abi): add storage variable examples to comprehensive test Add storage variable declarations to exercise the new ABI storage support: simple primitive, struct type, and vector type. Also updates all tests with `REWRITE_EXPECTATIONS` to include recent path, storage variables and is_async changes. * fix: Improve docs and panic msgs in abi lib.rs * feat(abi): include module struct definitions in ABI - Change Struct.name and Record.name to Struct.path and Record.path - Update generate() to accept full ast::Program - Iterate over ast.modules to collect module composites - Add is_record check for module composites - Add test with module struct (utils::Vector3) Module structs now appear in the ABI structs array with their full path (e.g., ["utils", "Vector3"] for utils::Vector3). * fix: Address clippy warnings * feat(abi): prune types not referenced in public interface Add `prune_non_interface_types()` to filter the generated ABI, removing structs and records that don't appear in transitions, mappings, or storage variables. The pruning: - Collects type paths from transition inputs/outputs, mapping keys/values, and storage variable types - Resolves transitive dependencies (e.g., Rectangle containing Point) - Retains only types referenced directly or transitively by the interface Add test case with an `Unused` struct to verify pruning behavior. * feat: Add abi::aleo module for .aleo ABIs. Always prune leo ABIs. * feat(abi): generate ABIs for imported programs Restructure `Compiled` to include ABIs for both primary and imported programs: - Add `CompiledProgram` struct with name, bytecode, and ABI fields. - Change `Compiled` to use `primary: CompiledProgram` and `imports: Vec<CompiledProgram>`. - Update `generate_abi()` to return ABIs for all stubs, handling both `FromLeo` and `FromAleo` variants. - Write `{program}.abi.json` files for each import to `build/imports/`. Output structure: build/ ├── main.aleo ├── abi.json └── imports/ ├── child.aleo └── child.abi.json ← NEW --------- Co-authored-by: Mohammad Fawaz <mohammadfawaz89@gmail.com> fix: drop `redundant filter_map` in remove_unreachable (#29072) Signed-off-by: GarmashAlex <garmasholeksii@gmail.com> Co-authored-by: Joshua Batty <josh@provable.com> perf: avoid Vec allocation in async finalize lookup (#29091) Signed-off-by: phrwlk <phrwlk7@gmail.com> feat(leo-fmt): implement declaration formatting (#29094) Adds formatting for functions, transitions, structs, records, imports, mappings, and annotations. Part of #28579. External storage accesses (#29092) Co-authored-by: Mohammad Fawaz <mohammadfawaz@gmail.com> feat(parser-rowan): implement core parsing logic (#29095) * feat(parser-rowan): implement recursive descent parser Add the core parsing module that transforms the lexer token stream into a rowan lossless green tree. This implements a recursive descent parser with Pratt-style expression parsing covering the full Leo grammar. Parser infrastructure (parser/mod.rs): - Parser struct wrapping a token stream with lookahead - Marker/CompletedMarker API for building the green tree - Error recovery with configurable recovery token sets - Trivia (whitespace/comment) skipping Grammar entry points (parser/grammar.rs): - parse_file: top-level file with multiple import/program blocks - parse_expression_entry, parse_statement_entry: standalone parsing Expressions (parser/expressions.rs): - Pratt parser with full operator precedence - Binary, unary, ternary, cast, method call, field access - Struct/tuple/array literals, parenthesized expressions - Async blocks (async { ... }), self.caller, block.height - Associated function calls (Foo::bar()), free function calls - ExprOpts to disable struct literals in ambiguous contexts (if/for conditions where `{` starts a block, not a struct) Statements (parser/statements.rs): - Let/const bindings with optional type annotations - Return, assert, assert_eq, assert_neq - If/else chains, for loops with range expressions - Block statements, expression statements, assignment Items (parser/items.rs): - Program declarations, import statements - Function/transition/inline definitions with all modifiers - Struct and record definitions with visibility modifiers (public, private, constant) - Mapping declarations, const declarations - Annotations accepting both identifiers and keywords (@program) - Multi-program file support (multiple import/program blocks) Types (parser/types.rs): - Primitive types (integer variants, bool, address, field, etc.) - Array types with expression-based lengths ([u8; N + M]) - Tuple types, optional types (Option<T>) - Future type, composite/named types, locator types Lexer updates (lexer.rs): - Integer tokens now include type suffixes (e.g. `1000u32`) - Unit tests for typed integer ranges SyntaxKind additions: - RECORD_DEF, IDENT_PATTERN, TUPLE_PATTERN, WILDCARD_PATTERN Achieves ~82.5% parse success rate across 1492 .leo files in the repository so far (using an unincluded integration test)... Remaining failures are primarily const generics syntax, bare test fragments, expected-failure files, and module separators. * feat(parser-rowan): add module file parsing support Add `parse_module_entry` as a standalone entry point for module files, which contain only `const`, `struct`, and `inline` declarations. Update `parse_file_items` to also accept module-level items at the top level, supporting multi-section test files where program declarations are followed by module content separated by `// --- Next Module:` comments. Wire `parse_module` in `lib.rs` to use the new `parse_module_entry` instead of delegating to `parse_file`. * feat(parser-rowan): add const generics support Implement const generic parsing for both declaration and use sites, covering the `::[]` bracket syntax used throughout Leo. Declaration sites (CONST_PARAM_LIST with CONST_PARAM nodes): - Function/transition/inline: `function foo::[N: u32]()` - Struct: `struct Foo::[N: u32] { ... }` - Record: `record Bar::[N: u32] { ... }` Use sites (CONST_ARG_LIST wrapping expression or type args): - Type annotations: `Foo::[N + 1]`, `Matrix::[M, K, N]` - Function calls: `foo::[5]()`, `bar::[N + 1]()` - Struct literals: `Foo::[8u32] { arr: x }` - Type arguments (intrinsics): `Deserialize::[u32]`, `Deserialize::[[u8; 4]]` - Locator paths: `child.aleo/Bar::[4]`, `child.aleo/foo::[3]()` Key changes: - types.rs: Add parse_const_param_list/parse_const_param for declarations. Rewrite parse_const_generic_args_bracket to wrap in CONST_ARG_LIST and parse args as expressions (with type heuristic for primitive/array types). Fix parse_named_type to check for :: after locator paths. - items.rs: Wire parse_const_param_list into parse_function_body, parse_struct_def, and parse_record_def. - expressions.rs: Fix parse_ident_expr to check for :: after locator paths, including struct literal support for locator+const generic paths. Add 20 new unit tests covering all const generic forms. 162 unit + 14 integration tests pass. [Feature] Leo Devnode CLI command (#29012) * updated branch from feat/leo-devnode * ran cargo fmt * made path to genesis block a const var with docblock * ran cargo fmt * reverted path for genesis block to use include_bytes * changed snarkVM rev * modified comment * listener_addr var passed to POST requests * changed var name listener_addr -> socket_addr * fixed merge conflicts * fixed build error * ran cargo fmt * fixed typo * rebasing * fixed Cargo.toml * fixed logging * added comments * updated rev * update lock file --------- Co-authored-by: Mohammad Fawaz <mohammadfawaz@gmail.com> fix: preserve and format HTTP response body in broadcast errors When broadcasting a transaction fails, users now see the actual error message from the server instead of just a status code. feat(parser-rowan): implement comprehensive error recovery (#29103) Add error recovery infrastructure to ensure the parser always produces a syntax tree, even for malformed input. This is essential for IDE tooling where partial/incomplete code must still be analyzed. Recovery enhancements: - Add recovery token sets (EXPR_RECOVERY, TYPE_RECOVERY, PARAM_RECOVERY) - Expression recovery: handle missing operands in unary, binary, and ternary expressions; recover from malformed call arguments and array elements - Statement recovery: handle missing types/expressions in let/const, missing conditions/blocks in if/for, malformed assert statements - Item recovery: handle malformed struct fields, missing types in mappings/storage/params, incomplete function definitions - Add delimiter balance tracking (paren/brace/bracket depth) with skip_to_balanced() helper for smarter recovery in nested contexts Add error recovery unit tests verifying: - Missing semicolons, types, expressions - Missing operands and unclosed delimiters - Malformed functions, structs, mappings - Multiple errors in a single file - Garbage input never causes panics Deprecate `Locator` expressions in favor of `Path` expressions (#29098) * deprecate locator expressions in favour of path expressions * Update compiler/passes/src/code_generation/expression.rs Co-authored-by: Joshua Batty <josh@provable.com> Signed-off-by: Mohammad Fawaz <mohammadfawaz89@gmail.com> --------- Signed-off-by: Mohammad Fawaz <mohammadfawaz89@gmail.com> Co-authored-by: Mohammad Fawaz <mohammadfawaz@gmail.com> Co-authored-by: Joshua Batty <josh@provable.com> Prepare for `testnet-v3.5.0` tag (#29105) * Prepare for testnet-v3.5.0 tag * update deps * fix a test_add cli test --------- Co-authored-by: Mohammad Fawaz <mohammadfawaz@gmail.com> update linux and macos releases (#29108) Co-authored-by: Mohammad Fawaz <mohammadfawaz@gmail.com> feat(leo-fmt): implement statement and expression formatting Add signature literals (#29102) Co-authored-by: Mohammad Fawaz <mohammadfawaz@gmail.com> Co-authored-by: Joshua Batty <josh@provable.com> feat(cli): add `leo abi` command for generating ABI from .aleo files (#29116) * feat(cli): add `leo abi` command for generating ABI from .aleo files Add a new CLI command to generate JSON ABI from compiled Aleo bytecode files without requiring Leo source code. Usage: leo abi program.aleo # Print ABI to stdout leo abi program.aleo -o out.json # Write to file leo abi program.aleo -n mainnet # Specify network The command disassembles the .aleo file and generates an ABI using the existing leo-abi infrastructure. Outputs to stdout by default for easy piping. Closes #29074 * test(abi): Add more comprehensive `leo abi` test with complex `.aleo` * test(cli): add --output flag coverage to abi command test Add test case that verifies the -o/--output flag correctly writes the ABI JSON to a file instead of stdout. refactor(leo-parser): change tests to use `Display` instead of `Serialize` (#29117) * refactor(leo-parser): change tests to use Display instead of Serialize Replace JSON serialization in parser tests with Display trait output. This produces clean Leo source code expectations instead of verbose JSON with node IDs and spans, making parser diffs meaningful and readable. Closes #29115 * fix(leo-ast): remove duplicate "sign" prefix in signature literal Display The Signature variant stores the full literal including the "sign" prefix, but Display was adding another "sign", producing "signsign..." output. Remove the redundant prefix to match how Address literals are handled. --------- Co-authored-by: Mohammad Fawaz <mohammadfawaz89@gmail.com> refactor(leo-fmt): switch from `leo-parser-lossless` to `leo-parser-rowan` (#29118) * refactor(leo-fmt): switch from leo-parser-lossless to leo-parser-rowan Rewrite the formatter to work directly with rowan SyntaxNode/SyntaxToken trees. Drop anyhow since rowan's error recovery means parsing never fails. * preserve ERROR nodes verbatim and add error recovery tests - Switch CLI tests to use leo devnode (#29120) - Make CLI tests individual rust unit tests using `build.rs` - Fix some issues in `leo devnode` related to an extra `\n` being emitted Turn CLI tests into separate unit test processes Turn CLI tests into separate unit test processes Co-authored-by: Mohammad Fawaz <mohammadfawaz@gmail.com> Surface missing private_key error for leo devnode (#29110) * Surface missing private_key error for leo devnode * Add CLI test --------- Co-authored-by: Victor Sint Nicolaas <vicsn@users.noreply.github.com> Co-authored-by: Mohammad Fawaz <mohammadfawaz@gmail.com> feat(cli): add leo fmt command Adds a `leo fmt` command emulating `cargo fmt` behaviour: silent on success, colored diffs on `--check`. Includes integration test and fixes constructor() spacing in leo-fmt. fix(cli): remove duplicate `EnvOptions` flatten in devnode subcommands (#29126) `EnvOptions` fields use `global = true`, so flattening at both `LeoDevnode` (parent) and `Start`/`Advance` (subcommands) caused clap's debug assertions to detect duplicate global arg registration, crashing `leo devnode start` in debug builds. Remove the `EnvOptions` flatten from `Start` and `Advance`. Pass the private key from the parent's `EnvOptions` through `Start::apply` via `type Input = Option<String>`, matching the pattern used by `LeoQuery`. use a prebuilt genesis block for execution tests
Summary
This PR adds ABI (Application Binary Interface) generation to the Leo compiler. The ABI describes the public interface of a Leo program, enabling downstream tooling to interact with deployed programs.
Changes
New Crates:
leo-abi-types- Pure type definitions for ABI (serializable with serde)leo-abi- ABI generation logic from the Leo ASTCompiler Integration
ProcessingAsyncpass (finalize functions exist) and before lowering passes (Leo types preserved).Compiledstruct returned fromcompile()containing bothbytecodeandabi.intermediate_passes()now returns the generated ABI.CLI Integration
leo buildwritesabi.jsonto the build directory alongsidemain.aleo.ABI Contents
Design Decisions
Option<T>) rather than lowered Aleo types (e.g., "T?" structs). We instead clearly document how types are lowered with doc comments on those that have a differing lowered representation (currently onlyOptional).ProcessingAsyncgeneration: Finalize functions are found by analyzing the async transition's return statement (which calls the generated async function).To-Do
Future Work