feat: expose LibTomMath modular arithmetic as primitives

[ggreif]

Summary

Expose LibTomMath modular arithmetic operations as Motoko primitives:

• Prim.intAddMod(a, b, m) — (a + b) mod m via mp_addmod
• Prim.intSubMod(a, b, m) — (a - b) mod m via mp_submod
• Prim.intMulMod(a, b, m) — (a * b) mod m via mp_mulmod
• Prim.intPowMod(base, exp, m) — base^exp mod m via mp_exptmod

All operate on Int (arbitrary precision). intPowMod is particularly useful for cryptographic applications (RSA, Diffie-Hellman, etc.) where modular exponentiation with large integers is needed.

Changes

Layer	File	What
RTS/C	rts/Makefile	Add libtommath source files + bindgen allowlist
RTS/Rust	rts/motoko-rts/src/bigint.rs	Wrapper functions calling mp_*
Compiler	compile_enhanced.ml, compile_classical.ml	Wasm imports + OtherPrim codegen
Interpreter	mo_values/prim.ml	OCaml reference implementation
Prelude	src/prelude/prim.mo	Public Prim declarations
Test	test/run/modular-arith.mo	Happy-path tests incl. Fermat's little theorem

Test plan

• nix build .#tests.unit passes
• nix build .#tests.release passes
• CI green

🤖 Generated with Claude Code

TODO

• Amend Changelog.md

[ggreif]

cc @timohanke — this was your request 🎯

[github-actions]

Comparing from 5e5ac5c to 49b538a:
In terms of gas, no changes are observed in 5 tests.
In terms of size, 5 tests regressed and the mean change is +4.0%.

[ggreif]

@timohanke — your modular arithmetic primitives are ready! 🎉

Prim.intAddMod, intSubMod, intMulMod, intPowMod backed by LibTomMath's mp_addmod, mp_submod, mp_mulmod, mp_exptmod. Build artifacts attached to the CI run.

Getting here was a bit of a Grothendieck experience — rather than cracking the nut with a hammer, we let the rising sea of understanding slowly submerge the problem. From missing libtommath transitive dependencies, through WASI allocator incompatibilities, to discovering that Motoko's compact BigNum representation requires proper unboxing for ternary operations — each wrong theory peeled back a layer until the fix emerged naturally from the architecture.

Build artifacts (macOS, Ubuntu, ARM) are available from the CI run linked above.

E.g. macOS ARM64 is here: https://github.com/caffeinelabs/motoko/actions/runs/24552026361/artifacts/6490288624

[timohanke]

Thanks @ggreif . This is awesome. Looking forward to try it out.

We should also consider exposing mp_sqr. And additionally, make the compiler detect x ** 2 or x * x in the source code and call it. (Off-topic for the feature, which is about modular arithmetic and mp_sqr isn't modular, but on-topic application-wise because it is useful for the same cryptographic applications.)

Further down the road we can also consider exposing

mp_montgomery_setup
mp_montgomery_calc_normalization
mp_montgomery_reduce

but let's see first how the ones already added perform in practice.

[ggreif]

We should also consider exposing mp_sqr. And additionally, make the compiler detect x ** 2 or x * x in the source code and call it. (Off-topic for the feature, which is about modular arithmetic and mp_sqr isn't modular, but on-topic application-wise because it is useful for the same cryptographic applications.)

Good point. We could do
We sure could:

• spot both args VarE and same -> call mp_sqr
• spot literal ** exponent 2 and call mp_sqr
• popcnt exponent and if =1 do iterated mp_sqr
  Both of the latter are possible, but does TomMath already do the second? Still it would be a win, because ** 2 is internally 0b100 and ctz gives 2 then iterate mp_sqr only once. For 0b1000 (a.k.a. 4) iterate twice. Etc. Maybe worth it. When heap pointer popcnt = 1 will never trigger.

[timohanke]

Also mp_invmod should be exposed as well. That fits right into this PR.

[timohanke]

After testing this in some benchmarks I would like to see this function exposed as well: mp_invmod.

The benchmarks showed an advantage of intPowMod = mp_exptmod over an implementation in user code. That's why I suspect that access to mp_invmod would be an advantage over implementing EEA in user code.

The benchmarks showed no advantage of intAddMod, intSubMod, intMulMod over using the operators +, -, *, % in user code.

[ggreif]

Also mp_invmod should be exposed as well. That fits right into this PR.

Go for it! Thanks for the feedback!

[timohanke]

Great, thank you. intInvMod is indeed faster than user code EEA. By how much differs widely depending on input. Hence it is more meaningful to benchmark in actual applications.

In raw EC arithmetic the results still differ, between 5-20% improvement. In the full final application of ECDSA-verify the improvement shrinks down to 1%.

I then tried using Montgomery form throughout ECDSA-verify and the result was a 20% improvement. That was by implementing everything in use code. So my next request would be to expose these four from libtommath:

• mp_montgomery_setup
• mp_montgomery_calc_normalization
• mp_montgomery_reduce
• mp_sqr

Then maybe we can speed up Montgomery form even further.