Skip to content

bench: add vec_nat, vec_nat32, vec_nat64 benchmarks#718

Merged
lwshang merged 4 commits intomasterfrom
sat-add-vec-benchmarks
Mar 15, 2026
Merged

bench: add vec_nat, vec_nat32, vec_nat64 benchmarks#718
lwshang merged 4 commits intomasterfrom
sat-add-vec-benchmarks

Conversation

@sasa-tomic
Copy link
Copy Markdown
Member

@sasa-tomic sasa-tomic commented Mar 13, 2026 •
edited
Loading

Summary

Adds three new canbench benchmarks to complement the existing vec_int16:

  • vec_nat — 262K Nat elements (variable-length LEB128 encoding). Exercises the Nat::encode/decode paths and the serde visitor round-trip for bignum types.
  • vec_nat32 — 2M u32 elements (fixed-width 4 bytes). Tests fixed-width primitive vector performance.
  • vec_nat64 — 2M u64 elements (fixed-width 8 bytes). Tests the largest fixed-width primitive vector.

Baseline results (master, ba72cf4)

Benchmark Encoding Decoding Total Heap
vec_nat 1.10B 1.49B 2.59B 172 pages
vec_nat32 136.28M 1.06B 1.20B 518 pages
vec_nat64 161.45M 1.06B 1.22B 1031 pages

These baselines make the impact of the performance work in sat-perf-improvements directly measurable:

Benchmark Metric Master Optimized Change
vec_nat Encoding 1.10B 66.54M -93.9% (16.5x)
vec_nat Decoding 1.49B 917.08M -38.5%
vec_nat Total 2.59B 983.62M -62.0%
vec_nat32 Encoding 136.28M 16.79M -87.7% (8.1x)
vec_nat32 Decoding 1.06B 406.87M -61.6%
vec_nat32 Total 1.20B 423.67M -64.7%
vec_nat64 Encoding 161.45M 33.57M -79.2% (4.8x)
vec_nat64 Decoding 1.06B 411.07M -61.2%
vec_nat64 Total 1.22B 444.64M -63.6%

Relates to #710

@sasa-tomic
bench: add vec_nat, vec_nat32, vec_nat64 benchmarks
291bfc6 
Cover variable-length (Nat/LEB128) and fixed-width (u32, u64) vector
serialization to complement the existing vec_int16 benchmark.

Made-with: Cursor
@sasa-tomic sasa-tomic requested a review from a team as a code owner March 13, 2026 14:39
@sasa-tomic sasa-tomic mentioned this pull request Mar 13, 2026
Closed
@github-actions
Copy link
Copy Markdown

github-actions Bot commented Mar 13, 2026 •
edited
Loading

Name Max Mem (Kb) Encode Decode
blob 4_224 4_207_487 2_122_431 ($\textcolor{green}{-0.00\%}$)
btreemap 73_856 531_976_003 ($\textcolor{red}{0.00\%}$) 13_022_440_615 ($\textcolor{green}{-0.27\%}$)
nns 192 2_021_253 5_665_159 ($\textcolor{green}{-0.21\%}$)
nns_list_proposal 1_216 7_014_781 ($\textcolor{red}{0.04\%}$) 64_250_335 ($\textcolor{green}{-0.19\%}$)
option_list 64 716_065 ($\textcolor{red}{0.01\%}$) 21_934_672 ($\textcolor{green}{-0.69\%}$)
text 6_336 4_204_384 7_877_758 ($\textcolor{green}{-0.00\%}$)
variant_list 64 711_213 ($\textcolor{red}{0.03\%}$) 20_749_188 ($\textcolor{red}{0.09\%}$)
vec_int16 12_480 8_404_689 408_970_098 ($\textcolor{green}{-0.00\%}$)
vec_nat (new) 9_664 67_095_666 842_209_893
vec_nat32 (new) 24_768 16_793_297 404_776_086
vec_nat64 (new) 49_344 33_570_495 408_970_650
  • Parser cost: 16_179_361
  • Extra args: 2_838_484 ($\textcolor{red}{0.00\%}$)
Click to see raw report 
---------------------------------------------------

Benchmark: blob
  total:
    instructions: 6.33 M (-0.00%) (change within noise threshold)
    heap_increase: 66 pages (no change)
    stable_memory_increase: 0 pages (no change)

  1. Encoding (scope):
    calls: 1 (no change)
    instructions: 4.21 M (no change)
    heap_increase: 66 pages (no change)
    stable_memory_increase: 0 pages (no change)

  2. Decoding (scope):
    calls: 1 (no change)
    instructions: 2.12 M (-0.00%) (change within noise threshold)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: btreemap
  total:
    instructions: 13.55 B (-0.26%) (change within noise threshold)
    heap_increase: 1154 pages (no change)
    stable_memory_increase: 0 pages (no change)

  1. Encoding (scope):
    calls: 1 (no change)
    instructions: 531.98 M (0.00%) (change within noise threshold)
    heap_increase: 159 pages (no change)
    stable_memory_increase: 0 pages (no change)

  2. Decoding (scope):
    calls: 1 (no change)
    instructions: 13.02 B (-0.27%) (change within noise threshold)
    heap_increase: 995 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: extra_args
  total:
    instructions: 2.84 M (0.00%) (change within noise threshold)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: nns
  total:
    instructions: 24.70 M (-0.05%) (change within noise threshold)
    heap_increase: 3 pages (no change)
    stable_memory_increase: 0 pages (no change)

  0. Parsing (scope):
    calls: 1 (no change)
    instructions: 16.18 M (no change)
    heap_increase: 3 pages (no change)
    stable_memory_increase: 0 pages (no change)

  1. Encoding (scope):
    calls: 1 (no change)
    instructions: 2.02 M (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

  2. Decoding (scope):
    calls: 1 (no change)
    instructions: 5.67 M (-0.21%) (change within noise threshold)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: nns_list_proposal
  total:
    instructions: 71.27 M (-0.16%) (change within noise threshold)
    heap_increase: 19 pages (no change)
    stable_memory_increase: 0 pages (no change)

  1. Encoding (scope):
    calls: 1 (no change)
    instructions: 7.01 M (0.04%) (change within noise threshold)
    heap_increase: 5 pages (no change)
    stable_memory_increase: 0 pages (no change)

  2. Decoding (scope):
    calls: 1 (no change)
    instructions: 64.25 M (-0.19%) (change within noise threshold)
    heap_increase: 14 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: option_list
  total:
    instructions: 22.65 M (-0.66%) (change within noise threshold)
    heap_increase: 1 pages (no change)
    stable_memory_increase: 0 pages (no change)

  1. Encoding (scope):
    calls: 1 (no change)
    instructions: 716.07 K (0.01%) (change within noise threshold)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

  2. Decoding (scope):
    calls: 1 (no change)
    instructions: 21.93 M (-0.69%) (change within noise threshold)
    heap_increase: 1 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: text
  total:
    instructions: 12.08 M (-0.00%) (change within noise threshold)
    heap_increase: 99 pages (no change)
    stable_memory_increase: 0 pages (no change)

  1. Encoding (scope):
    calls: 1 (no change)
    instructions: 4.20 M (no change)
    heap_increase: 66 pages (no change)
    stable_memory_increase: 0 pages (no change)

  2. Decoding (scope):
    calls: 1 (no change)
    instructions: 7.88 M (-0.00%) (change within noise threshold)
    heap_increase: 33 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: variant_list
  total:
    instructions: 21.46 M (0.09%) (change within noise threshold)
    heap_increase: 1 pages (no change)
    stable_memory_increase: 0 pages (no change)

  1. Encoding (scope):
    calls: 1 (no change)
    instructions: 711.21 K (0.03%) (change within noise threshold)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

  2. Decoding (scope):
    calls: 1 (no change)
    instructions: 20.75 M (0.09%) (change within noise threshold)
    heap_increase: 1 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: vec_int16
  total:
    instructions: 417.38 M (-0.00%) (change within noise threshold)
    heap_increase: 195 pages (no change)
    stable_memory_increase: 0 pages (no change)

  1. Encoding (scope):
    calls: 1 (no change)
    instructions: 8.40 M (no change)
    heap_increase: 130 pages (no change)
    stable_memory_increase: 0 pages (no change)

  2. Decoding (scope):
    calls: 1 (no change)
    instructions: 408.97 M (-0.00%) (change within noise threshold)
    heap_increase: 65 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: vec_nat (new)
  total:
    instructions: 909.31 M (new)
    heap_increase: 151 pages (new)
    stable_memory_increase: 0 pages (new)

  1. Encoding (scope):
    calls: 1 (new)
    instructions: 67.10 M (new)
    heap_increase: 33 pages (new)
    stable_memory_increase: 0 pages (new)

  2. Decoding (scope):
    calls: 1 (new)
    instructions: 842.21 M (new)
    heap_increase: 118 pages (new)
    stable_memory_increase: 0 pages (new)

---------------------------------------------------

Benchmark: vec_nat32 (new)
  total:
    instructions: 421.57 M (new)
    heap_increase: 387 pages (new)
    stable_memory_increase: 0 pages (new)

  1. Encoding (scope):
    calls: 1 (new)
    instructions: 16.79 M (new)
    heap_increase: 258 pages (new)
    stable_memory_increase: 0 pages (new)

  2. Decoding (scope):
    calls: 1 (new)
    instructions: 404.78 M (new)
    heap_increase: 129 pages (new)
    stable_memory_increase: 0 pages (new)

---------------------------------------------------

Benchmark: vec_nat64 (new)
  total:
    instructions: 442.54 M (new)
    heap_increase: 771 pages (new)
    stable_memory_increase: 0 pages (new)

  1. Encoding (scope):
    calls: 1 (new)
    instructions: 33.57 M (new)
    heap_increase: 514 pages (new)
    stable_memory_increase: 0 pages (new)

  2. Decoding (scope):
    calls: 1 (new)
    instructions: 408.97 M (new)
    heap_increase: 257 pages (new)
    stable_memory_increase: 0 pages (new)

---------------------------------------------------

Summary:
  instructions:
    status:   New benchmarks added ➕
    counts:   [total 12 | regressed 0 | improved 0 | new 3 | unchanged 9]
    change:   [max +18.68K | p75 -34 | median -50 | p25 -117.31K | min -35.65M]
    change %: [max +0.09% | p75 -0.00% | median -0.00% | p25 -0.16% | min -0.66%]

  heap_increase:
    status:   New benchmarks added ➕
    counts:   [total 12 | regressed 0 | improved 0 | new 3 | unchanged 9]
    change:   [max 0 | p75 0 | median 0 | p25 0 | min 0]
    change %: [max 0.00% | p75 0.00% | median 0.00% | p25 0.00% | min 0.00%]

  stable_memory_increase:
    status:   New benchmarks added ➕
    counts:   [total 12 | regressed 0 | improved 0 | new 3 | unchanged 9]
    change:   [max 0 | p75 0 | median 0 | p25 0 | min 0]
    change %: [max 0.00% | p75 0.00% | median 0.00% | p25 0.00% | min 0.00%]

---------------------------------------------------

Only significant changes:
| status | name                   | calls |     ins |  ins Δ% |  HI |  HI Δ% | SMI |  SMI Δ% |
|--------|------------------------|-------|---------|---------|-----|--------|-----|---------|
|  new   | vec_nat                |       | 909.31M |         | 151 |        |   0 |         |
|  new   | vec_nat32              |       | 421.57M |         | 387 |        |   0 |         |
|  new   | vec_nat32::1. Encoding |     1 |  16.79M |         | 258 |        |   0 |         |
|  new   | vec_nat32::2. Decoding |     1 | 404.78M |         | 129 |        |   0 |         |
|  new   | vec_nat64              |       | 442.54M |         | 771 |        |   0 |         |
|  new   | vec_nat64::1. Encoding |     1 |  33.57M |         | 514 |        |   0 |         |
|  new   | vec_nat64::2. Decoding |     1 | 408.97M |         | 257 |        |   0 |         |
|  new   | vec_nat::1. Encoding   |     1 |  67.10M |         |  33 |        |   0 |         |
|  new   | vec_nat::2. Decoding   |     1 | 842.21M |         | 118 |        |   0 |         |

ins = instructions, HI = heap_increase, SMI = stable_memory_increase, Δ% = percent change

---------------------------------------------------
Successfully persisted results to canbench_results.yml

lwshang and others added 3 commits March 15, 2026 14:11
@lwshang
Merge branch 'master' into sat-add-vec-benchmarks
0933cc3
@lwshang @claude
chore: remove generated canbench_results.yml and add to .gitignore
37c1435 
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@lwshang @claude
bench: raise COST quota to 25M and use it consistently in vec_nat64
e7eeb25 
vec_u64 decoding costs ~409M instructions but the quota only gates
memory-cost accounting, not wall time — 25M accommodates vec_nat64
while keeping all benchmarks on the same constant.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@lwshang lwshang merged commit 3759322 into master Mar 15, 2026
11 checks passed
@lwshang lwshang deleted the sat-add-vec-benchmarks branch March 15, 2026 18:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@lwshang lwshang lwshang approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@sasa-tomic @lwshang