real world test

[Licenser]

Heya it's friday so I had some time to toy around. For giggles I replaced snmalloc with smalloc in a allocation heavy service - largely parsing grcp data with prost (yay the allocations ...)

The results were not great. Compared to snmalloc time spend in the allocator and allocated memory went up 10 fold.

over all change:

top cpu usage (snmalloc left, smalloc right)

and graphic representation of that snmallco:

and smalloc:

[Licenser]

update after some more time running:

It's interesting that memory goes down but cpu keeps going up massively

[zooko]

Thank you so much for doing this and posting the results! How can I reproduce this experiment?

[Licenser]

I think the simplest thing is a service parsing metrics using the tremor-otelapi going over it and just doing some simple allocations after that (might not even be needed) should the main load seems to be in the parsing of the protobuf structure.

the Otel demo will probably do the trick on generating enough load to measure

[zooko]

Hm, okay I looked at https://github.com/tremor-rs/tremor-otelapis and https://opentelemetry.io/docs/demo/ and I don't immediately see how to set up this experiment that you ran. Could you show me a series of command-lines that do it?

[zooko]

I also didn't see any mention of benchmarks in either of those two repos that I could use.

[zooko]

Update: I found that https://github.com/tokio-rs/prost comes with benchmarks so I’ll try those.

[zooko]

Update: the benchmarks in https://github.com/tokio-rs/prost do not include allocations. :-)

GITCOMMIT: fb3994b0f3cdb0b95f86b331fcdf42a95ec35d3d
GITCLEANSTATUS: "Uncommitted changes"
CPUTYPE:                           AMD EPYC 9754 128-Core Processor
OSTYPE: linux-gnu
CPUCOUNT: 256

                                                        default           jemalloc          snmalloc          mimalloc          rpmalloc          smalloc
                                                    ----------------  ----------------  ----------------  ----------------  ----------------  ----------------
test                                        weight   time ms (diff%)   time ms (diff%)   time ms (diff%)   time ms (diff%)   time ms (diff%)   time ms (diff%)
----                                        ------  -------- -------  -------- -------  -------- -------  -------- -------  -------- -------  -------- -------
dataset/google_message1_proto2/encode          100      16.5 ( base)      16.7 (  +1%)      16.9 (  +2%)      15.9 (  -4%)      16.6 (   0%)      16.2 (  -2%)
dataset/google_message1_proto2/encoded_len     100       2.8 ( base)       2.8 (   0%)       2.8 (   0%)       2.8 (  -1%)       2.8 (   0%)       2.8 (  -1%)
dataset/google_message1_proto2/merge           100      26.7 ( base)      26.0 (  -3%)      25.7 (  -4%)      25.5 (  -5%)      28.1 (  +5%)      27.2 (  +2%)
dataset/google_message1_proto3/encode          100      13.1 ( base)      13.0 (   0%)      13.5 (  +3%)      13.3 (  +2%)      13.4 (  +2%)      13.2 (   0%)
dataset/google_message1_proto3/encoded_len     100       2.3 ( base)       2.2 (  -1%)       2.3 (   0%)       2.3 (  +1%)       2.3 (  +1%)       2.2 (  -1%)
dataset/google_message1_proto3/merge           100      20.2 ( base)      20.4 (  +1%)      20.6 (  +2%)      20.2 (   0%)      21.3 (  +5%)      20.4 (  +1%)
varint/large/decode                            100      52.6 ( base)      52.6 (   0%)      52.6 (   0%)      52.6 (   0%)      52.6 (   0%)      52.6 (   0%)
varint/large/encode                            100      65.6 ( base)      64.8 (  -1%)      65.5 (   0%)      64.9 (  -1%)      64.8 (  -1%)      64.8 (  -1%)
varint/large/encoded_len                       100       6.1 ( base)       6.1 (   0%)       6.1 (   0%)       6.1 (   0%)       6.1 (   0%)       6.1 (  -1%)
varint/medium/decode                           100      30.0 ( base)      30.0 (   0%)      30.0 (   0%)      30.0 (   0%)      30.0 (   0%)      33.1 ( +10%)
varint/medium/encode                           100      32.5 ( base)      32.5 (   0%)      32.5 (   0%)      32.5 (   0%)      32.5 (   0%)      33.0 (  +2%)
varint/medium/encoded_len                      100       6.1 ( base)       6.1 (   0%)       6.1 (   0%)       6.1 (   0%)       6.1 (   0%)       6.1 (  -1%)
varint/mixed/decode                            100      32.5 ( base)      32.5 (   0%)      32.5 (   0%)      32.5 (   0%)      32.5 (   0%)      34.1 (  +5%)
varint/mixed/encode                            100      35.7 ( base)      35.7 (   0%)      35.7 (   0%)      35.7 (   0%)      35.7 (   0%)      35.7 (   0%)
varint/mixed/encoded_len                       100       6.1 ( base)       6.1 (   0%)       6.1 (   0%)       6.1 (   0%)       6.1 (   0%)       6.0 (  -2%)
varint/small/decode                            100      16.6 ( base)      16.5 (  -1%)      16.5 (   0%)      16.6 (   0%)      16.6 (   0%)      16.4 (  -1%)
varint/small/encode                            100       6.5 ( base)       6.6 (   0%)       6.5 (   0%)       6.5 (   0%)       6.5 (   0%)       6.5 (   0%)
varint/small/encoded_len                       100       6.1 ( base)       6.1 (   0%)       6.1 (   0%)       6.1 (   0%)       6.1 (   0%)       6.0 (  -1%)
dataset/google_message2/encode                   1      71.6 ( base)      70.4 (  -2%)      71.2 (   0%)      70.2 (  -2%)      70.3 (  -2%)      69.6 (  -3%)
dataset/google_message2/encoded_len              1      10.3 ( base)      10.3 (   0%)      10.3 (   0%)      10.3 (   0%)      10.3 (   0%)      10.3 (   0%)
dataset/google_message2/merge                    1     128.3 ( base)     124.5 (  -3%)     128.1 (   0%)     106.5 ( -17%)     114.4 ( -11%)     117.3 (  -9%)
--------------------------------------------------------------------------------------------------------------------------------------------------------------
SUM                                                    588.3 ( base)     582.1 (  -1%)     587.8 (   0%)     562.7 (  -4%)     575.3 (  -2%)     579.6 (  -1%)
--------------------------------------------------------------------------------------------------------------------------------------------------------------

Tests compared: 21

Allocator       Weighted Sum vs Baseline
--------------- ------------ -----------
default                588.3    baseline
jemalloc               582.1     (  -1%)
snmalloc               587.8     (   0%)
mimalloc               562.7     (  -4%)
rpmalloc               575.3     (  -2%)
smalloc                579.6     (  -1%)

Graph saved to: ./benchmark-results/AMDEPYC9754128CoreProcessor.linuxgnu/prost.graph.svg

You can reproduce this with https://github.com/zooko/prost/blob/fb3994b0f3cdb0b95f86b331fcdf42a95ec35d3d/bench-allocators.sh

So, I'm back to square one on how to reproduce your experiment.

[Licenser]

Heya,

I've tried to synthetically reproduce this behavior today and[ I too have failed, for every test I put together with a synthetic consumer & producer smalloc and snmalloc are pretty much tied.

But there is a dragon hidden somewhere and I'm not sure where 😭

[zooko]

Huh! Okay, well there are only two ways that an allocator can affect performance. One is how long it takes to execute malloc, dealloc, and realloc, and the other is how well the allocator's arrangement of space aligns with the user code's use of that space over time.

The first one is what smalloc's micro-benchmarks measure and where smalloc really shines.

The second one is hard to analyze or optimize. The better those space and time patterns line up, the more cache and swap will help run your program efficiently. There could be some edge cases where those two things line up so badly that the process ends up waiting for the kernel to page memory in from swap space over and over. That's the only scenario I can think of, at the moment, where the difference of allocator could cause a 15X slowdown, and I think it would likely be rare, hard to reproduce, and may not even depend on which allocator.

One thing you can try is, if you're on linux put this before the command, and then after the command exits you'll get a nice printout of some statistics that might be a clue:

$ /usr/bin/time -f "Max RSS: %M KB\nMajor faults: %F\nMinor faults: %R" cargo build --quiet
Max RSS: 26624 KB
Major faults: 0
Minor faults: 6093