Simplify jemalloc setup

[madsmtm]

In the past, #[used] had to appear in the top-level crate to have a consistent effect on the linker. This has been fixed a while ago for ELF with the introduction of the symbols.o file in #95604, and more recently for Mach-O in #133832, which means that libraries can now implement the required workarounds themselves. This allows moving these #[used] declarations out of our main.rs.

Specifically, I have moved them into tikv-jemalloc-sys where they belong in tikv/jemallocator#109 and done the same for mimalloc in purpleprotocol/mimalloc_rust#146 (in case we want to experiment with switching to that one day).

Test with:

./x build library src/tools/rustdoc src/tools/clippy --set rust.jemalloc=true

# macOS
lldb -- ./build/host/stage1/bin/rustc -vV
(lldb) b _rjem_je_zone_register
(lldb) run
# Should breakpoint, this means that the allocator was properly linked

# Linux
lldb -- ./build/host/stage1/bin/rustc -vV
(lldb) b malloc
(lldb) run
# Should breakpoint, inspect that the `malloc` symbol comes from the `rustc` binary and not from `libc`

try-job: aarch64-gnu
try-job: dist-aarch64-linux
try-job: dist-x86_64-musl
try-job: dist-x86_64-apple
try-job: dist-aarch64-apple

[madsmtm]

tikv-jemalloc-sys v0.6.1 has been released with the mentioned PR, so this is now ready. I'm less certain about the Linux parts here, so please double-check that.

Also, in the second commit, I fixed librustdoc and clippy building with --features jemalloc / ./x build --set rust.jemalloc=true, unclear to me how this wasn't caught by CI? Are these built differently there?

@rustbot ready
r? compiler
@bors try

[rustbot]

Some changes occurred in src/tools/clippy

cc @rust-lang/clippy

The Miri subtree was changed

cc @rust-lang/miri

These commits modify the Cargo.lock file. Unintentional changes to Cargo.lock can be introduced when switching branches and rebasing PRs.

If this was unintentional then you should revert the changes before this PR is merged.
Otherwise, you can ignore this comment.

[rust-bors]

☀️ Try build successful (CI)
Build commit: 8d39231 (8d39231d22563844d4088a7af200b6885c749c85, parent: 79966ae420f38c5861d177356a3446023c090d6d)

[bors]

☔ The latest upstream changes (presumably #148412) made this pull request unmergeable. Please resolve the merge conflicts.

[jdonszelmann]

could you also link to this PR here? It's much easier for someone to accidentally change something in rustc and this reference to go stale, while if you link to this PR the reasoning always stays clear.

r=me after that

@rustbot author

[mati865]

Or perhaps pin the reference to a commit?

[madsmtm]

I went with:

/// See docs in https://github.com/rust-lang/rust/blob/HEAD/compiler/rustc/src/main.rs
/// and https://github.com/rust-lang/rust/pull/146627 for why we need this `use` statement.

(There wasn't really a commit I could pin it to while changing the contents in the same commit).

[rustbot]

Error: shortcut handler unexpectedly failed in this comment: failed to remove Label { name: "S-waiting-on-review" }

Please file an issue on GitHub at triagebot if there's a problem with this bot, or reach out on #triagebot on Zulip.

[bors]

⌛ Testing commit 65f0b7a with merge 0b329f8...

[Zalathar]

Bors, this has already been merged.

@bors r- retry

[Zalathar]

• Perf results from rollup: Rollup of 16 pull requests #148851 (comment)
• Comparison URL

Regressions in doc benchmarks across the board; this seems unintended?

[RalfJung]

Maybe rustdoc accidentally is not using jemalloc any more?

[Kobzol]

Some allocation changes have definitely happened there:

  496,710,988  ???:
  -408,481,523    realloc
   261,159,975    do_rallocx
   253,281,882    _rjem_je_arena_ralloc
  -193,667,679    tcache_bin_flush_small
   159,404,361    _rjem_je_arena_ralloc_no_move
    96,036,695    _rjem_je_tcache_bin_flush_small
    81,229,215    tcache_bin_flush_edatas_lookup
   -76,118,340    malloc_default
    67,153,310    rtree_read
    64,297,968    rtree_metadata_read
   -54,428,351    pa_alloc
    51,577,458    _rjem_je_arena_cache_bin_fill_small
    29,630,447    _rjem_je_malloc_default
    18,729,715    _rjem_je_emap_update_edata_state
    17,881,154    _rjem_je_eset_fit
    13,595,487    emap_try_acquire_edata_neighbor_impl
    11,825,510    _rjem_je_te_event_trigger
   -10,311,127    free_default
   -10,132,168    ecache_evict
     8,316,595    _rjem_je_eset_remove
     8,300,824    _rjem_je_tcache_bin_flush_stashed
     8,176,825    _rjem_je_eset_insert
     8,124,547    emap_rtree_leaf_elms_lookup
     7,930,250    base_alloc_impl
    -7,411,000    tcache_bin_flush_stashed
     7,385,380    _rjem_je_hook_invoke_dalloc
     7,385,320    _rjem_je_hook_invoke_alloc
     7,226,602    extent_recycle
    -7,008,483    eset_remove

I would suggest perf. testing any PR that touches things like memory allocation, for the next time 😅

I will post a revert, so that we can figure out what happened here.

[Kobzol]

#148896

[madsmtm]

Some allocation changes have definitely happened there:

[snip]

Could you tell me how you arrived at that table? I'm unfamiliar with Linux perf tooling.

I would suggest perf. testing any PR that touches things like memory allocation, for the next time 😅

Yeah, sorry about that, I should've done that!

[Kobzol]

Could you tell me how you arrived at that table? I'm unfamiliar with Linux perf tooling.

Yeah, sorry. If you open a benchmark row from the comparison table, it shows "Command for profiling locally", which you can copy paste in a checkout of https://github.com/rust-lang/rustc-perf to do a Cachegrind diff before/after the PR.