8354922: ZGC: Use MAP_FIXED_NOREPLACE when reserving memory

[sercher]

Hi all,

Please review the backport of JDK-8354922 to JDK 21.

There are versions of the Linux kernel that do not honor the address hint when mmapping memory without MAP_FIXED, that clobbers the older mappings overlapped with the requested range. A safer MAP_FIXED_NOREPLACE flag is used with ZGC since Java 25.

JDK 21 crashes with -XX:+UseZGC with RANDMMAP [1] kernel patch enabled, while JDK 25 works as expected.

$ ./25.0.1/bin/java -XX:+UseZGC -version
openjdk version "25.0.1" 2025-10-21 LTS
OpenJDK Runtime Environment (build 25.0.1+11-LTS)
OpenJDK 64-Bit Server VM (build 25.0.1+11-LTS, mixed mode, sharing)

$ ./21.0.9/bin/java -XX:+UseZGC -version
[0.070s][error][gc] Failed to reserve enough address space for Java heap
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

ZGC relies on kernel that respects address hints in mmap system call, which is not the case with RANDMMAP enabled kernels. ZVirtualMemoryManager::pd_reserve() always returns false at line 50 because the mmap returned value never matches the requested address.

jdk21u-dev/src/hotspot/os/posix/gc/z/zVirtualMemory_posix.cpp

Lines 40 to 51 in 634d3fa

	bool ZVirtualMemoryManager::pd_reserve(zaddress_unsafe addr, size_t size) {
	void* const res = mmap((void*)untype(addr), size, PROT_NONE, MAP_ANONYMOUS|MAP_PRIVATE|MAP_NORESERVE, -1, 0);
	if (res == MAP_FAILED) {
	// Failed to reserve memory
	return false;
	}
	if (res != (void*)untype(addr)) {
	// Failed to reserve memory at the requested address
	munmap(res, size);
	return false;
	}

Since Linux 4.17 there's MAP_FIXED_NOREPLACE flag in mmap, that satisfies the addr requests and reports failure when the requested range overlaps a pre-existing mapping.

The backport isn't clean. JDK 21 doesn't have JDK-8350441 that renamed zVirtualMemory_posix.cpp to zVirtualMemoryManager_posix.cpp, and JDK-8341692 that removed non-generational mode in ZGC. In absense of JDK-8341692 the same approach was taken on gc/x/ version of pd_reserve().

It is also proposed to backport JDK-8313319 that prevents unnecessary mmap-munmap cycle, that will follow in a separate PR (#2476 ).

[1] https://pax.grsecurity.net/docs/randmmap.txt

---

Progress

• Change must be properly reviewed (1 review required, with at least 1 Reviewer)
• Change must not contain extraneous whitespace
• Commit message must refer to an issue
• JDK-8354922 needs maintainer approval

Issue

• JDK-8354922: ZGC: Use MAP_FIXED_NOREPLACE when reserving memory (Enhancement - P4 - Requested)

Reviewers

• Stefan Karlsson (@stefank - Reviewer)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk21u-dev.git pull/2475/head:pull/2475
$ git checkout pull/2475

Update a local copy of the PR:
$ git checkout pull/2475
$ git pull https://git.openjdk.org/jdk21u-dev.git pull/2475/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 2475

View PR using the GUI difftool:
$ git pr show -t 2475

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk21u-dev/pull/2475.diff

Using Webrev

Link to Webrev Comment

[bridgekeeper]

👋 Welcome back schernyshev! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

[openjdk]

❗ This change is not yet ready to be integrated.
See the Progress checklist in the description for automated requirements.

[openjdk]

This backport pull request has now been updated with issue from the original commit.

[mlbridge]

Webrevs

• 00: Full (9c950cf0)

[stefank]

This looks good to me.

[openjdk]

⚠️ @sercher This change is now ready for you to apply for maintainer approval. This can be done directly in each associated issue or by using the /approval command.

[sercher]

Thanks @stefank

[sercher]

/approval request This fixes ZGC on systems that do not honor the address hint when mmapping memory without MAP_FIXED

[openjdk]

@sercher
8354922: The approval request has been created successfully.