8371046: Segfault in compiler/whitebox/StressNMethodRelocation.java with -XX:+UseZGC

[chadrako]

JDK-8371046

This pull request fixes two crashes (see below) and adds InvalidationReason::RELOCATED to better describe why an nmethod is marked not entrant during relocation.

---

1. Test Bug

It’s possible for an nmethod to be unloaded without its _state being explicitly set to not_entrant. Checking only is_in_use() isn’t sufficient, since the nmethod may already be in the process of unloading and therefore may not have a lock (as with ZGC, where nmethods are locked individually).

The fix adds an additional is_unloading() check in WhiteBox before acquiring the lock.

This issue was reproducible fairly consistently (every few runs) by executing compiler/whitebox/StressNMethodRelocation.java with -XX:+UseZGC -XX:ReservedCodeCacheSize=32m

After applying this patch, the original crash stopped occurring, though a more infrequent crash was still observed.

---

2. Implementation Bug

nmethod::relocate works by copying the instructions of an nmethod and then adjusting the call sites to account for new PC-relative offsets.

Previously, this fix-up happened after calling post_init(), which registers the nmethod and makes it visible to the GC. This introduced a race condition where the GC might attempt to resolve a call site before it had been fixed.

The fix ensures that all call sites are patched before the nmethod is registered.

In testing, the crash previously occurred roughly 60 times in 5,000 runs (~1.2%). With this patch, no crashes were observed in the same number of runs.

---

Progress

• Change must be properly reviewed (1 review required, with at least 1 Reviewer)
• Change must not contain extraneous whitespace
• Commit message must refer to an issue

Issue

• JDK-8371046: Segfault in compiler/whitebox/StressNMethodRelocation.java with -XX:+UseZGC (Bug - P4)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk.git pull/28241/head:pull/28241
$ git checkout pull/28241

Update a local copy of the PR:
$ git checkout pull/28241
$ git pull https://git.openjdk.org/jdk.git pull/28241/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 28241

View PR using the GUI difftool:
$ git pr show -t 28241

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/28241.diff

Using Webrev

Link to Webrev Comment

[bridgekeeper]

👋 Welcome back chadrako! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

[openjdk]

❗ This change is not yet ready to be integrated.
See the Progress checklist in the description for automated requirements.

[openjdk]

@chadrako The following labels will be automatically applied to this pull request:

• graal
• hotspot

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing lists. If you would like to change these labels, use the /label pull request command.

[mlbridge]

Webrevs

• 00: Full (dac140bb)