Add a stop-the-world, serial Compressor #1340

no-defun-allowed · 2025-07-15T05:25:14Z

This PR adds a Compressor-esque garbage collector, which is based on MarkCompact but uses a mark bitmap for forwarding metadata like the Compressor.

no-defun-allowed · 2025-07-15T05:57:24Z

The current failure to compile mock_test_allocator_info is a one-line fix, but I don't think the mock VM tests are going to work, as the mock VM uses header metadata and the Compressor requires mark bits to be in a side bitmap.

src/plan/compressor/global.rs

src/plan/compressor/mutator.rs

src/plan/compressor/global.rs

src/policy/compressor/compressorspace.rs

src/policy/compressor/forwarding.rs

k-sareen · 2025-07-15T06:07:38Z

Thanks for the PR @no-defun-allowed! The major thing I see is that currently the Compressor plan is not using the LOS. Was this intentional or accidental?

no-defun-allowed · 2025-07-15T06:23:53Z

Thanks for the PR @no-defun-allowed! The major thing I see is that currently the Compressor plan is not using the LOS. Was this intentional or accidental?

Definitely accidental. Thanks for the review.

qinsoon · 2025-07-16T01:22:22Z

The current failure to compile mock_test_allocator_info is a one-line fix, but I don't think the mock VM tests are going to work, as the mock VM uses header metadata and the Compressor requires mark bits to be in a side bitmap.

You can cherry pick this commit to the PR: qinsoon@ac90d4b.

It runs mock VM with side metadata. Compressor currently fails for tests that are not using contiguous space.

no-defun-allowed · 2025-07-16T04:39:34Z

You can cherry pick this commit to the PR: qinsoon@ac90d4b.

Thanks!

It runs mock VM with side metadata. Compressor currently fails for tests that are not using contiguous space.

Right. Should we skip such tests when running with the Compressor?

src/policy/compressor/forwarding.rs

src/policy/compressor/compressorspace.rs

qinsoon · 2025-07-16T04:52:08Z

It runs mock VM with side metadata. Compressor currently fails for tests that are not using contiguous space.

Right. Should we skip such tests when running with the Compressor?

Is there any intrinsic reason why the compressor cannot run with discontiguous spaces? I feel there is none. It would be better to allow running Compressor with discontiguous spaces.

At the same time, there might be practical reasons that this might be not easy, or it requires too much efforts. It is also okay to skip those tests.

src/plan/compressor/mutator.rs

no-defun-allowed · 2025-07-16T05:35:02Z

Is there any intrinsic reason why the compressor cannot run with discontiguous spaces? I feel there is none. It would be better to allow running Compressor with discontiguous spaces.

I feel that way too. The judgement was more about how closely we want to follow the Compressor paper - one simple approach to support discontiguous spaces would be to compress [sic] each chunk of a space, which would also give a simple way to parallelise (which is not the way in the paper).

The block offset vector is currently a Rust Vec<AtomicUsize> which I do the address-to-block calculation assuming a contiguous heap; I should be using some other metadata mechanism for that anyway.

k-sareen · 2025-07-17T01:30:03Z

The block offset vector is currently a Rust Vec which I do the address-to-block calculation assuming a contiguous heap; I should be using some other metadata mechanism for that anyway.

For what its worth, ART also uses a uint32_t* for the offset vector calculation.

This reverts commit ac90d4b.

no-defun-allowed · 2025-07-18T04:20:35Z

Now the block offset vector and the marks are in their own local metadata. I'm still undecided on if and how to support discontiguous spaces, since the forwarding algorithm in the Compressor paper seems very tied to having a contiguous space.

qinsoon · 2025-07-18T04:21:09Z

The failures for 32 bits was caused by this: #1342. However, you reverted the side metadata commit for mock VM, so the bug won't appear in the PR any way.

qinsoon · 2025-07-18T04:21:44Z

Now the block offset vector and the marks are in their own local metadata. I'm still undecided on if and how to support discontiguous spaces, since the forwarding algorithm in the Compressor paper seems very tied to having a contiguous space.

If that's the case, just skip the tests that use discontiguous space for Compressor.

src/util/metadata/side_metadata/spec_defs.rs

no-defun-allowed · 2025-07-18T06:11:45Z

I added a MMTK_PLAN=discontiguous set to the CI script, for tests which require discontiguous heaps. The script also removes the Compressor from all on i686, since i686 defaults to a discontiguous heap too; some tests like test_vm_layout_default should still be tested on x86_64 but will panic on i686.

k-sareen · 2025-07-18T09:35:37Z

I've made an issue for the commit that fixes the space mapping since it's a very subtle bug that I think can be checked with assertions (cd3bde3): #1343

qinsoon

I found the Transducer implementation is not easy to understand. A bit more comments would be helpful.

We probably also need a PR for OpenJDK that supports Compressor.

qinsoon · 2025-07-21T05:19:21Z

Cargo.toml

@@ -197,6 +197,9 @@ extreme_assertions = []
 # Enable multiple spaces for NoGC, each allocator maps to an individual ImmortalSpace.
 nogc_multi_space = []

+# Disable multiple spaces for Compressor.


Is there any reason that we want to keep this feature? Would it be useful to anyone?

This will be the most space-efficient an algorithm can be. I believe Steve mentioned it.

@no-defun-allowed Just mention in the comment there that this is configuration will move all objects. Don't use if you can't tolerate that. This is primarily for understanding how space-efficient we can be.

Right. Also the non-moving semantic can't be used with this feature.

It would be better if we can panic when the non-moving semantic is ever used. This probably would need a 'fake allocator' that panics when used, and the non-moving semantic is mapped to it. I am not sure if it is worth it.

qinsoon · 2025-07-21T05:45:32Z

src/policy/compressor/forwarding.rs

+/// each block by serialising the state using [`Transducer::encode`];
+/// the state can then be deserialised using [`Transducer::decode`].
+#[derive(Debug)]
+struct Transducer {


It is a bit hard to understand how this type works. More comments would help. For example, explain what each fields mean. Also you could comment step to say that address is always either the start of an object, or the end of an object, etc.

src/policy/compressor/forwarding.rs

qinsoon · 2025-07-21T05:57:16Z

I also requested reviews from @wks

…nt_metadata

k-sareen

Thanks for the updates! I think it looks pretty good now

src/policy/compressor/forwarding.rs

src/util/metadata/side_metadata/global.rs

no-defun-allowed added 5 commits July 15, 2025 15:23

Add a vaguely Compressor-esque GC

1ddbb5e

Clean up some

b80ba55

rustfmt

cd136b1

clippy fix

43b094a

cargo fmt

4e02f12

k-sareen self-requested a review July 15, 2025 05:37

k-sareen reviewed Jul 15, 2025

View reviewed changes

Review comments

10e5439

Capitalise sentences

e568453

qinsoon and others added 2 commits July 16, 2025 03:43

Allow running MockVM with side metadata

b30d542

Add a feature to disable LOS and immortal space for Compressor

237fbbf

qinsoon reviewed Jul 16, 2025

View reviewed changes

src/policy/compressor/forwarding.rs Outdated Show resolved Hide resolved

src/policy/compressor/compressorspace.rs Outdated Show resolved Hide resolved

k-sareen reviewed Jul 16, 2025

View reviewed changes

src/plan/compressor/mutator.rs Outdated Show resolved Hide resolved

no-defun-allowed added 3 commits July 18, 2025 03:28

Use MMTk metadata for the offset vector and a separate mark bitmap

efcf713

Revert "Allow running MockVM with side metadata"

def9e36

This reverts commit ac90d4b.

Fix some comments

4608807

k-sareen reviewed Jul 18, 2025

View reviewed changes

src/util/metadata/side_metadata/spec_defs.rs Outdated Show resolved Hide resolved

Skip MockVM tests with the Compressor on unsupported configurations

2470de7

no-defun-allowed added 3 commits July 18, 2025 06:29

Include common plan for create_space_mapping

cd3bde3

cargo fmt

a004cd6

i686 is spelled x86 in this instance

dfe9185

k-sareen mentioned this pull request Jul 18, 2025

Add more assertions checking space and allocator mappings #1343

Open

Clean up some more

b420216

k-sareen requested a review from qinsoon July 21, 2025 04:59

qinsoon reviewed Jul 21, 2025

View reviewed changes

qinsoon requested a review from wks July 21, 2025 05:56

no-defun-allowed added 2 commits July 21, 2025 06:53

Fix up some names and comments, and add SideMetadataSpec::are_differe…

1ef15cf

…nt_metadata

Add a warning about the compressor_single_space feature

b819763

k-sareen reviewed Jul 21, 2025

View reviewed changes

src/policy/compressor/forwarding.rs Outdated Show resolved Hide resolved

src/util/metadata/side_metadata/global.rs Outdated Show resolved Hide resolved

More spellings of things

fd84cbf

Add a stop-the-world, serial Compressor #1340

Are you sure you want to change the base?

Add a stop-the-world, serial Compressor #1340

Uh oh!

Conversation

no-defun-allowed commented Jul 15, 2025

Uh oh!

no-defun-allowed commented Jul 15, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

k-sareen commented Jul 15, 2025

Uh oh!

no-defun-allowed commented Jul 15, 2025

Uh oh!

qinsoon commented Jul 16, 2025

Uh oh!

no-defun-allowed commented Jul 16, 2025

Uh oh!

Uh oh!

Uh oh!

qinsoon commented Jul 16, 2025

Uh oh!

Uh oh!

no-defun-allowed commented Jul 16, 2025

Uh oh!

k-sareen commented Jul 17, 2025

Uh oh!

no-defun-allowed commented Jul 18, 2025

Uh oh!

qinsoon commented Jul 18, 2025

Uh oh!

qinsoon commented Jul 18, 2025

Uh oh!

Uh oh!

no-defun-allowed commented Jul 18, 2025

Uh oh!

k-sareen commented Jul 18, 2025

Uh oh!

qinsoon left a comment

Choose a reason for hiding this comment

Uh oh!

qinsoon Jul 21, 2025

Choose a reason for hiding this comment

Uh oh!

k-sareen Jul 21, 2025

Choose a reason for hiding this comment

Uh oh!

k-sareen Jul 21, 2025

Choose a reason for hiding this comment

Uh oh!

qinsoon Jul 21, 2025

Choose a reason for hiding this comment

Uh oh!

qinsoon Jul 21, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

qinsoon commented Jul 21, 2025

Uh oh!

k-sareen left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!