Online Indexer: replace the synchronized runner with a heartbeat #3530

jjezra · 2025-08-04T15:30:30Z

Each indexing session, for each index, will create a key-value heartbeat of the format:
[prefix, xid] -> [indexing-type, genesis time, heartbeat time]
Indexing session that are expected to be exclusive will throw an exception if another, active, session exists.

Motivation:
1. Represent the heartbeat in every index during multi target indexing (currently - only the master index has a sync lock)
2. Keep a heartbeat during mutual indexing, which can allow better automatic decision making
3. Decide about exclusiveness according to the indexing method (currently - user input)

With this change, the equivalent of a sync lock will be determined by the indexing type and cannot be set by the users. The index configuration function setUseSynchronizedSession will have no effect on the indexing process.

During graduate code upgrade on multiple servers, there may be a situation where one server is indexing with a synchronized session lock, while another server builds the same index with an exclusive heartbeat "lock". If that happens:
a) There will be no more than two concurrent active sessions
b) The indexing sessions will conflict each other until one of the indexers will give up. While this may be not optimal, the generated index will be valid.

Resolve #3529

ScottDugas

I didn't get a chance to look at all of the code, we can discuss some of the things offline.

ScottDugas · 2025-08-12T20:15:18Z

...st/java/com/apple/foundationdb/record/provider/foundationdb/OnlineIndexerBuildIndexTest.java

@@ -180,7 +178,6 @@ <M extends Message> void singleRebuild(
        if (!safeBuild) {


What does safeBuild here mean in this context? Do we need to continue to prevent concurrency?

The things that had left to non-safeBuild, as it seems, are clearing and marking the index as write-only and setting constrains that, IIUIC, should not be happening after the previous clearing.

After so many changes in the code, maybe it is time to re-design (and possibly simplify) the singleRebuild. Should that be a separate PR?

changing and cleaning up safeBuild should not be part of this PR.

...st/java/com/apple/foundationdb/record/provider/foundationdb/OnlineIndexerBuildIndexTest.java

...ain/java/com/apple/foundationdb/record/provider/foundationdb/OnlineIndexOperationConfig.java

...ore/src/main/java/com/apple/foundationdb/record/provider/foundationdb/IndexingHeartbeat.java

...st/java/com/apple/foundationdb/record/provider/foundationdb/OnlineIndexingHeartbeatTest.java

...yer-core/src/main/java/com/apple/foundationdb/record/provider/foundationdb/IndexingBase.java

...st/java/com/apple/foundationdb/record/provider/foundationdb/OnlineIndexingHeartbeatTest.java

...er-core/src/main/java/com/apple/foundationdb/record/provider/foundationdb/OnlineIndexer.java

ScottDugas

There were also two comments left on my previous review, that I want to followup on. I resolved everything else in that review, or answered in a (I hope) definitive way, so it should be easy to go back through.

...r-core/src/main/java/com/apple/foundationdb/record/provider/foundationdb/FDBRecordStore.java

...yer-core/src/main/java/com/apple/foundationdb/record/provider/foundationdb/IndexingBase.java

...st/java/com/apple/foundationdb/record/provider/foundationdb/OnlineIndexingHeartbeatTest.java

ScottDugas · 2025-08-18T19:55:49Z

...ore/src/main/java/com/apple/foundationdb/record/provider/foundationdb/IndexingHeartbeat.java

+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.atomic.AtomicInteger;
+
+public class IndexingHeartbeat {


We discussed offline, and decided to take a balanced approach of changing the method to info, so that this is, on disk, future-proofed for being upgraded to be more generic if we want to use it elsewhere. There would still need to be some substantial changes to the code, but it would need to take a subspace instead of a store and an index.

.../java/com/apple/foundationdb/record/provider/foundationdb/IndexingHeartbeatLowLevelTest.java

Each indexing session, for each index, will create a key-value heartbeat of the format: [prefix, xid] -> [indexing-type, genesis time, heartbeat time] Indexing session that are expected to be exclusive will throw an exception if another, active, session exists. Motivation: 1. Represent the heartbeat in every index during multi target indexing (currently - only the master index has a sync lock) 2. Keep a heartbeat during mutual indexing, which can allow better automatic decision making 3. Decide about exclusiveness according to the indexing method (currently - user input) Resolve FoundationDB#3529

...yer-core/src/main/java/com/apple/foundationdb/record/provider/foundationdb/IndexingBase.java

...ore/src/main/java/com/apple/foundationdb/record/provider/foundationdb/IndexingHeartbeat.java

.../java/com/apple/foundationdb/record/provider/foundationdb/IndexingHeartbeatLowLevelTest.java

...st/java/com/apple/foundationdb/record/provider/foundationdb/OnlineIndexingHeartbeatTest.java

...yer-core/src/main/java/com/apple/foundationdb/record/provider/foundationdb/IndexingBase.java

ScottDugas · 2025-09-05T19:41:38Z

...yer-core/src/main/java/com/apple/foundationdb/record/provider/foundationdb/IndexingBase.java

@@ -1259,11 +1214,11 @@ protected static <T> T findException(@Nullable Throwable ex, Class<T> classT) {
        return null;
    }

-    protected static boolean shouldLessenWork(@Nullable FDBException ex) {
+    protected static boolean doNotLessenWork(@Nullable FDBException ex) {


Why are you inverting this method, this seems completely unrelated to the change at hand.

Tried to please an idea warning. But you are right, it is unreadable. Converting back...

...ore/src/main/java/com/apple/foundationdb/record/provider/foundationdb/IndexingHeartbeat.java

ScottDugas · 2025-09-05T19:57:17Z

...ava/com/apple/foundationdb/record/provider/foundationdb/OnlineIndexerIndexFromIndexTest.java

+        FDBRecordStoreTestBase.RecordMetaDataHook hook = myHook(sourceIndex, tgtIndex);
+        openSimpleMetaData(hook);
+
+        Semaphore pauseMutualBuildSemaphore = new Semaphore(1);


Doing it in a followup PR sounds good. I think you could make it a little better by making a class ConfigLoaderThatPausesAfterOnePass that could then store all of these semaphores, and have some more helpful.
There might be something even better, but we're probably getting diminishing returns here, and trying to clean this up more isn't worth it.

To be clear, looking to see if some other tests could take advantage of pauseAfterOnePass in a followup PR sounds good, but it's probably not worth creating something more full-featured.

...st/java/com/apple/foundationdb/record/provider/foundationdb/OnlineIndexerBuildIndexTest.java

...ain/java/com/apple/foundationdb/record/provider/foundationdb/OnlineIndexOperationConfig.java

...c/test/java/com/apple/foundationdb/record/provider/foundationdb/OnlineIndexerSimpleTest.java

ScottDugas

Teamscale is complaining about two test gaps, these seem easy enough to add coverage in the test you already have.

There's also a few test methods that are beyond the length, or nesting depth threshold, probably worth addressing those.

…ions and tests length

jjezra added the breaking change Changes that are not backwards compatible label Aug 4, 2025

jjezra force-pushed the indexing_heartbeat branch from 89d0263 to f318f65 Compare August 7, 2025 15:55

jjezra requested a review from ScottDugas August 8, 2025 17:18

jjezra force-pushed the indexing_heartbeat branch from a680857 to 83830c6 Compare August 11, 2025 12:01

jjezra marked this pull request as ready for review August 11, 2025 14:13

jjezra marked this pull request as draft August 11, 2025 15:06

jjezra marked this pull request as ready for review August 11, 2025 20:23

ScottDugas requested changes Aug 12, 2025

View reviewed changes

jjezra force-pushed the indexing_heartbeat branch from 4cf86f4 to 97d02b6 Compare August 15, 2025 13:22

jjezra requested a review from ScottDugas August 15, 2025 13:52

ScottDugas requested changes Aug 18, 2025

View reviewed changes

jjezra force-pushed the indexing_heartbeat branch 2 times, most recently from 86ad032 to 2a9f7e5 Compare August 25, 2025 15:35

jjezra requested a review from ScottDugas August 25, 2025 15:36

jjezra added 16 commits September 2, 2025 11:22

Set/clear heartbeats at start/end

ca01fe8

Optimize heartbeat clear. Deprecate use-sync-indexing API

17f1aa7

false AvoidUsingHardCodedIP warning

a5891b0

Clear heartbeats after rebuild (whting the same transaction)

7e0fa30

Itarate targets explicitly after rebuild

afa990d

thenApply -> whenComplete

a9b5917

Add API get/clear heartbeats

747d76f

Add 1st test - low level

826dfa4

Add two more tests

8759309

Add a comment..

26d7f2c

Deprecated sync lock API functions

988f107

Some cleanup

bbf3af6

At genesis, first check indexing type, then heartbeat

587fdb2

Use subspace(...).unpack(..)

eff9c18

Verify no leftover heartbeats after any singleRebuild

b628851

jjezra added 2 commits September 2, 2025 11:22

Apply Scott's requested changes

f9c08b2

Apply (more of) Scott's requested changes

e5cbef1

ScottDugas requested changes Sep 2, 2025

View reviewed changes

jjezra force-pushed the indexing_heartbeat branch from 2a9f7e5 to d3b4baa Compare September 2, 2025 22:34

jjezra mentioned this pull request Sep 3, 2025

List of things to do post switching the Online Indexer for sync lock to heartbeats #3567

Open

Apply (even more of) Scott's requested changes

a8eff82

jjezra force-pushed the indexing_heartbeat branch from 921fd82 to a8eff82 Compare September 4, 2025 12:10

jjezra requested a review from ScottDugas September 4, 2025 12:14

jjezra added 2 commits September 4, 2025 12:32

Minor cleanup to make IndexingBase a little shorter

5e3d370

pmdMain: unused param

73442f0

ScottDugas requested changes Sep 5, 2025

View reviewed changes

Apply PR requested changes

fbfd5c5

jjezra requested a review from ScottDugas September 8, 2025 14:11

jjezra added enhancement New feature or request and removed breaking change Changes that are not backwards compatible labels Sep 8, 2025

ScottDugas reviewed Sep 9, 2025

View reviewed changes

Attempt to please some Teamscale croaks - untested (deprecated) funct…

48f0f07

…ions and tests length

jjezra requested a review from ScottDugas September 10, 2025 11:01

		@@ -180,7 +178,6 @@ <M extends Message> void singleRebuild(
		if (!safeBuild) {

Online Indexer: replace the synchronized runner with a heartbeat #3530

Are you sure you want to change the base?

Online Indexer: replace the synchronized runner with a heartbeat #3530

Uh oh!

Conversation

jjezra commented Aug 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ScottDugas left a comment

Choose a reason for hiding this comment

Uh oh!

ScottDugas Aug 12, 2025

Choose a reason for hiding this comment

Uh oh!

jjezra Aug 13, 2025

Choose a reason for hiding this comment

Uh oh!

ScottDugas Aug 18, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ScottDugas left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ScottDugas Aug 18, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ScottDugas Sep 5, 2025

Choose a reason for hiding this comment

Uh oh!

jjezra Sep 6, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ScottDugas Sep 5, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ScottDugas left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jjezra commented Aug 4, 2025 •

edited

Loading