Handle multiple updates to the same record in a transaction #3722
+201
−7
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…latest IndexWriter changes
# Conflicts: # fdb-record-layer-lucene/src/test/java/com/apple/foundationdb/record/lucene/LuceneIndexMaintenanceTest.java
jjezra
reviewed
Nov 11, 2025
|
|
||
| /** | ||
| * Try to find the document for the given record in the segment index. | ||
| * This method would first try to find the document using teh existing reader. If it can't, it will refresh the reader |
Contributor
There was a problem hiding this comment.
Suggested change
| * This method would first try to find the document using teh existing reader. If it can't, it will refresh the reader | |
| * This method would first try to find the document using the existing reader. If it can't, it will refresh the reader |
| * writer may cache the changes in NRT and the reader (created earlier) can't see them. Refreshing the reader from the | ||
| * writer can alleviate this. If the index can't find the document with the refresh reader, null is returned. | ||
| * Note that the refresh of the reader will do so at the {@link com.apple.foundationdb.record.lucene.directory.FDBDirectoryWrapper} | ||
| * and so has impact on the entire directory. |
ScottDugas
requested changes
Nov 11, 2025
| if (newReader != null) { | ||
| // previous reader instantiated but then writer changed | ||
| readersToClose.add(writerReader); | ||
| writerReader = LazyCloseable.supply(() -> newReader); |
Collaborator
There was a problem hiding this comment.
What happens if this is touched concurrently?
Contributor
Author
There was a problem hiding this comment.
Added test to update the readers concurrently
| ParameterizedTestUtils.booleans("isGrouped"), | ||
| Stream.of(0, 10), | ||
| Stream.of(0, 1, 4), | ||
| Stream.of(5365)); |
Collaborator
There was a problem hiding this comment.
Should this use RandomizedTestUtils.randomizedSeeds to run extra seeds during nightly?
| @ParameterizedTest | ||
| @MethodSource("multiUpdate") | ||
| void multipleUpdatesInTransaction(boolean isSynthetic, boolean isGrouped, int highWatermark, int updateCount, long seed) throws IOException { | ||
| final int documentCount = 15; |
Collaborator
There was a problem hiding this comment.
Would it make sense to have this be random, perhaps between 11 and 19?
As it is you either have 15 documents in the partition, or 10 and 5. Putting some randomness in there might find some interesting edge cases.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The issue at hand is that when running multiple update operations in a single transaction, the partition's document counts and the PK-segment index may get into an inconsistent state. The root cause is that the first update in the transaction clears the doc from the Lucene index and the PK index. Since the changes are not flushed, the IndexWriter has them cached in the NRT cache. The second record update would then not find the record in the PK index (because the segment has changed but the IndexReader does not yet reflect that) and therefore the delete is skipped, including updating the partition count. Note that it does attempt a delete-by-query that actually removes the doc from the Lucene index, but since we can't know that, the partition is not updated.
The solution is to
refreshtheDirectoryReaderwhen doing an update, so that any previously written changes are showing up. The refresh operation usesDirectoryReader.openIfChangedthat is more efficient in resources than using a brand newopencall.Resolve #3704