HDDS-14800. Guard RocksDB iterator against closed DB during volume

[priyeshkaratha]

…ilure

What changes were proposed in this pull request?

Provide a one-liner summary of the changes in the PR Title field above.
It should be in the form of HDDS-1234. Short summary of the change.

Please describe your PR in detail:

• What changes are proposed in the PR? and Why? It would be better if it is written from third person's
  perspective not just for the reviewer.
• Provide as much context and rationale for the pull request as possible. It could be copy-paste from
  the Jira's description if the jira is well defined.
• If it is complex code, describe the approach used to solve the issue. If possible attach design doc,
  issue investigation, github discussion, etc.

Examples of well-written pull requests:

• #3980
• #5265
• #4701
• #5283
• #5300

What is the link to the Apache JIRA

Please create an issue in ASF JIRA before opening a pull request, and you need to set the title of the pull
request which starts with the corresponding JIRA issue number. (e.g. HDDS-XXXX. Fix a typo in YYY.)

If you do not have an ASF Jira account yet, please follow the first-time contributor
instructions in the Jira guideline.

(Please replace this section with the link to the Apache JIRA)

How was this patch tested?

(Please explain how this patch was tested. Ex: unit tests, manual tests, workflow run on the fork git repo.)
(If this patch involves UI changes, please attach a screenshot; otherwise, remove this.)

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request resolves a critical race condition where RocksDB iterators could attempt to access a database that has been concurrently closed, leading to native crashes. The changes introduce robust checks within the iterator logic to determine if the database is closed before proceeding with operations, allowing for a clean exit. Furthermore, the database's close method now ensures that all active iterators have completed and released their resources before the physical database shutdown, significantly improving the stability and reliability of RocksDB interactions in scenarios like volume failures or concurrent shutdown procedures.

Highlights

• RocksDB Iterator Safety: Implemented robust guards within RocksDB iterators to prevent native crashes when the underlying database is concurrently closed, ensuring graceful termination of iteration.
• Synchronized DB Closure: Enhanced the database closure mechanism to wait for all active iterators to release their references before physically shutting down, preventing premature closure issues.
• Comprehensive Testing: Added extensive new test cases to validate the resilience of RocksDB iterators and database closure under various concurrent conditions, including simulated volume failures.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces a robust mechanism to prevent native crashes when a RocksDB instance is closed while iterators are still active. This is achieved through a combination of reference counting to delay the physical closing of the database and adding guards to iterator methods to handle the logically closed state gracefully. The implementation is solid and is accompanied by a comprehensive set of new unit and integration tests that simulate the race condition. The changes significantly improve the stability of database interactions. I found one minor typo in an exception message that I've commented on.

[gemini-code-assist]

There's a minor typo in the exception message string. A space is missing before "is closed", which will result in a message like "...tableNameis closed". Adding a space will improve readability.

Suggested change

	throw new RocksDatabaseException("Cannot removeFromDB: underlying RocksDB for table "
	+ rocksDBTable.getName() + "is closed");
	throw new RocksDatabaseException("Cannot removeFromDB: underlying RocksDB for table "
	+ rocksDBTable.getName() + " is closed");