[NAE-2261] SEARCH Node - fetch cases from mongo #385
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Introduced properties for Redis Sentinel and SSL configurations to improve connectivity options and security. - Updated session and security management to use enhanced Redis configuration structure. - Refactored Redis property handling and connection factory creation for better maintainability and support of new features.
- add ssl configuration for redis client
- Introduced `DEFAULT_SENTINEL_NODE` constant for default node configuration. - Added a helper method `hasCredentials` to improve credential checks. - Refactored sentinel node parsing to handle edge cases with better logging.
…performance and reliability Replaced direct Elasticsearch operations with ElasticQueueManager for better batch processing and asynchronous handling of indexing and deletion tasks. Updated configuration to include queue properties and standardized dependency handling with constructor injection. This improves performance, scalability, and maintainability of Elasticsearch operations.
- Corrected Redis session namespace property to align with enhanced configuration standards. - Ensured consistency across static and non-static session configurations.
Add a return statement for empty batches to prevent unnecessary processing. Implement error handling to requeue failed batches and log the exception details. Prevent scheduling new tasks when the scheduler is shut down.
Previously, the log message implied immediate indexing, which was misleading. The updated message correctly reflects that the case is added to the indexing queue. This improves clarity and aligns with the actual behavior of the method.
Added cancellation of the atomicDelayer to ensure proper task cleanup during shutdown. Implemented a timeout-based termination for the scheduler, with fallback to forced shutdown to handle interruptions and prevent potential resource leaks.
Replaced complex termination logic with a single `scheduler.shutdown()` call. This improves code readability and maintains the same functionality while reducing potential error points in the shutdown process.
Revised the shutdown method to ensure proper handling of queued tasks and prevent potential delays. Added cancellation of the atomic delayer and improved scheduler termination logic with a timeout and fallback to immediate shutdown. This enhances system stability and resource cleanup reliability.
Lowered the default scheduledExecutorPoolSize from 50 to 10. This change optimizes resource utilization and prevents over-allocation of threads, improving system performance under typical workloads.
- add keyValue translations to elastic case mapping
- Remove default value for `ClusterConnectionMode` in MongoDB properties
Introduce a new `queue.delay` parameter in the Elasticsearch configuration within `application-test.yaml`. This ensures better control over queue processing behavior during test execution.
…tId field Updated the code to use `role.processId` instead of `role.netId` when fetching the PetriNet object. This resolves potential mismatches and ensures accurate retrieval of process identifiers.
Introduced validation annotations (@Valid and @min) for queue-related properties to enforce valid configurations. Added a RefreshPolicy property with a default value of NONE to enhance control over Elasticsearch refresh behavior. These changes improve reliability and configurability of the queue settings.
Replaced ElasticsearchTemplate with ElasticsearchClient for bulk operations, improving efficiency and consistency. Updated ElasticQueueManager and related services to handle BulkOperation objects, eliminating deprecated methods and simplifying queue handling logic. Adjusted indexing and deletion logic to align with the new approach.
Removed unnecessary parameters from ElasticQueueManager constructor and updated related instantiations. Simplified timer reset logic, added scheduler shutdown, and cleaned up unused imports and repositories in ElasticServiceConfiguration.
Added a condition to flush the queue when its size exceeds the maximum threshold, ensuring more efficient management. The timer reset now occurs only if the flush is not triggered, improving resource utilization and robustness.
Implemented Serializable in ElasticTask with a serialVersionUID to ensure class compatibility during serialization. Introduced the _class field and getter method to store and retrieve the task's class name, improving type recognition.
Removed unused `_class` field from Elastic models and updated queue shutdown logic with `@PreDestroy`. Modified ElasticCaseService to use ElasticsearchConverter for mapping, ensuring better compatibility and maintainability.
The ElasticQueueManager class no longer uses a generic type parameter, simplifying its definition. This change also updates its usage across ElasticCaseService with non-generic instantiations, improving code maintainability and reducing unnecessary complexity.
ElasticsearchConverter was imported but never used in the code. Removing it improves code clarity and eliminates unnecessary dependencies.
Simplify the case deletion flow by removing unnecessary null checks and redundant exception handling. This enhances code readability and ensures a more consistent execution path.
The removeByPetriNetId method and its interface declaration were deleted as they are no longer required. This cleanup improves code clarity and removes unnecessary functionality, ensuring better maintainability.
Replaced direct document passing with mapped objects using ElasticsearchConverter. This ensures proper handling and conversion of objects before indexing or upserting them into Elasticsearch.
Added handling for `BulkResponse` in ElasticQueueManager to allow future response processing. Removed unused/commented code in ElasticTask classes to improve code clarity and maintainability.
Removed the unused `BulkResponse bulkResponse` variable to clean up the code and avoid potential confusion. This change does not impact functionality but improves code readability and maintenance.
Ensure the timer is reset when an indexing batch fails to prevent potential delays in subsequent retries. This change enhances the robustness of the error handling logic.
Removed redundant `@throws` tag in the javadoc to improve clarity and adhere to current documentation standards. No functional changes were made in the code.
- Updated `application-engine-parent` version from `7.0.0-RC8.1` to `7.0.0-RC8.2` across multiple modules.
# Conflicts: # application-engine/pom.xml # nae-object-library/pom.xml # nae-spring-core-adapter/pom.xml # nae-user-ce/pom.xml # nae-user-common/pom.xml # pom.xml
# Conflicts: # application-engine/pom.xml # nae-object-library/pom.xml # nae-spring-core-adapter/pom.xml # nae-user-ce/pom.xml # nae-user-common/pom.xml # pom.xml
# Conflicts: # application-engine/pom.xml # nae-object-library/pom.xml # nae-spring-core-adapter/pom.xml # nae-user-ce/pom.xml # nae-user-common/pom.xml # pom.xml
Updated exception handling to log failures during batch event publishing. Clarified retry mechanisms in class documentation, specifying scheduled reindexing or manual intervention for failed bulk operations. This improves transparency and resilience in error scenarios.
- Updated Redis session namespace initialization to handle default blank or null cases more effectively. - Aligned application properties and session configuration with the updated namespace standards. - Ensured consistency between YAML configuration and Java annotations.
# Conflicts: # application-engine/pom.xml # nae-object-library/pom.xml # nae-spring-core-adapter/pom.xml # nae-user-ce/pom.xml # nae-user-common/pom.xml # pom.xml
-fix according to PR
-fix according to PR
-fix according to PR
- Adjusted `redisNamespace` property to use `redisProperties.session.namespace` for enhanced configuration accuracy.
[NAE-2252] MongoClient ClusterConnectionMode
[NAE-2245] Fix role preference menu item mapping caused by removed ne…
[NAE-2250] Optimize Elasticsearch reindexing and bulk operations for performance and reliability
# Conflicts: # application-engine/src/main/java/com/netgrif/application/engine/configuration/properties/DataConfigurationProperties.java
[NAE-2246] Enable Redis TLS & Configure Redis Sentinel
[NAE-2251] NAE - Map field options are not translated
- add AuthPrincipalDto to object library - remove immediate data from elasticCase and add NodeInfo - rework of elastic Datafields to remove dependency to TextField
- remove code from 2251 - when the search node fetches cases from mongo, its not nessessery anymore to store translations of value into elastic
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Implements NAE-2261
Dependencies
none
Third party dependencies
Blocking Pull requests
Depends on #(373) and merging rev8 into rev9
How Has Been This Tested?
manually
Test Configuration
Checklist: