Skip to content

Expand tool to manage multiple source & target databases #60

@MaximumTrainer

Description

@MaximumTrainer

Act as a Senior Kotlin Backend Engineer and Software Architect. Your task is to refactor and expand the "OpenDataMask" tool to support multiple source/target database pairs.

Context

Currently, OpenDataMask is designed for a single source and destination database. We need to transition this into a multi-tenant-style architecture where multiple database environments (Source + Target) can be managed simultaneously, and their masking configurations remain strictly isolated.

Functional Requirements

  1. Connection Pair Management:
    • Define a 'ConnectionPair' entity that groups exactly one Source Database and one Destination Database.
    • Metadata for each database (JDBC URL, Driver, Credentials, Schema Name) must be stored in the application's internal database.
  2. Metadata Segregation:
    • All masking rules, transformation logic, column mappings, and execution logs must be linked to a specific ConnectionPairID.
    • Ensure that a user cannot apply rules from Pair A to the databases in Pair B.
  3. Dynamic Execution Context:
    • The masking engine must be refactored to accept a connectionPairId as an execution parameter.
    • Connections must be initialized dynamically from the stored metadata at runtime rather than being loaded from a static application.properties file.

Technical Tasks

  1. Schema Migration:
    • Design a Flyway/Liquibase migration to introduce connection_pairs, db_connections, and add pair_id foreign keys to all existing metadata tables (e.g., masking_rules, audit_logs).
  2. Persistence Layer:
    • Update the Repository layer (JPA/Exposed) to support CRUD operations for these pairs.
    • Implement "Soft Deletes" for database configurations to preserve audit history.
  3. Dynamic Connection Pooling:
    • Implement a Connection Manager that caches DataSource objects by pair_id to prevent memory leaks and redundant connection handshakes.
  4. Service Layer Refactoring:
    • Modify the core masking service to use a "Contextual Connection" pattern.
    • Ensure transaction management is handled per-pair.

Deliverables

  • A detailed Class Diagram showing the relationship between ConnectionPairs and Metadata.
  • Kotlin code for the new Entity models.
  • Refactored logic for the 'MaskingJob' runner to handle dynamic source/target selection.
  • A proposed API or CLI structure for switching between different database pairs.

Please provide the implementation strategy and the code for the core entity models first.


***

### Rationale: Why this prompt works
* **The "Connection Pair" Abstraction**: By explicitly asking to manage source and destination as a "Pair," you ensure the tool doesn't just support multiple databases but maintains the logical link between where data comes from and where it goes.
* **Architectural Guidance**: It instructs the AI to move away from static config (`application.properties`) toward dynamic storage, which is the biggest hurdle in this kind of expansion.
* **Metadata Integrity**: By emphasizing the "Pair ID" foreign key, the prompt ensures that your masking rules don't get mixed up between different clients or environments.
* **Resource Management**: It specifically mentions "Connection Pooling" and "Contextual Connections," which prevents the application from crashing when scaling to many database pairs.

### Recommended Next Steps for Implementation
If you are implementing this manually or reviewing the AI's output, look for these key changes:

1.  **Database Schema**: 2.  **The "Context" Pattern**: In Kotlin, you should use a `CoroutineContext` or a `ThreadLocal` (if not using coroutines) to hold the current `pair_id` so that the masking logic doesn't have to pass the ID through every single function call.
3.  **Encrypted Metadata**: Since you are moving credentials into the application database, ensure the prompt also includes a request for **encryption at rest** for the database passwords.

Metadata

Metadata

Labels

No labels
No labels

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions