Objective: Extend the existing Data Subsetting and Attribute-Level Column Filtering modules to support dynamic metadata loading and database-backed rule persistence.
- Preliminary Audit & Analysis:
Identify Core Logic: Locate the services responsible for "Data Subsetting" and "Column Filtering" (e.g., look for SubsettingService, MaskingEngine, or DataFilter in src/main/kotlin).
Assess Rule Lifecycle: Determine how rules are currently defined—are they hardcoded, passed via JSON/YAML files, or kept in-memory?
Tech Stack Verification: Confirm the UI framework (e.g., React, Vue, or a Kotlin-based web UI) and the database access layer (e.g., Spring Data JPA, Exposed, or JDBC).
- Task: Dynamic Attribute Loading (UI/Backend):
Backend: Create a MetadataService that utilizes JDBC metadata to fetch available schemas, tables, and columns from the connected target database.
API: Expose a REST endpoint (e.g., /api/metadata/tables) that returns a tree-like structure of database attributes.
UI Update: Enhance the filtering UI to replace manual text inputs with a searchable selection list or tree view populated by the new metadata endpoint. This allows users to "pick" columns for filtering/masking rather than typing them.
- Task: Database Rules Storage (Extract & Transform):
Persistence Layer: Implement a database schema to store "Subsetting Rules" and "Column Filter Rules." Key fields should include table_name, column_name, filter_expression, masking_type, and rule_set_id.
ETL Integration: Modify the "Extract & Transform" (ETL) engine to query this rule storage at runtime. Instead of using local config files, the engine should fetch active rules from the database to drive the masking and subsetting process.
CRUD Operations: Ensure the UI's "Save" or "Apply" button persists configurations to this storage.
- Technical Requirements:
Maintain the Kotlin coding standards used in the repository.
Ensure the solution respects the "blackbox" experiment nature—keep the database connection logic abstract enough to support multiple DB types (PostgreSQL, MySQL, etc.).
Update any existing unit/integration tests to account for the new persistence layer.
- Success Criteria:
The UI dynamically displays the target database's structure for selection.
Users can save filtering/subsetting configurations, and these rules persist across application restarts.
The "Extract & Transform" execution successfully applies filters retrieved from the database.
Objective: Extend the existing Data Subsetting and Attribute-Level Column Filtering modules to support dynamic metadata loading and database-backed rule persistence.
Identify Core Logic: Locate the services responsible for "Data Subsetting" and "Column Filtering" (e.g., look for SubsettingService, MaskingEngine, or DataFilter in src/main/kotlin).
Assess Rule Lifecycle: Determine how rules are currently defined—are they hardcoded, passed via JSON/YAML files, or kept in-memory?
Tech Stack Verification: Confirm the UI framework (e.g., React, Vue, or a Kotlin-based web UI) and the database access layer (e.g., Spring Data JPA, Exposed, or JDBC).
Backend: Create a MetadataService that utilizes JDBC metadata to fetch available schemas, tables, and columns from the connected target database.
API: Expose a REST endpoint (e.g., /api/metadata/tables) that returns a tree-like structure of database attributes.
UI Update: Enhance the filtering UI to replace manual text inputs with a searchable selection list or tree view populated by the new metadata endpoint. This allows users to "pick" columns for filtering/masking rather than typing them.
Persistence Layer: Implement a database schema to store "Subsetting Rules" and "Column Filter Rules." Key fields should include table_name, column_name, filter_expression, masking_type, and rule_set_id.
ETL Integration: Modify the "Extract & Transform" (ETL) engine to query this rule storage at runtime. Instead of using local config files, the engine should fetch active rules from the database to drive the masking and subsetting process.
CRUD Operations: Ensure the UI's "Save" or "Apply" button persists configurations to this storage.
Maintain the Kotlin coding standards used in the repository.
Ensure the solution respects the "blackbox" experiment nature—keep the database connection logic abstract enough to support multiple DB types (PostgreSQL, MySQL, etc.).
Update any existing unit/integration tests to account for the new persistence layer.
The UI dynamically displays the target database's structure for selection.
Users can save filtering/subsetting configurations, and these rules persist across application restarts.
The "Extract & Transform" execution successfully applies filters retrieved from the database.