feat: Data subsetting & attribute-level column filtering by Copilot · Pull Request #68 · MaximumTrainer/OpenDataMask

Copilot · 2026-04-16T11:29:25Z

- Add selectedAttributes field to TableConfiguration entity (stored as comma-separated TEXT) - Update DatabaseConnector.fetchData port signature with selectedAttributes parameter - Update PostgreSQL, AzureSQL, MySQL, MongoDB, File connectors to push column selection to source - Update DestinationSchemaService.mirrorSchema to filter schema by selectedAttributes - Update JobService to pass selectedAttributes to fetchData and mirrorSchema in all table modes - Update TableConfigurationRequest/Response DTOs with selectedAttributes field - Update TableConfigurationService to serialize/deserialize selectedAttributes list - Update WorkspaceConfigDto and WorkspaceExportService for export/import of selectedAttributes - Update frontend TypeScript types with selectedAttributes - Add column picker UI in TablesView.vue with comma-separated text input - Add unit tests for new selectedAttributes functionality - Fix existing test mocks to use updated 4-parameter fetchData signature - Update user-guide.md with subsetting and attribute filtering documentation Agent-Logs-Url: https://github.com/MaximumTrainer/OpenDataMask/sessions/9a3b22af-7e44-44d4-87a5-f806e0540af3 Co-authored-by: MaximumTrainer <1376575+MaximumTrainer@users.noreply.github.com>

Copilot

Pull request overview

Adds column-level extraction filtering (selectedAttributes) alongside existing row-level subsetting (whereClause), threading both through the API, persistence, job engine, schema mirroring, connectors, and the frontend Tables UI.

Changes:

Extend table configuration DTOs/entity and export/import to support selectedAttributes as a list (persisted as comma-separated text).
Push down column selection to source connectors (SQL SELECT col1,... / MongoDB projection / file row-map filtering) and mirror destination schema accordingly.
Update frontend types + table config modal to edit/display selected columns, and add/adjust tests for the new behavior/signature.

Reviewed changes

Copilot reviewed 24 out of 24 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
frontend/src/views/TablesView.vue	Adds UI input + payload normalization for selected columns and displays active column filters.
frontend/src/types/index.ts	Extends TS types and export DTO shape to include `selectedAttributes`.
docs/user-guide.md	Documents subsetting/WHERE and attribute-level column filtering plus UI instructions.
backend/src/main/kotlin/com/opendatamask/domain/port/output/DatabaseConnector.kt	Extends connector port with `selectedAttributes` parameter.
backend/src/main/kotlin/com/opendatamask/domain/port/input/dto/TableConfigurationDto.kt	Adds `selectedAttributes` to request/response DTOs.
backend/src/main/kotlin/com/opendatamask/domain/port/input/dto/WorkspaceConfigDto.kt	Adds `selectedAttributes` to table export DTO.
backend/src/main/kotlin/com/opendatamask/domain/model/TableConfiguration.kt	Persists `selectedAttributes` as comma-separated TEXT column.
backend/src/main/kotlin/com/opendatamask/application/service/TableConfigurationService.kt	Serializes/deserializes `selectedAttributes` between entity string and DTO list.
backend/src/main/kotlin/com/opendatamask/application/service/WorkspaceExportService.kt	Export/import handling for `selectedAttributes`.
backend/src/main/kotlin/com/opendatamask/application/service/JobService.kt	Threads `selectedAttributes` through schema mirroring and source reads across modes.
backend/src/main/kotlin/com/opendatamask/application/service/DestinationSchemaService.kt	Filters mirrored schema columns based on `selectedAttributes` (case-insensitive).
backend/src/main/kotlin/com/opendatamask/adapter/output/connector/PostgreSQLConnector.kt	Generates `SELECT col1,col2` (quoted identifiers) when attributes are provided.
backend/src/main/kotlin/com/opendatamask/adapter/output/connector/AzureSQLConnector.kt	Adds TOP + bracket-quoted column selection pushdown.
backend/src/main/kotlin/com/opendatamask/adapter/output/connector/MySQLConnector.kt	Adds backtick-quoted column selection pushdown.
backend/src/main/kotlin/com/opendatamask/adapter/output/connector/MongoDBConnector.kt	Adds projection-based field selection.
backend/src/main/kotlin/com/opendatamask/adapter/output/connector/FileConnector.kt	Filters parsed row maps to selected keys (case-insensitive).
backend/src/test/kotlin/com/opendatamask/application/service/DestinationSchemaServiceTest.kt	New tests for selectedAttributes filtering, empty list, and case-insensitivity.
backend/src/test/kotlin/com/opendatamask/adapter/output/connector/PostgreSQLConnectorTest.kt	Adds tests for selectedAttributes and WHERE+columns combined.
backend/src/test/kotlin/com/opendatamask/application/service/TableConfigurationServiceTest.kt	Adds tests around selectedAttributes serialization/deserialization.
backend/src/test/kotlin/com/opendatamask/application/service/JobServiceTest.kt	Updates mocks/verifications for new fetchData signature.
backend/src/test/kotlin/com/opendatamask/application/service/SubsetExecutionServiceTest.kt	Updates fetchData mock to new signature.
backend/src/test/kotlin/com/opendatamask/application/service/SensitivityScanServiceTest.kt	Updates fetchData mock to new signature.
backend/src/test/kotlin/com/opendatamask/application/service/SensitivityScanLogServiceTest.kt	Updates fetchData mocks to new signature.
backend/src/test/kotlin/com/opendatamask/adapter/input/rest/TableConfigurationControllerTest.kt	Updates response factory for new DTO field.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-04-16T12:11:43Z

+    // Normalise: send null when no attributes selected (empty list = select all)
+    const payload: TableConfigurationRequest = {
+      ...tableForm.value,
+      selectedAttributes:
+        tableForm.value.selectedAttributes?.length ? tableForm.value.selectedAttributes : undefined
+    }


The comment says the UI will "send null" when no attributes are selected, but the code sets selectedAttributes to undefined (omits the field). Either update the comment to match the behavior, or explicitly send null if the backend relies on distinguishing null vs missing.

Copilot · 2026-04-16T12:11:43Z

+        val filteredColumns = if (selectedAttributes.isEmpty()) {
+            sourceColumns
+        } else {
+            val normalizedSelected = selectedAttributes.map { it.lowercase() }.toSet()
+            sourceColumns.filter { it.name.lowercase() in normalizedSelected }
+        }


mirrorSchema() can end up with filteredColumns empty when selectedAttributes doesn't match any source column names. This will call createTable() with 0 columns (invalid SQL in some connectors, and a silent no-op in MySQL), causing confusing job failures later. Consider validating that at least one column matched and fail fast with a clear error listing the unknown attributes (or fall back to all columns).

Suggested change

val filteredColumns = if (selectedAttributes.isEmpty()) {

sourceColumns

} else {

val normalizedSelected = selectedAttributes.map { it.lowercase() }.toSet()

sourceColumns.filter { it.name.lowercase() in normalizedSelected }

}

val sourceColumnNames = sourceColumns.map { it.name }

val sourceColumnNamesNormalized = sourceColumnNames.map { it.lowercase() }.toSet()

val normalizedSelected = selectedAttributes.map { it.lowercase() }.toSet()

val filteredColumns = if (selectedAttributes.isEmpty()) {

sourceColumns

} else {

sourceColumns.filter { it.name.lowercase() in normalizedSelected }

}

if (selectedAttributes.isNotEmpty() && filteredColumns.isEmpty()) {

val unknownAttributes = selectedAttributes.filter { it.lowercase() !in sourceColumnNamesNormalized }

throw IllegalArgumentException(

"No selected attributes matched source columns for table '$tableName'. " +

"Unknown attributes: ${if (unknownAttributes.isEmpty()) selectedAttributes else unknownAttributes}. " +

"Available columns: $sourceColumnNames"

)

}

Copilot · 2026-04-16T12:11:43Z

+            val selectPart = if (!selectedAttributes.isNullOrEmpty()) {
+                selectedAttributes.joinToString(", ") { "`$it`" }
+            } else {
+                "*"
+            }
            val sql = buildString {
-                append("SELECT * FROM `$tableName`")
+                append("SELECT $selectPart FROM `$tableName`")
                if (whereClause != null) append(" WHERE $whereClause")


MySQL column identifiers in selectedAttributes are interpolated as `$it` without escaping/sanitizing. This can break queries for columns containing backticks and expands the injection surface compared to other connectors that strip quote characters. Sanitize/escape backticks in column names (e.g., remove or double them) before building selectPart.

Copilot · 2026-04-16T12:11:44Z

+            var query = if (limit != null) collection.find(filter).limit(limit) else collection.find(filter)
+            if (!selectedAttributes.isNullOrEmpty()) {
+                val projection = Document().apply {
+                    selectedAttributes.forEach { field -> put(field, 1) }


MongoDB projections that include specific fields will still return _id by default. With the current projection document, fetchData() will likely include _id even when it wasn't requested, which contradicts the "only requested columns" behavior used elsewhere. Consider explicitly excluding _id (set _id: 0) unless _id is present in selectedAttributes.

Suggested change

selectedAttributes.forEach { field -> put(field, 1) }

selectedAttributes.forEach { field -> put(field, 1) }

if (!selectedAttributes.contains("_id")) {

put("_id", 0)

}

Copilot · 2026-04-16T12:11:44Z

            rowLimit = request.rowLimit,
            whereClause = request.whereClause
        )
+        config.selectedAttributes = request.selectedAttributes?.filter { it.isNotBlank() }?.joinToString(",")
        return tableConfigurationRepository.save(config).toResponse()


When request.selectedAttributes is an empty list, joinToString(",") produces an empty string, so the entity stores "" rather than null. This makes responses/export potentially return an empty list instead of null and complicates the "empty/omitted = all columns" contract. Consider normalizing to null when the filtered+trimmed list is empty (e.g., takeIf { it.isNotEmpty() }).

Copilot · 2026-04-16T12:11:44Z

            table.mode = tableDto.mode
            table.rowLimit = tableDto.rowLimit
            table.whereClause = tableDto.whereClause
+            table.selectedAttributes = tableDto.selectedAttributes?.filter { it.isNotBlank() }?.joinToString(",")


Import currently persists selectedAttributes as a comma-separated string even when the list is empty, which results in an empty string instead of null. If the API contract is "null/empty = all columns", consider storing null when the incoming list is empty after trimming/filtering, to keep persistence/export consistent.

Suggested change

table.selectedAttributes = tableDto.selectedAttributes?.filter { it.isNotBlank() }?.joinToString(",")

val normalizedSelectedAttributes = tableDto.selectedAttributes

?.map { it.trim() }

?.filter { it.isNotBlank() }

table.selectedAttributes = normalizedSelectedAttributes

?.takeIf { it.isNotEmpty() }

?.joinToString(",")

Copilot · 2026-04-16T12:11:45Z

+        )
+        whenever(tableConfigurationRepository.save(any<TableConfiguration>())).thenReturn(saved)
+
+        val response = service.createTableConfiguration(10L, request)
+


This test name says it verifies persistence, but it only asserts the response mapping from the mocked saved entity. Consider verifying the argument passed to tableConfigurationRepository.save(...) (e.g., capture and assert selectedAttributes == "id,name,email") so the test actually covers the serialization behavior.

Suggested change

)

whenever(tableConfigurationRepository.save(any<TableConfiguration>())).thenReturn(saved)

val response = service.createTableConfiguration(10L, request)

)

val tableConfigurationCaptor = argumentCaptor<TableConfiguration>()

whenever(tableConfigurationRepository.save(any<TableConfiguration>())).thenReturn(saved)

val response = service.createTableConfiguration(10L, request)

verify(tableConfigurationRepository).save(tableConfigurationCaptor.capture())

assertEquals("id,name,email", tableConfigurationCaptor.firstValue.selectedAttributes)

MaximumTrainer · 2026-04-16T12:14:55Z

@copilot apply changes based on the comments in this thread

- Fix comment in TablesView.vue: omit field (undefined) rather than null when no columns selected - Normalize empty selectedAttributes list to null in TableConfigurationService and WorkspaceExportService using takeIf { isNotEmpty() }, preventing empty-string storage - Add fast-fail in DestinationSchemaService.mirrorSchema when selectedAttributes produces zero matching columns - Escape backticks in MySQL column identifiers (double-backtick escaping) to prevent SQL injection - Exclude _id from MongoDB projection unless explicitly requested in selectedAttributes - Improve TableConfigurationServiceTest to capture save() argument and verify comma-separated serialization - Add TableConfigurationServiceTest for empty-list normalization to null - Add DestinationSchemaServiceTest for unmatched selectedAttributes fast-fail path Agent-Logs-Url: https://github.com/MaximumTrainer/OpenDataMask/sessions/d68edfed-91a4-4db2-aa64-007b97334b8c Co-authored-by: MaximumTrainer <1376575+MaximumTrainer@users.noreply.github.com>

Copilot · 2026-04-16T12:22:33Z

@copilot apply changes based on the comments in this thread

Applied all 7 review comments in commit 0877d56:

TablesView.vue comment — updated to say "omit field" (undefined) instead of "send null".
DestinationSchemaService fast-fail — mirrorSchema now throws IllegalArgumentException when selectedAttributes matches zero source columns, reporting the unknown attribute names and listing available columns.
MySQL backtick escaping — column names now use it.replace("", "``")` before wrapping in backticks, closing the injection surface.
MongoDB _id exclusion — _id: 0 is added to the projection unless _id is explicitly listed in selectedAttributes.
TableConfigurationService empty-list normalization — both create and update paths use ?.takeIf { it.isNotEmpty() } so an empty list stores null instead of "".
WorkspaceExportService — same takeIf { isNotEmpty() } normalization applied consistently.
TableConfigurationServiceTest serialization — test now uses argumentCaptor<TableConfiguration>() to assert the saved entity's selectedAttributes field equals "id,name,email", plus a new test verifying an empty list normalizes to null.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

example.com
- Triggering command: /usr/lib/jvm/temurin-17-jdk-amd64/bin/java /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -Dorg.gradle.internal.worker.tmpdir=/home/REDACTED/work/OpenDataMask/OpenDataMask/backend/build/tmp/test/work -Dorg.gradle.native=false -javaagent:/home/REDACTED/work/OpenDataMask/OpenDataMask/backend/build/tmp/.cache/expanded/zip_0bd0851504e6286cb5fc6f37a81e25cc/jacocoagent.jar=destfile=build/jacoco/test.exec,append=true,inclnolocationclasses=false,dumponexit=true,output=file,jmx=false @/home/REDACTED/.gradle/.tmp/gradle-worker-classpath14013826460707004457txt -Xmx512m -Dfile.encoding=UTF-8 -Duser.country -Duser.language=en -Duser.variant -ea worker.org.gradle.process.internal.worker.GradleWorkerMain 'Gradle Test Executor 1' (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

Configure Actions setup steps to set up my environment, which run before the firewall is enabled
Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

Initial plan

1b615ea

Copilot AI assigned Copilot and MaximumTrainer Apr 16, 2026

Copilot started work on behalf of MaximumTrainer April 16, 2026 11:29 View session

Copilot AI linked an issue Apr 16, 2026 that may be closed by this pull request

Data Subsetting & Attribute Filtering #61

Closed

Copilot AI changed the title ~~[WIP] Add data subsetting and attribute filtering capability~~ feat: Data subsetting & attribute-level column filtering Apr 16, 2026

Copilot AI requested a review from MaximumTrainer April 16, 2026 11:53

Copilot finished work on behalf of MaximumTrainer April 16, 2026 11:53

MaximumTrainer requested review from Copilot and removed request for MaximumTrainer April 16, 2026 12:05

Copilot started reviewing on behalf of MaximumTrainer April 16, 2026 12:06 View session

Copilot AI reviewed Apr 16, 2026

View reviewed changes

Copilot started work on behalf of MaximumTrainer April 16, 2026 12:15 View session

Copilot finished work on behalf of MaximumTrainer April 16, 2026 12:24

Copilot AI requested a review from MaximumTrainer April 16, 2026 12:24

MaximumTrainer marked this pull request as ready for review April 16, 2026 12:34

MaximumTrainer merged commit 786b365 into main Apr 16, 2026
10 checks passed

MaximumTrainer deleted the copilot/implement-data-subsetting-attributes branch April 16, 2026 12:35

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Data subsetting & attribute-level column filtering#68

feat: Data subsetting & attribute-level column filtering#68
MaximumTrainer merged 3 commits intomainfrom
copilot/implement-data-subsetting-attributes

Copilot AI commented Apr 16, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Apr 16, 2026

Uh oh!

Copilot AI Apr 16, 2026

Uh oh!

Copilot AI Apr 16, 2026

Uh oh!

Copilot AI Apr 16, 2026

Uh oh!

Copilot AI Apr 16, 2026

Uh oh!

Copilot AI Apr 16, 2026

Uh oh!

Copilot AI Apr 16, 2026

Uh oh!

MaximumTrainer commented Apr 16, 2026

Uh oh!

Copilot AI commented Apr 16, 2026 •

edited

Loading

I tried to connect to the following addresses, but was blocked by firewall rules:

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

-            table.selectedAttributes = tableDto.selectedAttributes?.filter { it.isNotBlank() }?.joinToString(",")
+            val normalizedSelectedAttributes = tableDto.selectedAttributes
+                ?.map { it.trim() }
+                ?.filter { it.isNotBlank() }
+            table.selectedAttributes = normalizedSelectedAttributes
+                ?.takeIf { it.isNotEmpty() }
+                ?.joinToString(",")

Conversation

Copilot AI commented Apr 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Apr 16, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 16, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 16, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 16, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 16, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 16, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 16, 2026

Choose a reason for hiding this comment

Uh oh!

MaximumTrainer commented Apr 16, 2026

Uh oh!

Copilot AI commented Apr 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

I tried to connect to the following addresses, but was blocked by firewall rules:

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Copilot AI commented Apr 16, 2026 •

edited

Loading

Copilot AI commented Apr 16, 2026 •

edited

Loading