Add custom data mapping: per-column MIGRATE_AS_IS/MASK wizard with schema browsing by Copilot · Pull Request #72 · MaximumTrainer/OpenDataMask

Copilot · 2026-04-16T14:50:36Z

…hema browsing Agent-Logs-Url: https://github.com/MaximumTrainer/OpenDataMask/sessions/78fb0a16-9359-4a9d-a8ae-075229759b79 Co-authored-by: MaximumTrainer <1376575+MaximumTrainer@users.noreply.github.com>

…appingView, tests, docs Agent-Logs-Url: https://github.com/MaximumTrainer/OpenDataMask/sessions/78fb0a16-9359-4a9d-a8ae-075229759b79 Co-authored-by: MaximumTrainer <1376575+MaximumTrainer@users.noreply.github.com>

Agent-Logs-Url: https://github.com/MaximumTrainer/OpenDataMask/sessions/78fb0a16-9359-4a9d-a8ae-075229759b79 Co-authored-by: MaximumTrainer <1376575+MaximumTrainer@users.noreply.github.com>

Copilot

Pull request overview

Adds a new “Custom Data Mapping” capability to configure per-column actions (MIGRATE_AS_IS vs MASK with FAKE/HASH/NULL) and exposes it through a 3-step frontend wizard backed by new REST endpoints, including live connection schema browsing.

Changes:

Backend: introduce CustomDataMapping persistence model + CRUD/bulk APIs and a browseConnectionSchema endpoint on connections.
Frontend: add mapping types + API client + new DataMappingView wizard route and navigation entry points.
Tests/docs: add unit tests for the new services/types and document the new endpoints and bulk request format.

Reviewed changes

Copilot reviewed 19 out of 19 changed files in this pull request and generated 11 comments.

Show a summary per file

File	Description
frontend/src/views/tests/DataMappingView.test.ts	Adds vitest coverage for new mapping/schema types and basic structure expectations.
frontend/src/views/WorkspaceDetailView.vue	Adds “Data Mappings” tab and quick-action navigation entry.
frontend/src/views/DataMappingView.vue	Implements the 3-step mapping wizard UI, bulk save flow, and saved-mappings summary.
frontend/src/types/index.ts	Introduces `MappingAction`, `MaskingStrategy`, mapping DTO types, and schema browsing response types.
frontend/src/router/index.ts	Registers the new `/workspaces/:id/mappings` route.
frontend/src/api/mapping.ts	Adds API wrapper functions for mappings CRUD/bulk and schema browsing.
docs/user-guide.md	Documents the Custom Data Mapping feature and REST API including bulk example.
backend/src/test/kotlin/com/opendatamask/application/service/DataConnectionServiceTest.kt	Extends tests to cover schema browsing behavior.
backend/src/test/kotlin/com/opendatamask/application/service/CustomDataMappingServiceTest.kt	Adds unit tests covering mapping CRUD and bulk replace behavior.
backend/src/main/kotlin/com/opendatamask/domain/port/output/CustomDataMappingPort.kt	Defines the persistence port for mappings.
backend/src/main/kotlin/com/opendatamask/domain/port/input/dto/CustomDataMappingDto.kt	Adds request/response DTOs for mapping operations and connection schema responses.
backend/src/main/kotlin/com/opendatamask/domain/port/input/DataConnectionUseCase.kt	Extends the connection use case with schema browsing.
backend/src/main/kotlin/com/opendatamask/domain/port/input/CustomDataMappingUseCase.kt	Defines the mapping use case interface (CRUD + bulk).
backend/src/main/kotlin/com/opendatamask/domain/model/CustomDataMapping.kt	Adds the JPA entity + enums for mapping action and masking strategy.
backend/src/main/kotlin/com/opendatamask/application/service/DataConnectionService.kt	Implements schema browsing by calling connector table/column introspection.
backend/src/main/kotlin/com/opendatamask/application/service/CustomDataMappingService.kt	Implements mapping CRUD and bulk table-level replace logic.
backend/src/main/kotlin/com/opendatamask/adapter/output/persistence/CustomDataMappingRepository.kt	Adds Spring Data JPA repository implementing the new port.
backend/src/main/kotlin/com/opendatamask/adapter/input/rest/DataConnectionController.kt	Exposes `GET /connections/{id}/schema`.
backend/src/main/kotlin/com/opendatamask/adapter/input/rest/CustomDataMappingController.kt	Exposes mappings CRUD + bulk endpoints under `/api/workspaces/{id}/mappings`.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-04-16T15:35:27Z

+    setTimeout(() => {
+      saveSuccess.value = false
+      currentStep.value = 2
+    }, 1500)


saveMappings() uses setTimeout to update step state after saving but doesn’t clear the timer on unmount. Navigating away quickly can trigger state updates after unmount. Consider storing the timer id and clearing it in onUnmounted() (or avoiding the timer).

Copilot · 2026-04-16T15:35:28Z

+                    :class="{ active: columnMappings[col.name]?.action === 'MIGRATE_AS_IS' }"
+                    @click="setAction(col.name, MappingAction.MIGRATE_AS_IS)"


The template compares enum-typed fields against string literals (e.g. action === 'MIGRATE_AS_IS'). Prefer MappingAction.MIGRATE_AS_IS / MappingAction.MASK in comparisons to avoid duplicating enum values and to make refactors safer.

Copilot · 2026-04-16T15:35:28Z

+    @Mock private lateinit var customDataMappingRepository: CustomDataMappingRepository
+
+    @InjectMocks
+    private lateinit var service: CustomDataMappingService


This unit test mocks CustomDataMappingRepository (persistence adapter) even though the service depends only on CustomDataMappingPort. Mocking the port interface directly will keep the test focused on service behavior and reduce coupling to the Spring Data adapter.

Copilot · 2026-04-16T15:35:28Z

+                action = entry.action,
+                maskingStrategy = if (entry.action == MappingAction.MASK) entry.maskingStrategy else null,
+                fakeGeneratorType = if (entry.action == MappingAction.MASK) entry.fakeGeneratorType else null
+            )


saveBulkMappings() will persist entries with action = MASK even when maskingStrategy is null (or FAKE without a generator), which allows invalid mappings to be stored. Add per-entry validation (DTO @AssertTrue / explicit checks) before saving.

Copilot · 2026-04-16T15:35:29Z

+@Table(name = "custom_data_mappings")
+class CustomDataMapping(
+    @Id
+    @GeneratedValue(strategy = GenerationType.IDENTITY)
+    val id: Long = 0,
+
+    @Column(nullable = false)
+    var workspaceId: Long,
+
+    @Column(nullable = false)
+    var connectionId: Long,
+
+    @Column(nullable = false)
+    var tableName: String,
+
+    @Column(nullable = false)


CustomDataMapping has no uniqueness constraint for (workspaceId, connectionId, tableName, columnName), so creating/updating via the single-item endpoints can introduce duplicate rows for the same column. Adding a DB-level unique constraint (and handling conflicts with a 409/400) would prevent hard-to-debug duplication in the wizard and downstream masking logic.

Suggested change

@Table(name = "custom_data_mappings")

class CustomDataMapping(

@Id

@GeneratedValue(strategy = GenerationType.IDENTITY)

val id: Long = 0,

@Column(nullable = false)

var workspaceId: Long,

@Column(nullable = false)

var connectionId: Long,

@Column(nullable = false)

var tableName: String,

@Column(nullable = false)

@Table(

name = "custom_data_mappings",

uniqueConstraints = [

UniqueConstraint(

name = "uk_custom_data_mappings_workspace_connection_table_column",

columnNames = ["workspace_id", "connection_id", "table_name", "column_name"]

)

]

)

class CustomDataMapping(

@Id

@GeneratedValue(strategy = GenerationType.IDENTITY)

val id: Long = 0,

@Column(name = "workspace_id", nullable = false)

var workspaceId: Long,

@Column(name = "connection_id", nullable = false)

var connectionId: Long,

@Column(name = "table_name", nullable = false)

var tableName: String,

@Column(name = "column_name", nullable = false)

Copilot · 2026-04-16T15:35:29Z

+        val tables = connector.listTables().map { tableName ->
+            val columns = connector.listColumns(tableName).map { col ->
+                ConnectionSchemaResponse.ColumnSchemaInfo(
+                    name = col.name,
+                    type = col.type,


browseConnectionSchema() maps over all tables and calls connector.listColumns(tableName) for each one. For JDBC connectors, listColumns() opens a new DB connection per call, so schemas with many tables can cause a connection storm. Consider adding a connector API that returns tables+columns in one call / reusing a single connection, or limiting the number of tables returned.

Copilot · 2026-04-16T15:35:30Z

@@ -108,6 +109,33 @@ class DataConnectionService(
        }
    }



browseConnectionSchema() is missing a @Transactional(readOnly = true) annotation even though it reads from the repository (and other read methods in this service are marked read-only). Adding it keeps behavior consistent and avoids unnecessary transaction semantics.

Suggested change

@Transactional(readOnly = true)

Copilot · 2026-04-16T15:35:30Z

+
+    @field:NotNull(message = "Column mappings are required")
+    val columnMappings: List<ColumnMappingEntry>
+) {


BulkCustomDataMappingRequest.columnMappings is not annotated with @Valid, so the @NotBlank/@NotNull constraints on ColumnMappingEntry will not be applied to the list elements. Add @field:Valid (and optionally @field:NotEmpty if empty payloads should be rejected) to ensure invalid entries are caught at the API boundary.

Copilot · 2026-04-16T15:35:30Z

+            columnName = request.columnName,
+            action = request.action,
+            maskingStrategy = if (request.action == MappingAction.MASK) request.maskingStrategy else null,
+            fakeGeneratorType = if (request.action == MappingAction.MASK) request.fakeGeneratorType else null
+        )


The service persists action = MASK even when maskingStrategy/fakeGeneratorType are missing in the request, which allows invalid mappings to be stored (e.g., MASK with null strategy). Add cross-field validation (DTO @AssertTrue or explicit checks) to enforce valid combinations before saving.

Copilot · 2026-04-16T15:35:31Z

+        val result = if (connectionId != null && tableName != null) {
+            customDataMappingService.listMappingsForTable(workspaceId, connectionId, tableName)
+        } else {
+            customDataMappingService.listMappings(workspaceId)
+        }


GET /mappings falls back to listMappings(workspaceId) unless both connectionId and tableName are provided, so supplying only one filter is silently ignored. Consider returning 400 for partial filters, or supporting independent filtering explicitly.

MaximumTrainer · 2026-04-16T15:40:24Z

@copilot apply changes based on the comments in this thread

, enum comparisons, timer cleanup, bulk delete Agent-Logs-Url: https://github.com/MaximumTrainer/OpenDataMask/sessions/2758ddbd-e5a0-47b1-a915-5fcdf275497d Co-authored-by: MaximumTrainer <1376575+MaximumTrainer@users.noreply.github.com>

…tomDataMappingPort Agent-Logs-Url: https://github.com/MaximumTrainer/OpenDataMask/sessions/2758ddbd-e5a0-47b1-a915-5fcdf275497d Co-authored-by: MaximumTrainer <1376575+MaximumTrainer@users.noreply.github.com>

Copilot · 2026-04-16T15:55:07Z

@copilot apply changes based on the comments in this thread

Applied all review feedback across commits 73c6188 and dc01b92:

Backend:

CustomDataMapping — added @Table unique constraint on (workspace_id, connection_id, table_name, column_name) and explicit @Column(name=...) mappings
DataConnectionService.browseConnectionSchema() — added @Transactional(readOnly = true)
BulkCustomDataMappingRequest.columnMappings — added @field:Valid so nested @NotBlank/@NotNull constraints on each ColumnMappingEntry are applied
CustomDataMappingService — added validateMaskingCombination() that rejects: (a) MASK with no maskingStrategy, (b) FAKE without fakeGeneratorType; called from createMapping, updateMapping, and saveBulkMappings
CustomDataMappingController.listMappings — returns HTTP 400 when only one of connectionId/tableName is provided
CustomDataMappingServiceTest — mock renamed customDataMappingPort, typed as CustomDataMappingPort interface (not the JPA adapter); added 3 new validation tests

Frontend:

DataMappingView.vue — setTimeout timer stored in saveSuccessTimer and cleared in onUnmounted()
Template comparisons use MappingAction.MIGRATE_AS_IS, MappingAction.MASK, MaskingStrategy.FAKE enum constants instead of string literals
Delete table mappings now calls saveBulkMappings(…, { columnMappings: [] }) — a single atomic bulk-replace request instead of N per-row DELETEs

Initial plan

f3dcf20

Copilot AI assigned Copilot and MaximumTrainer Apr 16, 2026

Copilot started work on behalf of MaximumTrainer April 16, 2026 14:50 View session

Copilot AI linked an issue Apr 16, 2026 that may be closed by this pull request

add ability for custom data mapping #50

Closed

Copilot AI and others added 3 commits April 16, 2026 15:03

add CustomDataMapping backend: entity, ports, service, controller, sc…

6534744

…hema browsing Agent-Logs-Url: https://github.com/MaximumTrainer/OpenDataMask/sessions/78fb0a16-9359-4a9d-a8ae-075229759b79 Co-authored-by: MaximumTrainer <1376575+MaximumTrainer@users.noreply.github.com>

address code review: typed setFakeGeneratorType, rename test suite

480ccce

Agent-Logs-Url: https://github.com/MaximumTrainer/OpenDataMask/sessions/78fb0a16-9359-4a9d-a8ae-075229759b79 Co-authored-by: MaximumTrainer <1376575+MaximumTrainer@users.noreply.github.com>

Copilot AI changed the title ~~[WIP] Add ability for custom data mapping~~ Add custom data mapping: per-column MIGRATE_AS_IS/MASK wizard with schema browsing Apr 16, 2026

Copilot AI requested a review from MaximumTrainer April 16, 2026 15:24

Copilot finished work on behalf of MaximumTrainer April 16, 2026 15:24

MaximumTrainer requested review from Copilot and removed request for MaximumTrainer April 16, 2026 15:26

Copilot started reviewing on behalf of MaximumTrainer April 16, 2026 15:27 View session

Copilot AI reviewed Apr 16, 2026

View reviewed changes

Copilot started work on behalf of MaximumTrainer April 16, 2026 15:40 View session

Copilot AI and others added 2 commits April 16, 2026 15:53

fix review nits: import GeneratorType, remove FQN, rename mock to cus…

dc01b92

…tomDataMappingPort Agent-Logs-Url: https://github.com/MaximumTrainer/OpenDataMask/sessions/2758ddbd-e5a0-47b1-a915-5fcdf275497d Co-authored-by: MaximumTrainer <1376575+MaximumTrainer@users.noreply.github.com>

Copilot finished work on behalf of MaximumTrainer April 16, 2026 15:56

Copilot AI requested a review from MaximumTrainer April 16, 2026 15:56

MaximumTrainer marked this pull request as ready for review April 16, 2026 15:59

MaximumTrainer merged commit 1dd6849 into main Apr 16, 2026
10 checks passed

MaximumTrainer deleted the copilot/add-custom-data-mapping-again branch April 16, 2026 16:04

This was referenced Apr 16, 2026

Implement Dynamic Attribute Discovery and Persistent Rule Storage for OpenDataMask #57

Closed

add ability for custom PII attribute masking #49

Closed

		:class="{ active: columnMappings[col.name]?.action === 'MIGRATE_AS_IS' }"
		@click="setAction(col.name, MappingAction.MIGRATE_AS_IS)"

Conversation

Copilot AI commented Apr 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Apr 16, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 16, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 16, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 16, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 16, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 16, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 16, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 16, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 16, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 16, 2026

Choose a reason for hiding this comment

Uh oh!

MaximumTrainer commented Apr 16, 2026

Uh oh!

Copilot AI commented Apr 16, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Copilot AI commented Apr 16, 2026 •

edited

Loading