Skip to content

feat: Support mapper/mapping preservation in cross-workspace connection migration #980

@devin-ai-integration

Description

@devin-ai-integration

Summary

Ensure that stream mappers (hashing, field-renaming, row-filtering, encryption, field-filtering) are properly handled during cross-workspace connection migration workflows.

Requested by: Aaron ("AJ") Steers (@aaronsteers)
Related Devin session: https://app.devin.ai/sessions/eb636b8d14fc4895b3b894ea399f998e
Related issues:


Current State

What works today

Mappers are preserved during catalog dump/import via the existing tools:

  1. Catalog dump (dump_raw_catalog()web_backend/connections/get): The API response includes mappers per stream in the syncCatalog, with each mapper having type and mapperConfiguration.

  2. Catalog import (import_raw_catalog()web_backend/connections/update): The platform's CatalogConverter.toConfiguredInternal() correctly converts mappers from API model to internal model (lines 305-312 of CatalogConverter.kt), preserving type and mapperConfiguration.

So the basic mapper round-trip (dump from source connection → import to target connection) does work for mappers without secrets.

What does NOT work

  1. Mapper secrets are masked on dump: The platform's MapperSecretHelper.maskMapperSecrets() masks secret fields (e.g., encryption keys) before returning the catalog via the API. This means:

    • When dumping a catalog from the source connection, mapper secrets (e.g., AES encryption keys) will be replaced with "**********" or similar masked values
    • When importing the dumped catalog to the target connection, the masked values won't work
    • The user would need to manually re-enter mapper secrets for the target connection
  2. No MCP tools for mapper management: Neither Coral MCP nor Ops MCP has any tools for:

    • Listing mappers configured on a connection
    • Adding/removing/updating individual mappers
    • Validating mappers against the target connection's streams (the platform has a /v1/web_backend/mappers/validate endpoint, but it's not exposed via MCP)
  3. No mapper compatibility validation: When migrating, there's no check that the target workspace/platform version supports all the mapper types used by the source connection.


Supported Mapper Types

From the platform's StreamMapperType enum:

Mapper Type Has Secrets? Migration Concern
hashing No Should migrate cleanly
field-renaming No Should migrate cleanly
row-filtering No Should migrate cleanly
field-filtering No Should migrate cleanly
encryption Yes (encryption keys) Secrets will be masked, need re-entry

Proposed Improvements (P2 - not blocking initial migration)

1. Document mapper secret limitation

At minimum, document that mapper secrets (currently only encryption mappers) will need to be manually re-entered after migration.

2. Add mapper listing to describe_cloud_connection

Enhance the describe_cloud_connection MCP tool to include a summary of configured mappers per stream (type + non-secret config), so users can see what needs attention.

3. Add mapper validation tool

Expose the platform's /v1/web_backend/mappers/validate endpoint as an MCP tool to validate that mappers from a dumped catalog are compatible with the target connection.


Priority

P2 — Not blocking for initial migration. Basic mapper preservation works via catalog dump/import. Only encryption mapper secrets need manual re-entry, which is an edge case for most migrations.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions