[PROPOSAL] OpenSearch Flux — CLI and MCP server for migrating Dashboards saved objects between OpenSearch instances

[Maosaic]

What/Why

What are you proposing?

OpenSearch Flux is a CLI tool and Model Context Protocol (MCP) server that migrates OpenSearch Dashboards saved objects — dashboards, visualizations, index patterns, and saved searches — between OpenSearch instances. It handles the full export → inspect → repair → import pipeline, including data source remapping, index pattern translation, field compatibility checks, and workspace management.

The tool supports migrations across platform boundaries: OpenSearch managed domains (AOS), OpenSearch Serverless (AOSS), OpenSearch Dashboards Applications (OpenSearch UI), and self-managed OpenSearch clusters.

What users have asked for this feature?

There is no built-in way to migrate Dashboards saved objects between OpenSearch instances when the source and target use different data source configurations, index naming schemes, or platform types. Users currently resort to manual NDJSON editing, which is error-prone due to:

• Compound ID formats ({dataSourceId}_{objectId}) that must be stripped for cross-instance imports
• Data source references embedded inside nested JSON-within-JSON strings (searchSourceJSON.dataset.dataSource.id)
• TSVB and Vega visualization specs that require platform-specific data source injection
• Stale dataset blocks in dashboard searchSourceJSON that override visualization data source context
• Field compatibility mismatches that silently break filters on the target

Community discussions around saved object portability have surfaced in OpenSearch forums and GitHub issues on the OpenSearch-Dashboards repository, particularly from users migrating from self-managed clusters to AWS-managed services or consolidating multiple domains.

What problems are you trying to solve?

• When migrating dashboards from an AOS domain to an OpenSearch UI application, an operator wants to export all dashboard dependencies and re-import them with correct data source bindings, so they don't have to manually recreate dozens of visualizations.
• When consolidating OpenSearch instances across AWS accounts, a platform team wants to bulk-migrate all dashboards (40+) with zero manual intervention, so they can complete the migration in minutes instead of days.
• When source and target clusters use different index naming conventions, a user wants to remap index patterns during migration and validate field compatibility, so they don't end up with dashboards that show "No data".
• When migrating from AOSS to OpenSearch UI, a user wants the tool to handle the platform differences automatically (SigV4 signing service, collection semantics, permission model), so they don't need to understand the internal differences between AOSS and AOS.

What is the developer experience going to be?

CLI interface:

# List dashboards on source
opensearch-flux export --source <endpoint> --auth iam

# Export a dashboard with all dependencies
opensearch-flux export --source <endpoint> --auth iam --dashboard <id>

# Import to target with workspace creation
opensearch-flux import \
  --target <endpoint> --auth iam \
  --file ./migration/export.ndjson \
  --data-source-id <target-ds-id> \
  --source <source-endpoint> \
  --target-workspace new \
  --workspace-name "Migrated Dashboard" \
  --create-new-copies --yes

MCP server interface (15 tools):

Category	Tools
Connection	test_connection, list_dashboards, list_data_sources
Export	export_dashboard
Inspection	inspect_ndjson, extract_data_source_info, validate_index_patterns
Repair	strip_data_source_prefixes, remap_data_sources, strip_dashboard_dataset, remap_index_patterns, disable_missing_field_filters
Workspace	list_workspaces, create_workspace
Import	import_ndjson

REST API impact: None. The tool uses existing OpenSearch Dashboards saved objects APIs (_find, _import, _export) and does not introduce new APIs.

Are there any security considerations?

• The tool uses the standard AWS SDK credential chain for IAM authentication (SigV4). No credentials are stored or cached by the tool.
• Cookie-based authentication is supported for SAML/Cognito-protected instances. Session cookies are passed as CLI arguments and not persisted.
• Basic auth credentials are passed as CLI arguments and not persisted.
• The tool operates on saved object metadata only — it does not read or write index data.
• All network requests use HTTPS.

Are there any breaking changes to the API?

No. This is a new standalone tool. It does not modify any existing OpenSearch or OpenSearch Dashboards APIs.

What is the user experience going to be?

1. Export: User points the CLI at a source endpoint and selects a dashboard. The tool recursively resolves all dependencies (visualizations, index patterns, saved searches) and exports them as NDJSON.
2. Inspect: The tool automatically analyzes the export for missing references, platform mismatches, and permission gaps.
3. Import: User points the CLI at a target endpoint with a data source ID. The tool:
   • Strips compound ID prefixes
   • Remaps all data source references (including deep nested JSON)
   • Strips stale dataset blocks from dashboards
   • Validates index patterns against the target cluster
   • Suggests index pattern remappings when names differ
   • Checks field compatibility and auto-disables broken filters
   • Creates a workspace and associates the data source
   • Imports all objects
4. Verify: The tool reports success/error counts and provides a direct URL to the imported dashboard.

The MCP server enables AI agents to drive this entire workflow conversationally.

Are there breaking changes to the User Experience?

No. This is a new tool with no existing user base in the open-source community.

Why should it be built? Any reason not to?

Value:

• Eliminates hours of manual work for cross-platform dashboard migrations
• Handles edge cases that are nearly impossible to get right manually (nested JSON remapping, TSVB/Vega injection, compound ID stripping)
• Enables bulk migration of entire dashboard portfolios (tested with 43 dashboards, 568 objects, 0 errors)
• AI-native design makes it accessible to users who aren't familiar with NDJSON internals

Risk if not built:

• Users continue to manually edit NDJSON files, leading to broken dashboards and wasted time
• No standardized migration path between OpenSearch platform types

Reasons not to:

• The existing opensearch-migrations project handles data/cluster migrations, but it does not address Dashboards saved object migration. These are complementary, not overlapping.

What will it take to execute?

The tool is already built and production-tested:

• TypeScript/Node.js 22+ codebase
• 6 rounds of end-to-end testing across real AWS endpoints (AOS, AOSS, OpenSearch UI)
• 6 migration scenarios, all passing at 100% success rate
• 16 bugs found and fixed through automated testing
• Comprehensive documentation (migration workflow guide, build diary, CLI reference)

Open-sourcing requires:

• Code cleanup and removal of internal references
• Adding standard OpenSearch project files (CONTRIBUTING.md, CODE_OF_CONDUCT.md, NOTICE, etc.)
• CI/CD setup (GitHub Actions for build, test, lint)
• NPM publication setup (if desired)

Any remaining open questions?

1. Repository name: opensearch-flux or dashboards-migration-cli or another name that better fits the opensearch-project naming conventions?
2. Relationship to opensearch-migrations: Should this live as a sub-project within opensearch-migrations, or as a standalone repo? The scope is different (saved objects vs. data/cluster), but there may be organizational benefits to co-locating.
3. NPM publication: Should the CLI be published to NPM as @opensearch-project/opensearch-flux?
4. MCP server distribution: How should the MCP server be distributed for AI agent integration?
5. OSD plugin: We may consider building an OpenSearch Dashboards plugin in the future to enable an in-app migration experience — allowing users to import/export dashboards directly from the Dashboards UI without needing the CLI. The core migration logic (repair, remap, validate) is already modular and could be reused as a plugin backend.

[seraphjiang]

Supporting evidence for this proposal

I did some research across the OpenSearch ecosystem — GitHub issues, forum discussions, existing tooling, and official documentation — and the evidence strongly supports the need for this tool. Here's what I found:

1. Community pain points are well-documented

Multiple GitHub issues on OpenSearch-Dashboards highlight the gap:

• #5757 — "[Proposal][MD] Enable importing saved objects from external data sources" (still open, labeled v2.17.0). Acknowledges the multi-step pain of downloading from one instance and uploading to another. Has a PoC but remains unresolved for cross-platform scenarios.
• #2688 — Manual instruction needed for VisBuilder saved object migration.
• #1466 — Import API silently alters custom color info in visualizations.
• #6289 — Creating a datasource breaks the saved objects table entirely.

On the OpenSearch forum, the same frustration surfaces repeatedly:

Forum thread	Views	Year
How to download opensearch dashboard	5,300+	2022
Is there a Saved Objects API in OpenSearch?	2,500+	2024
API to script export/import dashboards	2,400+	2023
How to use curl to export dashboard objects?	—	2024
Copy dashboards from elasticsearch to opensearch	—	2024

In the scripting thread, an OpenSearch maintainer responded: "I wish I had a better answer, such as a link to some nice user docs. But those don't exist yet." That was Feb 2023 — and the situation hasn't materially changed.

2. No existing tool covers this scope

Tool	What it does	What it doesn't do
ndjson_manip (hmeine)	Unpacks/repacks NDJSON for readability and VCS diffs	No data source remapping, no cross-platform migration, no repair logic. 1 star, 21 commits.
dashboardsSanitizer (opensearch-migrations)	Basic sanitization of dashboard objects	Scope unclear; no comprehensive migration pipeline
AWS official docs (migration guide)	Describes manual export → import workflow	Acknowledges manual NDJSON editing is required; no automation
SocPrime blog (guide)	4-step manual process	Warns about index pattern duplicates; no data source handling

The OpenSearch project's own blog post — "The Saved Objects import feature improves information transfer for multiple data sources" (Feb 2024) — explicitly acknowledges the gap: importing objects without data source info causes visualization failures. But the 2.4 enhancement only handles the simple case of attaching a single dataSourceId. It does not handle compound ID stripping, nested JSON-within-JSON remapping, TSVB/Vega injection, field compatibility checks, or cross-platform differences.

3. Standalone repo, not a sub-project of opensearch-migrations

The opensearch-migrations repo (70 stars, 7,298 commits, v2.9.0) focuses on cluster and data migration: Elasticsearch → OpenSearch version upgrades, Reindex-from-Snapshot, traffic capture/replay, metadata migration, and Solr migration. Its scope is explicitly about indices, documents, and cluster configuration. It does not handle Dashboards saved object migration across platform types.

OpenSearch Flux fills the "last mile" that opensearch-migrations doesn't cover: once your data is migrated, you still need to migrate the dashboards that visualize it. But it should live as a standalone repo, not a sub-project:

• Different tech stack. opensearch-migrations is primarily Java/Gradle (46.5%) with Python and Go. It's infrastructure-heavy — CDK deployments, Docker orchestration, Kafka-based traffic capture, Fargate workers. OpenSearch Flux is a lightweight TypeScript/Node.js CLI. Embedding a Node.js project inside a Java/Gradle monorepo creates build system friction, confusing contributor onboarding, and CI/CD complexity.
• Different user persona. opensearch-migrations targets platform engineers doing full cluster migrations — CDK stacks, migration consoles, snapshot-based reindexing. OpenSearch Flux targets a broader audience: DevOps engineers, dashboard authors, and AI agents who just need to move dashboards between instances. A standalone npx opensearch-flux is far more approachable than cloning a 7,298-commit monorepo.
• Different release cadence. opensearch-migrations is on a weekly release cycle tightly coupled to OpenSearch version compatibility matrices (ES 1.x through OS 3.x). Dashboard saved object migration operates on the Dashboards saved objects API, which is stable across versions. Independent releases let Flux ship fixes without waiting for the migration assistant's release train.
• Precedent in the ecosystem. The OpenSearch project already maintains separate repos for tools with distinct scopes: opensearch-mcp-server-py is standalone despite being an OpenSearch integration tool. OpenSearch-Dashboards is separate from OpenSearch core.
• The DashboardsMigration folder is docs-only. The existing DashboardsMigration/docs in opensearch-migrations contains documentation, not a functional tool. And dashboardsSanitizer is a narrow utility, not a full migration pipeline. OpenSearch Flux replaces the need for them with a comprehensive solution.
• Cross-reference, don't co-locate. The right relationship is a link in opensearch-migrations' README pointing to OpenSearch Flux for dashboard migration. Users doing a full migration would use opensearch-migrations for data + Flux for dashboards — two tools, one workflow, documented together.

4 The manual alternative is genuinely hard

The proposal's description of the technical challenges isn't hypothetical — it's exactly what users hit:

• Compound IDs ({dataSourceId}_{objectId}) that must be stripped for cross-instance imports
• Data source references buried inside nested JSON-within-JSON strings (searchSourceJSON.dataset.dataSource.id)
• TSVB and Vega specs requiring platform-specific data source injection
• Stale dataset blocks that override visualization data source context

The ndjson_manip tool's README illustrates the problem perfectly — it shows a single NDJSON line that's 49KB of nested, escaped JSON. That's what users are expected to manually edit today.

---

Summary: This proposal addresses a real, well-documented gap in the OpenSearch ecosystem. The community has been asking for this since at least 2022, no existing tool covers the full scope, it's complementary to (not competing with) opensearch-migrations and should live as a standalone repo, and the MCP server design aligns with OpenSearch's AI-native direction. Strong +1.

[jkowall]

You say the tool is already built and production-tested, where is this magical tool?

Great idea but this is very broad and ambitious, so I ask:

• Why not a tighter MVP boundary. The proposal currently bundles a CLI, a 15-tool MCP server, workspace
  management, NPM publishing, and a future Dashboards plugin. That weakens the “atomic”
  criterion.
• Success criteria for acceptance. It lacks a crisp “v1 is done when…” section: required
  migration pairs, object types, validation guarantees, and failure/rollback behavior.

Thanks!

[Maosaic]

You say the tool is already built and production-tested, where is this magical tool?

Great idea but this is very broad and ambitious, so I ask:

• Why not a tighter MVP boundary. The proposal currently bundles a CLI, a 15-tool MCP server, workspace
  management, NPM publishing, and a future Dashboards plugin. That weakens the “atomic”
  criterion.
• Success criteria for acceptance. It lacks a crisp “v1 is done when…” section: required
  migration pairs, object types, validation guarantees, and failure/rollback behavior.

Thanks!

Thanks for the feedback — fair points on both.

On MVP scope: The core deliverable for v1 is the CLI only. The MCP server shares the same codebase (it wraps the same library functions), so it comes along for free, but it's not a separate effort. Workspace management is part of the import flow (not a standalone feature). NPM publishing and the Dashboards plugin are future considerations, not v1 scope. To be explicit:

• v1 scope: CLI + MCP server (same codebase, same library)
• Post-v1: NPM global install, Dashboards plugin

On success criteria — v1 is done when:

Criteria	Definition
Migration pairs	AOS → OpenSearch UI, AOSS → OpenSearch UI, OpenSearch UI → OpenSearch UI, self-managed → OpenSearch UI
Object types	dashboard, visualization, index-pattern, search (and their transitive dependencies)
Validation	Index pattern existence check against target cluster, field compatibility check, missing reference detection
Auto-repair	Compound ID stripping, data source remapping (including nested JSON), dashboard dataset stripping, broken filter disabling
Failure behavior	Non-destructive — all transformations happen on a local NDJSON copy, source is never modified. Import failures return error details per object. No rollback needed because the tool creates new copies by default (--create-new-copies)
Auth	IAM SigV4, basic auth, cookie/SAML, no-auth
Tested	6 E2E scenarios passing at 100% (UI→UI, AOS→UI, AOSS→UI, index remap, missing fields, bulk 43 dashboards)

All of the above is already built and tested — the source code is at https://github.com/Maosaic/opensearch-flux.

[anirudha]

Love this. Let's ship