[Repository Request] opensearch-flux

[Maosaic]

Are you requesting a new GitHub Repository within opensearch-project GitHub Organization?

Yes

GitHub Repository Proposal

• [PROPOSAL] OpenSearch Flux — CLI and MCP server for migrating Dashboards saved objects between OpenSearch instances #499

GitHub Repository Additional Information

1. What is the new GitHub repository name?

opensearch-flux

2. Project description and community value?

OpenSearch Flux is a CLI tool and MCP server for migrating OpenSearch Dashboards saved objects (dashboards, visualizations, index patterns, saved searches) between OpenSearch instances. It automates the full export → inspect → repair → import pipeline, handling cross-platform differences between AOS domains, AOSS serverless collections, OpenSearch Dashboards Applications, and self-managed OpenSearch clusters.

Community value: eliminates hours of manual NDJSON editing for dashboard migrations, handles edge cases that are nearly impossible to get right manually (nested JSON data source remapping, TSVB/Vega spec injection, compound ID stripping, field compatibility checks), and enables bulk migration of entire dashboard portfolios.

3. What user problem are you trying to solve with this new repository?

There is no built-in tool to migrate Dashboards saved objects between OpenSearch instances when the source and target use different data source configurations, index naming schemes, or platform types. Users currently resort to manual NDJSON editing, which is error-prone and time-consuming. This tool automates the entire process with validation and auto-repair.

4. Why do we create a new repo at this time?

The tool is production-ready after 6 rounds of end-to-end testing across real AWS endpoints. All 6 migration scenarios (UI→UI, AOS→UI, AOSS→UI, index remap, missing fields, bulk 43 dashboards) pass at 100% success rate with 0 critical failures. The codebase is stable and well-documented.

5. Is there any existing projects that is similar to your proposal?

opensearch-migrations handles data and cluster migrations (index data, cluster upgrades). OpenSearch Flux is complementary — it focuses specifically on Dashboards saved object migration (metadata, not data). The two tools address different layers of the migration problem.

6. Should this project be in OpenSearch Core/OpenSearch Dashboards Core? If no, why not?

No. This is a standalone CLI tool and MCP server, not a plugin or core feature. It operates externally against the existing Dashboards saved objects API and does not require changes to OpenSearch or OpenSearch Dashboards. Keeping it as a separate repository allows independent versioning and release cycles, and makes it usable with any OpenSearch version that supports the saved objects API.

7. Is this project an OpenSearch/OpenSearch Dashboards plugin to be included as part of the OpenSearch release?

Not at this time. OpenSearch Flux is currently a standalone Node.js CLI tool and MCP server that operates externally against the Dashboards saved objects API. However, we are considering building an OpenSearch Dashboards plugin in the future to enable an in-app migration experience. If that happens, we would evaluate inclusion in the OpenSearch Dashboards release at that point.

GitHub Repository Owners

1. Who will be supporting this repo going forward?

OpenSearch Dashboard maintainers will maintain the repository.

2. What is your plan (including staffing) to be responsive to the community?

• Triage and respond to new issues within 2 weeks
• Review PRs within 1 weeks
• Monitor OpenSearch forum threads related to dashboard migration on a monthly basis
• Maintain CI/CD pipeline and publish releases as needed

3. Initial Maintainers List (max 3 users, provide GitHub aliases):

1. @Maosaic
2. @xinruiba
3. @seraphjiang

GitHub Repository Source Code / License / Libraries

1. Please provide the URL to the source code.

https://github.com/Maosaic/opensearch-flux

2. What is the license for the source code?

Apache License 2.0

3. Does the source code include any third-party code that is not compliant with the Apache License 2.0?

No. All dependencies are Apache 2.0 compatible:

• @modelcontextprotocol/sdk (MIT)
• commander (MIT)
• zod (MIT)
• form-data (MIT)
• typescript (Apache 2.0)
• AWS SDK v3 packages (Apache 2.0)

What is the publication target(s)?

You can choose multiple targets from the list.

NPM

Notes (DO NOT CHANGE)

Next Steps:

• If this is about creating a new GitHub Repository

  • Build Interest Group (BIG) and its members will review your proposal and provide feedback
    • Review of Proposal, asking questions, adding comments
    • If there is any concern regarding the naming / IP, additional IP review will be requested
    • Involve Subject Matter Experts from other repositories on the proposed topics
    • Ensure new repositories align with the foundation’s charter
    • Review the provided source code if any
    • Send final feedback and recommendations to the Technical Steering Committee
  • Technical Steering Committee (TSC) will have a vote based on BIG feedback, and reply back the vote as a comment in this issue by a TSC member
  • At least three positive (+1) TSC members' votes are necessary, and no vetoes (-1) after a one week period, then Admin Team will open a repo creation ticket with Linux Foundation
  • Linux Foundation verify the votes and create repo
  • Admin Team setup automations on repo settings, secrets, scanning, add initial maintainers, and more
  • Repository delivered to the original requester

• If you already have a GitHub repo and just want to add new publication target(s)

  • Admin Team will review your request and follow up

Track the progress of your request here: Engineering Effectiveness Board (view).
Member of @opensearch-project/admin will take a look at the request soon.
Thanks!

[peterzhuamazon]

Hi @Maosaic ,

A few questions:

1. Seems like this is more related to OSD objects, so does it make sense to include that context in the naming, such as opensearch-dashboards-flux or similar?
2. We already have existing repos for similar migration purposes such as opensearch-migrations or opensearch-mcp-server-py, does it make sense to combine these efforts together? Or back to question 1 where we want to be more specific.
3. Does it make sense to include AOS related features in OSS? Just wondering.
4. Given the ask for another npm related repo, I will be working with you after github repo is delivered to set up the workflow for deployment and trusted publisher.

Adding @msfroh @mishail @anirudha @getsaurabh02 @bugmakerrrrrr to give their comments and votes.

Thanks.

[Maosaic]

Hi @Maosaic ,

A few questions:

1. Seems like this is more related to OSD objects, so does it make sense to include that context in the naming, such as opensearch-dashboards-flux or similar?
2. We already have existing repos for similar migration purposes such as opensearch-migrations or opensearch-mcp-server-py, does it make sense to combine these efforts together? Or back to question 1 where we want to be more specific.
3. Does it make sense to include AOS related features in OSS? Just wondering.
4. Given the ask for another npm related repo, I will be working with you after github repo is delivered to set up the workflow for deployment and trusted publisher.

Adding @msfroh @mishail @anirudha @getsaurabh02 @bugmakerrrrrr to give their comments and votes.

Thanks.

Thanks for the thoughtful questions.

1. Naming — should it include "dashboards"?

Good point. Open to renaming to opensearch-dashboards-migration or similar if that's clearer. The tool is specifically about Dashboards saved objects (dashboards, visualizations, index patterns, saved searches), not OpenSearch cluster data. Happy to go with whatever naming convention the team prefers.

2. Combining with existing repos?

I looked at both in detail:

opensearch-migrations — focuses on cluster-level data migration: index data replication, cluster upgrades, traffic capture/replay, and metadata migration (index settings, templates, aliases). It operates at the OpenSearch engine layer.

OpenSearch Flux operates at the Dashboards application layer — it migrates saved objects (dashboards, visualizations, index patterns, saved searches) via the Dashboards saved objects API (_find, _import). The challenges are completely different:

• Compound ID stripping ({dataSourceId}_{objectId} format unique to OpenSearch UI)
• Data source remapping inside nested JSON-within-JSON (searchSourceJSON.dataset.dataSource.id)
• TSVB/Vega visualization spec injection (platform-specific data source fields)
• Dashboard dataset block stripping (stale cached references)
• Field compatibility checks against target index mappings

None of these concepts exist in opensearch-migrations. It's also a different tech stack (Java/Python vs TypeScript) with different maintainers and release cycles.

opensearch-mcp-server-py — a general-purpose MCP server for OpenSearch cluster operations: searching indices, retrieving mappings, managing shards, cluster health, etc. It exposes generic OpenSearch APIs to AI agents.

Our MCP server is purpose-built for the migration workflow. The tools map to specific migration pipeline steps:

opensearch-mcp-server-py	opensearch-flux MCP
SearchIndexTool — search an index	export_dashboard — export a dashboard + all dependencies
IndexMappingTool — get mappings	validate_index_patterns — check patterns exist on target
ListIndexTool — list indices	list_dashboards — list Dashboards saved objects
Generic cluster operations	remap_data_sources — rewrite nested JSON references
N/A	strip_data_source_prefixes — remove compound IDs
N/A	strip_dashboard_dataset — remove stale cached refs
N/A	disable_missing_field_filters — auto-fix broken filters
N/A	import_ndjson — import to workspace with validation

There's zero tool overlap. The opensearch-mcp-server-py tools are read-only cluster introspection; ours are migration-specific transformations.

Combining either repo would mean merging unrelated codebases with different languages, dependencies, problem domains, and maintainers. I think standalone makes more sense, but open to discussion if there's a strong organizational preference.

3. AOS-related features in OSS?

The tool uses only public OpenSearch APIs (saved objects _find, _import, _export). The AOS/AOSS-specific logic is just endpoint detection for SigV4 signing service selection (es vs aoss vs opensearch) based on the public amazonaws.com URL patterns. There's no proprietary API usage — the same tool works against self-managed OpenSearch clusters with --auth basic or --auth none. The AWS-specific parts are purely about authentication, which is already how the OpenSearch clients work in other opensearch-project repos.

4. NPM workflow

Great — happy to work together on the trusted publisher setup once the repo is delivered.

[msfroh]

The repository sounds useful to me. I'm happy with either opensearch-dashboards-migrations or opensearch-dashboards-flux -- IMO, both accurately describe the purpose. (I guess -migrations is clearer from a "plain english" standpoint, but -flux conveys change / adapting to change.)

[mishail]

Sounds very similar to https://github.com/opensearch-project/opensearch-migrations/tree/main/dashboardsSanitizer , But in this case, it’s OpenSearch to OpenSearch, so to me that's not a "migration" but rather a "transfer”.

[anirudha]

Both names are ok. Let's build and ship. Also consider Claude code , kiro powers or plugins

[getsaurabh02]

Sounds good to me. The problem is well-defined as dashboard migration across different OpenSearch platform types (AOS, AOSS, self-managed) is a real pain point with no existing tooling.

Consider adding a compatibility matrix in the README showing which OpenSearch/Dashboards versions have been tested, since saved object formats can vary across versions.

[peterzhuamazon]

Thanks @msfroh @anirudha @getsaurabh02 for the votes.

Two more questions:

1. @mishail do you have more questions to @Maosaic regarding the RFC or creating repo for this?
2. @Maosaic do we want to use flux, opensearch-dashboards-flux, or opensearch-flux overall? Note this name is not easy to update after the initialization on both github and npm.

Let me know.

Thanks.

[Maosaic]

Thanks @msfroh @anirudha @getsaurabh02 for the votes.

Two more questions:

1. @mishail do you have more questions to @Maosaic regarding the RFC or creating repo for this?
2. @Maosaic do we want to use flux, opensearch-dashboards-flux, or opensearch-flux overall? Note this name is not easy to update after the initialization on both github and npm.

Let me know.

Thanks.

I'm leaning toward opensearch-flux if no objection from team.