[FEATURE] Marketplace upstreams: curated pass-through with allow-list governance

[sergio-sisternes-epam]

Problem

Today APM's marketplace has two disjoint sides:

• Authoring (apm.yml -> marketplace:): each packages[] entry's source must be a single git repo (owner/repo) or a local path. There is no way to express "include a plugin from another marketplace".
• Consumption (~/.apm/marketplaces.json): registers external marketplace.json URLs for browse / search / update. Read-only from the curator's perspective.

A curator (e.g. an enterprise running an internal marketplace) wants to selectively re-expose plugins from external marketplaces (e.g. a Claude Code marketplace such as abhigyanpatwari/GitNexus), with curator-side allow-list control over which plugins reach their consumers.

Proposed solution: marketplace upstreams

Add a first-class upstream concept to the curator authoring surface:

• New upstreams: block in apm.yml -> marketplace: registers external marketplace pointers under a curator-chosen alias.
• New upstream: field on packages[] entries declares an allow-listed entry that resolves to a plugin from a registered upstream.
• The builder fetches the upstream marketplace.json (atomic read at a pinned manifest SHA), strict-parses it, looks up the plugin, resolves its source to an immutable commit SHA, and emits a vanilla plugin entry into the curator's marketplace.json.

v1 product contract

v1 is curated upstream / governance-only, not artifact custody:

• Curator controls which plugins are exposed and at what version.
• Curator's lockfile pins manifest SHA + plugin SHA for reproducible curator builds.
• Consumers still fetch plugin content from the upstream git host at install time.
• Never called "mirror" in v1 prose -- "mirror" is reserved for a future v2 distribution: rehost mode that would add true artifact custody.

One-line pitch: "Allow-list governance over external plugins, pinned to immutable commits, with provenance baked into every build -- without running an artifact server."

Trust model

Concern	v1 status
Allow-list governance	YES (curator picks which plugins are exposed)
Build-time commit pinning	YES (curator's lockfile pins manifest SHA + plugin SHA)
Reproducible curator builds	YES (rebuild from lock = byte-identical output)
Consumer-side artifact custody	NO -- consumer clones from upstream git host at install
Resilience to upstream takedown / force-push	NO -- consumer install fails opaquely if upstream rewrites history
Defence against upstream repo rename / takeover	YES (canonical-owner check)

Sketch of the surface

marketplace:
  upstreams:
    - alias: gitnexus
      repo: abhigyanpatwari/GitNexus
      path: .claude-plugin/marketplace.json   # default
      ref: <sha-or-tag>                        # required for reproducibility
      branch: main                             # used only when ref absent and allow_head=true
      host: github.com                         # default
      allow_head: false                        # default
  packages:
    # Direct package (existing shape)
    - name: my-skill
      source: owner/repo
      version: ">=1.0.0"

    # Upstream-sourced package (new shape)
    - name: my-renamed-skill                  # display name in curator's marketplace
      upstream: gitnexus                      # references upstreams[].alias
      plugin: original-name                   # name in upstream marketplace; defaults to `name`
      version: ">=2.0.0"                      # optional curator override
      ref: <sha-or-tag>                       # optional curator override
      description: "ACME-curated ..."         # optional override
      tags: [acme, approved]                  # optional override
      allow_head: false                       # opt-in to mutable refs (per-entry)

apm marketplace upstream add <owner/repo> [--alias --path --ref --branch --host --allow-head]
apm marketplace upstream list [--verbose]
apm marketplace upstream remove <alias> [--yes]
apm marketplace upstream refresh [<alias>]   # shows diff before advancing lock pins
apm marketplace upstream browse <alias>      # lists plugins available in upstream
apm marketplace package add --upstream <alias> <plugin-name> [--name --version --ref --tags]

Key design decisions (panel-reviewed)

The plan was reviewed by an APM panel (supply-chain, architect, devx, primitives, test-coverage, CEO). Findings folded in:

• Provenance lives in apm.lock.yaml only. No metadata.apm.* keys in marketplace.json -- preserves the Anthropic-conformance hard rule at src/apm_cli/marketplace/builder.py:13-15.
• Type-level discriminated union: DirectPackageEntry / UpstreamPackageEntry as separate frozen dataclasses (not optional fields on a single class).
• Module decomposition: new marketplace/upstream_parser.py (strict parser, separate from lenient consumer parser), marketplace/upstream_resolver.py (extracted resolver -- avoids MarketplaceBuilder god-class), marketplace/upstream_cache.py (separate from consumer client.py).
• Cache key uses __ (double-underscore), not :: -- colons are illegal in Windows filenames (CI runs on windows-latest).
• Per-upstream-host auth: AuthResolver.resolve(upstream_host, org=upstream_owner) with unauth_first=True for public repos, never inheriting curator's marketplace-source PAT.
• Repo-rename / takeover guard: every fetch verifies GitHub Contents API repository.full_name matches configured upstreams[alias].repo; lockfile records canonical name.
• Cross-shape name uniqueness enforced builder-side (prevents dependency-confusion / name shadowing).
• Round-trip invariant: every emitted plugin must survive parse_marketplace_json (catches lenient-vs-strict parser drift).
• Atomic single-fetch invariant: manifest is fetched once at the pinned SHA; no re-fetch during resolution.
• Governance gate: new policy key marketplace.upstream.allow_unpinned_refs lets enterprises forbid allow_head: true.
• security.md update required in same PR: the existing "no intermediary registry, proxy, or mirror" claim becomes inaccurate.
• apm doctor integration: upstream reachability + pinned-vs-HEAD drift + manifest-SHA round-trip checks.
• Consumer-side passthrough notice: when apm install resolves a plugin with upstream provenance in the lockfile, output names the upstream so the trust boundary is visible.
• DevX: upstream_name renamed to plugin; CLI mutex enforcement (source xor --upstream); dual-shape help text; copy-pasteable next-step hints; exit code 2 for resolution errors; _authoring_commands registration.
• Cut from v1: apm marketplace upstream search (browse covers it).
• Terminology: "upstream" in user-facing copy; never "mirror" in v1 (reserved for v2 rehost).

Out of scope (deferred)

• Re-hosting / artifact custody / true air-gap (distribution: rehost) -- v2.
• Transitive upstreams (curator upstreams marketplace B which upstreams marketplace C).
• Install-time auto-sync of upstream changes.
• Cross-git-host upstreams (upstream and plugins on different host families).
• Mirroring non-APM/non-Claude formats (npm, PyPI, etc.).

Implementation plan (high-level, dependency-ordered)

1. Schema: type-level union dataclasses + upstreams: block + strict validation.
2. upstream_parser.py: strict parser with named rejection reasons.
3. upstream_cache.py: Windows-safe content-addressed cache with per-host auth.
4. upstream_resolver.py: extracted resolver (atomic fetch, precedence ladder, repo-rename guard).
5. MarketplaceBuilder integration: dispatch on union, name-shadowing check, round-trip invariant, exit code 2.
6. apm.lock.yaml upstream section: writer + reader; refresh diff.
7. CLI surface: upstream add/list/remove/refresh/browse + package add --upstream.
8. apm doctor upstream health checks.
9. Governance policy key marketplace.upstream.allow_unpinned_refs.
10. Consumer-side passthrough notice in apm install output.
11. Integration tests in scripts/test-integration.sh (incl. lock-holds, reproducibility).
12. Docs: new upstreams guide, security.md rewrite, policy-reference update, apm-guide skill resources, CHANGELOG.
13. Final lint + format gate.

Alternatives considered

• Mirror / proxy: rejected as v1 framing because no artifacts are re-hosted; reserve "mirror" for the future distribution: rehost mode.
• Optional source on PackageEntry: rejected -- breaks unguarded entry.source access across the builder; type-level discriminated union is the correct pattern.
• Provenance keys in marketplace.json: rejected -- violates the Anthropic-conformant emission contract documented in builder.py:13-15. All provenance lives in the lockfile.
• Extending consumer client.py for upstream cache: rejected -- consumer cache is wired to MarketplaceSource + consumer auth context. Sibling upstream_cache.py keeps boundaries clean.

Additional context

Plan validated with APM Panel

[github-actions]

Triage decision

accept

Proposed labels

theme/governance
area/marketplace
area/lockfile
type/feature
status/accepted
priority/high

Milestone

0.13.0

Suggested next action

Pick up the implementation plan from step 1 (schema: type-level union dataclasses + upstreams: block) and open a draft PR that includes the security.md rewrite in the same branch, since the existing "no intermediary registry" claim becomes inaccurate as soon as the builder emits upstream-sourced entries.

Suggested issue comment

Thanks for this detailed proposal and the pre-validation with the APM panel. The
marketplace upstreams feature is accepted.

The allow-list governance angle -- curators controlling which external plugins
reach their consumers, pinned to immutable commits -- is exactly the kind of
enterprise-facing capability on the roadmap. The design decisions folded into the
body (type-level discriminated union, per-upstream-host auth, canonical-owner
check, `allow_head: false` default, `marketplace.upstream.allow_unpinned_refs`
governance gate) are all the right calls.

A few implementation reminders before the first PR lands:

- The `security.md` rewrite (step 12) must ride in the same PR as the builder
  changes, not a follow-up. The existing "no intermediary registry, proxy, or
  mirror" claim will be inaccurate the moment step 5 (MarketplaceBuilder
  integration) merges.
- Per repo Rule 4, `packages/apm-guide/.apm/skills/apm-usage/` resources
  (commands.md for the new CLI surface, governance.md for the new policy key)
  must be updated in the same PR as the CLI changes.
- The acknowledged v1 limitations (consumer-side custody, resilience to
  upstream force-push) should be stated explicitly in the updated security.md
  as "out of scope for v1" so the trust boundary is visible to enterprise
  evaluators.

Suggested first PR scope: steps 1-2 (schema + strict parser), with the
round-trip invariant test from step 11 as the acceptance gate. This gives
reviewers a grounded diff before the resolver and CLI land.

Per-lens notes (collapsed)

DevX UX Expert -- User-Need Reviewer

The apm marketplace upstream add/list/remove/refresh/browse surface maps cleanly to
the cargo [registries] / Homebrew tap mental model: a curator registers an upstream
once, then allow-lists plugins from it. The plugin: field (defaulting to name) and
allow_head: false safe default are correct. The source xor --upstream CLI mutex
prevents dual-mode confusion. The refresh <alias> diff-before-commit flow mirrors
brew update and is appropriate for a lockfile-pinned tool. exit code 2 for
resolution errors and copy-pasteable next-step hints align with the repo UX standards.
No surface mismatch; _authoring_commands registration keeps the command tree consistent.

Supply Chain Security Expert -- Risk-Surface Reviewer

High governance and security relevance. Risk surfaces assessed: (1) name-shadowing --
cross-shape name uniqueness enforced builder-side; (2) lockfile pinning -- manifest
SHA + plugin SHA; (3) consumer-side fetch risk -- acknowledged v1 limitation (consumer
clones from upstream git host; no resilience to takedown/force-push); (4) token scoping
-- AuthResolver.resolve(upstream_host, ..., unauth_first=True) never inherits curator
PAT; (5) repo-rename/takeover guard via repository.full_name canonical check; (6)
allow_head: false default and marketplace.upstream.allow_unpinned_refs governance
gate. security.md rewrite is called out in the plan. Primary theme theme/governance;
area/lockfile rides as a secondary because the lockfile writer/reader is a first-class
implementation step.

OSS Growth Hacker -- Contributor-Tone Reviewer

Not activated -- Author is a COLLABORATOR with a panel-pre-validated proposal; tone
does not need first-time contributor adjustment.

Python Architect -- Architecture Reviewer

The design is implementation-ready. Type-level discriminated union (DirectPackageEntry
/ UpstreamPackageEntry as frozen dataclasses) avoids optional-field unguarded-access
bugs across the builder -- the correct Python pattern. Module decomposition
(upstream_parser.py, upstream_resolver.py, upstream_cache.py) cleanly avoids a
god-class and respects the existing client.py boundary. The __ cache key for Windows
filesystem compatibility is correctly identified. The 13-step dependency-ordered plan is
solid. Round-trip invariant and atomic single-fetch invariant are strong contracts. Body
states "Plan validated with APM Panel" -- design phase is complete; status/accepted is
correct, not status/needs-design.

Doc Writer -- Documentation Reviewer

Implementation plan step 12 explicitly requires a new upstreams guide, security.md
rewrite, policy-reference update, apm-guide skill resources, and CHANGELOG. The
security.md rewrite is load-bearing (the existing "no intermediary registry" claim
becomes inaccurate) and must not be deferred. The vocabulary in the issue body
("upstream", "curator", "allow-list", "pinned") is consistent with the marketplace
authoring mental model. The implementing PR must also update
packages/apm-guide/.apm/skills/apm-usage/commands.md and governance.md per
repo Rule 4.

APM CEO -- Triage Arbiter

This is a well-scoped, panel-pre-validated feature that fills a genuine enterprise gap.
The design decisions are correct and ready for implementation; no further design phase is
needed. Accepting at priority/high targeting 0.13.0. The security.md rewrite must
be non-negotiable in the first PR -- letting the security model drift from behavior is
the one unacceptable failure mode here.

{
  "decision": "accept",
  "decision_detail": "",
  "theme": "theme/governance",
  "areas": ["area/marketplace", "area/lockfile"],
  "type": "type/feature",
  "status": "status/accepted",
  "priority": "priority/high",
  "preserved_labels": [],
  "milestone": "0.13.0",
  "next_action": "Open a draft PR starting with steps 1-2 (schema + strict parser), carrying the security.md rewrite in the same branch.",
  "comment_markdown": "Thanks for this detailed proposal and the pre-validation with the APM panel. The marketplace upstreams feature is accepted.\n\nThe allow-list governance angle -- curators controlling which external plugins reach their consumers, pinned to immutable commits -- is exactly the kind of enterprise-facing capability on the roadmap. The design decisions folded into the body (type-level discriminated union, per-upstream-host auth, canonical-owner check, `allow_head: false` default, `marketplace.upstream.allow_unpinned_refs` governance gate) are all the right calls.\n\nA few implementation reminders before the first PR lands:\n\n- The `security.md` rewrite (step 12) must ride in the same PR as the builder changes, not a follow-up. The existing \"no intermediary registry, proxy, or mirror\" claim will be inaccurate the moment step 5 (MarketplaceBuilder integration) merges.\n- Per repo Rule 4, `packages/apm-guide/.apm/skills/apm-usage/` resources (commands.md for the new CLI surface, governance.md for the new policy key) must be updated in the same PR as the CLI changes.\n- The acknowledged v1 limitations (consumer-side custody, resilience to upstream force-push) should be stated explicitly in the updated security.md as \"out of scope for v1\" so the trust boundary is visible to enterprise evaluators.\n\nSuggested first PR scope: steps 1-2 (schema + strict parser), with the round-trip invariant test from step 11 as the acceptance gate. This gives reviewers a grounded diff before the resolver and CLI land."
}

---

Triage status: agentic proposal pending human ratification.
Silence is approval. Maintainers can:

• Override any label or milestone above by editing it directly --
  human edits are authoritative and will not be reverted on
  subsequent runs.
• Re-trigger triage by applying the status/needs-triage label, or
  by removing status/triaged to enroll the issue in the next
  daily sweep.

Posted by the Triage Panel workflow. See .apm/skills/apm-triage-panel for the panel skill.

Generated by Triage Panel for issue #1136 · ● 416.6K · ◷