Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 31 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,7 @@ The scanner evaluates only the surfaces a plugin actually exposes, then normaliz
| Security | 24 | `SECURITY.md`, `LICENSE`, hardcoded secret detection, dangerous MCP commands, MCP transport hardening, risky approval defaults |
| Operational Security | 20 | SHA-pinned GitHub Actions, `write-all`, privileged untrusted checkout patterns, Dependabot, dependency lockfiles |
| Best Practices | 15 | `README.md`, skills directory, `SKILL.md` frontmatter, committed `.env`, `.codexignore` |
| Marketplace | 15 | `marketplace.json` validity, policy fields, safe source paths |
| Marketplace | 15 | `.agents/plugins/marketplace.json` validity, legacy `marketplace.json` compatibility, policy fields, safe source paths |
| Skill Security | 15 | Cisco integration status, elevated skill findings, analyzability |
| Code Quality | 10 | `eval`, `new Function`, shell-injection patterns |

Expand Down Expand Up @@ -125,6 +125,7 @@ codex-plugin-scanner lint ./my-plugin --fix --profile strict-security

# Runtime readiness verification
codex-plugin-scanner verify ./my-plugin --format json
codex-plugin-scanner verify ./my-plugin --online --format text

# Artifact-backed submission gate
codex-plugin-scanner submit ./my-plugin --profile public-marketplace --attest dist/plugin-quality.json
Expand All @@ -133,6 +134,18 @@ codex-plugin-scanner submit ./my-plugin --profile public-marketplace --attest di
codex-plugin-scanner doctor ./my-plugin --component mcp --bundle dist/doctor.zip
```

## Codex Spec Alignment

The scanner follows the current Codex plugin packaging conventions more closely:

- local manifest paths should use `./` prefixes
- `.agents/plugins/marketplace.json` is the preferred marketplace manifest location
- root `marketplace.json` is still supported in compatibility mode
- `interface` metadata no longer requires an undocumented `type` field
- `verify` performs an MCP initialize handshake before probing declared capabilities

`lint --fix` preserves or adds the documented `./` prefixes instead of stripping them away.

## Config + Baseline Example

```toml
Expand Down Expand Up @@ -217,12 +230,24 @@ The scanner currently detects or validates:
Add the scanner to a plugin repository CI job:

```yaml
- name: Install scanner
run: pip install codex-plugin-scanner
permissions:
contents: read
security-events: write

- name: Scan plugin
run: codex-plugin-scanner ./my-plugin --fail-on-severity high --format sarif --output codex-plugin-scanner.sarif
continue-on-error: true
jobs:
scan-plugin:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
- uses: hashgraph-online/hol-codex-plugin-scanner-action@v1
with:
plugin_dir: "."
mode: scan
profile: public-marketplace
min_score: 80
fail_on_severity: high
format: sarif
upload_sarif: true
```

Local pre-commit style hook:
Expand Down
31 changes: 24 additions & 7 deletions action/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,12 +20,15 @@ This README is intentionally root-ready for a dedicated GitHub Marketplace actio
| Input | Description | Default |
|-------|-------------|---------|
| `plugin_dir` | Path to the plugin directory to scan | `.` |
| `mode` | Execution mode: `scan`, `lint`, `verify`, or `submit` | `scan` |
| `format` | Output format: `text`, `json`, `markdown`, `sarif` | `text` |
| `output` | Write report to this file path | `""` |
| `profile` | Policy profile: `default`, `public-marketplace`, or `strict-security` | `default` |
| `config` | Optional path to `.codex-plugin-scanner.toml` | `""` |
| `baseline` | Optional path to a baseline suppression file | `""` |
| `online` | Enable live network probing for `verify` mode | `false` |
| `upload_sarif` | Upload the generated SARIF report to GitHub code scanning when `mode: scan` | `false` |
| `sarif_category` | SARIF category used during GitHub code scanning upload | `codex-plugin-scanner` |
| `write_step_summary` | Write a concise markdown summary to the GitHub Actions job summary | `true` |
| `registry_payload_output` | Write a machine-readable Codex ecosystem payload JSON file for registry or awesome-list automation | `""` |
| `min_score` | Fail if score is below this threshold (0-100) | `0` |
Expand All @@ -51,6 +54,8 @@ This README is intentionally root-ready for a dedicated GitHub Marketplace actio
| `score` | Numeric score (0-100) |
| `grade` | Letter grade (A-F) |
| `grade_label` | Human-readable grade label |
| `policy_pass` | `true` when the selected policy profile passed |
| `verify_pass` | `true` when runtime verification passed |
| `max_severity` | Highest finding severity, or `none` |
| `findings_total` | Total number of findings across all severities |
| `report_path` | Path to the rendered report file, if `output` was set |
Expand Down Expand Up @@ -82,12 +87,22 @@ Mode notes:
### SARIF output for GitHub Code Scanning

```yaml
- uses: your-org/hol-codex-plugin-scanner-action@v1
with:
plugin_dir: "."
format: sarif
output: codex-plugin-scanner.sarif
fail_on_severity: high
permissions:
contents: read
security-events: write

jobs:
scan:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
- uses: your-org/hol-codex-plugin-scanner-action@v1
with:
plugin_dir: "."
mode: scan
format: sarif
fail_on_severity: high
upload_sarif: true
```

### With Cisco skill scanning
Expand All @@ -109,7 +124,7 @@ Mode notes:
with:
plugin_dir: "."
format: sarif
output: codex-plugin-scanner.sarif
upload_sarif: true
registry_payload_output: codex-plugin-registry-payload.json

- name: Show trust signals
Expand Down Expand Up @@ -205,3 +220,5 @@ Set `mode` to one of `scan`, `lint`, `verify`, or `submit`.
```

For `submit` mode, use `registry_payload_output` to control artifact path.

For `scan` mode, set `upload_sarif: true` to emit and upload SARIF automatically instead of wiring a separate upload step by hand.
22 changes: 22 additions & 0 deletions action/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,14 @@ inputs:
description: "Enable live network probing for verify mode"
required: false
default: "false"
upload_sarif:
description: "Upload the generated SARIF report to GitHub code scanning. Requires security-events: write in the calling workflow."
required: false
default: "false"
sarif_category:
description: "SARIF category used when upload_sarif is enabled"
required: false
default: "codex-plugin-scanner"
write_step_summary:
description: "Write a concise markdown summary to the GitHub Actions job summary"
required: false
Expand Down Expand Up @@ -114,6 +122,12 @@ outputs:
grade_label:
description: "The human-readable grade label"
value: ${{ steps.scan.outputs.grade_label }}
policy_pass:
description: "Whether the selected policy profile passed"
value: ${{ steps.scan.outputs.policy_pass }}
verify_pass:
description: "Whether runtime verification passed"
value: ${{ steps.scan.outputs.verify_pass }}
max_severity:
description: "The most severe finding in the scan result, or none"
value: ${{ steps.scan.outputs.max_severity }}
Expand Down Expand Up @@ -184,6 +198,7 @@ runs:
CONFIG: ${{ inputs.config }}
BASELINE: ${{ inputs.baseline }}
ONLINE: ${{ inputs.online }}
UPLOAD_SARIF: ${{ inputs.upload_sarif }}
WRITE_STEP_SUMMARY: ${{ inputs.write_step_summary }}
REGISTRY_PAYLOAD_OUTPUT: ${{ inputs.registry_payload_output }}
MIN_SCORE: ${{ inputs.min_score }}
Expand All @@ -202,3 +217,10 @@ runs:
SUBMISSION_AUTHOR: ${{ inputs.submission_author }}
GITHUB_STEP_SUMMARY: ${{ env.GITHUB_STEP_SUMMARY }}
run: python3 -m codex_plugin_scanner.action_runner

- name: Upload SARIF
if: ${{ inputs.upload_sarif == 'true' && inputs.mode == 'scan' && steps.scan.outputs.report_path != '' }}
uses: github/codeql-action/upload-sarif@51f77329afa6477de8c49fc9c7046c15b9a4e79d
with:
sarif_file: ${{ steps.scan.outputs.report_path }}
category: ${{ inputs.sarif_category }}
86 changes: 86 additions & 0 deletions docs/codex-spec-alignment-todo.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
# Codex Spec Alignment Todo

## Implementation workstreams

### Marketplace model and compatibility

- Add a shared marketplace loader that resolves the preferred path `.agents/plugins/marketplace.json` and a deprecated fallback `marketplace.json`.
- Normalize marketplace context around two roots:
- repository root
- marketplace file parent directory
- Replace string-only `plugin.source` handling with support for `source.source` and `source.path`.
- Add validation helpers for:
- required `plugins[].policy.installation`
- required `plugins[].policy.authentication`
- required `plugins[].category`
- optional marketplace `interface.displayName`
- `source.path` `./` prefix
- `source.path` staying within the marketplace root
- Keep legacy root marketplace parsing but mark it as compatibility mode in messages/findings.

### Manifest and autofix alignment

- Remove `interface.type` from the required interface metadata set.
- Centralize path normalization logic so manifest and marketplace checks share the same `./`-prefixed relative-path policy.
- Update autofix behavior to:
- preserve existing valid `./` prefixes
- add `./` for eligible local paths in plugin and marketplace JSON
- avoid mutating remote URLs or non-path string fields

### MCP verification lifecycle

- Introduce a small JSON-RPC transport helper for newline-delimited stdio MCP sessions.
- Send `initialize` with:
- `protocolVersion`
- `capabilities`
- `clientInfo`
- Parse the initialize result and, when successful, send `notifications/initialized`.
- Probe optional capabilities with:
- `tools/list`
- `resources/list`
- `prompts/list`
- Record requests and responses in runtime traces for `doctor`.
- Preserve strict timeouts and guaranteed subprocess cleanup.

### Action ergonomics

- Extend `action/action.yml` inputs with:
- `upload_sarif`
- `sarif_category`
- Extend Action outputs with:
- `policy_pass`
- `verify_pass`
- Update `action_runner.py` to emit those outputs and write a default SARIF path when upload is requested.
- Add a conditional `github/codeql-action/upload-sarif` step pinned by SHA.

### Documentation and fixtures

- Update README examples for:
- `.agents/plugins/marketplace.json`
- `lint --fix`
- SARIF upload usage
- Add or update fixtures for:
- valid Codex marketplace repo layout
- legacy marketplace compatibility
- MCP stdio stub server handshake

## Test plan

- `tests/test_marketplace.py`
- preferred marketplace path
- legacy fallback
- `source.path` prefix and containment
- required category and policy fields
- `tests/test_manifest.py`
- interface metadata passes without `type`
- `tests/test_verification.py`
- MCP initialize + initialized
- capability enumeration traces
- doctor bundle includes real lifecycle traces
- `tests/test_cli.py`
- `lint --fix` preserves or adds `./`
- `tests/test_action_runner.py`
- SARIF upload path preparation
- `policy_pass` and `verify_pass` outputs
- `tests/test_action_bundle.py`
- Action metadata and new upload inputs
102 changes: 102 additions & 0 deletions docs/prd-v2-codex-spec-alignment.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,102 @@
# PRD: v2 Codex Spec Alignment and Runtime Hardening

## Summary

`codex-plugin-scanner` should become the default readiness gate for Codex plugins by aligning its validation and verification logic to the current Codex plugin and marketplace conventions, hardening MCP runtime verification, and reducing GitHub Action adoption friction.

This release is intentionally scoped to the highest-confidence gaps identified in [code-scanner-research.md](/Users/michaelkantor/CascadeProjects/hashgraph-online/code-scanner-research.md):

- first-class support for Codex repo marketplaces at `.agents/plugins/marketplace.json`
- path semantics that preserve documented `./`-prefixed relative paths instead of stripping them
- manifest validation that stops requiring undocumented `interface.type`
- MCP stdio verification that performs a lifecycle-compliant initialize flow
- GitHub Action ergonomics for SARIF upload and policy/verification outputs

## Problem

The scanner already has strong policy, scoring, suppression, and CI foundations, but it still diverges from the current Codex packaging contract in a few critical ways:

- marketplace validation assumes a legacy root `marketplace.json` shape instead of the Codex repo marketplace path and schema
- safe autofix rewrites documented `./` path prefixes away
- runtime MCP verification sends an empty `initialize` payload and does not send `notifications/initialized`
- Action adoption still requires manual SARIF wiring even though SARIF is a first-class output

Those gaps create false negatives, false positives, and trust issues for plugin authors who are following the docs correctly.

## Goals

- Validate Codex marketplaces at `.agents/plugins/marketplace.json` using the documented `plugins[].source.path` object shape.
- Keep legacy root `marketplace.json` support only as a compatibility fallback with an explicit deprecation signal.
- Preserve and autofix documented `./`-prefixed relative paths for manifest and marketplace references.
- Perform a protocol-grade MCP stdio initialize flow and capture richer traces for `doctor`.
- Let plugin authors opt into SARIF upload directly from the scanner Action with least-privilege guidance.

## Non-goals

- Replacing `$plugin-creator`
- Adding network-on-by-default verification
- Reworking the existing score model or registry artifact schemas beyond what is required for this spec-alignment release
- Building a generic MCP remote inspector beyond safe reachability and stdio lifecycle verification

## Users

- Codex plugin authors shipping repository-local plugins
- Teams maintaining repo-local marketplaces of plugins
- Registry maintainers consuming scanner artifacts and SARIF

## Scope

### 1. Marketplace spec alignment

- Default marketplace location becomes `.agents/plugins/marketplace.json`.
- Validation accepts:
- `name: string`
- optional `interface.displayName`
- `plugins: []`
- each plugin entry with:
- `source: { source: string, path: string }`
- `policy.installation`
- `policy.authentication`
- `category`
- `source.path` must:
- start with `./`
- resolve inside the marketplace root
- legacy root `marketplace.json` remains supported in v2 with a compatibility warning path in validation/verification/docs.

### 2. Manifest and path semantics

- `interface.type` is no longer required for publishability.
- interface asset paths and manifest-declared local paths must preserve `./` prefixes where Codex expects them.
- autofix upgrades eligible local paths to documented `./` form instead of stripping prefixes.

### 3. MCP verification hardening

- stdio MCP verification sends:
- `initialize` with `protocolVersion`, `capabilities`, and `clientInfo`
- `notifications/initialized`
- optional capability probes for `tools/list`, `resources/list`, and `prompts/list` when declared by the server
- traces recorded for `doctor` include the full request/response sequence and timeout classification.

### 4. Action ergonomics

- composite Action adds optional SARIF upload support:
- `upload_sarif`
- `sarif_category`
- Action outputs explicitly include `policy_pass` and `verify_pass`.
- docs provide the required `security-events: write` guidance for SARIF upload.

## Acceptance criteria

- A plugin repo with `.agents/plugins/marketplace.json` and `./`-prefixed `source.path` passes marketplace checks.
- `lint --fix` never strips a valid `./` prefix from manifest or marketplace paths.
- a manifest with a publishable `interface` object but no `interface.type` passes interface metadata checks.
- MCP stdio verification records a successful initialize + initialized exchange against a compliant stub server.
- `doctor --bundle` contains real stdio trace output for the MCP lifecycle exchange.
- the GitHub Action can scan in SARIF mode and optionally upload the generated SARIF when permissions are present.

## Verification plan

- unit tests for marketplace parsing, path normalization, manifest interface metadata, MCP lifecycle traces, and Action runner outputs
- CLI tests covering `lint --fix`, `verify`, and `doctor --bundle`
- targeted Action tests for `upload_sarif`, `policy_pass`, and `verify_pass`
- full `pytest`, `ruff check`, `ruff format --check`, and `python -m build`
Loading
Loading