-
Notifications
You must be signed in to change notification settings - Fork 2
Granular Exit Codes & CI Gate Modes #20
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
00af518
8db0dc8
30362dc
0483d31
8c9c539
81200cf
47d16b2
40aee7c
f2bf4cf
e342a69
4f2abe3
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -202,6 +202,185 @@ bash clawpinch.sh --config-dir /path/to/openclaw/config | |
|
|
||
| # Print auto-fix commands (read-only -- does not execute them) | ||
| bash clawpinch.sh --fix | ||
|
|
||
| # CI/CD gate modes -- fail only on critical findings | ||
| bash clawpinch.sh --severity-threshold critical | ||
|
|
||
| # Fail only on specific checks | ||
| bash clawpinch.sh --fail-on CHK-CFG-001,CHK-SEC-002 | ||
|
|
||
| # Combine threshold and specific checks | ||
| bash clawpinch.sh --severity-threshold warn --fail-on CHK-NET-005 | ||
| ``` | ||
|
|
||
| --- | ||
|
|
||
| ## Exit Codes | ||
|
|
||
| ClawPinch provides granular exit codes for CI/CD pipeline integration: | ||
|
|
||
| | Exit Code | Meaning | Description | | ||
| |-----------|---------|-------------| | ||
| | `0` | Success | No findings above the severity threshold | | ||
| | `1` | Critical | Critical findings detected | | ||
| | `2` | Warning | Warning-level findings detected (no critical) | | ||
| | `3` | Error | Scan error or incomplete execution | | ||
|
|
||
| ### Severity Threshold | ||
|
|
||
| Use `--severity-threshold` to control which findings trigger non-zero exit codes: | ||
|
|
||
| ```bash | ||
| # Fail only on critical findings (exit 1) -- warnings and info are ignored | ||
| bash clawpinch.sh --severity-threshold critical | ||
|
|
||
| # Fail on warnings or critical (exit 1 or 2) -- info findings are ignored | ||
| bash clawpinch.sh --severity-threshold warn | ||
|
|
||
| # Fail on any findings including info (default behavior) | ||
| bash clawpinch.sh --severity-threshold info | ||
| ``` | ||
|
Comment on lines
+240
to
+242
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Docs contradict default behavior The README says Prompt To Fix With AIThis is a comment left during a code review.
Path: README.md
Line: 240:242
Comment:
**Docs contradict default behavior**
The README says `--severity-threshold info` is “default behavior”, but `clawpinch.sh` defaults to backward-compatible behavior where warnings/info exit `0` unless `--severity-threshold` is explicitly provided. This will mislead CI users expecting non-zero on info by default. Either update the docs to match the implemented default, or change the code to make `info` the default threshold.
How can I resolve this? If you propose a fix, please make it concise. |
||
|
|
||
| **Use cases:** | ||
| - **Production deployments:** `--severity-threshold critical` blocks only on critical vulnerabilities | ||
| - **Staging environments:** `--severity-threshold warn` catches warnings before production | ||
| - **Development:** `--severity-threshold info` enforces all best practices | ||
|
|
||
| ### Fail on Specific Checks | ||
|
|
||
| Use `--fail-on` to enforce specific checks as mandatory regardless of severity threshold: | ||
|
|
||
| ```bash | ||
| # Always fail if auth is not required (even with --severity-threshold critical) | ||
| bash clawpinch.sh --severity-threshold critical --fail-on CHK-CFG-001 | ||
|
|
||
| # Fail on multiple specific checks (comma-separated) | ||
| bash clawpinch.sh --fail-on CHK-CFG-001,CHK-SEC-002,CHK-NET-005 | ||
| ``` | ||
|
|
||
| **Use cases:** | ||
| - Enforce organization-specific security policies | ||
| - Make specific checks mandatory for compliance | ||
| - Gradually tighten security gates over time | ||
|
|
||
| --- | ||
|
|
||
| ## Practical Examples | ||
|
|
||
| ### Example 1: Critical-Only Gate for Production | ||
|
|
||
| Block deployments only on critical vulnerabilities, allowing warnings to pass through: | ||
|
|
||
| ```bash | ||
| # Scan and fail only on critical findings | ||
| bash clawpinch.sh --severity-threshold critical --json | ||
|
|
||
| # Exit codes: | ||
| # 0 = no critical findings (deployment proceeds even with warnings) | ||
| # 1 = critical findings detected (deployment blocked) | ||
| # 3 = scan error (deployment blocked) | ||
| ``` | ||
|
|
||
| **Workflow:** | ||
| 1. Run scan with `--severity-threshold critical` | ||
| 2. If exit code = 0, deploy to production | ||
| 3. If exit code = 1, block deployment and alert security team | ||
| 4. Warnings/info findings are logged but don't block deployment | ||
|
|
||
| **Use case:** Production deployments where you want to move fast but block on serious vulnerabilities. | ||
|
|
||
| --- | ||
|
|
||
| ### Example 2: Enforce Specific Checks | ||
|
|
||
| Make specific security checks mandatory regardless of severity: | ||
|
|
||
| ```bash | ||
| # Always fail if auth is disabled or secrets are exposed | ||
| bash clawpinch.sh --fail-on CHK-CFG-001,CHK-SEC-003,CHK-SEC-004 | ||
|
|
||
| # Combine with severity threshold for layered security | ||
| bash clawpinch.sh --severity-threshold warn --fail-on CHK-CFG-001 | ||
| ``` | ||
|
|
||
| **Workflow:** | ||
| 1. Identify your organization's mandatory checks (e.g., auth, secrets, TLS) | ||
| 2. Add them to `--fail-on` in your CI pipeline | ||
| 3. Even if a check is downgraded to `info`, it still blocks deployment | ||
| 4. Gradually expand the mandatory check list over time | ||
|
|
||
| **Use case:** Enforce compliance requirements or organization-specific security policies. | ||
|
|
||
| --- | ||
|
|
||
| ### Example 3: Progressive Adoption Pattern | ||
|
|
||
| Start with loose gates and tighten over time to avoid disruption: | ||
|
|
||
| **Phase 1: Discovery (Week 1-2)** | ||
| ```bash | ||
| # Audit mode -- scan but don't fail builds | ||
| bash clawpinch.sh --json || true | ||
| ``` | ||
| Run scans in CI but ignore exit codes. Review findings and prioritize fixes. | ||
|
|
||
| **Phase 2: Critical-Only Gate (Week 3-4)** | ||
| ```bash | ||
| # Fail only on critical findings | ||
| bash clawpinch.sh --severity-threshold critical --json | ||
| ``` | ||
| Fix critical vulnerabilities. Warnings are visible but don't block. | ||
|
|
||
| **Phase 3: Add Mandatory Checks (Month 2)** | ||
| ```bash | ||
| # Critical gate + enforce specific checks | ||
| bash clawpinch.sh --severity-threshold critical --fail-on CHK-CFG-001,CHK-SEC-003 | ||
| ``` | ||
| Add organization-specific mandatory checks (auth, secrets, etc.). | ||
|
|
||
| **Phase 4: Tighten to Warnings (Month 3+)** | ||
| ```bash | ||
| # Fail on warnings or critical | ||
| bash clawpinch.sh --severity-threshold warn --json | ||
| ``` | ||
| Once critical/mandatory checks are clean, tighten to include warnings. | ||
|
|
||
| **Phase 5: Full Enforcement (Month 6+)** | ||
| ```bash | ||
| # Fail on any findings | ||
| bash clawpinch.sh --severity-threshold info --json | ||
| ``` | ||
| Enforce all best practices including informational findings. | ||
|
|
||
| **Use case:** Adopt ClawPinch without disrupting existing workflows. Progressive tightening builds security culture. | ||
|
|
||
| --- | ||
|
|
||
| ### CI/CD Integration Examples | ||
|
|
||
| **GitHub Actions:** | ||
| ```yaml | ||
| - name: Security audit | ||
| run: npx clawpinch --json --severity-threshold critical | ||
| # Fails workflow only on critical findings (exit 1) | ||
| ``` | ||
|
|
||
| **GitLab CI:** | ||
| ```yaml | ||
| security_audit: | ||
| script: | ||
| - npx clawpinch --severity-threshold warn --fail-on CHK-CFG-001 | ||
| # Fails on warnings or if auth check fails | ||
| ``` | ||
|
|
||
| **Jenkins:** | ||
| ```groovy | ||
| stage('Security Scan') { | ||
| steps { | ||
| sh 'npx clawpinch --json --severity-threshold critical' | ||
| // Pipeline fails only on critical findings | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| --- | ||
|
|
||
| Original file line number | Diff line number | Diff line change | ||||||
|---|---|---|---|---|---|---|---|---|
|
|
@@ -28,6 +28,8 @@ QUIET=0 | |||||||
| NO_INTERACTIVE=0 | ||||||||
| REMEDIATE=0 | ||||||||
| CONFIG_DIR="" | ||||||||
| SEVERITY_THRESHOLD="" | ||||||||
| FAIL_ON_CHECKS="" | ||||||||
|
|
||||||||
| # ─── Usage ─────────────────────────────────────────────────────────────────── | ||||||||
|
|
||||||||
|
|
@@ -36,18 +38,22 @@ usage() { | |||||||
| Usage: clawpinch [OPTIONS] | ||||||||
|
|
||||||||
| Options: | ||||||||
| --deep Run thorough / deep scans | ||||||||
| --json Output findings as JSON array only | ||||||||
| --fix Show auto-fix commands in report | ||||||||
| --quiet Print summary line only | ||||||||
| --no-interactive Disable interactive post-scan menu | ||||||||
| --remediate Run scan then pipe findings to Claude for AI remediation | ||||||||
| --config-dir PATH Explicit path to openclaw config directory | ||||||||
| -h, --help Show this help message | ||||||||
| --deep Run thorough / deep scans | ||||||||
| --json Output findings as JSON array only | ||||||||
| --fix Show auto-fix commands in report | ||||||||
| --quiet Print summary line only | ||||||||
| --no-interactive Disable interactive post-scan menu | ||||||||
| --remediate Run scan then pipe findings to Claude for AI remediation | ||||||||
| --config-dir PATH Explicit path to openclaw config directory | ||||||||
| --severity-threshold LEVEL Minimum severity to trigger non-zero exit (critical|warn|info|ok) | ||||||||
| --fail-on CHECK_IDS Comma-separated list of check IDs to fail on | ||||||||
| -h, --help Show this help message | ||||||||
|
|
||||||||
| Exit codes: | ||||||||
| 0 No critical findings | ||||||||
| 1 One or more critical findings detected | ||||||||
| 0 No findings above severity threshold (all checks passed) | ||||||||
| 1 Critical findings detected | ||||||||
| 2 Warning findings detected (no critical) | ||||||||
| 3 Scan error or incomplete | ||||||||
|
Comment on lines
+55
to
+56
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The description for exit code
Suggested change
|
||||||||
| EOF | ||||||||
| exit 0 | ||||||||
| } | ||||||||
|
|
@@ -68,6 +74,25 @@ while [[ $# -gt 0 ]]; do | |||||||
| exit 2 | ||||||||
| fi | ||||||||
| CONFIG_DIR="$2"; shift 2 ;; | ||||||||
| --severity-threshold) | ||||||||
| if [[ -z "${2:-}" ]]; then | ||||||||
| log_error "--severity-threshold requires a severity level argument" | ||||||||
| exit 2 | ||||||||
| fi | ||||||||
| case "$2" in | ||||||||
| critical|warn|info|ok) | ||||||||
| SEVERITY_THRESHOLD="$2" ;; | ||||||||
| *) | ||||||||
| log_error "--severity-threshold must be one of: critical, warn, info, ok" | ||||||||
| exit 2 ;; | ||||||||
| esac | ||||||||
| shift 2 ;; | ||||||||
| --fail-on) | ||||||||
| if [[ -z "${2:-}" ]]; then | ||||||||
| log_error "--fail-on requires a comma-separated list of check IDs" | ||||||||
| exit 2 | ||||||||
| fi | ||||||||
| FAIL_ON_CHECKS="$2"; shift 2 ;; | ||||||||
| -h|--help) usage ;; | ||||||||
| -v|--version) | ||||||||
| node -e "console.log('clawpinch v' + require('$CLAWPINCH_DIR/package.json').version)" 2>/dev/null \ | ||||||||
|
|
@@ -83,6 +108,8 @@ done | |||||||
| export CLAWPINCH_DEEP="$DEEP" | ||||||||
| export CLAWPINCH_SHOW_FIX="$SHOW_FIX" | ||||||||
| export CLAWPINCH_CONFIG_DIR="$CONFIG_DIR" | ||||||||
| export CLAWPINCH_SEVERITY_THRESHOLD="$SEVERITY_THRESHOLD" | ||||||||
| export CLAWPINCH_FAIL_ON_CHECKS="$FAIL_ON_CHECKS" | ||||||||
| export QUIET | ||||||||
|
|
||||||||
| # ─── Detect OS ─────────────────────────────────────────────────────────────── | ||||||||
|
|
@@ -144,6 +171,7 @@ ALL_FINDINGS="[]" | |||||||
| scanner_count=${#scanners[@]} | ||||||||
| scanner_idx=0 | ||||||||
| _SPINNER_PID="" | ||||||||
| SCAN_HAD_ERRORS=0 | ||||||||
|
|
||||||||
| # Record scan start time | ||||||||
| _scan_start="${EPOCHSECONDS:-$(date +%s)}" | ||||||||
|
|
@@ -181,6 +209,7 @@ for scanner in "${scanners[@]}"; do | |||||||
| stop_spinner "$local_name" 0 0 | ||||||||
| fi | ||||||||
| log_warn "Skipping $scanner_name (python not found)" | ||||||||
| SCAN_HAD_ERRORS=1 | ||||||||
| continue | ||||||||
| fi | ||||||||
| fi | ||||||||
|
|
@@ -195,6 +224,7 @@ for scanner in "${scanners[@]}"; do | |||||||
| ALL_FINDINGS="$(echo "$ALL_FINDINGS" "$output" | jq -s '.[0] + .[1]')" | ||||||||
| else | ||||||||
| log_warn "Scanner $scanner_name did not produce a valid JSON array" | ||||||||
| SCAN_HAD_ERRORS=1 | ||||||||
| fi | ||||||||
| fi | ||||||||
|
|
||||||||
|
|
@@ -236,6 +266,25 @@ count_warn="$(echo "$SORTED_FINDINGS" | jq '[.[] | select(.severity == "warn | |||||||
| count_info="$(echo "$SORTED_FINDINGS" | jq '[.[] | select(.severity == "info")] | length')" | ||||||||
| count_ok="$(echo "$SORTED_FINDINGS" | jq '[.[] | select(.severity == "ok")] | length')" | ||||||||
|
|
||||||||
| # ─── Check --fail-on enforcement ──────────────────────────────────────────── | ||||||||
|
|
||||||||
| if [[ -n "$FAIL_ON_CHECKS" ]]; then | ||||||||
| # Convert comma-separated list to jq array format | ||||||||
| fail_on_array="$(echo "$FAIL_ON_CHECKS" | jq -R 'split(",") | map(gsub("^\\s+|\\s+$";""))')" | ||||||||
|
|
||||||||
| # Count findings matching any of the specified check IDs | ||||||||
| count_fail_on="$(echo "$SORTED_FINDINGS" | jq --argjson ids "$fail_on_array" ' | ||||||||
| [.[] | select(.id as $id | $ids | any(. == $id))] | length | ||||||||
| ')" | ||||||||
|
|
||||||||
| if [[ "$count_fail_on" -gt 0 ]]; then | ||||||||
| if [[ "$JSON_OUTPUT" -eq 0 ]] && [[ "$QUIET" -eq 0 ]]; then | ||||||||
| log_error "Found $count_fail_on finding(s) matching --fail-on check IDs: $FAIL_ON_CHECKS" | ||||||||
| fi | ||||||||
| exit 1 | ||||||||
| fi | ||||||||
|
Comment on lines
+271
to
+285
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. --fail-on always exits 1
Prompt To Fix With AIThis is a comment left during a code review.
Path: clawpinch.sh
Line: 271:285
Comment:
**--fail-on always exits 1**
`--fail-on` currently forces `exit 1` for *any* matching check ID (even if the matching finding is only `warn`/`info`). Given the new exit code semantics, this collapses “explicitly failed checks” into the same signal as “critical findings”, so CI can’t distinguish between “critical” vs “policy-mandated check failed”. If the intent is to preserve granular meaning, `--fail-on` should map to an exit code consistent with the matched finding severities (or introduce a distinct exit code), otherwise the documented granularity becomes ambiguous for this mode.
How can I resolve this? If you propose a fix, please make it concise. |
||||||||
| fi | ||||||||
|
|
||||||||
| # ─── Output ────────────────────────────────────────────────────────────────── | ||||||||
|
|
||||||||
| if [[ "$JSON_OUTPUT" -eq 1 ]]; then | ||||||||
|
|
@@ -308,8 +357,44 @@ fi | |||||||
|
|
||||||||
| # ─── Exit code ─────────────────────────────────────────────────────────────── | ||||||||
|
|
||||||||
| # Exit 3 if scan had errors | ||||||||
| if (( SCAN_HAD_ERRORS > 0 )); then | ||||||||
| exit 3 | ||||||||
| fi | ||||||||
|
Comment on lines
+360
to
+363
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Scan errors hidden by JSON mode
Prompt To Fix With AIThis is a comment left during a code review.
Path: clawpinch.sh
Line: 360:363
Comment:
**Scan errors hidden by JSON mode**
`SCAN_HAD_ERRORS` is set when a python scanner is skipped or a scanner emits invalid JSON, but in `--json` mode those warnings are never printed. This means CI will just see exit code `3` with no diagnostic output, making failures hard to debug. Consider emitting an error message to stderr even under `--json`/`--quiet` when `SCAN_HAD_ERRORS` is set (or at least print which scanners were skipped/invalid).
How can I resolve this? If you propose a fix, please make it concise. |
||||||||
|
|
||||||||
| # Exit 1 if critical findings exist (always, regardless of threshold) | ||||||||
| if (( count_critical > 0 )); then | ||||||||
| exit 1 | ||||||||
| fi | ||||||||
|
|
||||||||
| # Apply severity threshold logic for non-critical findings | ||||||||
| if [[ -n "$SEVERITY_THRESHOLD" ]]; then | ||||||||
| # Exit 2 if we have warn findings and threshold is warn or lower | ||||||||
| if [[ "$SEVERITY_THRESHOLD" == "warn" || "$SEVERITY_THRESHOLD" == "info" || "$SEVERITY_THRESHOLD" == "ok" ]]; then | ||||||||
| if (( count_warn > 0 )); then | ||||||||
| exit 2 | ||||||||
| fi | ||||||||
| fi | ||||||||
|
|
||||||||
| # Exit 2 if we have info findings and threshold is info or lower | ||||||||
| if [[ "$SEVERITY_THRESHOLD" == "info" || "$SEVERITY_THRESHOLD" == "ok" ]]; then | ||||||||
| if (( count_info > 0 )); then | ||||||||
| exit 2 | ||||||||
| fi | ||||||||
| fi | ||||||||
|
|
||||||||
| # Exit 2 if we have ok findings and threshold is ok | ||||||||
| if [[ "$SEVERITY_THRESHOLD" == "ok" ]]; then | ||||||||
| if (( count_ok > 0 )); then | ||||||||
| exit 2 | ||||||||
| fi | ||||||||
| fi | ||||||||
| else | ||||||||
| # Default behavior when no threshold specified: maintain backward compatibility | ||||||||
| # Original behavior: only critical findings cause non-zero exit (exit 1 handled above) | ||||||||
| # Users who want warnings to fail must explicitly use --severity-threshold warn | ||||||||
| exit 0 | ||||||||
| fi | ||||||||
|
|
||||||||
| # No findings above threshold | ||||||||
| exit 0 | ||||||||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,26 @@ | ||
| { | ||
| "project_type": "single", | ||
| "services": { | ||
| "clawpinch": { | ||
| "path": ".", | ||
| "tech_stack": ["bash", "shell", "jq", "python3"], | ||
| "port": null, | ||
| "dev_command": "./clawpinch.sh", | ||
| "test_command": "bash scripts/helpers/test_e2e.sh" | ||
| } | ||
| }, | ||
| "infrastructure": { | ||
| "docker": false, | ||
| "database": null, | ||
| "ci_cd": "none" | ||
| }, | ||
| "conventions": { | ||
| "linter": null, | ||
| "formatter": null, | ||
| "testing": "bash test scripts", | ||
| "exit_codes": { | ||
| "current": "0=success, 1=critical findings, 2=errors", | ||
| "target": "0=clean, 1=critical, 2=warnings, 3=scan error" | ||
| } | ||
|
Comment on lines
+21
to
+24
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Exit code mapping outdated
Prompt To Fix With AIThis is a comment left during a code review.
Path: project_index.json
Line: 21:24
Comment:
**Exit code mapping outdated**
`conventions.exit_codes.current` still states `2=errors`, but the implementation and README define `2=warning-level findings` and `3=scan error`. This metadata will be incorrect for any tooling consuming `project_index.json` unless it’s updated to reflect the new semantics.
How can I resolve this? If you propose a fix, please make it concise. |
||
| } | ||
| } | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The exit code table in
README.mddescribes exit code2as "Warning-level findings detected (no critical)". This is inconsistent with theclawpinch.shimplementation, whereinfoandokfindings can also trigger exit code2if they are the highest severity found and meet the specified--severity-threshold. Please update this description to match the script's behavior for clarity.