Permission gap analyzer — find unprotected agent actions

[rodchalski]

Goal

Analyze agent codebases to find tool calls, API accesses, and actions that lack permission boundaries. This is the core scan engine.

The 20 Checks (v1)

Goal Integrity (GI)

1. No system prompt boundaries — Agent/prompt files with no role definition, task scope, or behavioral constraints
2. Unbounded tool access — Agent can call all tools without restriction list
3. No output validation — Agent responses not validated before delivery

Tool Safety (TS)

4. Raw database access — SQL/NoSQL queries without parameterization or access control
5. Unvalidated tool inputs — Tool functions that accept params without validation/sanitization
6. Shell/exec access — subprocess, exec, eval, or shell commands accessible to agent
7. Unrestricted file system — File read/write without path restrictions
8. HTTP calls without allowlist — Outbound API calls with no domain/URL restrictions

Data Leakage (DL)

9. PII in prompts/context — Hardcoded PII patterns (email, SSN, phone) in prompt templates
10. Shared memory between users — Memory/context stores without user isolation
11. Logging sensitive data — Agent responses or tool outputs logged without redaction
12. No output filtering — Missing PII/secret filtering on agent responses

Human Oversight (HO)

13. No approval gate — Tool calls that modify external state with no requireApproval or equivalent
14. No kill switch — No mechanism to halt agent execution
15. Autonomous external actions — Sends emails, makes payments, deploys code with no human check

Identity & Access (IA)

16. Hardcoded API keys — Secrets in source files, env examples with real values
17. No auth on agent endpoint — HTTP endpoint serving agent without authentication
18. Overprivileged service account — Cloud IAM roles broader than needed (heuristic)

MCP Specific (MCP)

19. Unpinned MCP server versions — npx without version pinning (supply chain risk)
20. MCP tool without description validation — Tools whose runtime description can differ from registered description (rug-pull vector)

Finding Interface

interface Finding {
  id: string;              // e.g. "PP-TS-004"
  domain: "goal-integrity" | "tool-safety" | "data-leakage" | "human-oversight" | "identity-access" | "mcp";
  severity: "critical" | "high" | "medium" | "low" | "info";
  title: string;
  description: string;
  file: string;
  line?: number;
  snippet?: string;        // The offending code (truncated)
  fix: string;             // Recommended fix
  ppFix?: string;          // How PP specifically solves this (e.g. "Wrap with requireApproval()")
  compliance: string[];    // e.g. ["OWASP:ASI01", "NIST:MAP-1.1"]
}

interface AnalysisResult {
  findings: Finding[];
  scores: Record<string, number>;  // domain -> 0-100 score
  overall: { score: number; grade: string };
  summary: {
    critical: number;
    high: number;
    medium: number;
    low: number;
    info: number;
  };
}

Key Design Principle

Every finding must include a ppFix — the specific PP SDK call or pattern that resolves it. This is the conversion funnel: scan shows the problem, PP is the solution.

Examples:

• "No approval gate on deploy tool" → ppFix: "Wrap with @requireApproval({ resource: \"production\", action: \"deploy\" })"
• "Raw database access" → ppFix: "Gate with pp.authorize({ action: \"query\", resource: \"database\" }) before execution"

Acceptance Criteria

• All 20 checks implemented
• Works against Python and TypeScript codebases
• Each finding includes compliance tags (from issue Compliance mapping data — OWASP/NIST/ISO tags for all findings #5 mapping data)
• Each finding includes ppFix recommendation
• Scoring algorithm: 100 base per domain, deduct by severity (CRIT: -20, HIGH: -12, MED: -6, LOW: -3)
• Overall grade: A (90+), B (80+), C (70+), D (60+), F (<60)
• Unit tests with sample agent code snippets

Depends On

• Issue Repo scaffold + CLI package setup #1 (scaffold)
• Issue Framework detector — identify agent frameworks in codebase #2 (framework detector — for entry point context)