Skip to content

Add modular agent skill with progressive disclosure for token-efficient security reviews#7

Open
SrFlipFlop wants to merge 2 commits intoArcanum-Sec:mainfrom
SrFlipFlop:modular-agent-skill
Open

Add modular agent skill with progressive disclosure for token-efficient security reviews#7
SrFlipFlop wants to merge 2 commits intoArcanum-Sec:mainfrom
SrFlipFlop:modular-agent-skill

Conversation

@SrFlipFlop
Copy link
Copy Markdown

Why?

From the moment I read about the project in an Executive Offense email earlier this year, I thought it was a very interesting idea, even though the context windows were more limited back then. After last week's Attacking AI training, I was working on a similar project at work to see how we could help developers introduce fewer issues into the codebase, and the sec-context project popped into my mind. From the start, I had planned to use only certain parts of the project, but after playing around with it a bit, I thought it might be useful to slightly modify the structure so that agents could automatically use whatever patterns they needed depending on the project.

What?

The monolithic files (ANTI_PATTERNS_BREADTH.md at ~65K tokens and ANTI_PATTERNS_DEPTH.md at ~100K tokens) are excellent references, but loading ~165K tokens for every task is expensive when an agent only needs, say, injection patterns for a SQL-heavy file. This PR adds a modular agent skill that restructures the same content into a progressive-disclosure format while keeping the originals untouched.

Architecture

SKILL.md                          ← Agent entry point (~174 lines, ~500 tokens)
references/
  {surface}-breadth.md            ← Concise BAD/GOOD patterns + CWE mappings
  {surface}-depth.md              ← Full examples, edge cases, common mistakes (6 surfaces)

10 security surfaces are covered with breadth files (concise patterns):
Secrets, Injection, XSS, Authentication, Cryptography, Input Validation, Config & Deployment, Dependencies, API Security, File Handling.

6 of those also have depth files (the top-priority surfaces from the original DEPTH document):
Secrets, Injection, XSS, Authentication, Cryptography, Input Validation.

All content was extracted verbatim from the source files — reorganized, not rewritten. Each reference file includes a CC BY 4.0 attribution header pointing back to the originals.

Routing

SKILL.md acts as the router. It contains:

  1. Routing table — Maps trigger keywords (e.g., SQL, query, execute) to specific reference files, so agents know which surfaces to load.
  2. Quick Reference Table — CWE-to-surface mapping for fast lookup.
  3. Security Surface Identification Checklist — A safety net that catches surfaces keyword matching might miss.
  4. Workflow — Step-by-step instructions: identify surfaces → load breadth → escalate to depth if needed → produce structured output.
  5. "Load at most 2-3 surfaces per task" guideline to prevent agents from over-loading all references.

An agent reading SKILL.md can identify the relevant surfaces for any given code file and load only the matching references. Token cost drops from ~165K to ~2-8K per task depending on how many surfaces are relevant.

What's changed

  • Added: SKILL.md (agent skill entry point with router)
  • Added: references/ directory with 16 modular files (10 breadth + 6 depth)
  • Added: examples/mockups/ with 5 deliberately vulnerable test files
  • Added: examples/results/ with security reviews produced by the skill
  • Updated: README.md with Option 5 documentation and testing results
  • Unchanged: ANTI_PATTERNS_BREADTH.md and ANTI_PATTERNS_DEPTH.md (fully backward compatible)

Tests

To validate the skill, I created 5 deliberately vulnerable mockup files spanning different languages and security surfaces, then ran each through the modular skill using OpenCode (Claude Sonnet) as the reviewing agent.

The 5 Mockups

# File Language What's planted
01 01-user-search-api.py Python/Flask SQL injection via f-strings, no auth, debug mode, SSN/salary exposure
02 02-auth-service.js Node.js/Express Hardcoded secrets, MD5 hashing, Math.random() tokens, timing-vulnerable comparison
03 03-profile-page.tsx React/TSX dangerouslySetInnerHTML, client-side admin gate, IDOR, open redirect
04 04-file-upload-server.go Go Path traversal, unrestricted uploads, predictable temp files, world-writable perms, debug endpoints
05 05-infra-deploy.dockerfile Dockerfile+Bash+JSON Unpinned base images, USER root, curl|bash, typosquattable packages, hardcoded tokens

How they were tested

Each mockup was reviewed with a prompt like: "Using the sec-context skill, perform a security review of examples/mockups/01-user-search-api.py". The agent loaded SKILL.md, identified relevant surfaces, loaded matching reference files, and produced a structured review.

Results

Test Findings Pattern Match Surfaces Loaded Depth Used Rating
01 (Python API) 7 100% 4 breadth No 4.5/5
02 (Auth service) 10 100% 3 breadth No 4.5/5
03 (React TSX) 6 100% 3 breadth No 4.4/5
04 (Go file server) 12 100% 3 breadth + 1 depth Yes 8.5/10
05 (Dockerfile/infra) 18 ~95% 3 breadth No 4/5

53 total findings across 5 reviews. Zero irrelevant surfaces loaded.

What worked

  • Routing accuracy — Trigger keywords correctly identified relevant surfaces in every test. No false loads.
  • BAD/GOOD pattern pairs — Every vulnerability found had a near-exact match to a BAD example. This was the most valuable feature.
  • Progressive disclosure — Breadth-only was sufficient in 4/5 tests. Only test 04 escalated to a depth file.
  • CWE mapping — Pre-mapped CWEs eliminated lookup time and ensured accurate classification.
  • Token efficiency — 3-4 files loaded per review (~1,200-3,500 lines) vs ~14,000+ lines for the monolithic approach.

Known gaps / improvement ideas

  • No language-specific guidance — Pseudocode patterns require manual translation. Language-specific function mappings would speed up remediation.
  • "2-3 surfaces" guideline slightly restrictive — Tests 01 and 04 needed 4 surfaces.
  • Missing container/Dockerfile patterns — No dedicated surface for USER root, --privileged, unpinned base images, mutable tags.
  • Missing postinstall/lifecycle script pattern — Dependencies breadth covers typosquatting but not npm lifecycle script RCE.
  • Absence-of-code not a trigger — When auth code is entirely missing, none of the auth trigger keywords match.
  • No composite vulnerability scoring — No guidance on how compound vulnerabilities escalate aggregate risk.
  • Cross-surface overlaps not signposted — e.g., rate limiting appears in both API Security and Input Validation.
  • CWE-601 (Open Redirect) missing from the Quick Reference Table.

Restructure the two monolithic anti-pattern files (~165K tokens combined)
into a modular, agent-friendly skill format that achieves 20-100x token
cost reduction through progressive disclosure.

- Add SKILL.md as the skill entry point and router
- Add references/ with 16 modular files (10 breadth + 6 depth)
  covering all 10 security surfaces and 25+ vulnerability classes
- Update README.md with Option 5 documenting the new skill format

Original monolithic files are preserved unchanged for backward
compatibility. Content in references/ is extracted verbatim from the
source files, reorganized by security surface.
Add 5 deliberately vulnerable mockups (Python, JS, TSX, Go, Dockerfile)
and their security review results demonstrating the modular skill's
routing accuracy, progressive disclosure, and token efficiency.

Document findings in README: 53 vulnerabilities found across 5 reviews,
100% pattern match rate, zero false surfaces loaded, along with known
gaps for future improvement.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant