diff --git a/docs/getting-started/quick-start.md b/docs/getting-started/quick-start.md index 584e235..b555f4e 100644 --- a/docs/getting-started/quick-start.md +++ b/docs/getting-started/quick-start.md @@ -30,10 +30,10 @@ Create and iteratively refine custom topic guardrails using an LLM-driven feedba ```bash # Interactive — prompts for all inputs -daystrom generate +daystrom runtime topics generate # Non-interactive -daystrom generate \ +daystrom runtime topics generate \ --profile my-security-profile \ --topic "Block discussions about building explosives" \ --intent block \ @@ -97,10 +97,10 @@ Evaluate all topics in a security profile at once, with conflict detection. ```bash # Terminal output -daystrom audit my-security-profile +daystrom runtime profiles audit my-security-profile # HTML report -daystrom audit my-security-profile --format html --output audit-report.html +daystrom runtime profiles audit my-security-profile --format html --output audit-report.html ``` [Full audit docs](../features/profile-audits.md) @@ -111,11 +111,11 @@ daystrom audit my-security-profile --format html --output audit-report.html ```bash # Resume a paused or failed guardrail run -daystrom resume +daystrom runtime topics resume # View a run report -daystrom report +daystrom runtime topics report # List all saved runs -daystrom list +daystrom runtime topics runs ``` diff --git a/docs/index.md b/docs/index.md index 7205c99..2c8dcd5 100644 --- a/docs/index.md +++ b/docs/index.md @@ -19,94 +19,89 @@ title: Home --- -Daystrom is a CLI tool that provides full operational coverage over **Palo Alto Prisma AIRS** AI security capabilities. Generate and iteratively refine custom topic guardrails with LLM-driven feedback loops, run adversarial red team scans against AI targets, manage ML model supply chain security, and audit entire security profiles for coverage gaps and cross-topic conflicts. Named after Star Trek's Dr. Richard Daystrom, it automates the tedious so you can focus on intent. +Daystrom is a CLI tool that provides full operational coverage over **Palo Alto Prisma AIRS** AI security capabilities — runtime prompt scanning and configuration management, LLM-driven guardrail generation, adversarial AI red teaming, ML model supply chain security, and multi-topic profile audits with conflict detection. Six LLM provider configurations are supported out of the box. -
+--- -- :material-refresh-auto:{ .lg .middle } **Iterative Refinement** +## Capabilities - --- - - Analyzes false positives and negatives after each iteration, feeding structured feedback to the LLM until coverage meets your threshold. +
-- :material-brain:{ .lg .middle } **Multi-Provider LLM** +- :material-shield-search:{ .lg .middle } **Runtime Security** --- - Six provider configs out of the box — Claude API, Claude Vertex, Claude Bedrock, Gemini API, Gemini Vertex, and Gemini Bedrock. + Scan prompts against live AIRS security profiles (sync and async), and manage runtime configuration — profiles, topics, API keys, customer apps, deployment/DLP profiles, and scan logs. -- :material-memory:{ .lg .middle } **Cross-Run Memory** + [:octicons-arrow-right-24: Runtime Security](features/runtime-security.md) + +- :material-refresh-auto:{ .lg .middle } **Guardrail Generation** --- - Persists learnings across runs so the LLM avoids repeating past mistakes. Budget-aware injection keeps prompts focused. + LLM-driven feedback loop that generates, deploys, tests, evaluates, and iteratively refines custom topic guardrails until coverage meets your target threshold. -- :material-play-pause:{ .lg .middle } **Resumable Runs** + [:octicons-arrow-right-24: Guardrail Generation](features/guardrail-generation.md) + +- :material-sword:{ .lg .middle } **AI Red Teaming** --- - Every iteration checkpoints to disk. Resume failed or paused runs from exactly where they left off — no wasted API calls. + Launch static, dynamic, and custom adversarial scans against AI targets. Full CRUD for targets, prompt sets, and individual prompts with attack category filtering. -- :material-shield-check:{ .lg .middle } **Block & Allow Intent** + [:octicons-arrow-right-24: AI Red Teaming](features/red-team.md) + +- :material-shield-lock:{ .lg .middle } **Model Security** --- - First-class support for both block (blacklist) and allow (whitelist) guardrails with intent-aware test generation and analysis. + ML model supply chain security — manage security groups, browse rules, configure rule instances, create scans, and review evaluations, violations, and file results. + + [:octicons-arrow-right-24: Model Security](features/model-security.md) -- :material-test-tube:{ .lg .middle } **Test Accumulation** +- :material-clipboard-check:{ .lg .middle } **Profile Audits** --- - Optionally carry forward test prompts across iterations with dedup, catching regressions that fresh tests might miss. + Evaluate all topics in a security profile at once. Per-topic and composite metrics, cross-topic conflict detection, with terminal, JSON, and HTML output formats. -- :material-shield-search:{ .lg .middle } **Runtime Security** + [:octicons-arrow-right-24: Profile Audits](features/profile-audits.md) - --- +
- Scan prompts against live security profiles and manage AIRS configuration — profiles, topics, API keys, customer apps, and scan logs via `daystrom runtime`. +--- - [:octicons-arrow-right-24: Runtime Security](features/runtime-security.md) +## Platform Features -- :material-sword:{ .lg .middle } **AI Red Teaming** +
+ +- :material-brain:{ .lg .middle } **Multi-Provider LLM** --- - Launch static, dynamic, and custom adversarial scans against AI targets. Full CRUD on targets, prompt sets, and prompts via `daystrom redteam`. + Six provider configs — Claude API, Claude Vertex, Claude Bedrock, Gemini API, Gemini Vertex, Gemini Bedrock. - [:octicons-arrow-right-24: Red Team](features/red-team.md) + [:octicons-arrow-right-24: LLM Providers](providers/overview.md) -- :material-clipboard-check:{ .lg .middle } **Profile Audits** +- :material-memory:{ .lg .middle } **Cross-Run Memory** --- - Evaluate all topics in a security profile at once. Per-topic metrics, composite scores, and cross-topic conflict detection via `daystrom audit`. + Persists learnings across guardrail runs so the LLM avoids repeating past mistakes. Budget-aware injection keeps prompts focused. -- :material-shield-lock:{ .lg .middle } **Model Security** +- :material-play-pause:{ .lg .middle } **Resumable Runs** --- - Manage ML model supply chain security — security groups, rules, scans, evaluations, violations, and labels via `daystrom model-security`. + Every guardrail iteration checkpoints to disk. Resume failed or paused runs from exactly where they left off. - [:octicons-arrow-right-24: Model Security](features/model-security.md) +- :material-test-tube:{ .lg .middle } **Test Composition** -
+ --- ---- + Carried failures, regression tiers, and fresh LLM-generated tests are composed each iteration with weighted category generation. -## How It Works - -```mermaid -flowchart LR - A["Describe\nwhat to block\nor allow"] --> B["LLM generates\ntopic definition"] - B --> C["Deploy\nto AIRS"] - C --> D["Generate\ntest prompts"] - D --> E["Scan against\nlive service"] - E --> F["Evaluate\nTPR · TNR · F1"] - F --> G{Coverage\nmet?} - G -->|No| H["Analyze\nFP / FN"] - H --> B - G -->|Yes| I["Done"] -``` +
--- @@ -142,7 +137,7 @@ flowchart LR --- - Core loop, AIRS integration, memory system, and design decisions. + System overview, core loop, memory system, and design decisions. [:octicons-arrow-right-24: Architecture](architecture/overview.md) diff --git a/mkdocs.yml b/mkdocs.yml index d483d3e..9bb927b 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -73,19 +73,15 @@ nav: - Configuration: getting-started/configuration.md - Quick Start: getting-started/quick-start.md - Capabilities: - - Guardrail Generation: - - Overview: features/guardrail-generation.md - - Core Loop: architecture/core-loop.md - - Memory System: features/memory-system.md - - Metrics & Evaluation: features/metrics.md - - Topic Constraints: features/topic-constraints.md - - Resumable Runs: features/resumable-runs.md - Runtime Security: features/runtime-security.md + - Guardrail Generation: features/guardrail-generation.md - AI Red Teaming: features/red-team.md - Model Security: features/model-security.md - Profile Audits: features/profile-audits.md - Architecture: - Overview: architecture/overview.md + - Core Loop: architecture/core-loop.md + - Memory System: features/memory-system.md - Design Decisions: architecture/design-decisions.md - LLM Providers: - Overview: providers/overview.md @@ -98,7 +94,10 @@ nav: - Configuration Options: reference/configuration.md - Environment Variables: reference/environment-variables.md - AIRS Constraints: reference/airs-constraints.md - - Examples: + - Metrics & Evaluation: features/metrics.md + - Topic Constraints: features/topic-constraints.md + - Resumable Runs: features/resumable-runs.md + - Guides: - Guardrail to Red Team: examples/guardrail-to-redteam.md - Managing Targets: examples/managing-targets.md - Managing Prompt Sets: examples/managing-prompt-sets.md