Implement production DSPy skills registry with JSON Schema validation, JSONL evals, and AgenticFleet taxonomy

[Copilot]

Builds a curated registry for versioned DSPy Module/Signature skills with strict contracts, safety permissions, and golden evaluations. Enables skill discovery via semantic tags and deterministic catalog generation.

Core Infrastructure

packages/skills_core/ - Validation and catalog library:

• types.py: Pydantic models enforcing JSON Schema contract (SkillMetadata, SafetyLevel, Permissions)
• loader.py: Auto-discovery from skill.yaml files with validation
• validator.py: JSON Schema + structural validation + DSPy import verification
• catalog.py: Deterministic catalog generation with stable sort order
• dspy_contract.py: Reflection utilities for DSPy Module/Signature verification
• evals.py: JSONL golden evaluation framework (3 match types: exact, contains, schema-valid)

CLI Tools

tools/validate.py - Validates all skills, regenerates catalog/skills.json, fails if diff detected
tools/new_skill.py - Scaffolds complete skill structure with boilerplate
tools/run_eval.py - Runs golden evals in dry-run (format check) or execution mode

Schemas

catalog/schema.skill.json - Authoritative JSON Schema for skill.yaml (validates IDs, versions, DSPy paths, permissions, safety)
catalog/schema.catalog.json - Schema for generated catalog

Skill Contract

Each skill requires:

skills/{skill_id}/
├── skill.yaml              # Metadata (validated against JSON Schema)
├── src/skill.py            # DSPy Module + Signature + SIGNATURES export
├── tests/test_contract.py  # Contract validation tests
├── eval/golden.jsonl       # Golden examples with match types
└── examples/minimal.py     # Runnable example

AgenticFleet Tag Taxonomy

6 categories for semantic routing:

• Core reasoning: reasoning, planning, decision, decomposition, reflection
• Memory/knowledge: memory, retrieval, knowledge, embedding
• Execution: orchestration, tool_use, workflow, routing
• IO/transform: transform, summarization, extraction, classification
• Safety: safety, validation, moderation
• System: agent, skill, planner, meta

Safety Framework

Three-tier classification with explicit permissions:

safety:
  level: "low|medium|high"
  risks: ["..."]
  mitigations: ["..."]
permissions:
  network: false
  filesystem_read: false
  filesystem_write: false
  external_tools: []

Golden Evaluations

JSONL format with three match types:

{"name": "test1", "input": {...}, "expected": {...}, "match": "exact_match"}
{"name": "test2", "input": {...}, "expected": {...}, "match": "contains"}
{"name": "test3", "input": {...}, "expected": {"type": "object", ...}, "match": "json_schema_valid"}

Usage

# Create skill
python tools/new_skill.py sentiment_analyzer --description "..." --tags nlp classification

# Validate all
python tools/validate.py

# Run evals
python tools/run_eval.py --dry-run

# Search
python tools/validate.py --list --tag nlp

Documentation

• docs/skill_contract.md: Complete DSPy skill specification
• docs/tagging.md: AgenticFleet taxonomy
• docs/versioning.md: Semver rules for skills
• docs/evaluation.md: Golden eval framework
• docs/safety.md: 3-tier safety model

Governance

Added CODE_OF_CONDUCT, SECURITY, CONTRIBUTING, CHANGELOG, GitHub templates (PR + 3 issue types), CODEOWNERS, .editorconfig, .gitattributes.

Template System

skills/_templates/ provides complete boilerplate for rapid skill creation.

Original prompt

Create a Python-first registry of DSPy Module/Signature skills for agent orchestration. Each skill must include skill.yaml metadata, DSPy Module, signatures, IO schemas, safety permissions, tests, and golden evals. Generate catalog/skills.json from metadata and validation tooling. Add starter skills (web_summarizer, doc_transformer, task_planner), CI with pytest + validation, and a reusable skill template scaffold.

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

[Zochory]

@junie review