ship 0.5.0: bug-fix sweep and foundational agent capabilities

[cchinchilla-dev]

Description

Master tracking issue for the 0.5.0 release. Scope is deliberately narrow: the security and correctness fixes from the recent code review, plus the foundational LLM capabilities that transform AgentLoom from a "YAML workflow orchestrator with LLM calls" into a framework that can express real agents.

Current version: 0.4.0 → Target: 0.5.0.

The version jump is justified by the scope:

• Security: closing the router RCE and sandbox bypass surface (fix router expression sandbox to prevent arbitrary code execution #104, harden tool sandbox against command, path, and URL-scheme bypasses #105).
• Correctness: gateway resilience, record/replay race, DAG skip propagation, provider adapter normalization, state lock bypass, subworkflow observability — eight bug-fix issues in total.
• Cost correctness: reasoning tokens (o1/o3/Claude thinking) are unbilled today; fixing the accounting changes reported costs.
• Foundational LLM primitives: tool calling, structured output, embeddings, conversation history, Agent primitive — none of which exist today, and all of which are prerequisites for any modern agent workflow.
• Observability schema: align with OTel GenAI semantic conventions, centralize span/metric names, and expose custom dimensions needed by downstream trace consumers.

Out of scope for 0.5.0 (deferred to 0.6.x or later) is listed at the bottom.

How to use this issue

• Each phase is a coherent body of work with shared dependencies.
• Issues within a phase are parallelizable unless explicitly noted.
• GitHub renders the task lists below as tracked tasks: checkboxes update automatically when each linked issue closes, and the parent shows aggregate progress.

---

Phase 0 — Bug-fix sweep and cost correctness

The bug-fix backlog from the security and correctness review, plus the reasoning-tokens cost bug. Everything downstream depends on a clean baseline.

• fix router expression sandbox to prevent arbitrary code execution #104 fix router expression sandbox to prevent arbitrary code execution
• harden tool sandbox against command, path, and URL-scheme bypasses #105 harden tool sandbox against command, path, and URL-scheme bypasses
• fix gateway resilience: CB/RL ordering, stream cancellation, retry jitter, rate-limiter edge cases #106 fix gateway resilience: CB/RL ordering, stream cancellation, retry jitter, rate-limiter edge cases
• fix record/replay: concurrent write race, streaming capture, hash key coverage #107 fix record/replay: concurrent write race, streaming capture, hash key coverage
• fix DAG skip propagation, parallel cancellation, and budget overshoot #108 fix DAG skip propagation, parallel cancellation, and budget overshoot
• normalize provider adapters: kwargs, 429, streaming usage, pricing, connection cleanup #109 normalize provider adapters: kwargs, 429, streaming usage, pricing, connection cleanup
• fix state manager lock bypass, template rendering gaps, and approval gate flow #110 fix state manager lock bypass, template rendering gaps, and approval gate flow
• wire subworkflow observability, checkpoint serialization, observer surface, metrics cardinality, webhook blocking #111 wire subworkflow observability, checkpoint serialization, observer surface, metrics cardinality, webhook blocking
• add reasoning tokens tracking across providers (o1/o3, Claude thinking, Gemini, Ollama) #127 add reasoning tokens tracking (o1, o3, Claude extended thinking) (cost-correctness bug)

Parallelization: all 9 simultaneously. Suggested split — security duo (#104, #105), providers/resilience trio (#106, #107, #109), core trio (#108, #110, #111), correctness solo (#127).

---

Phase 1 — Internal hygiene refactor

Foundation work for the LLM-primitive phase — collaborator protocols and shared retry primitives so subsequent issues (#116 tool calling, #120 Agent) have stable seams to plug into.

• add typing protocols, retryable status codes, and shared retry primitives #112 add typing protocols, retryable status codes, and shared retry primitives

Sequencing: single PR. Scope is the typing protocols, heapq topo sort, retry-primitive sharing, and retryable_status_codes. The collaborator-class split (LayerRunner / RouterActivation / StepRetryLoop / BudgetEnforcer) is deferred — the current engine.py size is acceptable until #116 / #120 prove a refactor is needed.

Parallelization with Phase 0: can start once #108 lands — the typing work uses the iterative DFS rewrite as its baseline.

---

Phase 2 — Observability foundation

Stabilize the observability contract before adding features that emit data. Every feature landed after this phase will emit with the correct schema from day one — no second migration later. This phase also unblocks downstream trace consumers that need custom dimensions like version or run tags.

• expand OTel span schema, align with GenAI semantic conventions, centralize names #125 expand OTel span schema, align with GenAI semantic conventions, centralize names (adds run_id propagation, prompt metadata, tool-call details, provider-level child spans)
• add experiment metadata logging per execution #77 add experiment metadata logging per execution (custom workflow labels propagate to all spans and metrics)
• attach quality scores to OTel trace spans #59 attach quality scores to OTel trace spans

Parallelization: #125 lands the schema.py module first; #77 and #59 follow once the namespace exists.

---

Phase 3 — LLM core primitives

The three capabilities AgentLoom currently lacks structurally. Foundation for every agentic workflow.

• add native tool/function calling with streaming and parallel-call support #116 add native tool/function calling with streaming and parallel-call support
• add structured output (JSON schema, response_format) across providers #117 add structured output (JSON schema, response_format) across providers
• add embeddings API across providers #118 add embeddings API across providers

Parallelization: 3 independent PRs in parallel. #116 is the largest (likely 4 sub-PRs: one per provider plus the unified surface).

---

Phase 4 — Composition primitives

Step types and conversation primitive. Most are independent of Phase 3; #119 strictly depends on #116.

• add bounded retry loops with feedback #38 add bounded retry loops with feedback (loop step — required by the agent loop pattern)
• add evaluator step type for output quality assessment #39 add evaluator step type for output quality assessment
• add explicit fan-out / fan-in and map operations #43 add explicit fan-out / fan-in and map operations
• add conversation history primitive with token-budget trimming and multi-agent threads #119 add conversation history primitive with token-budget trimming and multi-agent threads (depends on add native tool/function calling with streaming and parallel-call support #116)

Parallelization: #38, #39, #43 in parallel from day one. #119 starts when #116 closes.

---

Phase 5 — Agent + typed state + templating

The capability layer. #120 is the deliverable that makes AgentLoom express real agents end-to-end: system prompt plus tools plus memory plus a bounded decision loop plus typed output.

• add Pydantic-typed state schemas with validation on reads and writes #128 add Pydantic-typed state schemas with validation on reads and writes (depends on add structured output (JSON schema, response_format) across providers #117)
• add safe conditional, loop, and filter constructs to template engine #129 add safe conditional, loop, and filter constructs to template engine (pairs with add Pydantic-typed state schemas with validation on reads and writes #128, shares AST validator with fix router expression sandbox to prevent arbitrary code execution #104)
• add Agent high-level primitive (system prompt + tools + memory + loop policy) #120 add Agent high-level primitive (system prompt + tools + memory + loop policy) (closes Phase 5; depends on add native tool/function calling with streaming and parallel-call support #116, add structured output (JSON schema, response_format) across providers #117, add conversation history primitive with token-budget trimming and multi-agent threads #119, add bounded retry loops with feedback #38, add evaluator step type for output quality assessment #39)

Parallelization: #128 and #129 in parallel; #120 last.

---

Phase 6 — Testing infrastructure

Primitives that make workflows testable at scale: deterministic failure replay, stress-test chaos injection, cross-provider benchmarking, and a deterministic inter-agent replay contract.

• add MockProvider fault modes for deterministic failure replay #123 add MockProvider fault modes for deterministic failure replay (depends on fix gateway resilience: CB/RL ordering, stream cancellation, retry jitter, rate-limiter edge cases #106, fix record/replay: concurrent write race, streaming capture, hash key coverage #107; complements add chaos/fault injection testing mode #62)
• add chaos/fault injection testing mode #62 add chaos/fault injection testing mode (complements add MockProvider fault modes for deterministic failure replay #123)
• add benchmark mode to compare workflow across N provider configs #124 add benchmark mode to compare workflow across N provider configs (depends on add embeddings API across providers #118 for output similarity)
• add deterministic inter-agent message record/replay with JSONL spec and verifier CLI #135 add deterministic inter-agent message record/replay with JSONL spec and verifier CLI (depends on fix record/replay: concurrent write race, streaming capture, hash key coverage #107)

Parallelization: #62 and #124 immediately. #123 after Phase 0 (#106, #107). #135 after #107.

---

Cross-phase dependency map

Phase 0 (parallel sweep)
  |
  +--> Phase 1 (#112)
  |
  +--> Phase 2 (#125 -> #77, #59)
  |      |
  |      +--> Phase 3 (#116, #117, #118 parallel)
  |             |
  |             +--> #119 (#116)
  |             +--> #128 (#117)
  |             +--> #124 (#118)
  |
  +--> Phase 4 (parallel: #38, #39, #43)
  |
  +--> Phase 5 (#128, #129 parallel; #120 closes)
  |
  +--> Phase 6 (#62, #124 anytime; #123 and #135 after Phase 0)

What is deliberately not in 0.5.0

The deferred items are tracked but not gated by 0.5.0:

Provider catalog (defer to 0.6.0):

• add DeepSeek provider #16 DeepSeek
• add Azure OpenAI provider #46 Azure OpenAI
• add AWS Bedrock provider #47 AWS Bedrock
• add generic OpenAI-compatible endpoint provider (vLLM / TGI) #48 OpenAI-compatible endpoint
• add MCP (Model Context Protocol) support for tool_exec steps #67 MCP

Production scale beyond cost correctness (defer to 0.6.x):

• add provider-native prompt caching (Anthropic cache_control, OpenAI cached_tokens) #126 prompt caching
• add distributed budget enforcement with Redis-backed tracker #131 distributed budget
• add shared rate limiting across workflows (Redis-backed) #69 Redis rate limiting
• add RedisCheckpointer and S3Checkpointer backends #115 Redis/S3 checkpointers
• add gateway request hedging, coalescing, health checks, admission control #130 hedging/coalescing
• add LLM response caching #51 response cache
• add strategy-based provider selection (cost, latency, priority) #50 strategy-based selection
• add multi-key round-robin for provider rate limit distribution #49 multi-key round-robin
• pre-flight budget estimation before LLM calls #12 pre-flight estimation
• add soft budget alerts and semantic-aware retry classification #132 soft alerts + semantic retry
• add Grafana dashboards for cost analytics #56 Grafana dashboards
• add Prometheus AlertManager rules #57 AlertManager rules
• add webhook / event-driven workflow triggering (agentloom serve) #65 agentloom serve

Security hardening for regulated industries (defer to 0.6.x or 0.7.x):

• add input validation hooks for prompt injection prevention #52 prompt injection hooks
• add state scoping and access control per step #53 access control
• add secrets management interface with file backend #54 secrets management
• add immutable audit logging for compliance #55 audit logging

DX polish (defer to 0.6.x+):

• add interactive debug/step-through mode #64 interactive debugger
• add workflow versioning and content hash tracking #45 workflow versioning
• add agentloom lint for semantic workflow validation #66 lint
• add agentloom test CLI command for workflow testing #60 agentloom test CLI
• add snapshot (golden file) testing for LLM outputs #63 snapshot golden testing
• add workflow templating and composition (includes/inheritance) #44 workflow composition

Step types and primitives not on the 0.5.0 critical path:

• add transform, embed, moderate, delay step types #121 transform/embed/moderate/delay
• add A2A (Agent-to-Agent) protocol adapter #122 A2A protocol
• add logprob / confidence access in router conditions #74 logprob in router
• add priority ordering to router expressions #13 router priority
• subworkflow config inheritance + CLI global override #33 subworkflow inheritance
• selective state merge for subworkflows #10 selective state merge

What 0.5.0 unlocks

After 0.5.0 ships:

• AgentLoom is production-deployable: all known critical bugs closed, cost reporting correct.
• Agent workflows are expressible natively: tool calling, structured output, conversation state, and the Agent primitive cover the modern agent pattern end-to-end.
• Comparative benchmarks possible: agentloom bench enables side-by-side evaluation across provider configs with unified reporting.
• Observability is contract-stable: schemas centralized in schema.py, gen_ai.* aligned, custom dimensions propagating. External consumers can build trace-parsing code that survives future releases without breakage.

The 32 deferred issues remain valuable but are not on the critical path for those four outcomes.

Issue inventory (27 total)

Phase	Count	Issues
0 — Bug fix sweep	9	#104, #105, #106, #107, #108, #109, #110, #111, #127
1 — Internal hygiene	1	#112
2 — Observability	3	#125, #77, #59
3 — LLM core	3	#116, #117, #118
4 — Composition	4	#38, #39, #43, #119
5 — Agent + typing	3	#128, #129, #120
6 — Testing infra	4	#123, #62, #124, #135

Total in 0.5.0: 27 issues (8 critical bugs + 1 cost bug + 1 refactor + 17 features).
Deferred to 0.6.x+: 32 issues.

Notes

• This is a release-tracking issue, not an implementation issue. Each child issue retains its own discussion, scope, and PR linkage.
• Progress is visible at the top of this issue once any child closes.
• Adjust phase membership freely as scope clarifies — this is a living plan, not a contract.
• If a deferred issue turns out to be blocking a consumer after all, promote it into the appropriate phase; if a 0.5.0 issue turns out to be smaller than expected, ship it early.