Skip to content

developer-guide.md

Latest commit

 

History

History
712 lines (533 loc) · 18.4 KB

developer-guide.md

File metadata and controls

712 lines (533 loc) · 18.4 KB

Forge Developer Guide

Everything you need to run Forge locally, test it, observe what it's doing, and debug problems.

Table of Contents

  1. Prerequisites
  2. Initial Setup
  3. Environment Configuration
  4. Running Services
  5. Running Tests
  6. Testing with Payloads
  7. GitHub Webhook Testing
  8. Prometheus Metrics
  9. Langfuse Tracing
  10. Debugging Tools
  11. Common Workflows
  12. Service Reference

1. Prerequisites

  • Python 3.11+ with uv
  • Podman — for running task containers (dnf install podman / brew install podman)
  • Docker Compose — for Redis and API gateway (dnf install docker-compose / included with Docker Desktop)
  • Jira Cloud account with API access
  • GitHub account with a Personal Access Token (scopes: repo, read:org)
  • Claude API key (Anthropic direct) OR Google Cloud project with Vertex AI enabled

2. Initial Setup

# Clone and install Python dependencies
git clone https://github.com/your-org/forge.git
cd forge
uv sync

# Copy and configure environment
cp .env.example .env
# Edit .env — at minimum fill in Jira, GitHub, and LLM credentials

# Build the task container image
podman build -t forge-dev:latest -f containers/Containerfile containers/

3. Environment Configuration

All settings live in .env. The most important ones for local development:

Required

# Jira
JIRA_BASE_URL=https://your-org.atlassian.net
JIRA_USER_EMAIL=you@example.com
JIRA_API_TOKEN=your-jira-api-token

# GitHub
GITHUB_TOKEN=github_pat_your_token
GITHUB_KNOWN_REPOS=org/repo1,org/repo2   # repos Forge can work on
GITHUB_DEFAULT_REPO=org/repo1

# LLM — choose one backend

# Option A: Anthropic direct
ANTHROPIC_API_KEY=sk-ant-your-key

# Option B: Google Vertex AI (supports Claude + Gemini)
ANTHROPIC_VERTEX_PROJECT_ID=your-gcp-project
ANTHROPIC_VERTEX_REGION=us-east5

# Which model to use
LLM_MODEL=claude-opus-4-5@20251101

For Local Testing (skip webhook validation)

JIRA_WEBHOOK_SECRET=          # empty = no signature check
GITHUB_WEBHOOK_SECRET=        # empty = no signature check

Redis

REDIS_URL=redis://localhost:6380/0   # matches docker-compose port mapping

Container execution

CONTAINER_IMAGE=localhost/forge-dev:latest   # built with podman above
CONTAINER_TIMEOUT=7200                        # 2 hours max
CONTAINER_MEMORY=4g
CONTAINER_CPUS=2

Observability

# Langfuse (LLM call tracing)
LANGFUSE_ENABLED=true
LANGFUSE_PUBLIC_KEY=pk-lf-...
LANGFUSE_SECRET_KEY=sk-lf-...
LANGFUSE_HOST=https://cloud.langfuse.com

# CI behaviour
CI_FIX_MAX_RETRIES=5
CI_IGNORED_CHECKS=tide          # comma-separated substrings of checks to ignore

4. Running Services

Start Redis

Redis is the only service that runs in Docker for local development.

docker compose up redis -d

This starts Redis Stack on localhost:6380 — state, queue, and checkpoints.

Start the API server

uv run uvicorn forge.main:app --reload --port 8000 --host 0.0.0.0

The --reload flag restarts the server automatically when source files change — useful during development.

Start the worker

The worker spawns Podman containers for task execution; it must run on the host.

uv run forge worker

Why not in Docker? The worker spawns Podman containers. Running it inside Docker would require socket mounting which is not supported in this setup.

Start Prometheus (optional, for metrics)

docker compose up prometheus -d
# Dashboard at http://localhost:9092

Full local stack

# Terminal 1 — Redis (and optionally Prometheus)
docker compose up redis prometheus -d

# Terminal 2 — API server
uv run uvicorn forge.main:app --reload --port 8000 --host 0.0.0.0

# Terminal 3 — Worker
uv run forge worker

Health check

curl http://localhost:8000/api/v1/health

5. Running Tests

# Full test suite
uv run pytest

# Unit tests only (fast)
uv run pytest tests/unit/ -v

# Flow/scenario tests
uv run pytest tests/flows/ -v

# Single file
uv run pytest tests/unit/workflow/test_ci_gate_skip.py -v

# With output (useful for debugging)
uv run pytest tests/unit/ -s

# Linting
uv run ruff check src/

# Type checking
uv run mypy src/forge/

The pre-existing failures in tests/flows/status_transitions/ and a few other files are known issues unrelated to current work — ignore them.

6. Testing with Payloads

Sample Jira webhook payloads live in tests/payloads/. They simulate the full feature workflow from creation to task approval.

Endpoint

POST http://localhost:8000/api/v1/webhooks/jira
Content-Type: application/json

Complete feature workflow sequence

# 1. Create a feature ticket → Forge generates PRD
curl -X POST http://localhost:8000/api/v1/webhooks/jira \
  -H "Content-Type: application/json" \
  -d @tests/payloads/01-feature-created.json

# 2. Request PRD revision (comment with feedback)
curl -X POST http://localhost:8000/api/v1/webhooks/jira \
  -H "Content-Type: application/json" \
  -d @tests/payloads/02-prd-revision-requested.json

# 3. Approve PRD → Forge generates spec
curl -X POST http://localhost:8000/api/v1/webhooks/jira \
  -H "Content-Type: application/json" \
  -d @tests/payloads/03-prd-approved.json

# 4–9. Continue: spec, plan, tasks (same pattern)
curl -X POST http://localhost:8000/api/v1/webhooks/jira \
  -H "Content-Type: application/json" \
  -d @tests/payloads/05-spec-approved.json

curl -X POST http://localhost:8000/api/v1/webhooks/jira \
  -H "Content-Type: application/json" \
  -d @tests/payloads/07-plan-approved.json

curl -X POST http://localhost:8000/api/v1/webhooks/jira \
  -H "Content-Type: application/json" \
  -d @tests/payloads/09-task-approved.json
# → Implementation starts

Q&A mode (ask without triggering regeneration)

Comments starting with ? or @forge ask ask a question instead of requesting a revision:

# Ask about the PRD
curl -X POST http://localhost:8000/api/v1/webhooks/jira \
  -H "Content-Type: application/json" \
  -d @tests/payloads/10-prd-question.json

Supported prefixes (case-insensitive):

  • ?Why did you choose REST?
  • @forge ask explain the auth approach

Editing ticket keys in payloads

The payloads use TEST-123 as a placeholder. Replace it with your actual ticket:

sed 's/TEST-123/AISOS-999/g' tests/payloads/01-feature-created.json | \
  curl -X POST http://localhost:8000/api/v1/webhooks/jira \
  -H "Content-Type: application/json" \
  -d @-

Bug workflow

# Create a bug → Forge generates RCA
# (use a payload with issuetype: Bug)

7. GitHub Webhook Testing

GitHub webhooks require the X-GitHub-Event header. Always include it.

CI check result (triggers CI evaluation)

curl -X POST http://localhost:8000/api/v1/webhooks/github \
  -H "Content-Type: application/json" \
  -H "X-GitHub-Event: check_run" \
  -d '{
    "action": "completed",
    "check_run": {
      "status": "completed",
      "conclusion": "failure",
      "head_sha": "YOUR_HEAD_SHA",
      "pull_requests": [
        {"number": 42, "head": {"ref": "forge/ticket-key"}}
      ]
    },
    "repository": {"full_name": "org/repo"},
    "sender": {"login": "your-username"}
  }'

Get the current head SHA for a PR:

gh pr view 42 --repo org/repo --json headRefOid,headRefName

CI gate skip (skip a failing check by name)

Post this as a PR comment:

curl -X POST http://localhost:8000/api/v1/webhooks/github \
  -H "Content-Type: application/json" \
  -H "X-GitHub-Event: issue_comment" \
  -d '{
    "action": "created",
    "issue": {
      "number": 42,
      "title": "[TICKET-123] Your PR title",
      "pull_request": {"url": "https://api.github.com/repos/org/repo/pulls/42"}
    },
    "comment": {"body": "/forge skip-gate e2e-openstack"},
    "repository": {"full_name": "org/repo"},
    "sender": {"login": "your-username"}
  }'

To remove a skip: change skip-gate to unskip-gate.

The check name is matched as a case-insensitive substring — e2e-openstack skips any check whose name contains that string.

PR review (triggers human review gate)

# Approved review → advances to complete_tasks
curl -X POST http://localhost:8000/api/v1/webhooks/github \
  -H "Content-Type: application/json" \
  -H "X-GitHub-Event: pull_request_review" \
  -d '{
    "action": "submitted",
    "review": {
      "state": "approved",
      "body": "LGTM",
      "commit_id": "YOUR_HEAD_SHA"
    },
    "pull_request": {
      "number": 42,
      "title": "[TICKET-123] Your PR title",
      "head": {"ref": "forge/ticket-123", "sha": "YOUR_HEAD_SHA"},
      "state": "open"
    },
    "repository": {"full_name": "org/repo"},
    "sender": {"login": "your-username"}
  }'

# Changes requested / comment → triggers implement_review
# Same payload but: "state": "changes_requested" or "state": "commented"

Note: Comments (state: "commented") are treated as changes_requested — if your review has comments, Forge will address them.

8. Prometheus Metrics

Endpoints

Source URL
API server http://localhost:8000/metrics
Worker http://localhost:8001/metrics
Prometheus UI http://localhost:9092

Key metrics to watch

# Workflow throughput
forge_workflows_started_total
forge_workflows_completed_total
forge_workflows_failed_total

# CI fixing
forge_ci_fix_attempts_total

# Agent performance
forge_agent_duration_seconds     # histogram of agent execution time
forge_phase_duration_seconds     # time per workflow phase

# Webhook health
forge_webhooks_received_total
forge_webhooks_processed_total
forge_webhooks_failed_total

# External API latency
forge_external_api_latency_seconds{service="jira"}
forge_external_api_latency_seconds{service="github"}
forge_external_api_latency_seconds{service="claude"}

Adding worker metrics to Prometheus

The default prometheus.yml only scrapes the API server. Add this to scrape the worker:

# prometheus.yml
scrape_configs:
  - job_name: 'forge-worker'
    static_configs:
      - targets: ['host.docker.internal:8001']   # on macOS/Windows
      # or on Linux:
      # - targets: ['172.17.0.1:8001']           # docker bridge gateway
    metrics_path: /metrics

Reload Prometheus after editing:

curl -X POST http://localhost:9092/-/reload

Quick spot check

# Check workflow counts
curl -s http://localhost:8000/metrics | grep forge_workflows

# Check if worker is serving metrics
curl -s http://localhost:8001/metrics | grep forge_agent

9. Langfuse Tracing

Langfuse records every LLM call: prompt, response, latency, cost, token count.

Setup

  1. Create an account at cloud.langfuse.com
  2. Create a project, get your keys
  3. Add to .env:
LANGFUSE_ENABLED=true
LANGFUSE_PUBLIC_KEY=pk-lf-...
LANGFUSE_SECRET_KEY=sk-lf-...
LANGFUSE_HOST=https://cloud.langfuse.com
  1. Restart the worker — traces appear in the Langfuse dashboard immediately

Self-hosted Langfuse (optional)

Run Langfuse locally with Docker:

git clone https://github.com/langfuse/langfuse.git
cd langfuse
docker compose up -d

Then set LANGFUSE_HOST=http://localhost:3000 in .env.

What gets traced

  • PRD / spec / epic / task generation calls
  • Q&A answer calls
  • PR description sync calls
  • CI fix analysis and implementation calls
  • All calls have the ticket key, task type, and token counts attached

Disabling tracing

LANGFUSE_ENABLED=false

10. Debugging Tools

Patch a workflow checkpoint

Directly edit Redis state for a ticket — useful when a workflow gets stuck due to a bug or incorrect state:

uv run python devtools/patch_checkpoint.py <ticket-key> <field=value> [field=value ...]

Examples:

# Reset a stuck workflow to ci_evaluator
uv run python devtools/patch_checkpoint.py AISOS-376 \
  current_node=ci_evaluator \
  is_paused=false \
  is_blocked=false \
  last_error=null \
  ci_fix_attempts=0

# Resume at wait_for_ci_gate after patching from escalated state
uv run python devtools/patch_checkpoint.py AISOS-376 \
  current_node=wait_for_ci_gate \
  is_paused=true \
  is_blocked=false \
  last_error=null

# Skip e2e checks for a ticket
uv run python devtools/patch_checkpoint.py AISOS-376 \
  'ci_skipped_checks=["e2e-openstack"]'

# Add new state fields introduced by a code change
uv run python devtools/patch_checkpoint.py AISOS-376 \
  'ci_skipped_checks=[]' \
  'review_comments=[]'

JSON parsing: values are parsed as JSON if possible, otherwise as strings. Use quotes for lists and booleans: true/false/null are parsed correctly.

Common checkpoint patches

Situation Patch
Workflow wrongly escalated to blocked current_node=ci_evaluator is_blocked=false last_error=null ci_fix_attempts=0
Restart CI from scratch current_node=wait_for_ci_gate is_paused=true ci_fix_attempts=0
Skip a flaky CI check 'ci_skipped_checks=["check-name-substring"]'
New field added in code new_field=default_value
Force retry after fix add forge:retry label in Jira instead

forge:retry label

Add the forge:retry label to a Jira ticket to resume a blocked workflow. Forge will:

  • Clear last_error
  • Clear is_blocked
  • Reset ci_fix_attempts to 0
  • Resume from the node that failed

Worker logs

The worker logs to stdout. Useful log entries to grep for:

# Watch for a specific ticket
uv run forge worker 2>&1 | grep AISOS-376

# See what signals are detected
uv run forge worker 2>&1 | grep "Detected"

# See CI evaluation results
uv run forge worker 2>&1 | grep "CI"

Check Redis state directly

# Connect to Redis
redis-cli -p 6380

# List all checkpoints
KEYS checkpoint:*

# Get a specific checkpoint (LangGraph stores them as JSON)
GET checkpoint:AISOS-376:...

11. Common Workflows

Start a new feature end-to-end (local test)

# 1. Make sure worker and API are running
docker compose up redis forge-api -d
uv run forge worker &

# 2. Create the feature
curl -X POST http://localhost:8000/api/v1/webhooks/jira \
  -H "Content-Type: application/json" \
  -d @tests/payloads/01-feature-created.json

# 3. Watch worker logs, approve each stage as it completes
curl -X POST http://localhost:8000/api/v1/webhooks/jira \
  -H "Content-Type: application/json" \
  -d @tests/payloads/03-prd-approved.json
# ... and so on through task approval

Test CI gate skip

# 1. Get a PR with failing CI checks
gh pr checks 42 --repo org/repo

# 2. Patch the checkpoint to wait_for_ci_gate if not already there
uv run python devtools/patch_checkpoint.py TICKET-123 \
  current_node=wait_for_ci_gate is_paused=true ci_fix_attempts=0

# 3. Send the skip-gate command
curl -X POST http://localhost:8000/api/v1/webhooks/github \
  -H "Content-Type: application/json" \
  -H "X-GitHub-Event: issue_comment" \
  -d '{
    "action": "created",
    "issue": {
      "number": 42,
      "title": "[TICKET-123] Title",
      "pull_request": {"url": "https://api.github.com/repos/org/repo/pulls/42"}
    },
    "comment": {"body": "/forge skip-gate failing-check-name"},
    "repository": {"full_name": "org/repo"},
    "sender": {"login": "you"}
  }'

# 4. Send a CI webhook to trigger re-evaluation
curl -X POST http://localhost:8000/api/v1/webhooks/github \
  -H "Content-Type: application/json" \
  -H "X-GitHub-Event: check_run" \
  -d '{"action":"completed","check_run":{"status":"completed","conclusion":"failure","head_sha":"SHA","pull_requests":[{"number":42,"head":{"ref":"forge/ticket-123"}}]},"repository":{"full_name":"org/repo"},"sender":{"login":"you"}}'

Trigger the human review flow

# After CI passes, the workflow pauses at human_review_gate.
# Send a review webhook to trigger implement_review:
curl -X POST http://localhost:8000/api/v1/webhooks/github \
  -H "Content-Type: application/json" \
  -H "X-GitHub-Event: pull_request_review" \
  -d '{
    "action": "submitted",
    "review": {
      "state": "commented",
      "body": "Please fix the jitter constant comment in controller.go.",
      "commit_id": "YOUR_HEAD_SHA"
    },
    "pull_request": {
      "number": 42,
      "title": "[TICKET-123] Title",
      "head": {"ref": "forge/ticket-123", "sha": "YOUR_HEAD_SHA"},
      "state": "open"
    },
    "repository": {"full_name": "org/repo"},
    "sender": {"login": "you"}
  }'

12. Service Reference

Ports

Service Port URL
Forge API 8000 http://localhost:8000
API metrics 8000 http://localhost:8000/metrics
Worker metrics 8001 http://localhost:8001/metrics
Redis 6380 redis://localhost:6380/0
Prometheus 9092 http://localhost:9092

API endpoints

Endpoint Method Purpose
/api/v1/health GET Health check
/api/v1/webhooks/jira POST Jira webhook receiver
/api/v1/webhooks/github POST GitHub webhook receiver
/metrics GET Prometheus metrics

GitHub PR comment commands

Command Effect Active at
/forge skip-gate <name> Skip named CI check CI stages
/forge unskip-gate <name> Remove a skip CI stages

Jira labels

Label Meaning
forge:managed Ticket managed by Forge
forge:prd-pending Awaiting PRD approval
forge:prd-approved PRD approved
forge:spec-pending Awaiting spec approval
forge:spec-approved Spec approved
forge:plan-pending Awaiting epic plan approval
forge:plan-approved Plan approved
forge:task-pending Awaiting task approval
forge:task-approved Tasks approved, implementation starts
forge:blocked Workflow blocked, needs intervention
forge:retry Resume a blocked workflow

Useful .env knobs for development

LOG_LEVEL=DEBUG              # verbose logging
CONTAINER_LANGCHAIN_VERBOSE=true  # verbose container agent logs
LANGFUSE_ENABLED=false       # disable tracing for speed
JIRA_WEBHOOK_SECRET=         # skip signature validation
GITHUB_WEBHOOK_SECRET=       # skip signature validation
CI_FIX_MAX_RETRIES=1         # fail fast during CI testing