Prompt Tuning: PR-creating agents lack local verification step and kelos-config-update missing PR template instructions

[kelos-bot]

🤖 Kelos Agent @gjkim42

Area: Prompt Tuning

Summary

Two PR-creating TaskSpawner prompts (kelos-workers and kelos-config-update) are missing instructions to run local verification before pushing, and kelos-config-update entirely lacks PR template compliance instructions. This results in unnecessary CI round-trips and PRs that require manual label/release-note fixups.

Findings

1. Neither PR-creating prompt instructs agents to run make verify or make test locally

The agentconfig.yaml agentsMD field tells agents about Makefile targets:

- make verify — run all verification checks (lint, fmt, vet, etc.)
- make test — run all unit tests

However, neither kelos-workers.yaml nor kelos-config-update.yaml prompt templates instruct agents to actually run these before pushing. The worker prompt's only testing instruction is the vague:

Make sure the PR passes all CI tests.

(kelos-workers.yaml, steps 8a/7b)

This means agents push code, wait for CI, get failures, then iterate — wasting time and compute on issues that make verify would catch locally in seconds (lint errors, formatting, vet warnings).

Evidence from recent PRs:

• Multiple agent PRs show patterns of push → CI failure → fix → re-push cycles. For example, PR Support GitHub-based plugins in AgentConfig #507 went through 5 /reset-worker cycles. While not all failures are lint/test issues, a local make verify && make test step would catch a subset of them upfront.
• The agentconfig.yaml already provides the knowledge of these targets but the prompts don't operationalize it into a concrete step.

2. kelos-config-update.yaml has no PR template instructions

The kelos-config-update agent creates PRs (step 4 in its prompt), but its PR creation instructions are:

4. **Create a PR with the changes**
   - Create a branch and commit your changes
   - Push to origin and create a PR with labels `generated-by-kelos` and `ok-to-test`
   - The PR description should:
     - List the PRs and review comments that motivated each change
     - Explain what was changed and why

This does not mention:

• Following .github/PULL_REQUEST_TEMPLATE.md
• Including a /kind label (from: bug, cleanup, docs, feature)
• Including a release-note block
• Filling in all template sections

Evidence: PR #561 (created by kelos-config-update on 2026-03-05) follows a custom format rather than the PR template. It includes a /kind cleanup label, but this appears to come from the agent reading the template file on its own — not from the prompt instructing it to do so. This is unreliable.

By contrast, AGENTS.md (and CLAUDE.md) explicitly require:

## Pull Requests
- Always follow `.github/PULL_REQUEST_TEMPLATE.md` when creating PRs.
- Fill in every section of the template.
- Choose exactly one `/kind` label from: `bug`, `cleanup`, `docs`, `feature`.
- If the PR does not introduce a user-facing change, write "NONE" in the `release-note` block.

3. Relationship to existing issues

• Workflow: AgentConfig drifted from CLAUDE.md and kelos-workers missing explicit issue filters #515 (AgentConfig drifted from CLAUDE.md): Covers the agentconfig.yaml agentsMD missing the Pull Requests section. Does NOT cover the prompt-level gaps in kelos-config-update.yaml or the missing local verification step in either prompt.
• Prompt Tuning: kelos-workers prompt missing PR template compliance (kind labels, release-note blocks) #538 (PR template compliance in workers): Was closed as COMPLETED on 2026-03-05, but the worker prompt on main still does not contain the proposed PR template instructions. The changes from Prompt Tuning: kelos-workers prompt missing PR template compliance (kind labels, release-note blocks) #538 appear to have never been merged — the issue may have been prematurely closed. This is a separate concern from the current issue.

Proposed Changes

Change 1: Add local verification step to kelos-workers.yaml

In the "If no PR exists" branch, add a verification step between implementation and PR creation. In the "If PR exists" branch, add it before pushing.

For the "no PR exists" branch (between steps 3b and 4b):

- 3c. Before pushing, run local verification:

make update && make verify && make test

Fix any issues found before proceeding.

For the "PR exists" branch (between steps 4a and 5a):

- 4c. Before pushing, run local verification:

make update && make verify && make test

Fix any issues found before proceeding.

Change 2: Add local verification step to kelos-config-update.yaml

Add a verification step before PR creation (between steps 3 and 4):

3b. Before creating a PR, run local verification:

make update && make verify && make test

Fix any issues found before proceeding.

Change 3: Add PR template instructions to kelos-config-update.yaml

Replace the current step 4 with:

4. **Create a PR with the changes**
   - Create a branch and commit your changes
   - Push to origin and create a PR with labels `generated-by-kelos` and `ok-to-test`
   - The PR must follow `.github/PULL_REQUEST_TEMPLATE.md`:
     - `#### What type of PR is this?` with exactly one `/kind` (cleanup, docs, bug, or feature)
     - `#### What this PR does / why we need it:` listing the PRs/reviews that motivated each change
     - `#### Which issue(s) this PR is related to:` (use "N/A" if none)
     - `#### Special notes for your reviewer:` (use "N/A" if none)
     - `#### Does this PR introduce a user-facing change?` with a `release-note` block (use "NONE")

Impact

• Prompt-only changes — no code or CRD modifications
• Reduces CI round-trips — local make verify && make test catches lint/test failures before push
• Ensures consistent PR formatting from the config-update agent
• Complements Workflow: AgentConfig drifted from CLAUDE.md and kelos-workers missing explicit issue filters #515 — which covers agentconfig.yaml drift but not these prompt-level gaps

Not Covered

• Worker prompt PR template compliance (covered by Prompt Tuning: kelos-workers prompt missing PR template compliance (kind labels, release-note blocks) #538, which should be reopened or a new issue filed)
• AgentConfig agentsMD drift from CLAUDE.md (Workflow: AgentConfig drifted from CLAUDE.md and kelos-workers missing explicit issue filters #515, still open)
• Cross-spawner dedup (Workflow: Cron agents have weak cross-spawner dedup — widen search window and scope focus areas #553, has open PR self-development: widen cross-spawner dedup and reduce focus area overlap #556)

/kind cleanup

[kelos-bot]

🤖 Kelos Agent @gjkim42

Kelos Triage Report

Kind: kind/cleanup
Priority: priority/important-longterm — Reduces CI round-trips and improves PR consistency, but not blocking adoption or user-facing
Actor: actor/kelos — All three changes are concrete, scoped YAML prompt edits with objectively verifiable acceptance criteria

Status: STILL VALID

Evidence:

• Verified kelos-workers.yaml on main (lines 41–83): no make verify, make update, or make test step exists before pushing. The only testing instruction is "Make sure the PR passes all CI tests" (steps 8a/7b), which is a post-push CI check, not a local pre-push verification.
• Verified kelos-config-update.yaml on main (lines 71–76): PR creation instructions lack any reference to .github/PULL_REQUEST_TEMPLATE.md, /kind labels, or release-note blocks. No local verification step exists either.
• agentconfig.yaml agentsMD documents make verify and make test targets, but neither prompt operationalizes them into a concrete step — confirming the issue's analysis.
• No merged PRs or commits reference Prompt Tuning: PR-creating agents lack local verification step and kelos-config-update missing PR template instructions #566.
• Issue Prompt Tuning: kelos-workers prompt missing PR template compliance (kind labels, release-note blocks) #538 (worker PR template compliance) was closed as COMPLETED on 2026-03-05 but no PR was ever merged for it — the worker prompt on main still lacks PR template instructions. This is a separate concern from Prompt Tuning: PR-creating agents lack local verification step and kelos-config-update missing PR template instructions #566 (which focuses on kelos-config-update template compliance and the verification step for both prompts).

Duplicates found:

• Workflow: AgentConfig drifted from CLAUDE.md and kelos-workers missing explicit issue filters #515 (AgentConfig drifted from CLAUDE.md) — Partial overlap on the agentconfig.yaml agentsMD missing the Pull Requests section, but Workflow: AgentConfig drifted from CLAUDE.md and kelos-workers missing explicit issue filters #515 does NOT cover the prompt-level gaps in kelos-config-update.yaml or the missing local verification step. Both issues can coexist.
• Prompt Tuning: kelos-workers prompt missing PR template compliance (kind labels, release-note blocks) #538 (Worker PR template compliance, CLOSED) — Covers worker prompt PR template instructions, but was closed without a merge. Prompt Tuning: PR-creating agents lack local verification step and kelos-config-update missing PR template instructions #566 explicitly scopes itself to kelos-config-update PR template and the verification step for both prompts, treating worker PR template as a separate concern.

Recommendation: KEEP OPEN — The three proposed changes are well-scoped, evidence-backed, and address real gaps in the current prompt templates. The issue clearly delineates its scope from #515 and #538.