Hard to troubleshoot, agent stays running for hour

[hpe-ykoehler]

Question

What is the correct way to troubleshoot things in this system, I have a big job going on, it was in the middle of it, then I submitted a couple more job, and everything stop working and task were running but nothing was getting done. There is a need for more visibility, I try to see if LiteLLM was still working it was frozen, restarting it didn't help, I then check the SWE-AF agents there was no easy way to see if they were doing anything. I restarted them but the system didn't unblock.

I restarted agentfield, and still, they show up as running and nothing were really getting things done. I had to delete the request but now the question is, how do I resume that work?

I think a queue system, with a max execution in concurrent is needed, and some way to understand what is the problem is it the agent, the llm, maybe have a agent health system, and an llm health system. Even the agent node it not consistently showing up as up/down properly.

It feels like the UI was put on with AI but not really tested/used in real world scenario. I do feel this project to be very interesting and would even like to a participate.

Context

[AbirAbbas]

Hey, thanks for writing this up — these are all valid points.

Right now when the LLM backend goes unresponsive, executions just hang waiting on responses that never come. Agents look "running" but they're stuck on dead upstream calls, and there's nothing that detects that or breaks the cycle. The node status inconsistency doesn't help either.

Going to investigate and push proper fixes — LLM health checks/circuit breakers, per-agent concurrency limits, better visibility into what executions are actually doing, and retry/recovery for stalled work.

For now, bumping LOG_LEVEL=debug on the control plane and using af dev --verbose for agents will at least give you more to work with when things go sideways. The stale execution reaper (stale_execution_timeout in config) can also be tuned down from the default 10m if you want faster automatic cleanup.

Re: contributing — always welcome, feel free to open PRs or follow up here.

[santoshkumarradha]

Also to add, the main ones being use in SWE right now, which is .harness() for harness orchestration is at its first version, so logging at harness level might be a bit janky, feel free to open up issues as you find, would love to prioritize hiccups you get. Meanwhile to note the progress etc.. inside the workflow view, there should be a tab for notes, and that should help you see where you are and ofcourse feel free to add more app.notes and if you think more precise notes are needed feel free to open an issue in https://github.com/Agent-Field/SWE-AF and we can add them for sure.

[hpe-ykoehler]

I do have a hard time with output, like, having a live refresh of the output would be amazing, the ui tend to display all the data in json which is not really human friendly. I do wish that those were presented in a more polished way, often having a single sentence display when there is a lot of text is confusing the button to expand the text, I keep missing it and assume it was a short string sent.

I have to use it more and understand it better, before making more comment, I like the idea and the back-and-forth aspect. I am not clear why it would say agent has 0 skills or mcp, no idea where those would need to be configured, kind of lack self-help docs (click on skill, get how they work, how to install them - folder location, etc.)

I like the visibility part and I think that is great, I am trying to understand why in a multi-repo situation it still try to write to my project's .artifacts folder... where I was expecting those to be within the repo themselves.

[AbirAbbas]

regarding swe-af multi repo -> This is intended behavior — .artifacts is shared across all repos because execution state (checkpoints, DAG, plan, logs) is per-build, not per-repo. Individual issues already track their target repo, so the build remains one logical unit even when spanning multiple repositories

[AbirAbbas]

These issues are addressed in #319. Here's what was added:

Fixes:

• LLM health monitoring with circuit breaker — detects when your LLM backend (e.g. LiteLLM) goes down and fails fast with HTTP 503 instead of letting executions hang
• Per-agent concurrency limits — prevents job pile-up, rejects excess work with HTTP 429
• Error classification — every failure now returns an error_category (llm_unavailable, concurrency_limit, agent_unreachable, agent_timeout, etc.) so you can tell what's wrong
• Stale execution auto-retry — timed-out executions get retried before being marked failed
• Real-time log streaming — SSE endpoint for live execution events

New API endpoints:

• GET /api/ui/v1/llm/health — circuit breaker state for all LLM endpoints
• GET /api/ui/v1/queue/status — per-agent concurrency slot usage
• GET /api/ui/v1/executions/:id/logs/stream — live execution event stream

To reproduce and test locally:

cd examples/e2e_resilience_tests

# 1. Start a controllable mock LLM server
python3 mock_llm_server.py 4000

# 2. Start the control plane with health monitoring enabled
cd control-plane
AGENTFIELD_LLM_HEALTH_ENABLED=true \
AGENTFIELD_LLM_HEALTH_ENDPOINT=http://localhost:4000/health \
AGENTFIELD_LLM_HEALTH_ENDPOINT_NAME=litellm \
AGENTFIELD_LLM_HEALTH_CHECK_INTERVAL=5s \
AGENTFIELD_LLM_HEALTH_FAILURE_THRESHOLD=3 \
AGENTFIELD_MAX_CONCURRENT_PER_AGENT=3 \
./dev.sh

# 3. Start test agents
cd examples/e2e_resilience_tests
AGENTFIELD_URL=http://localhost:8080 python3 agent_healthy.py &
AGENTFIELD_URL=http://localhost:8080 python3 agent_slow.py &
AGENTFIELD_URL=http://localhost:8080 python3 agent_flaky.py &

# 4. Simulate the LLM freeze scenario from this issue
curl -X POST localhost:4000/freeze    # LLM hangs
# Wait ~15s, then:
curl http://localhost:8080/api/ui/v1/llm/health  # circuit_state: open
curl -X POST localhost:4000/unfreeze  # LLM recovers

# Or run the full automated suite (27 assertions):
./run_tests.sh

The mock LLM server supports: POST /freeze, POST /unfreeze, POST /error, POST /recover to simulate different failure modes.

Note: UI changes to surface this data in the dashboard are not yet included — the backend APIs are available but the frontend doesn't consume them yet.

[AbirAbbas]

leaving this open til I make the UI changes as well