OpenHands integration: ACP CLI path does not expose stable token/cost metrics; need SDK-runner or equivalent metrics bridge

[AmyTao]

Summary

Benchflow's current openhands integration uses the OpenHands CLI ACP path (openhands acp ...). This works for install / launch / ACP execution, but it does not reliably expose the token / cost metrics that the older Harbor + OpenHands SDK runner path provided.

This becomes a problem if Benchflow is expected to replace Harbor in the OpenHands evaluation harness while preserving metrics such as:

• agent_result.n_input_tokens
• agent_result.n_cache_tokens
• agent_result.n_output_tokens
• agent_result.cost_usd

Current behavior

Benchflow can run OpenHands through ACP successfully, but the ACP path does not provide a stable source of cumulative LLM metrics.

What we can get reliably from the current ACP session:

• agent name / version from initialize
• tool calls
• message / thought / tool trajectory
• stop reason
• timing

What we cannot rely on the ACP path to provide today:

• cumulative prompt tokens
• cumulative completion tokens
• cumulative cache tokens
• cumulative cost in USD

Benchflow now has result schema slots for these metrics, but under the current OpenHands CLI ACP integration they will usually remain null unless some future trajectory source includes usage/cost.

Why this matters

The older Harbor/OpenHands SDK-runner path could collect these values directly inside the agent process via:

• llm.metrics.accumulated_token_usage
• llm.metrics.accumulated_cost

That means Harbor output could populate richer trial results, while the current Benchflow ACP integration cannot provide equivalent metrics with the same reliability.

Expected behavior

Benchflow should have a supported way to run OpenHands that can emit stable metrics comparable to the old Harbor/OpenHands SDK-runner results.

At minimum, OpenHands runs in Benchflow should be able to populate:

"agent_result": {
  "n_input_tokens": ...,
  "n_cache_tokens": ...,
  "n_output_tokens": ...,
  "cost_usd": ...
}

Suggested direction
Add an OpenHands SDK runner execution path in Benchflow, and collect metrics directly from the SDK runtime.
Or add an equivalent telemetry/metrics bridge for the CLI ACP path, if OpenHands CLI can emit cumulative usage/cost in a supported machine-readable way.
From an accuracy and maintenance perspective, the SDK-runner path appears preferable, since it reads metrics from the same execution layer that is actually calling the LLM.

Notes
This issue is specifically about metrics parity, not basic OpenHands ACP functionality.
OpenHands CLI ACP works for task execution, but not for reliable token/cost accounting.
If Benchflow is meant to replace Harbor inside the OpenHands evaluation harness, this gap blocks lossless migration of result metadata.

Related local findings
Current Benchflow openhands agent launches via openhands acp --always-approve --override-with-envs
ACP initialize provides agentInfo but not cumulative token/cost usage
Older Harbor/OpenHands SDK runner gathered metrics from llm.metrics.accumulated_token_usage and llm.metrics.accumulated_cost
Benchflow result schema has been extended locally to include agent_info, agent_result, verifier_result, and exception_info, but the ACP path still lacks a reliable source for OpenHands token/cost values

[xdotli]

Implementation Plan: HTTP Proxy-Based Token/Cost Metrics

Architecture: Host-Side HTTP Proxy

The proxy runs on the BenchFlow host (not inside the container). The container reaches it via host.docker.internal:{port} (Docker) or localhost (Daytona). This intercepts LLM API calls by:

1. Starting a TrajectoryProxy on the host before the agent launches
2. Overriding the agent's *_BASE_URL env vars to point at the proxy
3. The proxy forwards to the real API endpoint, recording usage from each response
4. After the agent finishes, extracting aggregate token counts from the proxy's Trajectory

This works for all HTTP-based agents (claude-code, codex, gemini, openhands, pi, openclaw) because they all route API calls through env-var-configurable base URLs. The env_mapping system in registry.py already translates BENCHFLOW_PROVIDER_BASE_URL to agent-specific vars. Zero agent code changes required.

Implementation: 6 files, ~225 LOC

1. models.py (+15 LOC) — Add token/cost fields to RunResult

# After n_prompts in RunResult.__init__
input_tokens: int = 0,
output_tokens: int = 0,
cache_tokens: int = 0,
cost_usd: float | None = None,

These get written to result.json and aggregated by metrics.py.

2. _proxy.py (+80 LOC, new file) — Proxy lifecycle manager

async def start_usage_proxy(
    agent: str,
    agent_env: dict[str, str],
    model: str | None,
    session_id: str = "",
) -> TrajectoryProxy | None:
    """Start a usage-tracking proxy and rewrite agent_env to route through it.
    
    Returns None if no target URL can be determined (e.g., oracle mode).
    Mutates agent_env in-place to redirect API calls through the proxy.
    """

def extract_usage(proxy: TrajectoryProxy | None) -> dict[str, int | float | None]:
    """Extract aggregate token/cost metrics from a stopped proxy.
    
    Returns dict with keys: input_tokens, output_tokens, cache_tokens, cost_usd.
    """

Target URL resolution order:

1. BENCHFLOW_PROVIDER_BASE_URL (set by resolve_provider_env)
2. Agent-native vars via env_mapping reverse lookup
3. Default well-known URLs based on model name

Env var rewriting: After starting, sets BENCHFLOW_PROVIDER_BASE_URL to http://host.docker.internal:{port}. Also updates any agent-specific vars already set so env_mapping doesn't clobber the proxy URL.

3. sdk.py (+30 LOC) — Wire proxy into run loop

Three insertion points:

# After resolve_agent_env(), before agent launch
usage_proxy = None
if agent != "oracle":
    usage_proxy = await start_usage_proxy(agent, agent_env, model, session_id=trial_name)

# In finally block, after ACP client close
usage_metrics = {"input_tokens": 0, "output_tokens": 0, "cache_tokens": 0, "cost_usd": None}
if usage_proxy:
    try:
        await usage_proxy.stop()
        usage_metrics = extract_usage(usage_proxy)
    except Exception as e:
        logger.warning(f"Usage proxy stop failed: {e}")

# Pass to _build_result
return self._build_result(..., **usage_metrics)

4. trajectories/types.py (+10 LOC) — Add cache token extraction

@property
def total_cache_tokens(self) -> int:
    total = 0
    for ex in self.exchanges:
        usage = ex.response.body.get("usage", {})
        total += usage.get("cache_read_input_tokens", 0) or 0
        total += usage.get("cache_creation_input_tokens", 0) or 0
    return total

5. metrics.py (+30 LOC) — Aggregate token/cost metrics

Add to TaskMetrics: input_tokens, output_tokens, cost_usd

Add to BenchmarkMetrics:

• total_tokens — sum across all tasks
• total_cost_usd — sum across all tasks
• avg_cost_per_task — average over completed (non-errored) tasks

6. tests/test_usage_proxy.py (+60 LOC)

• test_extract_usage_none_proxy — returns zeros when proxy is None
• test_extract_usage_with_exchanges — verify extraction from known usage data
• test_start_proxy_rewrites_env — verify env var mutation
• test_no_proxy_for_oracle — returns None for oracle agent

Verified Against Codebase

• ✅ TrajectoryProxy already handles Anthropic + OpenAI streaming formats (_reconstruct_response)
• ✅ env_mapping system already translates BENCHFLOW_PROVIDER_BASE_URL to agent-native vars
• ✅ Subscription auth (OAuth) works — proxy forwards auth headers transparently
• ✅ Custom providers (zai, vllm) work — proxy reads existing BENCHFLOW_PROVIDER_BASE_URL as target
• ✅ ATIF schema already defines Metrics(input_tokens, output_tokens, cache_tokens, cost_usd) — can wire in later

Risks

Risk	Mitigation
Linux Docker needs host.docker.internal	Docker 20.10+ supports --add-host=host.docker.internal:host-gateway
Gemini uses usageMetadata not usage	Add ~10 LOC branch in extract_usage
Daytona networking differs	Defer to phase 2; proxy can run inside sandbox via SSH
Proxy adds latency	Negligible — simple passthrough, sub-ms for non-streaming
Agent already has custom BENCHFLOW_PROVIDER_BASE_URL	Proxy reads existing value as target, then replaces — naturally handles custom providers

Why This Is General

The solution covers all agents because:

1. All HTTP-based agents use configurable base URLs → proxy intercepts transparently
2. The env_mapping registry pattern means no agent-specific code in the proxy
3. Usage data comes from the API response, not the agent — format is standardized (Anthropic usage, OpenAI usage, Gemini usageMetadata)
4. Non-HTTP agents (future) can fall back to agent-native log parsing as a secondary path