Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 15 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,20 @@ Add this action after your AI agent step to:

The action **never fails your workflow** — all API calls and comment posts use `core.warning()` for errors, not `core.setFailed()`.

<p align="center">
<img src="images/screenshot-agentmeter-github-comment.png" alt="AgentMeter GitHub PR comment showing cost summary" width="700" />
<br/><sub>AgentMeter posts a cost summary directly on the PR.</sub>
</p>

<p align="center">
<img src="images/screenshot-agentmeter-run-detail.png" alt="Run detail with token breakdown" width="48%" />
&nbsp;
<img src="images/screenshot-agentmeter-runs.png" alt="Runs feed dashboard" width="48%" />
</p>
<p align="center">
<sub>Full token breakdown per run &nbsp;·&nbsp; All runs in one dashboard</sub>
</p>

---

## Quickstart
Expand Down Expand Up @@ -281,7 +295,7 @@ Replace `$INPUT_TOKENS` etc. with however your agent exposes token counts (step
| `api_key` | ✅ | — | Your AgentMeter API key (`am_sk_…`). Get it from [agentmeter.app/dashboard/settings](https://agentmeter.app/dashboard/settings). |
| `model` | ❌ | `''` | The AI model used (e.g. `claude-sonnet-4-5`). Used for per-token cost display. |
| `engine` | ❌ | `claude` | The AI engine (`claude`, `codex`). |
| `status` | ❌ | `success` | Run status: `success`, `failed`, `timed_out`, `cancelled`, `needs_human`. |
| `status` | ❌ | `success` | Run outcome. In companion `workflow_run` mode this is resolved automatically from the triggering workflow's conclusion. In inline mode pass `${{ steps.agent.outcome }}` or a custom value like `needs_human`. See [docs/status-values.md](docs/status-values.md). |
| `agent_output` | ❌ | `''` | Raw stdout from the agent step. Used to auto-extract token counts from JSON. |
| `input_tokens` | ❌ | `''` | Explicit input token count. Overrides extraction from `agent_output`. |
| `output_tokens` | ❌ | `''` | Explicit output token count. |
Expand Down
37 changes: 37 additions & 0 deletions __tests__/comment.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -163,6 +163,43 @@ describe('buildCommentBody', () => {
expect(body).toContain('approximate');
});

it('calculates cache hit rate using reads / (reads + writes + input)', () => {
const body = buildCommentBody({
apiPricing: testPricing,
existingBody: null,
runData: {
...baseRun,
tokens: {
inputTokens: 50,
outputTokens: 2172,
cacheWriteTokens: 55569,
cacheReadTokens: 124794,
isApproximate: false,
},
},
});
// 124794 / (50 + 55569 + 124794) = 124794 / 180413 ≈ 69%
expect(body).toContain('69% cache hit rate');
});

it('does not show cache hit rate when cacheReadTokens is 0', () => {
const body = buildCommentBody({
apiPricing: testPricing,
existingBody: null,
runData: {
...baseRun,
tokens: {
inputTokens: 1000,
outputTokens: 500,
cacheWriteTokens: 0,
cacheReadTokens: 0,
isApproximate: false,
},
},
});
expect(body).not.toContain('cache hit rate');
});

it('skips token details when tokens are not provided', () => {
const body = buildCommentBody({
apiPricing: testPricing,
Expand Down
1 change: 0 additions & 1 deletion action.yml
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
name: 'AgentMeter'
description: 'Track token usage and cost for AI agent runs in GitHub Actions'
author: 'Foo.software'
branding:
icon: 'zap'
color: 'green'
Expand Down
2 changes: 1 addition & 1 deletion dist/index.js

Large diffs are not rendered by default.

89 changes: 89 additions & 0 deletions docs/status-values.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,89 @@
# Status Values

Documents how the `status` input works in the AgentMeter Action and what values are valid.

---

## How status is determined

### Companion workflow mode (`workflow_run` trigger)

When `workflow_run_id` is set, status is resolved **automatically** from the GitHub Actions conclusion of the triggering workflow. The user does not need to pass `status` — the action reads it from the `workflow_run` event payload and normalizes it internally via `normalizeConclusion()`.

```yaml
- uses: foo-software/agentmeter-action@main
with:
api_key: ${{ secrets.AGENTMETER_API_KEY }}
workflow_run_id: ${{ github.event.workflow_run.id }}
# status is resolved automatically — no need to set it
```

### Inline mode (direct / same-workflow)

When running inline (no `workflow_run_id`), the user passes `status` explicitly. It defaults to `'success'` if omitted.

```yaml
- uses: foo-software/agentmeter-action@main
if: always()
with:
api_key: ${{ secrets.AGENTMETER_API_KEY }}
status: ${{ steps.agent.outcome }}
```

---

## Built-in GitHub conclusion mappings

`normalizeConclusion()` maps GitHub's standard conclusion strings to AgentMeter's internal status enum:

| GitHub conclusion | AgentMeter status | Notes |
|---|---|---|
| `success` | `success` | |
| `failure` | `failed` | |
| `timed_out` | `timed_out` | |
| `cancelled` | `cancelled` | |
| `skipped` | *(not ingested)* | Run is skipped entirely — nothing is sent to the API |

**Source of truth:** `normalizeConclusion()` in `src/workflow-run.ts`.

---

## Custom statuses

Any value **not** in the mapping table above is passed through to the API unchanged. This is intentional — unrecognized values are preserved so custom statuses are not silently replaced with `failed`.

### `needs_human`

The primary custom status. Use it when an agent run completes but requires human review before the result can be acted on (e.g. low-confidence output, a tool call was blocked, or the agent explicitly flagged escalation).

**Example — conditionally set based on an agent output flag:**

```yaml
- uses: anthropics/claude-code-action@v1
id: agent
with:
anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
prompt: "..."

- uses: foo-software/agentmeter-action@main
if: always()
with:
api_key: ${{ secrets.AGENTMETER_API_KEY }}
model: claude-sonnet-4-5
status: ${{ steps.agent.outputs.needs_human == 'true' && 'needs_human' || job.status }}
```

When `steps.agent.outputs.needs_human` is `'true'`, the run is recorded as `needs_human`. Otherwise it falls back to the job's actual status (`success` or `failure`).

---

## All valid AgentMeter status values

| Value | Set via action | Description |
|---|---|---|
| `success` | ✅ | Agent run completed successfully |
| `failed` | ✅ | Agent run failed |
| `timed_out` | ✅ | Agent run exceeded its time limit |
| `cancelled` | ✅ | Agent run was cancelled |
| `needs_human` | ✅ | Run completed but requires human review |
| `running` | ❌ internal only | Run is currently in progress — not settable via action |
Binary file added images/screenshot-agentmeter-github-comment.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added images/screenshot-agentmeter-run-detail.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added images/screenshot-agentmeter-runs.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
7 changes: 4 additions & 3 deletions src/comment.ts
Original file line number Diff line number Diff line change
Expand Up @@ -155,10 +155,11 @@ function buildTokenDetails({
const { tokens, model, turns } = run;
if (!tokens) return null;

// Cache hit rate = reads / (reads + writes + input). Cache writes are tokens processed
// fresh to populate the cache — they belong in the denominator alongside input tokens.
const totalPromptTokens = tokens.cacheReadTokens + tokens.cacheWriteTokens + tokens.inputTokens;
const cacheHitRate =
tokens.cacheReadTokens + tokens.inputTokens > 0
? Math.round((tokens.cacheReadTokens / (tokens.cacheReadTokens + tokens.inputTokens)) * 100)
: 0;
totalPromptTokens > 0 ? Math.round((tokens.cacheReadTokens / totalPromptTokens) * 100) : 0;

const pricing = getPricing({ apiPricing, model });
const perM = (count: number, pricePerM: number | null | undefined): string => {
Expand Down
Loading