Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,15 @@

All notable changes to Plarix Scan will be documented in this file.

## [0.6.0] - 2026-01-04

### Added
- Docker support: `Dockerfile`, `docker-compose.yaml`, and `plarix-scan proxy` daemon mode
- Real-time streaming token usage capture (stream_options injection support)
- Upstream override support (PLARIX_UPSTREAM_*)
- Pricing update (Jan 2026) with sources
- Documentation overhaul (README, Go docs)

## [0.5.0] - 2026-01-03

### Added
Expand Down
10 changes: 10 additions & 0 deletions DEV_NOTES.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
# Dev Notes

**Date**: Jan 4, 2026
**Goal**: Hardening + Docker + Pricing Refresh + Docs

## Objective
Make plarix-scan production-grade for real projects:
- Accurate real-time recording of LLM usage + costs based on provider-reported usage fields.
- Clear documentation.
- Dockerized runtime option.
37 changes: 37 additions & 0 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
# Build stage
# Note: Using latest stable Go for build environment
FROM golang:1.24-alpine AS builder
WORKDIR /app

# Copy source
COPY go.mod ./
COPY cmd ./cmd
COPY internal ./internal
COPY prices ./prices

# Build binary
# CGO_ENABLED=0 for static binary
RUN CGO_ENABLED=0 go build -o plarix-scan ./cmd/plarix-scan

# Runtime stage
# minimal alpine image
FROM alpine:latest
WORKDIR /app

# Install certificates for HTTPS (needed for provider calls)
RUN apk --no-cache add ca-certificates

# Copy binary and assets
COPY --from=builder /app/plarix-scan .
COPY --from=builder /app/prices ./prices

# Setup user
RUN adduser -D -g '' plarix
USER plarix

EXPOSE 8080
VOLUME /data

ENTRYPOINT ["./plarix-scan"]
# Default to proxy mode
CMD ["proxy", "--port", "8080", "--ledger", "/data/plarix-ledger.jsonl"]
141 changes: 89 additions & 52 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,25 +1,25 @@
# Plarix Scan

Free CI cost recorder for LLM API usage — records tokens and costs from real provider responses.
**Free CI cost recorder for LLM API usage.**
Records tokens and costs from *real* provider responses (no estimation).

## What It Does
## Use Cases
- **CI/CD**: Block PRs that exceed cost allowance.
- **Local Dev**: Measure cost of running your test suite.
- **Production**: Monitor LLM sidecar traffic via Docker.

Plarix Scan is a GitHub Action that:
---

1. Starts a local HTTP forward-proxy (no TLS MITM, no custom certs)
2. Runs your test/build command
3. Intercepts LLM API calls when SDKs support base-URL overrides to plain HTTP
4. Records usage from real provider responses (not estimated)
5. Posts a cost summary to your PR
## Quick Start (GitHub Action)

## Quick Start
Add this to your `.github/workflows/cost.yml`:

```yaml
name: LLM Cost Tracking
name: LLM Cost Check
on: [pull_request]

permissions:
pull-requests: write
pull-requests: write # Required for PR comments

jobs:
scan:
Expand All @@ -29,67 +29,104 @@ jobs:

- uses: plarix-ai/scan@v1
with:
command: "pytest -q"
command: "pytest -v" # Your test command
fail_on_cost_usd: 1.0 # Optional: fail if > $1.00
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
```

## Inputs
## How It Works in 3 Steps
1. **Starts a Proxy** on `localhost`.
2. **Injects Env Vars** (e.g. `OPENAI_BASE_URL`) so your SDK routes traffic to the proxy.
3. **Records Usage** from the actual API response body before passing it back to your app.

| Input | Required | Default | Description |
|-------|----------|---------|-------------|
| `command` | Yes | — | Command to run (e.g., `pytest -q`, `npm test`) |
| `fail_on_cost_usd` | No | — | Exit non-zero if total cost exceeds threshold |
| `pricing_file` | No | bundled | Path to custom pricing JSON |
| `providers` | No | `openai,anthropic,openrouter` | Providers to intercept |
| `comment_mode` | No | `both` | Where to post: `pr`, `summary`, or `both` |
| `enable_openai_stream_usage_injection` | No | `false` | Opt-in for OpenAI streaming usage |
### Supported Providers
The proxy sets these environment variables:

## Supported Providers (v1)
| Provider | Env Var Injected | Notes |
|----------|------------------|-------|
| **OpenAI** | `OPENAI_BASE_URL` | Chat Completions + Responses |
| **Anthropic** | `ANTHROPIC_BASE_URL` | Messages API |
| **OpenRouter**| `OPENROUTER_BASE_URL` | OpenAI-compatible endpoint |

- **OpenAI** (Chat Completions + Responses API)
- **Anthropic** (Messages API)
- **OpenRouter** (OpenAI-compatible)
> **Requirement**: Your LLM SDK must respect these standard environment variables or allow configuring the `base_url`.

## How It Works
---

The action sets base URL environment variables to route SDK calls through the local proxy:
## Output Files

Artifacts are written to the working directory:

### `plarix-ledger.jsonl`
One entry per API call.
```json
{"ts":"2026-01-04T12:00:00Z","provider":"openai","model":"gpt-4o","input_tokens":50,"output_tokens":120,"cost_usd":0.001325,"cost_known":true}
```
OPENAI_BASE_URL=http://127.0.0.1:<port>/openai
ANTHROPIC_BASE_URL=http://127.0.0.1:<port>/anthropic
OPENROUTER_BASE_URL=http://127.0.0.1:<port>/openrouter

### `plarix-summary.json`
Aggregated totals.
```json
{
"total_calls": 5,
"total_known_cost_usd": 0.045,
"model_breakdown": {
"gpt-4o": {"calls": 5, "known_cost_usd": 0.045}
}
}
```

**Requirements:**
- Your SDK must support base URL overrides via environment variables
- SDKs that require HTTPS or hardcode endpoints won't work
---

## Limitations
## Usage Guide

### Fork PRs
Secrets are usually unavailable on PRs from forks. In this case, Plarix Scan will report: "No provider secrets available; no real usage observed."
### 1. Local Development
Run the binary to wrap your test command:

### Stubbed Tests
Many test suites stub LLM calls. If no real API calls are made, observed cost will be $0.
```bash
# Build (or download)
go build -o plarix-scan ./cmd/plarix-scan

# Run
./plarix-scan run --command "npm test"
```

### SDK Compatibility
Not all SDKs support HTTP base URLs. If interception fails, the project is marked "Not interceptable".
### 2. Production (Docker Sidecar)
Run Plarix as a long-lived proxy sidecar.

**docker-compose.yaml:**
```yaml
services:
plarix:
image: plarix-scan:latest # (Build locally provided Dockerfile)
ports:
- "8080:8080"
volumes:
- ./ledgers:/data
command: proxy --port 8080 --ledger /data/plarix-ledger.jsonl

app:
image: my-app
environment:
- OPENAI_BASE_URL=http://plarix:8080/openai
- ANTHROPIC_BASE_URL=http://plarix:8080/anthropic
```

## Output
### 3. CI Configuration

- **PR Comment** (idempotent, updated each run)
- **GitHub Step Summary**
- `plarix-ledger.jsonl` — one JSON line per API call
- `plarix-summary.json` — aggregated totals
**Inputs:**
- `command` (Required): The command to execute.
- `fail_on_cost_usd` (Optional): Exit code 1 if cost exceeded.
- `pricing_file` (Optional): Path to custom `prices.json`.
- `enable_openai_stream_usage_injection` (Optional, default `false`): Forces usage reporting for OpenAI streams.

## Cost Calculation
---

Costs are computed **only** from provider-reported usage fields:
- No token estimation or guessing
- Unknown costs are reported explicitly
- Pricing from bundled `prices.json` (with staleness warnings)
## Accuracy Guarantee

## License
Plarix Scan prioritizes **correctness over estimation**.
- **Provider Reported**: We ONLY record costs if the provider returns usage fields (e.g., `usage: { prompt_tokens: ... }`).
- **Real Streaming**: We intercept streaming bodies to parse usage chunks (e.g. OpenAI `stream_options`).
- **Unknown Models**: If a model is not in our pricing table, we record usage but mark cost as **Unknown**. We do not guess.

MIT
> **Note on Stubs**: If your tests use stubs/mocks (e.g. VCR cassettes), Plarix won't see any traffic, and cost will be $0. This is expected.
2 changes: 1 addition & 1 deletion VERSION
Original file line number Diff line number Diff line change
@@ -1 +1 @@
0.5.0
0.6.0
57 changes: 40 additions & 17 deletions action.yml
Original file line number Diff line number Diff line change
@@ -1,41 +1,41 @@
name: 'Plarix Scan'
description: 'Free CI cost recorder for LLM API usage — records tokens and costs from real provider responses'
author: 'plarix-ai'
name: "Plarix Scan"
description: "Free CI cost recorder for LLM API usage — records tokens and costs from real provider responses"
author: "plarix-ai"

branding:
icon: 'activity'
color: 'purple'
icon: "activity"
color: "purple"

inputs:
command:
description: 'Command to run (e.g., pytest -q, npm test)'
description: "Command to run (e.g., pytest -q, npm test)"
required: true
fail_on_cost_usd:
description: 'Exit non-zero if total known cost exceeds this threshold (USD)'
description: "Exit non-zero if total known cost exceeds this threshold (USD)"
required: false
pricing_file:
description: 'Path to custom pricing JSON file (default: bundled prices.json)'
description: "Path to custom pricing JSON file (default: bundled prices.json)"
required: false
providers:
description: 'Comma-separated list of providers to intercept (default: openai,anthropic,openrouter)'
description: "Comma-separated list of providers to intercept (default: openai,anthropic,openrouter)"
required: false
default: 'openai,anthropic,openrouter'
default: "openai,anthropic,openrouter"
comment_mode:
description: 'Where to post results: pr, summary, or both (default: both)'
description: "Where to post results: pr, summary, or both (default: both)"
required: false
default: 'both'
default: "both"
enable_openai_stream_usage_injection:
description: 'Opt-in: inject stream_options to enable usage reporting on OpenAI streaming (default: false)'
description: "Opt-in: inject stream_options to enable usage reporting on OpenAI streaming (default: false)"
required: false
default: 'false'
default: "false"

runs:
using: 'composite'
using: "composite"
steps:
- name: Setup Go
uses: actions/setup-go@v5
with:
go-version: '1.22'
go-version: "1.22"
cache: true
cache-dependency-path: ${{ github.action_path }}/go.sum

Expand All @@ -55,4 +55,27 @@ runs:
INPUT_COMMENT_MODE: ${{ inputs.comment_mode }}
INPUT_ENABLE_OPENAI_STREAM_USAGE_INJECTION: ${{ inputs.enable_openai_stream_usage_injection }}
run: |
${{ github.action_path }}/plarix-scan run --command "$INPUT_COMMAND"

CMD="${{ github.action_path }}/plarix-scan run --command \"$INPUT_COMMAND\""

if [ -n "$INPUT_FAIL_ON_COST_USD" ]; then
CMD="$CMD --fail-on-cost $INPUT_FAIL_ON_COST_USD"
fi

if [ -n "$INPUT_PRICING_FILE" ]; then
CMD="$CMD --pricing \"$INPUT_PRICING_FILE\""
fi

if [ -n "$INPUT_PROVIDERS" ]; then
CMD="$CMD --providers \"$INPUT_PROVIDERS\""
fi

if [ -n "$INPUT_COMMENT_MODE" ]; then
CMD="$CMD --comment \"$INPUT_COMMENT_MODE\""
fi

if [ "$INPUT_ENABLE_OPENAI_STREAM_USAGE_INJECTION" == "true" ]; then
CMD="$CMD --enable-openai-stream-usage-injection=true"
fi

eval "$CMD"
Loading
Loading