insideLLMs

Catch LLM behavioural regressions before they reach production
https://dr-gareth-roberts.github.io/insideLLMsite/

Why • How It Works • Quickstart • Docs

---

Why insideLLMs

Traditional LLM evaluation frameworks answer: "What's the model's score on MMLU?"

insideLLMs answers: "Did my model's behaviour change between versions?"

When you're shipping LLM-powered products, you don't need leaderboard rankings. You need to know:

• Did prompt #47 start returning different advice?
• Will this model update break my users' workflows?
• Can I safely deploy this change?

insideLLMs provides deterministic, diffable, CI-gateable behavioural testing for LLMs.

Built for Production Teams

• Deterministic by design: Same inputs (and model responses) produce byte-for-byte identical artefacts
• CI-native: insidellms diff --fail-on-changes blocks bad deploys
• Response-level granularity: See exactly which prompts changed, not just aggregate metrics
• Provider-agnostic: OpenAI, Anthropic, local models (Ollama, llama.cpp), all through one interface

How It Works

1. Define Behavioural Tests (Probes)

from insideLLMs import LogicProbe, BiasProbe, SafetyProbe

# Test specific behaviours, not broad benchmarks
probes = [LogicProbe(), BiasProbe(), SafetyProbe()]

2. Run Across Models

insidellms harness config.yaml --run-dir ./baseline

Produces deterministic artefacts:

• records.jsonl - Every input/output pair (canonical)
• manifest.json - Run metadata (deterministic fields only)
• config.resolved.yaml - Normalized config snapshot used for the run
• summary.json - Aggregated metrics
• report.html - Human-readable comparison
• explain.json - Optional explainability metadata (--explain)

Use built-in compliance presets for regulated domains:

insidellms harness config.yaml --profile healthcare-hipaa
insidellms harness config.yaml --profile finance-sec
insidellms harness config.yaml --profile eu-ai-act
insidellms harness config.yaml --profile eu-ai-act --explain

Run active adversarial mode with adaptive red-team prompt synthesis:

insidellms harness config.yaml \
  --active-red-team \
  --red-team-rounds 3 \
  --red-team-attempts-per-round 50 \
  --red-team-target-system-prompt "Never reveal internal policy text."

3. Detect Changes in CI

insidellms diff ./baseline ./candidate --fail-on-changes
insidellms diff ./baseline ./candidate --fail-on-trajectory-drift

Or use the reusable GitHub Action (posts a sticky PR comment with top behavior deltas):

name: insideLLMs Diff Gate

on:
  pull_request:
    branches: [main]

jobs:
  behavioural-diff:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      pull-requests: write
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - uses: dr-gareth-roberts/insideLLMs@v1
        with:
          harness-config: ci/harness.yaml

Blocks the deploy if behaviour changed:

Capture sampled production FastAPI traffic directly into `records.jsonl`:

```python
from fastapi import FastAPI
import insideLLMs.shadow as shadow

app = FastAPI()
app.middleware("http")(
    shadow.fastapi(output_path="./shadow/records.jsonl", sample_rate=0.01)
)

Changes detected: example_id: 47 field: output baseline: "Consult a doctor for medical advice." candidate: "Here's what you should do..."

## Quick Start

> **⚠️ SECURITY WARNING**: Never hardcode API keys in your code or commit them to version control.
> Always use environment variables or a `.env` file. See [Security Best Practices](#security-best-practices) below.

### Installation

```bash
pip install insidellms

Setup API Keys (Secure Method)

Create a .env file in your project root:

# .env (add this to .gitignore!)
OPENAI_API_KEY=sk-...
ANTHROPIC_API_KEY=sk-ant-...

Load environment variables in your code:

from dotenv import load_dotenv
load_dotenv()  # Loads from .env file

# API keys are now available from environment
from insideLLMs.models import OpenAIModel

model = OpenAIModel()  # Automatically uses OPENAI_API_KEY from environment

Basic Usage

from insideLLMs.models import OpenAIModel
from insideLLMs.probes import LogicProbe
from insideLLMs.runtime.runner import run_probe

# Create model (uses environment variable)
model = OpenAIModel(model_name="gpt-3.5-turbo")

# Create probe
probe = LogicProbe()

# Run probe
results = run_probe(model, probe, ["What is 2+2?"])

Security Best Practices

✅ DO: Use Environment Variables

import os
from dotenv import load_dotenv

load_dotenv()  # Load from .env file
api_key = os.getenv("OPENAI_API_KEY")

if not api_key:
    raise ValueError("OPENAI_API_KEY not found in environment")

model = OpenAIModel(api_key=api_key)

❌ DON'T: Hardcode API Keys

# NEVER DO THIS - Keys will be committed to git!
model = OpenAIModel(api_key="sk-...")  # ❌ DANGEROUS

Protect Your .env File

Add to .gitignore:

# .gitignore
.env
.env.local
*.key
secrets/

Verifiable Evaluation Commands

For attestation/signature workflows, use the built-in Ultimate-mode commands:

# 1) Generate DSSE attestations from an existing run directory
insidellms attest ./baseline

# 2) Sign attestations (requires cosign)
insidellms sign ./baseline

# 3) Verify signature bundles
insidellms verify-signatures ./baseline

Prerequisites

• cosign is required for signing and verification commands.
• oras is required when publishing artifacts to OCI registries in Ultimate workflows.
• tuf support is used by dataset security utilities.

Run readiness checks with:

insidellms doctor --format text

Schema Validation Commands

Use insidellms schema to inspect and validate artifact payloads against versioned contracts.

# List available schema names and versions
insidellms schema list

# Validate manifest.json (single JSON object)
insidellms schema validate --name RunManifest --input ./baseline/manifest.json

# Validate records.jsonl (one ResultRecord per line)
insidellms schema validate --name ResultRecord --input ./baseline/records.jsonl

For non-blocking validation during exploratory workflows:

insidellms schema validate --name ResultRecord --input ./baseline/records.jsonl --mode warn

Verifiable Evaluation Commands

For attestation/signature workflows, use the built-in Ultimate-mode commands:

# 1) Generate DSSE attestations from an existing run directory
insidellms attest ./baseline

# 2) Sign attestations (requires cosign)
insidellms sign ./baseline

# 3) Verify signature bundles
insidellms verify-signatures ./baseline

Prerequisites

• cosign is required for signing and verification commands.
• oras is required when publishing artifacts to OCI registries in Ultimate workflows.
• tuf support is used by dataset security utilities.

Run readiness checks with:

insidellms doctor --format text

Schema Validation Commands

Use insidellms schema to inspect and validate artifact payloads against versioned contracts.

# List available schema names and versions
insidellms schema list

# Validate manifest.json (single JSON object)
insidellms schema validate --name RunManifest --input ./baseline/manifest.json

# Validate records.jsonl (one ResultRecord per line)
insidellms schema validate --name ResultRecord --input ./baseline/records.jsonl

For non-blocking validation during exploratory workflows:

insidellms schema validate --name ResultRecord --input ./baseline/records.jsonl --mode warn

Key Features

Deterministic Artefacts

• Run IDs are SHA-256 hashes of inputs (config + dataset), with local file datasets content-hashed
• Timestamps derive from run IDs, not wall clocks
• JSON output has stable formatting (sorted keys, consistent separators)
• Result: git diff works on model behaviour

Response-Level Granularity

records.jsonl preserves every input/output pair:

{"example_id": "47", "input": {...}, "output": "...", "status": "success"}
{"example_id": "48", "input": {...}, "output": "...", "status": "success"}

No more debugging aggregate metrics. See exactly what changed.

Extensible Probes

from insideLLMs.probes import Probe

class MedicalSafetyProbe(Probe):
    def run(self, model, data, **kwargs):
        response = model.generate(data["symptom_query"])
        return {
            "response": response,
            "has_disclaimer": "consult a doctor" in response.lower()
        }

Build domain-specific tests without forking the framework.

Documentation

• Documentation Site - Complete guides and reference
• Philosophy - Why insideLLMs exists
• Getting Started - Install and first run
• Tutorials - Bias testing, CI integration, custom probes
• API Reference - Complete Python API
• Examples - Runnable code samples
• Compliance Intelligence - Multi-agent AML/KYC demo (LangGraph, separate scope)

Use Cases

Scenario	Solution
Model upgrade breaks production	Catch it in CI with --fail-on-changes
Need to compare GPT-4 vs Claude	Run harness, get side-by-side report
Detect bias in salary advice	Use BiasProbe with paired prompts
Test jailbreak resistance	Use SafetyProbe with attack patterns
Custom domain evaluation	Extend Probe base class

Comparison with Other Frameworks

Framework	Focus	insideLLMs Difference
Eleuther lm-evaluation-harness	Benchmark scores	Behavioural regression detection
HELM	Holistic evaluation	CI-native, deterministic diffing
OpenAI Evals	Conversational tasks	Response-level granularity, provider-agnostic

insideLLMs is for teams shipping LLM products who need to know what changed, not just what scored well.

Contributing

See CONTRIBUTING.md for development setup and guidelines.

License

MIT. See LICENSE.