Skip to content

README.md

Latest commit

 

History

History
158 lines (106 loc) · 4.79 KB

README.md

File metadata and controls

158 lines (106 loc) · 4.79 KB

Fairvisor Benchmark Suite

Single-script benchmark suite for fairvisor/edge with reproducible latency and throughput tests. The primary workflow is now a local controller that orchestrates two remote Linux hosts:

  • one remote Fairvisor host
  • one remote k6 load-generator host

Script: run-all.sh

What It Tests

run-all.sh runs 5 scenarios:

  1. Raw nginx baseline latency (:8082)
  2. Fairvisor decision_service latency (POST /v1/decision)
  3. Fairvisor reverse_proxy latency
  4. Max throughput: simple policy (1 rule)
  5. Max throughput: complex policy (5 rules + JWT + loop detection)

Each run prints a summary table and stores raw artifacts.

Environment

  • Designed for Linux hosts (Amazon Linux / Ubuntu)
  • Installs dependencies automatically:
    • OpenResty
    • k6
    • jq, bc, git, python3, pip3
  • Clones https://github.com/fairvisor/edge into /opt/fairvisor

Quick Start

Recommended: local controller + two remote hosts

Run from your local machine:

FAIRVISOR_REMOTE=ubuntu@fairvisor-host \
LOADGEN_REMOTE=ubuntu@loadgen-host \
FAIRVISOR_TARGET_HOST=10.0.0.42 \
bash run-all.sh

Meaning of the variables:

  • FAIRVISOR_REMOTE: SSH target for the Fairvisor host
  • LOADGEN_REMOTE: SSH target for the k6 load-generator host
  • FAIRVISOR_TARGET_HOST: IP or DNS name that the load-generator host should use for HTTP traffic to Fairvisor
  • SSH_OPTS: optional extra ssh/scp flags used by the controller, for example -o StrictHostKeyChecking=accept-new

What the controller does:

  • copies run-all.sh to both hosts under /tmp/fv-bench/
  • installs only the dependencies each role needs
  • starts/stops baseline nginx, backend nginx, and Fairvisor on the Fairvisor host
  • generates and runs k6 workloads on the load-generator host
  • pulls raw k6 result JSON files back to the local controller for summary rendering

FAIRVISOR_TARGET_HOST is optional if the SSH hostname of FAIRVISOR_REMOTE is also the correct network address from the load-generator host.

On a first-time connection, prefer setting SSH_OPTS so the controller does not block on host key confirmation:

SSH_OPTS="-o StrictHostKeyChecking=accept-new" \
FAIRVISOR_REMOTE=ubuntu@fairvisor-host \
LOADGEN_REMOTE=ubuntu@loadgen-host \
bash run-all.sh

Local single-host fallback

For quick local validation or development:

bash run-all.sh

This keeps the old single-host behavior where Fairvisor and k6 share one machine.

Dry-run validation

To validate orchestration commands without touching real hosts:

DRY_RUN=1 \
FAIRVISOR_REMOTE=ubuntu@fairvisor-host \
LOADGEN_REMOTE=ubuntu@loadgen-host \
FAIRVISOR_TARGET_HOST=10.0.0.42 \
bash run-all.sh

CPU Pinning

For setups where SUT and k6 share one host, optional taskset pinning is supported. In remote-controller mode, pinning is applied independently on whichever host is running that role.

  • ORESTY_CPUSET for OpenResty/backend processes
  • K6_CPUSET for k6

Example:

ORESTY_CPUSET=0-3 K6_CPUSET=4-7 bash run-all.sh

If taskset exists and host has >=8 cores, defaults are auto-split 50/50.

Outputs

On the load-generator host:

  • Raw k6 summaries: /tmp/fv-bench/results/
  • Generated k6 scripts: /tmp/fv-bench/scripts/

On the Fairvisor host:

  • Service logs: /tmp/fv-bench/run/*/logs/

On the local controller:

  • Pulled result JSONs: /tmp/fv-bench/controller-results/

Reference Results

Measured on 2 × AWS c7i.xlarge (4 vCPU, 8 GB RAM each), cluster placement group, eu-central-1, Ubuntu 24.04 LTS. k6 v0.54.0, constant-arrival-rate, 10 000 RPS / 60 s / 10 s warmup. Fairvisor and k6 run on separate hosts.

Latency @ 10 000 RPS

Percentile Decision Service Reverse Proxy Raw nginx
p50 304 μs 302 μs 235 μs
p90 543 μs 593 μs 409 μs
p99 2 000 μs 1 790 μs 1 950 μs
p99.9 4 000 μs 5 120 μs 3 620 μs

Max Sustained Throughput

Configuration RPS
Simple rate limit (1 rule) 195 000
Complex policy (5 rules, JWT + loop detect) 195 000

Your numbers will vary by instance type, network topology, and OS.

Notes on Interpretation

  • decision_service and reverse_proxy are different paths:
    • decision_service: decision API only
    • reverse_proxy: decision + upstream proxy round-trip
  • In single-host runs, k6 and SUT can contend for CPU unless pinned or separated.
  • The recommended path for production-like proxy latency is the two-host controller mode.

Repository

This benchmark suite is intended to be published in:

  • https://github.com/fairvisor/benchmark