Implement Tests/HA

[swalkerhppr]

High availability and federated keystore

Summary

Adds support for running multiple Stoke replicas behind a load balancer with a shared database and federated keystore. When HA is enabled, each replica keeps signing keys in memory only (no key persistence). Peers discover each other via a static list, merge their public keys, and serve a single logical JWKS at /api/pkeys so tokens issued by any replica can be verified by any replica and by resource servers.

Goals

• HA without DB key writes: Multiple replicas share the same claims DB; each has its own in-memory signing keys. Key persistence is disabled when cluster.enabled is true.
• Federated JWKS: Each replica discovers peers (static list), fetches each peer's JWKS from /api/pkeys?local=true, merges with its own keys (deduplicated by kid), and serves the merged JWKS at /api/pkeys. Verification uses this merged set so tokens from any replica are accepted.
• Join/leave at runtime: Discovery is refreshed periodically (cluster.refresh_sec, default 30s); updating static_peers and restarting (or config reload) reflects new or removed replicas.
• Documented path: HA requirements (shared Postgres/MySQL, no SQLite), configuration, and Helm scaling are documented.

Implemented changes

Configuration

• internal/cfg/cluster.go – New cluster config: enabled, discovery (e.g. "static"), static_peers (base URLs), refresh_sec (default 30), instance_id (optional, for distinct kids in merged JWKS). Cluster is attached to context via WithContext; ClusterFromContext(ctx) available where the issuer is built.
• internal/cfg/config.go – Config extended with Cluster; cluster wired into WithContext.
• internal/cfg/tokens.go – When ClusterFromContext(ctx).Enabled is true: PersistKeys is forced to false, and the issuer is wrapped in key.NewFederatedTokenIssuer(...) so the token issuer in context is the federated one.

Discovery and merge

• internal/cluster/discovery.go – Discoverer interface: Peers(ctx) ([]string, error).
• internal/cluster/static.go – StaticDiscoverer implementing Discoverer from cluster.static_peers. Tests in static_test.go.
• internal/cluster/federated.go – MergeJWKS(localJWKS, peerURLs, httpClient) parses local JWKS, GETs each peer at {base}/api/pkeys?local=true, merges keys by kid, and returns combined JWKSet JSON. Peer fetch failures skip that peer; merge still succeeds. Tests in federated_test.go.

Federated issuer

• internal/key/federated_issuer.go – FederatedTokenIssuer wraps a TokenIssuer: delegates IssueToken, RefreshToken, WithContext to inner; PublicKeys(ctx) returns merged JWKS (cached until merged JWKS exp minus a short buffer, or RefreshSec). ParseClaims uses the merged key set so tokens from any replica are valid. When LocalKeysOnly(ctx) is set (e.g. request with ?local=true), returns only inner keys to avoid recursion when peers fetch our keys. Tests in federated_issuer_test.go.

Web

• internal/web/ctx.go – When handling /api/pkeys, requests with ?local=true get key.WithLocalKeysOnly(ctx) so peer fetches receive only that replica's keys.

Helm

• helm/values.yaml – server.replicaCount: 1.
• helm/templates/stoke-server-deployment.yaml – replicas: {{ .Values.server.replicaCount | default 1 }}.
• helm/README.md – HA section: use shared DB and cluster.enabled + cluster.static_peers; link to docs/high-availability.md.

Documentation

• docs/high-availability.md – Requirements (shared Postgres/MySQL, no key persistence in HA), how to enable HA (cluster config, static_peers, instance_id), behaviour (issuance, verification, DB), config reference table, and pointer to Helm/values.

Testing

• test/e2e/specs/ha.spec.ts – E2E: GET /api/pkeys returns 200 and valid JWKS; with HA profile, replica 2's /api/pkeys returns merged JWKS with at least two keys. HA run uses docker-compose.e2e-ha.yaml and configs config-ha-replica1.yaml / config-ha-replica2.yaml.

Configuration example

cluster:
enabled: true
discovery: static
static_peers:
- https://stoke-0:8080
- https://stoke-1:8080
refresh_sec: 30
instance_id: "stoke-0" # optional; keeps kids distinct in merged JWKS