Add standalone shadow testing app server

[eazyhozy]

Background

Shadow testing validates a new Actionbase artifact against a running one before deployment — by replaying the same traffic to all three instances and comparing responses. The key challenge is non-deterministic noise: fields like timestamps or trace IDs will always differ even when the two artifacts are functionally identical.

A Diffy-style 3-way architecture eliminates this automatically:

• Primary and Secondary both run the current artifact — differences between them identify non-deterministic noise
• Candidate runs the artifact under test — after stripping noise, remaining differences are real behavioral changes

graph LR
LOG["Access Log Processor"]
subgraph SCOPE ["Shadow Testing App Server ← this task"]
RELAY["Request Relay & Compare"]
end
P["Primary (artifact A)"]
S["Secondary (artifact A)"]
C["Candidate (artifact B)"]
LOG -->|request| RELAY
RELAY -->|expected| P
RELAY -->|remove noise| S
RELAY -->|actual| C

Loading

In scope: a standalone shadow testing app server that accepts a structured access log entry, relays it to Primary/Secondary/Candidate, and returns a comparison result.

Out of scope: the access log publisher (e.g., nginx), the stream processor (e.g., Kafka consumer, Spark Streaming), and provisioning of Primary/Secondary/Candidate instances.

Task

• Add a standalone shadow testing app server module
• Accept a structured access log entry as input; return a comparison result as output
• Relay requests to configured Primary, Secondary, and Candidate endpoints
• Compare three responses: detect noise (P vs. S), strip it, diff P vs. C
• Report mismatches with structured field paths; cap the number of returned entries
• Unit tests for core comparison scenarios
• Document the interface

Done When

• The shadow testing app server is deployable as a standalone module
• Accepts a structured access log entry and returns a comparison result (match verdict, noise fields, mismatch fields)
• No dependency on any specific log publisher, stream processor, or instance provisioning
• Unit tests pass

Notes

• Reference architecture: https://github.com/opendiffy/diffy
• Related: Expose StorageBackend operation details in mutation response #253 — Expose StorageBackend operation details in mutation response (relevant for write-path comparison accuracy)