Skip to content

fix: prevent tracestate/traceparent header duplication in upstream requests#42

Merged
frac merged 1 commit intomainfrom
fix/tracestate-header-duplication
Apr 23, 2026
Merged

fix: prevent tracestate/traceparent header duplication in upstream requests#42
frac merged 1 commit intomainfrom
fix/tracestate-header-duplication

Conversation

@thspinto
Copy link
Copy Markdown

@thspinto thspinto commented Apr 22, 2026

Problem

When responses.js forwards a request to the AI gateway, the tracestate and traceparent headers are duplicated. For example, an incoming tracestate: origin=test arrives at the upstream as tracestate: origin=test, origin=test.

Evidence

Trace showing the duplicated tracestate in Grafana:
Grafana Tempo trace

Root Cause

Two independent mechanisms both add W3C trace context headers to the outgoing request:

  1. Manual header forwarding — In innerStream.ts, all incoming request headers (except those in NOT_FORWARDED_HEADERS) are passed as defaultHeaders to the OpenAI SDK client. Since traceparent and tracestate were not in the exclusion list, they were forwarded verbatim.

  2. OTel auto-instrumentation — In stateful_responses, responses.js is started with --import ./instrumentation.mjs (see Dockerfile.responses-js). That file sets up @opentelemetry/auto-instrumentations-node via getNodeAutoInstrumentations(), which includes UndiciInstrumentation and HttpInstrumentation. These instrumentations hook into the outgoing request pipeline and call propagation.inject() to add traceparent/tracestate from the active span context.

The combination of both results in each header value appearing twice.

Fix

Add traceparent and tracestate to the NOT_FORWARDED_HEADERS set in src/routes/responses/utils.ts. This lets OTel instrumentation be the single owner of trace context propagation to upstream services, which is the correct behavior — it ensures the child span ID (not the original caller's) is propagated.

Before fix:

tracestate: ["origin=test, origin=test"]

After fix:

tracestate: ["origin=test"]

…quests

Add traceparent and tracestate to NOT_FORWARDED_HEADERS so they are not
manually forwarded to the upstream AI gateway. OTel auto-instrumentation
(UndiciInstrumentation) already injects these W3C trace-context headers
from the active span context, so forwarding them manually caused each
value to appear twice in the outgoing request.
@github-actions
Copy link
Copy Markdown

⚠️ Deprecation Warning: The deny-licenses option is deprecated for possible removal in the next major release. For more information, see issue 997.

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Scanned Files

None

@frac frac merged commit ff4afce into main Apr 23, 2026
9 checks passed
@frac frac deleted the fix/tracestate-header-duplication branch April 23, 2026 07:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants