Observability: per-RPC tracing spans and OpenTelemetry integration

[iainmcgin]

There is currently no per-RPC observability hook. The runtime emits tracing events at debug/trace level for connection state and envelope decoding, but no span scoped to an individual RPC. Users who want a span carrying rpc.system, rpc.service, rpc.method, and rpc.grpc.status_code (the OpenTelemetry RPC semantic conventions) have nowhere to put it.

The recommended middleware pattern (tower_http::trace::TraceLayer::new_for_http() from the middleware example) produces HTTP spans (POST /pkg.Service/Method 200), not RPC spans. The HTTP layer doesn't know the RPC kind, can't see the Connect/gRPC error code (which is in the trailer or JSON body, not the HTTP status for most codes), and doesn't compose with downstream services that propagate W3C trace context through metadata.

For comparison: connect-go ships connectrpc.com/otelconnect (an Interceptor that creates spans and metrics per the OTel RPC conventions and propagates trace context); tonic users reach for tonic-tracing-opentelemetry's OtelGrpcLayer. Both depend on having a typed RPC-level interception point.

Proposed direction

Once a typed RPC-level interceptor surface exists (tracked separately), ship a connectrpc-otel companion crate (or a feature-gated module) providing:

• a server-side interceptor that opens a span per RPC with rpc.system = "connect_rpc" (or "grpc"/"grpc_web" depending on the negotiated protocol), rpc.service, rpc.method, and records rpc.grpc.status_code on completion
• a client-side interceptor that propagates W3C traceparent/tracestate (and grpc-trace-bin for gRPC peers) via request headers and opens a client span
• optional metrics (request count, duration histogram, message size histogram) keyed by service/method/code

Even before the interceptor work lands, two cheap improvements are possible:

• expose service name, method name, and negotiated protocol on RequestContext (or a Spec struct) so a user-written tower layer can at least get the labels right by reading extensions
• a TracingLayer that wraps the existing HTTP-level path in a span named pkg.Service/Method rather than POST /pkg.Service/Method

Scope

• depends on typed interceptor surface (separate issue)
• connectrpc-otel crate or otel feature, server + client interceptors
• example showing OTLP export end to end