observability: tagged auth-key rejected despite correct tagOwners (#139 followup)

[gerchowl]

Context

While testing the docs/observability/docker-compose.yml sidecar locally (PR coming shortly), the tailscale/tailscale:stable sidecar consistently fails to join the tailnet with:

backend error: requested tags [tag:observability] are invalid or not permitted

…even though:

1. tagOwners on the tailnet correctly lists "tag:observability": ["autogroup:admin"] (verified via GET /api/v2/tailnet/-/acl).
2. The auth-key (reusable, user-owned, generated via admin UI → Add Linux server) shows tag:ci + tag:observability in its "Tags" column at https://login.tailscale.com/admin/settings/keys.
3. The authenticated user is the tailnet owner (admin role by default).
4. Multiple cycles of ACL-save + fresh-key-regen hit the same error. Propagation lag was ruled out (waited 5+ min).

A read-only API token (Policy File Read) was used to confirm the ACL is correctly saved. We don't yet know if the admin UI's "Tags" column reflects the key's requested tags (from the wizard form) or its actually-bound capabilities after server validation. Diagnostic GET /api/v2/tailnet/-/keys/{id} would clarify; needs Auth Keys Read scope.

Workaround

The sidecar compose in the upcoming PR ships with --advertise-tags commented out. Operator spins it up as an untagged user-owned node, then applies tag:observability manually via Machines → Edit tags in the admin UI. That works (different validation path). Good enough for a single-host observability stack; doesn't block anything.

Resolution options

• Reproduce with a read-enabled API token (Policy File Read + Auth Keys Read + Devices Read) and diff the key's actual bound capabilities vs the UI column. Likely next step.
• Switch to OAuth-client-minted ephemeral keys (same pattern as devc-remote.sh for tag:devc). Higher setup cost, but we know that flow works end-to-end.
• File upstream (tailscale/tailscale issue tracker) if we confirm a server-side bug.

What this does NOT block

• The committed compose stack works as-is (with the manual-tag workaround).
• CI-side OTLP (the otlp-tailnet composite action) is unrelated — it uses tailscale/github-action@v2, a different code path that we haven't tried yet.
• Merging the observability PR.

Refs

• PR #TBD (observability sidecar compose)
• ADR-0010 (sharded pipeline + Dagger)
• Epic [epic] Migrate bake + derive + CI to Dagger pipeline (handoff) #140 (closed) — this is post-epic polish on feat(ci): OTLP spans from Dagger + baker → self-hosted grafana/otel-lgtm #139's deliverable.