Registry client doesn't preserve effective URL after redirects - breaks tunnel/proxy scenarios

[cx-shaked-karta]

Registry client doesn't preserve effective URL after redirects - breaks tunnel/proxy scenarios

Description

The Stereoscope registry client fails to preserve the effective URL after HTTP redirects, causing issues when accessing registries through tunnels, proxies, or load balancers. Instead of using the effective URL from resp.Request.URL, it appears to reconstruct URLs from the original reference.Named authority.

Impact on Downstream Projects

This affects Syft and potentially other projects that depend on Stereoscope for OCI registry operations.

Problem Details

When a registry is accessed through a tunnel/proxy:

1. Original reference: tunnel.example.com/repo/image:tag
2. Registry returns auth challenge referencing original domain: original.registry.com
3. Stereoscope reconstructs URLs using original domain instead of tunnel
4. Subsequent requests fail because they bypass the tunnel

Expected Behavior

The registry client should:

• Preserve the effective URL from resp.Request.URL after any redirects
• Use the effective host for all subsequent registry operations
• Only fall back to reference authority if no redirects occurred

Actual Behavior

Error shows original registry hostname instead of tunnel URL:

Get "https://jfrog.lumodev.com/api/docker/mult/v2/token?scope=repository%3Atest-docker%2Falpine%3Apull&service=docker&realm=https://jfrog.lumodev.com": dial tcp: lookup jfrog.lumodev.com: no host address

When the request should have been made to: ua25m32e2lnmahwbxbo0uw7icem1vhj9.zrok.cx

Technical Details

The issue likely occurs in the registry client code where:

// PROBLEMATIC: Reconstructing from reference authority
url := fmt.Sprintf("https://%s/v2/%s/manifests/%s", ref.Registry(), ref.Repository(), ref.Tag())

// CORRECT: Preserve effective URL after redirects
effectiveURL := resp.Request.URL  // Use this for subsequent requests

Suggested Fix Areas

1. HTTP Client Configuration:

   client := &http.Client{
     CheckRedirect: func(req *http.Request, via []*http.Request) error {
       // Preserve authorization headers and effective URL
       return nil // Default behavior
     },
   }

2. URL Construction:

   // After any auth/redirect flow, use effective URL
   if effectiveHost := getEffectiveHost(lastResponse); effectiveHost != "" {
     baseURL = fmt.Sprintf("https://%s", effectiveHost)
   } else {
     baseURL = fmt.Sprintf("https://%s", ref.Registry())
   }

Environment

• Stereoscope version: v0.1.10
• Use case: JFrog Artifactory through zrok tunnel
• Client tools: Docker (works), Syft via Stereoscope (fails)
• OS: macOS

Reproduction Scenario

1. Set up registry tunnel: original.registry.com → tunnel.proxy.com
2. Docker pull works: docker pull tunnel.proxy.com/repo/image
3. Stereoscope-based tools fail, reverting to original.registry.com

Example Setup

• Original registry: jfrog.lumodev.com
• Tunnel URL: ua25m32e2lnmahwbxbo0uw7icem1vhj9.zrok.cx
• Working: docker pull ua25m32e2lnmahwbxbo0uw7icem1vhj9.zrok.cx/test-docker/alpine:3.15
• Failing: Stereoscope-based tools trying to access jfrog.lumodev.com directly

Related Standards

This aligns with OCI Distribution Spec and Docker Registry HTTP API v2, where clients should follow redirects and preserve effective URLs for subsequent requests.

Related Issues

• Related Syft issue: [will be filed separately]

[cx-shaked-karta]

Related issue:
anchore/syft#4226

[kzantow]

Hi @cx-shaked-karta, we talked about this on our livestream this week. We definitely would like to fix any issues if we're not being spec complaint in the registry provider. There are some complications to make tests for this and verify that stereoscope is actually doing the right thing, which we will want to do for any changes -- if we have a registry that's available we would somehow have to make it available only to the proxy and not to the test directly if we wanted to make sure it works properly. Maybe we could validate this with a handwritten httptest server.

I also wanted to mention the standard Go environment variables: HTTP_PROXY and HTTPS_PROXY, which Go programs support to use the configured proxies. Does this potentially provide a solution for you without making code changes?

Either way, PRs are definitely welcome! 👍

[cx-shaked-karta]

Added a PR
#461