google-cloud-auth: permanent credential failure after transient network failures

[johnsonj]

Summary

The google-cloud-auth library can get into a state where authentication credentials are permanently marked as failed after transient token refresh failures (for example, DNS fails for a period of time). Once in this state, the client never recovers, even after the underlying network issue is resolved.

The actual authentication used to create the client is still valid, so it would be fine to retry, but the only workaround due to this behavior is to recreate the entire client.

This error shows up as: cannot create the authentication headers All retry attempts to fetch the token were exhausted. Subsequent calls with this credential will also fail. and future attempts will not succeed

Reproduction

This repro uses a mock GCE Metadata Service (MDS) server that proxies to real ADC credentials but with a shorter expiration time. Initially, it returns fine, but for a period of time it immediately closes the TCP connection.

gist here

[0-4s]  ✓ SUCCESS - Real GCS requests work
[5-19s] ✗ OUTAGE - Mock server drops connections
[20s+]  ✗ RECOVERY - Server works, but client never asks for tokens!

^^^ BUG: Client is permanently poisoned ^^^

Root Cause

In src/auth/src/retry.rs:

// After exhausting retries, the error is marked as non-transient
let transient = remaining > 0;
return Err(CredentialsError::new(transient, source));

The problem: once transient = false, this flag is permanent. The cached credential state is never cleared, so the library never attempts to refresh credentials again.

Workaround

Detect permanent auth failures and recreate the GCS client:

if is_permanent_auth_failure(&error) {
     client = Storage::builder().build().await?; // Recreate!
}

...
fn is_permanent_auth_failure(error: &GcsError) -> bool {
    use std::error::Error;

    if !error.is_authentication() {
        return false;
    }

    let mut source = error.source();
    while let Some(err) = source {
        if let Some(cred_err) = err.downcast_ref::<CredentialsError>() {
            return !cred_err.is_transient();
        }
        source = err.source();
    }

    false
}

Environment

• google-cloud-auth: 1.x
• google-cloud-gax: 1.4.x
• google-cloud-storage: 1.x
• Rust: 1.75+

[alvarowolfx]

@johnsonj thanks for the report and great reproduction gist. We managed to convert it to a test case (#4551) and we are discussing the best course of action to solve the issue.

The issue stems from the fact that our TokenCache implementation has a logic to refresh the token and cache it in the background, but in some scenarios, it exits it's loop (like when it exhausted retries) and gets into that poisoned state like you've describe. But also we discussing that the TokenCache can also store an expired Token indefinitely, so there are more layers to it that can be causing the issue.

We'll keep you up to date on the best solution here.