Intermittent AmazonSQSException: The security token included in the request is invalid in EKS + IRSA with AddAWSService<IAmazonSQS>() and BackgroundService

[felixextraordinarypay]

Summary

Runtime: .NET 9 (ASP.NET Core worker services using BackgroundService)
AWS SDK for .NET: AWSSDK.SQS (and other AWS packages) – latest version
Hosting: AWS EKS, using IRSA (IAM Roles for Service Accounts)
Credential source: IRSA only (no static AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_SESSION_TOKEN in pod env)

After upgrading our applications from AWS SDK for .NET v3 to v4, our EKS background workers that use IAM Roles for Service Accounts (IRSA) started intermittently failing with:

Amazon.SQS.AmazonSQSException: The security token included in the request is invalid.
ErrorCode: InvalidClientTokenId
StatusCode: 403 (Forbidden)

Full stack trace example:

Amazon.SQS.AmazonSQSException: The security token included in the request is invalid.
ErrorCode: InvalidClientTokenId
StatusCode: Forbidden
InnerException: Amazon.Runtime.Internal.HttpErrorResponseException:
    Exception of type 'Amazon.Runtime.Internal.HttpErrorResponseException' was thrown.
...
   at Amazon.Runtime.HttpWebRequestMessage.ProcessHttpResponseMessage(HttpResponseMessage responseMessage)
   at Amazon.Runtime.HttpWebRequestMessage.GetResponseAsync(CancellationToken cancellationToken)
   at Amazon.Runtime.Internal.HttpHandler`1.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.Unmarshaller.InvokeAsync[T](IExecutionContext executionContext)
   ...
   at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext)
   ...
   at HealthNow.Workers.Common.ConsumerServiceBase.ExecuteAsync(CancellationToken stoppingToken) in .../ConsumerServiceBase.cs:line XX

This issue never occurred on v3, and no infrastructure/configuration changes were made — only NuGet updates (including .NET runtime packages and AWS SDK packages).

The application is a long-running background worker that receives messages from SQS. After running successfully for some time, SQS API calls begin to fail with the above error. Restarting the pod temporarily resolves it.

Additional Notes / Diagnostics

Our DI configuration for AWS clients looks like:

// Called during startup
services.AddDefaultAWSOptions(new AWSOptions
{
    Region = Amazon.RegionEndpoint.GetBySystemName(
        Environment.GetEnvironmentVariable("AWS_REGION") ?? "ap-southeast-2")
});

services.AddAWSService<IAmazonSQS>();

We have investigated:

✔ Verified IRSA is correctly configured

• Trust relationship, OIDC provider, sub claim, and token file are all correct.
• Pods show correct AWS_WEB_IDENTITY_TOKEN_FILE and AWS_ROLE_ARN.

✔ Verified no other credential sources

• No AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_PROFILE, or AWS config files present.

✔ Enabled AWS SDK logging

• Logs confirm that once the client enters the failing state, every SQS call is signed with credentials that AWS rejects.

✔ Scoped DI vs Singleton

Changing from:

services.AddAWSService<IAmazonSQS>();

to:

services.AddAWSService<IAmazonSQS>(ServiceLifetime.Scoped);

✔ No code changes

• The same codebase works flawlessly under AWS SDK for .NET v3.

[normj]

Can you provide the exact version number of the AWSSDK.Core that is being used? Since it is inherited dependency it might be easier to look in the *.deps.json file in your build output to see what version was resolved to.

In version 4.0.3.0 of AWSSDK.Core we pushed a fix that rework the background refreshing of credentials. If you are before that version I would start with upgrading to the latest to see if the problem continues. The 4.0.3.0 of AWSSDK.Core was released recently on November 7th 2025.

[felixextraordinarypay]

We tried a few versions. v4.0.2 at the moment. Will update to the latest now.

[felixextraordinarypay]

Looks like the upgrade to the latest version fixed the issue. thank you

[normj]

That is great to hear, thanks for the validation about the rework we did in that version.