PROJQUAY-5850: feat(operator): standardized STS configuration via OLM and CCO#40
PROJQUAY-5850: feat(operator): standardized STS configuration via OLM and CCO#40quay-devel wants to merge 4 commits intoquay:mainfrom
Conversation
Design proposal for integrating the Quay operator with OpenShift's Cloud Credential Operator (CCO) CredentialRequest flow, enabling STS-based (short-lived) AWS credentials in place of static keys. Targets ROSA and OSD customers who require IAM-role-only policies. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
| - "@dmesser" | ||
| - "@doconnor" |
There was a problem hiding this comment.
Fixed — removed the Jira reporter/assignee handles which were not necessarily the document authors. Set to TBD pending the correct GitHub handles from the team.
There was a problem hiding this comment.
Fixed — removed the Jira reporter/assignee handles which were not necessarily the document authors. Set to TBD pending the correct GitHub handles from the team.
… (PROJQUAY-5850) - Replace open questions with concrete resolved answers for all four items - Fix credential flow: CCO writes a web-identity credentials file (role_arn + web_identity_token_file), not static keys; operator mounts this into Quay pods via AWS_SHARED_CREDENTIALS_FILE, avoiding any rotation reconcile loop - Fix CredentialRequest spec: add stsIAMRoleARN field (available OCP 4.14+), correct serviceAccountNames to quay-app (not operator SA), add ownerRef - Expand cluster detection: check CCO credentialsMode, OIDC issuer presence, and CredentialRequest CRD availability - not just platform type - Add full IAM permission analysis derived from static analysis of storage/cloud.py boto3 call sites (GetObject, PutObject, DeleteObject, HeadObject, ListBucket, HeadBucket, GetBucketLocation, AbortMultipartUpload, ListBucketMultipartUploads, GetBucketCors, PutBucketCors, multipart actions) with example policy document and trust policy template - Add dedicated section on unmanaged ObjectStorage: explain why the operator cannot create a scoped CredentialRequest without the bucket name, and document three user-facing alternatives (IRSA SA annotation, STSS3Storage, AWS_ROLE_ARN env var via override) - Expand RHEL guidance: document all three options (EC2 instance profile, STSS3Storage cross-account assume-role, manual web identity token) - Scope enhancement to managed ObjectStorage only; unmanaged deferred Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Remove placeholder authors derived from Jira ticket reporter/assignee; those are not necessarily the document authors. Set to TBD pending correct GitHub handles from the team. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
jbpratt
left a comment
jbpratt
left a comment
There was a problem hiding this comment.
Hmm this doesn't make sense yet.. the operator doesn't manage the aws s3 bucket (unmanaged object store) so why should it manage the credentials and what should that look like?
… (PROJQUAY-5850) Major corrections based on codebase research and OLM/CCO documentation: - Scope to unmanaged ObjectStorage only. Managed = NooBaa/ODF OBC which produces internal credentials unrelated to AWS IAM. Customers who want real AWS S3 on ROSA must set ObjectStorage: managed: false. - Fix role ARN input mechanism: use ROLEARN env var injected by OLM via Subscription spec.config.env (OCPSTRAT-171 standard), not a QuayRegistry annotation. OLM propagates ROLEARN to all operator-managed pods. - Fix CredentialRequest target: serviceAccountNames: [quay-app] not the operator SA. The quay-app pods call S3, not the operator. The operator acts as a credential broker, creating the CredentialRequest on behalf of the application SA. serviceAccountNames is a required enforcement field in CCO 4.14+ (not just metadata). - Fix credential flow: CCO produces a web-identity credentials file (role_arn + web_identity_token_file path), not static keys. Operator mounts this into quay-app pods as AWS_SHARED_CREDENTIALS_FILE. boto handles AssumeRoleWithWebIdentity transparently on each S3 call. - Note that CredentialRequest must be created at runtime (OKD docs explicitly state bundled CredentialRequests are not supported). - Drop statementEntries bucket scoping: operator does not know the bucket name for unmanaged storage; resource: '*' is used since CCO in STS mode does not create or enforce IAM policies (the actual policy is the customer's responsibility on the IAM role). - Expand cluster detection to check CCO credentialsMode (skip STS if Mint or Passthrough), CRD availability, and OIDC issuer presence. - Add warning behavior when ROLEARN is set but ObjectStorage is managed. - Correct config.yaml section: for unmanaged storage the operator does not generate storage config at all (customer provides it). Operator only ensures AWS_SHARED_CREDENTIALS_FILE is set on quay-app pods. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
| name: <quayregistry-name>-quay-app-aws | ||
| namespace: <quayregistry-namespace> | ||
| serviceAccountNames: | ||
| - quay-app |
There was a problem hiding this comment.
The cloudTokenPath is missing from the spec (w/o it, CCO cannot verify the operator's identity).
The guide ("How To Short-Lived Token Auth", step 3 for AWS) requires setting cloudTokenPath: /var/run/secrets/openshift/serviceaccount/token, so CCO knows where to find the operator pod's projected OIDC token when processing the CredentialRequest.
https://docs.google.com/document/d/1iFNpyycby_rOY1wUew-yl3uPWlE00krTgr9XHDZOTNo/edit?tab=t.0#heading=h.ugf6s9d5c2k2
serviceAccountNames:
- quay-app
cloudTokenPath: /var/run/secrets/openshift/serviceaccount/token # add this
| ### CSV Changes | ||
|
|
||
| ```yaml | ||
| features.operators.openshift.io/token-auth-aws: "true" # changed from "false" |
There was a problem hiding this comment.
If I read the guide correctly, it also mentions two additional CSV changes for the operator pod alongside the annotation:
-
A
bound-sa-tokenprojected volume mount so the operator itself has an OIDC token at/var/run/secrets/openshift/serviceaccount/token<--this is whatcloudTokenPathin theCredentialRequestpoints to. -
RBAC for
config.openshift.io/infrastructuresandoperator.openshift.io/cloudcredentials.
This volume is needed for the operator pod's cloudTokenPath directory to exist. CCO cannot validate the CredentialRequest without it.
# Add to operator Deployment in CSV:
volumeMounts:
- name: bound-sa-token
mountPath: /var/run/secrets/openshift/serviceaccount
readOnly: true
volumes:
- name: bound-sa-token
projected:
sources:
- serviceAccountToken:
path: token
audience: openshift
3 participants
Summary
Design proposal for integrating the Quay operator with the standardized OpenShift STS configuration flow using OLM and the Cloud Credential Operator (CCO)
CredentialRequestAPI (OCPSTRAT-171 / OCPSTRAT-6).The enhancement enables Quay on STS-enabled OpenShift clusters (ROSA, OSD) to authenticate to AWS object storage using short-lived STS credentials instead of static IAM keys, by delegating credential management to CCO.
What's included
enhancements/sts-olm-cco-support.md— new enhancement proposal covering:quay-operator/aws-sts-role-arnannotation onQuayRegistryCredentialRequestCR lifecycle (creation, watching, deletion)controllers/quay/features.goandpkg/kustomize/secrets.gofor STS credential injectiontoken-auth-aws: "true") and new RBAC rules for CCO APIsDegradedcondition when CCO fails to reconcileAWS_ROLE_ARN/ Web Identity Token file)Related
🤖 Generated with Claude Code