Skip to content

Updating api server env to production#1398

Closed
maknop wants to merge 55 commits intoambient-code:mainfrom
RedHatInsights:api_server_flag
Closed

Updating api server env to production#1398
maknop wants to merge 55 commits intoambient-code:mainfrom
RedHatInsights:api_server_flag

Conversation

@maknop
Copy link
Copy Markdown
Contributor

@maknop maknop commented Apr 21, 2026

This change updates the environment to be production which should resolve a crashing pod in OpenShift containing the following error:

panic: runtime error: invalid memory address or nil pointer dereference

Summary by CodeRabbit

  • New Features

    • Added automated CI/CD pipeline configurations for building and testing containerized services.
    • Introduced new routes and authentication mechanisms for API and frontend services.
    • Added operator framework with custom resource definitions for session and project management.
  • Chores

    • Updated database security configurations and credentials management.
    • Enhanced service deployments with improved SSL/TLS connectivity and OAuth proxy integration.

red-hat-konflux and others added 30 commits April 6, 2026 20:22
Signed-off-by: red-hat-konflux <konflux@no-reply.konflux-ci.dev>
Signed-off-by: red-hat-konflux <konflux@no-reply.konflux-ci.dev>
Signed-off-by: red-hat-konflux <konflux@no-reply.konflux-ci.dev>
Signed-off-by: red-hat-konflux <konflux@no-reply.konflux-ci.dev>
Signed-off-by: red-hat-konflux <konflux@no-reply.konflux-ci.dev>
…d-main

Red Hat Konflux update ambient-code-backend-main
…nd-main

Red Hat Konflux update ambient-code-frontend-main
…or-main

Red Hat Konflux update ambient-code-operator-main
…-api-main

Red Hat Konflux update ambient-code-public-api-main
…t-api-server-main

Red Hat Konflux update ambient-code-ambient-api-server-main
Signed-off-by: red-hat-konflux <konflux@no-reply.konflux-ci.dev>
…nt-runner-main

Red Hat Konflux update ambient-code-ambient-runner-main
Creates kustomize overlay for deploying to hcmais01ue1 via app-interface:
- Uses Konflux images from redhat-services-prod/hcm-eng-prod-tenant
- Scales down in-cluster databases (using external RDS from app-interface Phase 2)
- Scales down MinIO (using external S3 from app-interface Phase 2)
- Includes CRDs, RBAC, routes, and all application components
- Patches operator to use Konflux runner image

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Creates kustomize overlay for deploying to hcmais01ue1 via app-interface:
- Uses Konflux images from redhat-services-prod/hcm-eng-prod-tenant
- Scales down in-cluster databases (using external RDS from app-interface Phase 2)
- Scales down MinIO (using external S3 from app-interface Phase 2)
- Includes CRDs, RBAC, routes, and all application components
- Patches operator to use Konflux runner image

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
…-overlay

Add app-interface overlay for AppSRE platform deployment
Convert kustomize overlay to OpenShift Template format for app-interface
SaaS deployment. Split into two templates:

1. template-operator.yaml (CRDs, ClusterRoles, operator deployment)
   - Operator and ambient-runner images
   - Cluster-scoped resources (CRDs, RBAC)
   - Operator deployment and its ConfigMaps

2. template-services.yaml (Application services)
   - Backend, frontend, public-api, ambient-api-server images
   - All deployments, services, routes, configmaps
   - Scales in-cluster services to 0 (minio, postgresql, unleash)

Both templates use IMAGE_TAG parameter (auto-generated from git commit SHA)
and support Konflux image gating through app-interface.

This allows app-interface to use provider: openshift-template with
proper parameter substitution instead of the directory provider which
doesn't run kustomize build.
The objects field must be a YAML array with proper list indicators.
Previous version was missing the '-' prefix on array items, causing:
'unable to decode STDIN: json: cannot unmarshal object into Go struct
field Template.objects of type []runtime.RawExtension'

Changes:
- Rebuild templates using Python yaml library for correct formatting
- Objects now properly formatted as YAML array with '- apiVersion:'
- Add validate.sh script for testing with oc process
- Both templates validated successfully

Generated from kustomize overlay output with proper YAML structure.
fix: correct OpenShift Template objects array format
Remove minio, postgresql, unleash, ambient-api-server-db.
Using external RDS and S3 from app-interface.

Removed 12 resources (4 Deployments, 4 Services, 3 PVCs, 1 Secret)
Remaining: ambient-api-server, backend-api, frontend, public-api
refactor: remove in-cluster services from template
Disables OTEL metrics export by commenting out OTEL_EXPORTER_OTLP_ENDPOINT
environment variable in operator deployment manifests.

The operator was configured to send metrics to otel-collector.ambient-code.svc:4317,
but this service does not exist in the cluster, causing repeated gRPC connection
failures every 30 seconds with error:
"failed to upload metrics: context deadline exceeded: rpc error: code = Unavailable
desc = name resolver error: produced zero addresses"

With OTEL_EXPORTER_OTLP_ENDPOINT unset, InitMetrics() will skip metrics export
and log "metrics export disabled" instead of throwing connection errors.

Changes:
- Comment out OTEL_EXPORTER_OTLP_ENDPOINT in base operator deployment
- Comment out OTEL_EXPORTER_OTLP_ENDPOINT in OpenShift template
- Add clarifying comment about re-enabling when collector is deployed

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
wcmitchell and others added 25 commits April 15, 2026 15:58
fix: disable OpenTelemetry metrics export in operator
Changes:
- Add oauth-proxy component to frontend deployment (dashboard-ui port on 8443)
- Enable SSL for ambient-api-server RDS connection (db-sslmode=require)
- Set AMBIENT_ENV to 'stage' for ambient-api-server
- Enable OpenShift service-ca for ambient-api-server TLS cert provisioning
- Regenerate templates with new oauth-proxy and api-server patches

This enables:
- Authenticated access to frontend via OpenShift OAuth
- Secure connections to external RDS database
- Automatic TLS certificate rotation for ambient-api-server

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Add OAuth proxy and SSL/TLS configuration for app-interface overlay
Remove postgresql, minio, unleash, and ambient-api-server-db resources
from the services template. These services are scaled to 0 via kustomize
patches because we use external RDS and S3 instead.

Including them in the template causes app-interface to try deploying
them, which fails imagePattern validation and wastes resources.

Excluded resources:
- Deployment/postgresql, Service/postgresql
- Deployment/minio, Service/minio, PVC/minio-data
- Deployment/unleash, Service/unleash
- Deployment/ambient-api-server-db, Service/ambient-api-server-db

Template now has 21 service resources (down from 30).

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
…ices

Remove in-cluster services from template-services.yaml
Switch from custom vault secrets to OpenShift service account-based OAuth:
- Use Red Hat's official ose-oauth-proxy-rhel9 image
- Use service account token for cookie secret (no vault needed)
- Enable HTTPS on OAuth proxy with OpenShift service-ca auto-generated certs
- Add system:auth-delegator ClusterRoleBinding for OAuth delegation
- Add OAuth redirect reference annotation to frontend ServiceAccount
- Fix service account reference from 'nginx' to 'frontend'
- Add missing NAMESPACE and UPSTREAM_TIMEOUT parameters

Benefits:
- No manual vault secret management
- Automatic TLS cert rotation via service-ca
- Standard OpenShift OAuth integration pattern
- Follows app-interface team recommendations

Files changed:
- frontend-rbac.yaml: Added OAuth annotations and auth-delegator binding
- oauth-proxy component patches: Updated to new configuration
- Templates: Regenerated with OAuth fixes (27 operator, 21 service resources)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Fix OAuth proxy configuration to use OpenShift service account auth
The RDS credentials secret should not be in the OpenShift template - it's
provided by the external resource provider (terraform) in app-interface.

The namespace's externalResources section already defines:
  - provider: rds
    output_resource_name: ambient-code-rds

This automatically creates the secret with the correct RDS credentials.
Including the secret in the template with VAULT_INJECTED placeholders
caused deployment failures.

Changes:
- Excluded ambient-code-rds secret from template generation
- Template now has 20 service resources (down from 21)
- Deployment still references the secret via volumeMount (correct)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Exclude ambient-code-rds secret from services template
Signed-off-by: Chris Mitchell <cmitchel@redhat.com>
…ination

fix: fix frontent route termination
Signed-off-by: Chris Mitchell <cmitchel@redhat.com>
fix: revert https changes for oauth pods
Changes GCP service account configuration to align with app-interface
deployment where credentials are provided via Vault.

Changes:
- template-services.yaml: Update backend vertex-credentials secret name
  from 'ambient-vertex' to 'stage-gcp-creds' (matches Vault secret)
- template-operator.yaml: Update GOOGLE_APPLICATION_CREDENTIALS path
  to match Vault secret key name 'itpc-gcp-hcm-pe-eng.json'

The secret is provided by app-interface via:
  path: engineering-productivity/ambient-code/stage-gcp-creds

This allows the backend and operator to use Vertex AI for Claude and
Gemini API calls with the service account configured with
roles/aiplatform.user permissions.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

Signed-off-by: Chris Mitchell <cmitchel@redhat.com>
Update Vertex AI credentials to use app-interface Vault secret
Configure OAuth proxy sidecar to inject authentication token into
forwarded requests, fixing 401 errors on /api/projects endpoints.

Changes:
- Add --pass-access-token=true flag to inject X-Forwarded-Access-Token header
- Change upstream from frontend-service:3000 to localhost:3000 (correct sidecar pattern)
- Remove --request-logging to reduce log noise

Backend logs showed:
  tokenSource=none hasAuthHeader=false hasFwdToken=false

The backend expects the X-Forwarded-Access-Token header, which is now
injected by the OAuth proxy for all authenticated requests.

Flow:
1. User authenticates via OpenShift OAuth ✓
2. OAuth proxy injects token header ✓ (new)
3. Frontend forwards token to backend API ✓ (fixed)

This resolves the 401 authentication errors while maintaining the
working OpenShift OAuth integration.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Fix OAuth proxy to pass access token to backend API
@netlify
Copy link
Copy Markdown

netlify Bot commented Apr 21, 2026

Deploy Preview for cheerful-kitten-f556a0 failed.

Name Link
🔨 Latest commit e56ee1b
🔍 Latest deploy log https://app.netlify.com/projects/cheerful-kitten-f556a0/deploys/69e7d31fb15f1700083c7cd4

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented Apr 21, 2026

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: b9364844-21a9-4c76-a1e2-a2ead95887a9

📥 Commits

Reviewing files that changed from the base of the PR and between b993135 and e56ee1b.

📒 Files selected for processing (50)
  • .tekton/ambient-code-ambient-api-server-main-pull-request.yaml
  • .tekton/ambient-code-ambient-api-server-main-push.yaml
  • .tekton/ambient-code-ambient-runner-main-pull-request.yaml
  • .tekton/ambient-code-ambient-runner-main-push.yaml
  • .tekton/ambient-code-backend-main-pull-request.yaml
  • .tekton/ambient-code-backend-main-push.yaml
  • .tekton/ambient-code-frontend-main-pull-request.yaml
  • .tekton/ambient-code-frontend-main-push.yaml
  • .tekton/ambient-code-operator-main-pull-request.yaml
  • .tekton/ambient-code-operator-main-push.yaml
  • .tekton/ambient-code-public-api-main-pull-request.yaml
  • .tekton/ambient-code-public-api-main-push.yaml
  • components/ambient-api-server/Dockerfile
  • components/ambient-api-server/templates/db-template.yml
  • components/manifests/README.md
  • components/manifests/base/core/ambient-api-server-service.yml
  • components/manifests/base/core/operator-deployment.yaml
  • components/manifests/base/platform/ambient-api-server-db.yml
  • components/manifests/base/platform/ambient-api-server-secrets.yml
  • components/manifests/base/rbac/frontend-rbac.yaml
  • components/manifests/components/ambient-api-server-db/ambient-api-server-db-json-patch.yaml
  • components/manifests/components/ambient-api-server-db/ambient-api-server-init-db-patch.yaml
  • components/manifests/components/ambient-api-server-db/kustomization.yaml
  • components/manifests/components/oauth-proxy/frontend-oauth-deployment-patch.yaml
  • components/manifests/components/oauth-proxy/frontend-oauth-service-patch.yaml
  • components/manifests/overlays/app-interface/ambient-api-server-db-secret-patch.yaml
  • components/manifests/overlays/app-interface/ambient-api-server-env-patch.yaml
  • components/manifests/overlays/app-interface/ambient-api-server-route.yaml
  • components/manifests/overlays/app-interface/ambient-api-server-service-ca-patch.yaml
  • components/manifests/overlays/app-interface/ambient-api-server-ssl-patch.yaml
  • components/manifests/overlays/app-interface/backend-route.yaml
  • components/manifests/overlays/app-interface/kustomization.yaml
  • components/manifests/overlays/app-interface/namespace-patch.yaml
  • components/manifests/overlays/app-interface/namespace.yaml
  • components/manifests/overlays/app-interface/operator-config-openshift.yaml
  • components/manifests/overlays/app-interface/operator-runner-image-patch.yaml
  • components/manifests/overlays/app-interface/public-api-route.yaml
  • components/manifests/overlays/app-interface/route.yaml
  • components/manifests/overlays/kind/api-server-db-security-patch.yaml
  • components/manifests/overlays/kind/api-server-no-jwt-patch.yaml
  • components/manifests/overlays/local-dev/ambient-api-server-db-credentials-patch.yaml
  • components/manifests/overlays/local-dev/ambient-api-server-db-json-patch.yaml
  • components/manifests/overlays/local-dev/ambient-api-server-init-db-patch.yaml
  • components/manifests/overlays/production/ambient-api-server-jwt-args-patch.yaml
  • components/manifests/overlays/production/ambient-api-server-migration-ssl-patch.yaml
  • components/manifests/overlays/production/kustomization.yaml
  • components/manifests/templates/template-operator.yaml
  • components/manifests/templates/template-services.yaml
  • components/manifests/templates/validate.sh
  • components/runners/ambient-runner/Dockerfile

Cache: Disabled due to data retention organization setting

Knowledge base: Disabled due to data retention organization setting


📝 Walkthrough

Walkthrough

Pull request adds comprehensive Tekton CI/CD pipelines for automated container image builds across six service components with conditional security scanning, renames the database secret to ambient-code-rds across manifests, introduces OpenShift templates for operator and service deployments, updates OAuth proxy configuration, and configures platform routing and RBAC infrastructure.

Changes

Cohort / File(s) Summary
Tekton CI/CD Pipelines
.tekton/ambient-code-*-main-{pull-request,push}.yaml (8 files)
New PipelineRun manifests for automated container builds triggered on PR and push events for API server, runner, backend, frontend, operator, and public API components. Each pipeline orchestrates repository cloning, optional dependency prefetch, buildah-based container image builds, conditional security scans (clair, SAST, ClamAV, Coverity), and artifact publishing via OCI artifacts and task result references.
Database Secret Renaming
components/manifests/base/core/ambient-api-server-service.yml, components/manifests/base/platform/ambient-api-server-db.yml, components/manifests/base/platform/ambient-api-server-secrets.yml, components/manifests/components/ambient-api-server-db/*.yaml, components/manifests/overlays/{app-interface,kind,local-dev,production}/*.yaml (13 files)
Renamed database secret from ambient-api-server-db to ambient-code-rds in Deployment secretKeyRef.name, Secret metadata, and environment variable bindings across base and overlay kustomize configurations.
OpenShift Template Definitions
components/manifests/templates/template-operator.yaml, components/manifests/templates/template-services.yaml, components/manifests/templates/validate.sh
New OpenShift Templates for operator deployment (CRDs, RBAC, operator Deployment) and full service stack (namespace, Postgres/backend PVCs, API server/backend/frontend/public-api Deployments, Routes, Services). Added validation script for template syntax.
App-Interface Overlay Manifests
components/manifests/overlays/app-interface/{ambient-api-server-*,backend-route,public-api-route,route,kustomization,namespace*,operator-config-openshift,operator-runner-image-patch}.yaml (12 files)
New overlay resources for staged platform deployment: namespace definitions, Routes for frontend/backend/public-api, SSL/TLS Service patches, environment configuration, secret definitions, Kustomize aggregation, and operator configuration with Vertex AI settings.
OAuth Proxy Configuration
components/manifests/components/oauth-proxy/frontend-oauth-{deployment-patch,service-patch}.yaml
Updated OAuth proxy from OpenShift 4.14 to RHEL9 image, switched from client-secret to service-account-based auth, added explicit TLS cert/key paths with OpenShift-managed frontend-proxy-tls secret, increased upstream timeout to 5m, adjusted liveness/readiness probe timing and scheme to HTTPS.
RBAC & Networking
components/manifests/base/rbac/frontend-rbac.yaml
Added OpenShift OAuth redirect annotation to frontend ServiceAccount and created system:auth-delegator ClusterRoleBinding for OAuth delegation flow.
Operator Configuration
components/manifests/base/core/operator-deployment.yaml
Disabled OTEL_EXPORTER_OTLP_ENDPOINT environment variable (pending OpenTel collector availability).
Build Manifest Updates
components/ambient-api-server/Dockerfile, components/runners/ambient-runner/Dockerfile
Removed vendor="Ambient" label from API server Dockerfile; changed ambient-runner build to copy entire context (.) instead of just ambient-runner/ directory to image.
Miscellaneous Configuration
components/ambient-api-server/templates/db-template.yml, components/manifests/README.md, components/manifests/overlays/production/kustomization.yaml
Updated template parameter default for DATABASE_SERVICE_NAME to ambient-code-rds; updated documentation notes; added production migration SSL patch to kustomization.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
✨ Simplify code
  • Create PR with simplified code

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@maknop maknop closed this Apr 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants