-
Notifications
You must be signed in to change notification settings - Fork 98
feat(runner,manifests): alpha migration PR 6+7 — runners and Kustomize overlays #1379
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
f99f331
c586206
fdf839c
31269ff
73fee5f
f240cd9
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,55 @@ | ||
| --- | ||
| apiVersion: apps/v1 | ||
| kind: Deployment | ||
| metadata: | ||
| name: ambient-control-plane | ||
| labels: | ||
| app: ambient-control-plane | ||
| spec: | ||
| replicas: 1 | ||
| selector: | ||
| matchLabels: | ||
| app: ambient-control-plane | ||
| template: | ||
| metadata: | ||
| labels: | ||
| app: ambient-control-plane | ||
| spec: | ||
| serviceAccountName: ambient-control-plane | ||
| securityContext: | ||
| runAsNonRoot: true | ||
| seccompProfile: | ||
| type: RuntimeDefault | ||
| containers: | ||
| - name: ambient-control-plane | ||
| image: quay.io/ambient_code/vteam_control_plane:latest | ||
| imagePullPolicy: Always | ||
| securityContext: | ||
| allowPrivilegeEscalation: false | ||
| readOnlyRootFilesystem: true | ||
| capabilities: | ||
| drop: ["ALL"] | ||
| env: | ||
| - name: AMBIENT_API_TOKEN | ||
| valueFrom: | ||
| secretKeyRef: | ||
| name: ambient-control-plane-token | ||
| key: token | ||
| - name: AMBIENT_API_SERVER_URL | ||
| value: "https://ambient-api-server.ambient-code.svc:8000" | ||
| - name: AMBIENT_GRPC_SERVER_ADDR | ||
| value: "ambient-api-server.ambient-code.svc:9000" | ||
| - name: AMBIENT_GRPC_USE_TLS | ||
| value: "true" | ||
| - name: MODE | ||
| value: "kube" | ||
| - name: LOG_LEVEL | ||
| value: "info" | ||
| resources: | ||
| requests: | ||
| cpu: 50m | ||
| memory: 64Mi | ||
| limits: | ||
| cpu: 200m | ||
| memory: 256Mi | ||
| restartPolicy: Always | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,27 @@ | ||
| apiVersion: rbac.authorization.k8s.io/v1 | ||
| kind: ClusterRole | ||
| metadata: | ||
| name: ambient-control-plane | ||
| rules: | ||
| # AgenticSession custom resources (full lifecycle management) | ||
| - apiGroups: ["vteam.ambient-code"] | ||
| resources: ["agenticsessions"] | ||
| verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] | ||
| - apiGroups: ["vteam.ambient-code"] | ||
| resources: ["agenticsessions/status"] | ||
| verbs: ["update", "patch"] | ||
| # Namespaces (create and label per-project namespaces) | ||
| - apiGroups: [""] | ||
| resources: ["namespaces"] | ||
| verbs: ["get", "list", "watch", "create", "update", "patch"] | ||
| # RoleBindings (reconcile group access from ProjectSettings) | ||
| - apiGroups: ["rbac.authorization.k8s.io"] | ||
| resources: ["rolebindings"] | ||
| verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] | ||
| # Session runner resources (provision/deprovision per-session workloads in project namespaces) | ||
| - apiGroups: [""] | ||
| resources: ["secrets", "serviceaccounts", "services", "pods"] | ||
| verbs: ["get", "list", "watch", "create", "update", "patch", "delete", "deletecollection"] | ||
|
Comment on lines
+22
to
+24
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Cluster-wide secret/service mutation is over-privileged Lines 22–24 grant create/update/delete (plus read/list/watch) on As per coding guidelines, 🧰 Tools🪛 Trivy (0.69.3)[error] 22-24: Manage secrets ClusterRole 'ambient-control-plane' shouldn't have access to manage resource 'secrets' Rule: KSV-0041 (IaC/Kubernetes) [error] 22-24: Manage Kubernetes networking ClusterRole 'ambient-control-plane' should not have access to resources ["services", "endpoints", "endpointslices", "networkpolicies", "ingresses"] for verbs ["create", "update", "patch", "delete", "deletecollection", "impersonate", "*"] Rule: KSV-0056 (IaC/Kubernetes) 🤖 Prompt for AI Agents |
||
| - apiGroups: ["batch"] | ||
| resources: ["jobs"] | ||
| verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,12 +1,12 @@ | ||
| apiVersion: rbac.authorization.k8s.io/v1 | ||
| kind: ClusterRoleBinding | ||
| metadata: | ||
| name: readonly-admin-cluster-reader | ||
| name: ambient-control-plane | ||
| roleRef: | ||
| apiGroup: rbac.authorization.k8s.io | ||
| kind: ClusterRole | ||
| name: cluster-reader | ||
| name: ambient-control-plane | ||
| subjects: | ||
| - kind: ServiceAccount | ||
| name: readonly-admin | ||
| namespace: ambient-code | ||
| - kind: ServiceAccount | ||
| name: ambient-control-plane | ||
| namespace: ambient-code |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,14 @@ | ||
| apiVersion: v1 | ||
| kind: ServiceAccount | ||
| metadata: | ||
| name: ambient-control-plane | ||
| namespace: ambient-code | ||
| --- | ||
| apiVersion: v1 | ||
| kind: Secret | ||
| metadata: | ||
| name: ambient-control-plane-token | ||
| namespace: ambient-code | ||
| annotations: | ||
| kubernetes.io/service-account.name: ambient-control-plane | ||
| type: kubernetes.io/service-account-token |
This file was deleted.
This file was deleted.
This file was deleted.
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -19,3 +19,5 @@ spec: | |
| value: "IfNotPresent" | ||
| - name: POD_FSGROUP | ||
| value: "0" | ||
| - name: RUNNER_LOG_LEVEL | ||
| value: "debug" | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,101 @@ | ||
| # MPP OpenShift Overlay | ||
|
|
||
| Kustomize overlay for the Managed Platform Plus (MPP) OpenShift environment: `ambient-code--runtime-int`. | ||
|
|
||
| ## Apply | ||
|
|
||
| ```bash | ||
| kubectl apply -k components/manifests/overlays/mpp-openshift/ | ||
| ``` | ||
|
|
||
| ## What This Overlay Does | ||
|
|
||
| - Targets namespace `ambient-code--runtime-int` | ||
| - Sets `PLATFORM_MODE=mpp` so the CP uses `MPPNamespaceProvisioner` (namespaces as `ambient-code--<project>`) | ||
| - Configures OIDC client credentials auth (no static K8s SA token) | ||
| - Adds `--grpc-jwk-cert-url` so the api-server validates RH SSO tokens on gRPC | ||
| - Mounts `tenantaccess-ambient-control-plane-token` for the CP's project kube client | ||
| - Mounts `ambient-runner-api-token` for runner pods to authenticate as service callers on gRPC | ||
| - Adds `allow-ambient-tenant-ingress` NetworkPolicy (ports 8000/9000 from all `ambient-code` tenant namespaces) | ||
|
|
||
| ## ⚠️ One-Time Manual Bootstrap | ||
|
|
||
| Two secrets must be created manually once per cluster. They are **not** managed by kustomize (to avoid committing secret values) and are **not** required per session — only per cluster. | ||
|
|
||
| ### Step A — TenantServiceAccount | ||
|
|
||
| Grants the CP's service account `namespace-admin` in every current and future tenant namespace via the tenant-access-operator. | ||
|
|
||
| ```bash | ||
| # Apply the TenantServiceAccount CR to ambient-code--config (NOT via kustomize) | ||
| kubectl apply -f components/manifests/overlays/mpp-openshift/ambient-cp-tenant-sa.yaml | ||
| ``` | ||
|
|
||
| Wait ~30s for the operator to create `tenantaccess-ambient-control-plane-token` in `ambient-code--config`, then copy it to the runtime namespace: | ||
|
|
||
| ```bash | ||
| kubectl get secret tenantaccess-ambient-control-plane-token \ | ||
| -n ambient-code--config \ | ||
| -o json \ | ||
| | python3 -c " | ||
| import json, sys | ||
| s = json.load(sys.stdin) | ||
| del s['metadata']['namespace'] | ||
| del s['metadata']['resourceVersion'] | ||
| del s['metadata']['uid'] | ||
| del s['metadata']['creationTimestamp'] | ||
| s['metadata'].pop('ownerReferences', None) | ||
| s['metadata'].pop('annotations', None) | ||
| s['type'] = 'Opaque' | ||
| print(json.dumps(s)) | ||
| " | kubectl apply -n ambient-code--runtime-int -f - | ||
| ``` | ||
|
|
||
| **Effect:** The operator automatically injects a `namespace-admin` RoleBinding into every `ambient-code--*` namespace, including ones created after this step. The CP mounts this token as its `projectKube` client for all namespace-scoped operations. | ||
|
|
||
| ### Step B — Static Runner API Token | ||
|
|
||
| The runner uses a static token to authenticate as a gRPC service caller, bypassing the per-user session ownership check on `WatchSessionMessages`. | ||
|
|
||
| ```bash | ||
| # Generate a random token — record this value; you will need it for Step C | ||
| STATIC_TOKEN=$(python3 -c "import secrets; print(secrets.token_urlsafe(32))") | ||
|
|
||
| kubectl create secret generic ambient-runner-api-token \ | ||
| --from-literal=token=${STATIC_TOKEN} \ | ||
| -n ambient-code--runtime-int | ||
| ``` | ||
|
|
||
| **Do not commit the token value.** | ||
|
|
||
| ### Step C — Set AMBIENT_API_TOKEN on the api-server | ||
|
|
||
| The api-server must know the static token so it can recognise the runner as a service caller: | ||
|
|
||
| ```bash | ||
| # Patch the api-server args to include the token file | ||
| # (or set AMBIENT_API_TOKEN directly if your deployment supports it) | ||
| # The token value must match what was set in Step B | ||
| ``` | ||
|
|
||
| > **Note:** Step C is currently pending implementation — see the open gap `WatchSessionMessages PERMISSION_DENIED` in `docs/internal/design/control-plane.guide.md`. | ||
|
|
||
| ## Files in This Overlay | ||
|
|
||
| | File | Purpose | | ||
| |------|---------| | ||
| | `kustomization.yaml` | Root kustomize config; sets namespace, images, patches | | ||
| | `ambient-control-plane.yaml` | CP Deployment — OIDC env, `PROJECT_KUBE_TOKEN_FILE`, project-kube volume mount | | ||
| | `ambient-api-server.yaml` | api-server Deployment base | | ||
| | `ambient-api-server-args-patch.yaml` | api-server command args — db, grpc, OIDC JWKS URL | | ||
| | `ambient-api-server-service-ca-patch.yaml` | Service CA annotation for TLS | | ||
| | `ambient-api-server-db.yaml` | PostgreSQL Deployment + Service | | ||
| | `ambient-api-server-route.yaml` | OpenShift Route for external access | | ||
| | `ambient-control-plane-sa.yaml` | ServiceAccount for the CP | | ||
| | `ambient-control-plane-rbac.yaml` | RBAC for the CP SA | | ||
| | `ambient-tenant-ingress-netpol.yaml` | NetworkPolicy allowing runner→api-server traffic | | ||
| | `ambient-cp-tenant-sa.yaml` | TenantServiceAccount CR (applied manually — see Step A) | | ||
|
|
||
| ## Re-Bootstrap Required? | ||
|
|
||
| Only if `ambient-code--runtime-int` is destroyed, which MPP should never do to runtime/config namespaces. Session namespaces (`ambient-code--<project>`) are created and destroyed per session with no manual action required. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,46 @@ | ||
| apiVersion: apps/v1 | ||
| kind: Deployment | ||
| metadata: | ||
| name: ambient-api-server | ||
| spec: | ||
| template: | ||
| spec: | ||
| containers: | ||
| - name: api-server | ||
| command: | ||
| - /usr/local/bin/ambient-api-server | ||
| - serve | ||
| - --db-host-file=/secrets/db/db.host | ||
| - --db-port-file=/secrets/db/db.port | ||
| - --db-user-file=/secrets/db/db.user | ||
| - --db-password-file=/secrets/db/db.password | ||
| - --db-name-file=/secrets/db/db.name | ||
| - --enable-authz=false | ||
| - --enable-https=false | ||
| - --api-server-bindaddress=:8000 | ||
| - --metrics-server-bindaddress=:4433 | ||
| - --health-check-server-bindaddress=:4434 | ||
| - --db-sslmode=disable | ||
| - --db-max-open-connections=50 | ||
| - --enable-db-debug=false | ||
| - --enable-metrics-https=false | ||
| - --http-read-timeout=5s | ||
| - --http-write-timeout=30s | ||
| - --cors-allowed-origins=* | ||
| - --cors-allowed-headers=X-Ambient-Project | ||
| - --jwk-cert-file=/configs/authentication/jwks.json | ||
| - --enable-grpc=true | ||
| - --grpc-server-bindaddress=:9000 | ||
| - --grpc-enable-tls=true | ||
| - --grpc-tls-cert-file=/etc/tls/tls.crt | ||
| - --grpc-tls-key-file=/etc/tls/tls.key | ||
| - --alsologtostderr | ||
| - -v=4 | ||
| volumeMounts: | ||
| - name: tls-certs | ||
| mountPath: /etc/tls | ||
| readOnly: true | ||
| volumes: | ||
| - name: tls-certs | ||
| secret: | ||
| secretName: ambient-api-server-tls |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🧩 Analysis chain
🏁 Script executed:
Repository: ambient-code/platform
Length of output: 1164
Add
metadata.namespace: ambient-codeto DeploymentThis Deployment references
serviceAccountName: ambient-control-planeand secretambient-control-plane-token(lines 18, 36), both of which exist in namespaceambient-code. Without an explicit namespace in the Deployment metadata and no namespace transformer in kustomization, pod startup will fail with service account resolution errors. Addnamespace: ambient-codeto lines 4–7 metadata block.Also applies to: 33-37
🤖 Prompt for AI Agents