feat(ws): add `spec.podTemplate.ports[]` to WorkspaceKind #507

harshad16 · 2025-07-29T05:10:57Z

related: #37

This PR adds spec.podTemplate.ports[] to workspaceKind CRD, which lets users include ports httpproxy setting for their workspaces.

WorkspaceKind CRD changes

spec:
  podTemplate:
    ports:
      - id: "jupyterlab"
        protocol: HTTP
        displayname: "Jupyterlab"
        httpProxy: {}

Following changes are included:

moved protocol from imageconfig.spec.ports to podtemplates.ports
included the podtemplates.ports with defaultdisplayname
add validation webhook for podtemplate.ports
update the sample workspacekind with ports reference
referencing same id for portid in imageconfig and podtemplate.ports

These changes would be consider while setting the routing for proper traffic controller/routing to the pods.

thesuperzapper · 2025-07-31T16:11:49Z

/ok-to-test

workspaces/controller/internal/webhook/suite_test.go

andyatmiami

I think we need to update the samples/ workspacekind.yaml to reflect these changes.

https://github.com/harshad16/kf-notebooks/blob/port-in-wsk-podtemplate/workspaces/controller/config/samples/jupyterlab_v1beta1_workspacekind.yaml#L140

workspaces/controller/config/crd/bases/kubeflow.org_workspacekinds.yaml

workspaces/controller/internal/controller/suite_test.go

workspaces/controller/internal/webhook/suite_test.go

workspaces/controller/api/v1beta1/workspacekind_types.go

andyatmiami · 2025-08-01T17:01:33Z

workspaces/controller/api/v1beta1/workspacekind_types.go

+	// ports that the container listens on
 	// +kubebuilder:validation:Optional
-	HTTPProxy *HTTPProxy `json:"httpProxy,omitempty"`
+	Ports []WorkspaceKindPort `json:"ports,omitempty"`


Out of curiosity - what is the practical purpose of defining a WorkspaceKind with no Ports ? Why would someone want to do that ?

This is a great question , we should rethink this 🤔

We should require at least one port, just so that the frontend does not have to deal with the possibility of a workspace with no ports.

+kubebuilder:validation:MinItems:=1

workspaces/controller/internal/webhook/workspacekind_webhook.go

andyatmiami · 2025-08-06T20:01:01Z

/ok-to-test

andyatmiami · 2025-08-06T21:06:35Z

/lgtm

testing these changes on a cluster and was able to:

create a workspacekind (using samples/)
create a workspace referencing the workspacekind (using samples/)
view the YAML representation of workspacekind and see the ports: changes

 $ kubectl get workspacekinds.kubeflow.org/jupyterlab -o yaml
apiVersion: kubeflow.org/v1beta1
kind: WorkspaceKind
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"kubeflow.org/v1beta1","kind":"WorkspaceKind","metadata":{"annotations":{},"name":"jupyterlab"},"spec":{"podTemplate":{"containerSecurityContext":{"allowPrivilegeEscalation":false,"capabilities":{"drop":["ALL"]},"runAsNonRoot":true},"culling":{"activityProbe":{"jupyter":{"lastActivity":true}},"enabled":true,"maxInactiveSeconds":86400},"extraEnv":[{"name":"NB_PREFIX","value":"{{ httpPathPrefix \"jupyterlab\" }}"}],"extraVolumeMounts":[{"mountPath":"/dev/shm","name":"dshm"}],"extraVolumes":[{"emptyDir":{"medium":"Memory"},"name":"dshm"}],"options":{"imageConfig":{"spawner":{"default":"jupyterlab_scipy_190"},"values":[{"id":"jupyterlab_scipy_180","redirect":{"message":{"level":"Info","text":"This update will change..."},"to":"jupyterlab_scipy_190"},"spawner":{"description":"JupyterLab, with SciPy Packages","displayName":"jupyter-scipy:v1.8.0","hidden":true,"labels":[{"key":"python_version","value":"3.11"}]},"spec":{"image":"ghcr.io/kubeflow/kubeflow/notebook-servers/jupyter-scipy:v1.8.0","imagePullPolicy":"IfNotPresent","ports":[{"displayName":"JupyterLab","id":"jupyterlab","port":8888,"protocol":"HTTP"}]}},{"id":"jupyterlab_scipy_190","spawner":{"description":"JupyterLab, with SciPy Packages","displayName":"jupyter-scipy:v1.9.0","labels":[{"key":"python_version","value":"3.11"}]},"spec":{"image":"ghcr.io/kubeflow/kubeflow/notebook-servers/jupyter-scipy:v1.9.0","imagePullPolicy":"IfNotPresent","ports":[{"displayName":"JupyterLab","id":"jupyterlab","port":8888,"protocol":"HTTP"}]}}]},"podConfig":{"spawner":{"default":"tiny_cpu"},"values":[{"id":"tiny_cpu","spawner":{"description":"Pod with 0.1 CPU, 128 Mb RAM","displayName":"Tiny CPU","labels":[{"key":"cpu","value":"100m"},{"key":"memory","value":"128Mi"}]},"spec":{"resources":{"requests":{"cpu":"100m","memory":"128Mi"}}}},{"id":"small_cpu","spawner":{"description":"Pod with 1 CPU, 2 GB RAM","displayName":"Small CPU","hidden":false,"labels":[{"key":"cpu","value":"1000m"},{"key":"memory","value":"2Gi"}]},"spec":{"affinity":{},"nodeSelector":{},"resources":{"requests":{"cpu":"1000m","memory":"2Gi"}},"tolerations":[]}},{"id":"big_gpu","spawner":{"description":"Pod with 4 CPU, 16 GB RAM, and 1 GPU","displayName":"Big GPU","hidden":false,"labels":[{"key":"cpu","value":"4000m"},{"key":"memory","value":"16Gi"},{"key":"gpu","value":"1"}]},"spec":{"affinity":{},"nodeSelector":{},"resources":{"limits":{"nvidia.com/gpu":1},"requests":{"cpu":"4000m","memory":"16Gi"}},"tolerations":[{"effect":"NoSchedule","key":"nvidia.com/gpu","operator":"Exists"}]}}]}},"podMetadata":{"annotations":{"my-workspace-kind-annotation":"my-value"},"labels":{"my-workspace-kind-label":"my-value"}},"ports":[{"httpProxy":{"removePathPrefix":false,"requestHeaders":{}},"portId":"jupyterlab"}],"probes":null,"securityContext":{"fsGroup":100},"serviceAccount":{"name":"default-editor"},"volumeMounts":{"home":"/home/jovyan"}},"spawner":{"deprecated":false,"deprecationMessage":"This WorkspaceKind will be removed on 20XX-XX-XX, please use another WorkspaceKind.","description":"A Workspace which runs JupyterLab in a Pod","displayName":"JupyterLab Notebook","hidden":false,"icon":{"url":"https://jupyter.org/assets/favicons/apple-touch-icon-152x152.png"},"logo":{"url":"https://upload.wikimedia.org/wikipedia/commons/3/38/Jupyter_logo.svg"}}}}
  creationTimestamp: "2025-08-06T21:02:15Z"
  finalizers:
  - notebooks.kubeflow.org/workspacekind-protection
  generation: 2
  name: jupyterlab
  resourceVersion: "31521"
  uid: 22488214-4fa7-410d-b41c-50d8624c5134
spec:
  podTemplate:
    containerSecurityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop:
        - ALL
      runAsNonRoot: true
    culling:
      activityProbe:
        jupyter:
          lastActivity: true
      enabled: true
      maxInactiveSeconds: 86400
    extraEnv:
    - name: NB_PREFIX
      value: '{{ httpPathPrefix "jupyterlab" }}'
    extraVolumeMounts:
    - mountPath: /dev/shm
      name: dshm
    extraVolumes:
    - emptyDir:
        medium: Memory
      name: dshm
    options:
      imageConfig:
        spawner:
          default: jupyterlab_scipy_190
        values:
        - id: jupyterlab_scipy_180
          redirect:
            message:
              level: Info
              text: This update will change...
            to: jupyterlab_scipy_190
          spawner:
            description: JupyterLab, with SciPy Packages
            displayName: jupyter-scipy:v1.8.0
            hidden: true
            labels:
            - key: python_version
              value: "3.11"
          spec:
            image: ghcr.io/kubeflow/kubeflow/notebook-servers/jupyter-scipy:v1.8.0
            imagePullPolicy: IfNotPresent
            ports:
            - displayName: JupyterLab
              id: jupyterlab
              port: 8888
              protocol: HTTP
        - id: jupyterlab_scipy_190
          spawner:
            description: JupyterLab, with SciPy Packages
            displayName: jupyter-scipy:v1.9.0
            hidden: false
            labels:
            - key: python_version
              value: "3.11"
          spec:
            image: ghcr.io/kubeflow/kubeflow/notebook-servers/jupyter-scipy:v1.9.0
            imagePullPolicy: IfNotPresent
            ports:
            - displayName: JupyterLab
              id: jupyterlab
              port: 8888
              protocol: HTTP
      podConfig:
        spawner:
          default: tiny_cpu
        values:
        - id: tiny_cpu
          spawner:
            description: Pod with 0.1 CPU, 128 Mb RAM
            displayName: Tiny CPU
            hidden: false
            labels:
            - key: cpu
              value: 100m
            - key: memory
              value: 128Mi
          spec:
            resources:
              requests:
                cpu: 100m
                memory: 128Mi
        - id: small_cpu
          spawner:
            description: Pod with 1 CPU, 2 GB RAM
            displayName: Small CPU
            hidden: false
            labels:
            - key: cpu
              value: 1000m
            - key: memory
              value: 2Gi
          spec:
            affinity: {}
            resources:
              requests:
                cpu: "1"
                memory: 2Gi
        - id: big_gpu
          spawner:
            description: Pod with 4 CPU, 16 GB RAM, and 1 GPU
            displayName: Big GPU
            hidden: false
            labels:
            - key: cpu
              value: 4000m
            - key: memory
              value: 16Gi
            - key: gpu
              value: "1"
          spec:
            affinity: {}
            resources:
              limits:
                nvidia.com/gpu: "1"
              requests:
                cpu: "4"
                memory: 16Gi
            tolerations:
            - effect: NoSchedule
              key: nvidia.com/gpu
              operator: Exists
    podMetadata:
      annotations:
        my-workspace-kind-annotation: my-value
      labels:
        my-workspace-kind-label: my-value
    ports:
    - httpProxy:
        removePathPrefix: false
        requestHeaders: {}
      portId: jupyterlab
    securityContext:
      fsGroup: 100
    serviceAccount:
      name: default-editor
    volumeMounts:
      home: /home/jovyan
  spawner:
    deprecated: false
    deprecationMessage: This WorkspaceKind will be removed on 20XX-XX-XX, please use
      another WorkspaceKind.
    description: A Workspace which runs JupyterLab in a Pod
    displayName: JupyterLab Notebook
    hidden: false
    icon:
      url: https://jupyter.org/assets/favicons/apple-touch-icon-152x152.png
    logo:
      url: https://upload.wikimedia.org/wikipedia/commons/3/38/Jupyter_logo.svg
status:
  podTemplateOptions:
    imageConfig:
    - id: jupyterlab_scipy_180
      workspaces: 0
    - id: jupyterlab_scipy_190
      workspaces: 1
    podConfig:
    - id: tiny_cpu
      workspaces: 1
    - id: small_cpu
      workspaces: 0
    - id: big_gpu
      workspaces: 0
  workspaces: 1

andyatmiami · 2025-09-11T14:52:49Z

/lgtm

Testing Methodology

using a kind cluster created by a script I have
- https://github.com/andyatmiami/kubeflow-notebooks/blob/experimental/remote-deployment/workspaces/develop/scripts/setup-kubeflow-notebooks.sh
➜ controller/ git:((HEAD detached at harshad/port-in-wsk-podtemplate)) $ gmake
➜ controller/ git:((HEAD detached at harshad/port-in-wsk-podtemplate)) $ gmake install
➜ controller/ git:((HEAD detached at harshad/port-in-wsk-podtemplate)) $ gmake test
➜ controller/ git:((HEAD detached at harshad/port-in-wsk-podtemplate)) $ DOCKER_BUILDKIT=0 gmake docker-build IMG=quay.io/rh-ee-astonebe/kubeflow-notebooks-v2:controller-ports-harshad-review
➜ controller/ git:((HEAD detached at harshad/port-in-wsk-podtemplate)) $ KIND_EXPERIMENTAL_PROVIDER=podman kind load docker-image quay.io/rh-ee-astonebe/kubeflow-notebooks-v2:controller-ports-harshad-review --name kubeflow
➜ controller/ git:((HEAD detached at harshad/port-in-wsk-podtemplate)) $ kubectl -n workspace-controller-system edit deployment.apps/workspace-controller-controller-manager
- replace container image with one built off this branch
➜ controller/ git:((HEAD detached at harshad/port-in-wsk-podtemplate)) $ kubectl -n workspace-controller-system get deployment.apps/workspace-controller-controller-manager -o yaml
- verify desired image in place and deployment healthy
➜ workspaces/ git:((HEAD detached at harshad/port-in-wsk-podtemplate)) $ cd controller/config/samples/common
➜ common/ git:((HEAD detached at harshad/port-in-wsk-podtemplate)) $ kubectl -n kubeflow-default-profile apply -f workspace_home_pvc.yaml
➜ common/ git:((HEAD detached at harshad/port-in-wsk-podtemplate)) $ kubectl -n kubeflow-default-profile apply -f workspace_data_pvc.yaml
➜ common/ git:((HEAD detached at harshad/port-in-wsk-podtemplate)) $ cd ~/Development/Test/notebooks-v2/manifests/harshad-ports
- directory with manifests:
  - /samples for valid workspacekind + workspace
  - modified samples/ workspacekind that has:
    - unreferenced ports element (i.e. no image using it)
    - invalid ports.id reference in imageConfig
➜ harshad-ports/ $ kubectl -n kubeflow-default-profile apply -f jupyterlab_v1beta1_workspacekind.yaml
➜ harshad-ports/ $ kubectl -n kubeflow-default-profile apply -f jupyterlab_v1beta1_workspace.yaml
➜ harshad-ports/ $ kubectl get all -n kubeflow-default-profile
- confirm pod created from Workspace CR creation

➜ harshad-ports/ $ kubectl -n kubeflow-default-profile apply -f jupyterlab_v1beta1_workspacekind-bad.yaml

confirm rejected by API server

The WorkspaceKind "jupyterlab" is invalid: spec.podTemplate.ports: Invalid value: "does-not-exist": port ID "does-not-exist" is referenced in imageConfig but not defined in ports

thesuperzapper

Thanks @harshad16 here are comments.

workspaces/backend/internal/models/workspaces/funcs.go

workspaces/backend/api/suite_test.go

thesuperzapper · 2025-09-11T20:12:03Z

workspaces/controller/api/v1beta1/workspacekind_types.go

+	// ports that the container listens on
 	// +kubebuilder:validation:Optional
-	HTTPProxy *HTTPProxy `json:"httpProxy,omitempty"`
+	Ports []WorkspaceKindPort `json:"ports,omitempty"`


We should require at least one port, just so that the frontend does not have to deal with the possibility of a workspace with no ports.

+kubebuilder:validation:MinItems:=1

workspaces/controller/internal/controller/suite_test.go

workspaces/controller/internal/controller/workspace_controller.go

workspaces/controller/internal/webhook/workspacekind_webhook.go

andyatmiami

A couple (minor) last tweaks being suggested here. However, none of them are really functionality impacting - so I will note that I also tested this code against a kind cluster (making sure I updated both the controller AND backend deployments - and I can happily confirm:

creating workspace kinds + workspaces through kubectl works
creating workspace kinds + workspaces through frontend UI also work

Additionally confirmed common negative scenarios as well as ensuring "falling back" to the DefaultDisplayName is handled appropriately.

workspaces/controller/api/v1beta1/workspacekind_types.go

workspaces/backend/internal/models/workspaces/funcs.go

workspaces/controller/internal/webhook/workspacekind_webhook.go

- moved protocal from imageconfig.spec.ports to podtemplates.ports - included the podtemplates.ports with defaultdisplayname - add validation webhook for podtemplate.ports - update the sample workspacekind with ports reference - referencing same id for portid in imageconfig and podtemplate.ports Signed-off-by: Harshad Reddy Nalla <hnalla@redhat.com>

andyatmiami · 2025-09-25T12:16:13Z

/lgtm

workspaces/backend/internal/models/workspaces/funcs.go

workspaces/controller/api/v1beta1/workspacekind_types.go

workspaces/controller/config/samples/jupyterlab_v1beta1_workspacekind.yaml

workspaces/controller/internal/webhook/workspacekind_webhook.go

- adjusted method buildService with details from ports Signed-off-by: Harshad Reddy Nalla <hnalla@redhat.com>

thesuperzapper

Small changes, then good to go.

workspaces/controller/internal/webhook/workspacekind_webhook.go

workspaces/backend/internal/models/workspaces/funcs.go

workspaces/controller/api/v1beta1/workspacekind_types.go

Signed-off-by: Harshad Reddy Nalla <hnalla@redhat.com>

thesuperzapper

@harshad16 thanks for bearing with us while we went back and forward on these reviews, great to have this merged.

/approve
/lgtm

google-oss-prow · 2025-10-02T18:30:21Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: thesuperzapper

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~workspaces/backend/OWNERS~~ [thesuperzapper]
~~workspaces/controller/OWNERS~~ [thesuperzapper]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

github-project-automation bot added this to Kubeflow Notebooks Jul 29, 2025

google-oss-prow bot added the do-not-merge/work-in-progress label Jul 29, 2025

github-project-automation bot moved this to Needs Triage in Kubeflow Notebooks Jul 29, 2025

google-oss-prow bot added area/controller area - related to controller components area/v2 area - version - kubeflow notebooks v2 labels Jul 29, 2025

google-oss-prow bot requested review from andyatmiami, kimwnasptd and thesuperzapper July 29, 2025 05:11

google-oss-prow bot added the size/L label Jul 29, 2025

harshad16 mentioned this pull request Jul 31, 2025

feat(ws): setup virtualservice to route traffic for workspaces #500

Open

harshad16 marked this pull request as ready for review July 31, 2025 05:14

google-oss-prow bot removed the do-not-merge/work-in-progress label Jul 31, 2025

google-oss-prow bot added the ok-to-test label Jul 31, 2025