Release3.0 and Kubernetes v1.23

[drvgautam]

Dear Alexandre,
I needed to migrate my abcdesktop deployment from a DigitalOcean cluster to a Openstack-magnum based cluster with Kubernetes COE. Unfortunately, my infrastructure does not support cluster templates with K8 version =>1.24. So, I have only K8-v1.23. Sadly, the Openstack-Magnum is not known for keeping up with the kubernetes versions.

Based on (https://www.abcdesktop.io/2.0/features/), It is not very explicitly clear to me if I can deploy version3.0 on K8-v1.23 or not. Would you please clarify this?

best regards,
Vinay

[alexandredevely]

Hello Vinay,

In release 2.0 an application runs as docker container or as kubernetes pod. So the pyos need the socket to dockerd.
In release 3.0 the dependancy of dockerd has been remove, and all you need is a kubernetes cluster. An application runs as ephemeral container or as pod. Why did i change that ? because dockershim is removed by google https://kubernetes.io/blog/2020/12/02/dockershim-faq/

about your case :
Ephemeral Containers are available as a beta feature in Kubernetes 1.23, it should work.

For you information https://endoflife.date/kubernetes
Kubernetes 1.24 Ended (28 Jul 2023)

See you,

Alexandre

[drvgautam]

Hi Alexandre,

Thanks for the note. I tried deploying the abcdesktop with the openstack-magnum-k8 and it seems working, except that the nginx-od pod is repeatedly failing.

loginansatt01:~/abc-promo$ kubectl get pods -n abcdesktop
NAME                           READY   STATUS             RESTARTS       AGE
memcached-od-78578c879-bb8qq   1/1     Running            0              164m
mongodb-od-5b4dd4765f-ptw2j    1/1     Running            0              164m
nginx-od-788c97cdc9-b4gbq      0/1     CrashLoopBackOff   36 (57s ago)   164m
openldap-od-65759b74dc-tbvfg   1/1     Running            0              164m
pyos-od-7d5d9457cf-jw6nk       1/1     Running            0              164m
speedtest-od-c94b56c88-48cvq   1/1     Running            0              164m

The log read (attached below) shows an error cp: cannot create regular file '/etc/nginx/sites-enabled/default': Read-only file system. I checked the manifest file abcdesktop-3.0.yaml and mounted container path below. I understand that the filetype is set as readOnly:true, which is standard practice to minimize the attack surface for containerized workloads in Kubernetes.

- name: default-config
            mountPath: /etc/nginx/sites-enabled/default
            subPath: default
            readOnly: true

I could not resolve this issue, and I also remember that there was no such issue in the DO deployment. The logs of nginx-od are attached below. I would appreciate if you can see what's wrong here.

loginansatt01:~/abc-promo$ kubectl logs nginx-od-788c97cdc9-b4gbq -n abcdesktop
Resolver for nginx is 10.254.0.10
New site configuration
cp: cannot create regular file '/etc/nginx/sites-enabled/default': Read-only file system
lua_package_path "/usr/local/share/lua/5.1/?.lua;;";
types {
    # Web fonts
    application/font-woff2               woff2;
    application/-font-ttf                ttc ttf;
    font/opentype                        otf;
}
server {
    resolver 'kube-dns.kube-system.svc.cluster.local';
    set $my_speedtest 'speedtest.abcdesktop.svc.cluster.local';
    set $my_proxy 'pyos.abcdesktop.svc.cluster.local';
    listen 80;
    server_name _;
    root /var/webModules;
    index index.html index.htm;
    # default abcdesktop.io oc.user tcp port
    set $pulseaudio_http_port               4714;
    set $ws_tcp_bridge_tcp_port             6081;
    set $api_service_tcp_port               8000;
    set $filemanager_bridge_tcp_port        29780;
    set $xterm_tcp_port                     29781;
    set $printerfile_service_tcp_port       29782;
    set $file_service_tcp_port              29783;
    set $broadcast_tcp_port                 29784;
    set $lync_service_tcp_port              29785;
    set $spawner_service_tcp_port           29786;
    set $janus_service_tcp_port		29787; 
    # uncomment to use env var
    # set_by_lua  $filemanager_bridge_tcp_port 'return os.getenv("FILEMANAGER_BRIDGE_TCP_PORT")';
    # set_by_lua  $broadcast_tcp_port 'return os.getenv("BROADCAST_SERVICE_TCP_PORT")';
    # set_by_lua  $ws_tcp_bridge_tcp_port 'return os.getenv("WS_TCP_BRIDGE_SERVICE_TCP_PORT")';
    # set_by_lua  $spawner_service_tcp_port 'return os.getenv("SPAWNER_SERVICE_TCP_PORT")';
    # set_by_lua  $xterm_tcp_port 'return os.getenv("XTERM_TCP_PORT")';
    # set_by_lua  $file_service_tcp_port 'return os.getenv("FILE_SERVICE_TCP_PORT")';
    # set_by_lua  $pulseaudio_http_port 'return os.getenv("PULSEAUDIO_HTTP_PORT")';
    location /nstatus {
             # allow 127.0.0.1;
             # deny all;
             stub_status;
    }
    
    include route.conf;
}
running standart configuration file

[alexandredevely]

Hi Vinay,

I agree with your security point of view.
The nginx configuration is generated to update the resolver ip address using a simple sed command

	RESOLVER=$(grep -m 1 nameserver /etc/resolv.conf | awk '{ print $2 }')
	echo Resolver for nginx is $RESOLVER
	cp /etc/nginx/sites-enabled/default /tmp/default
        sed -i "s/127.0.0.11/$RESOLVER/g" /tmp/default
	# sed -i "s/pyos/pyos.abcdesktop.svc.cluster.local./g" /tmp/default
	echo New site configuration
	cp  /tmp/default /etc/nginx/sites-enabled/default

Out-of-the-box, Nginx doesn't support using environment variables inside most configuration blocks.
I will try to use the set_by_lua feature to build the nginx template config

Thank you for you secure point of view

See you

Alexandre

[alexandredevely]

Hi Vinay,

I agree with this issue. If we define the nginx config file as a configmap, we don't need to update it.
I've fixed that point in abcdesktopio/oc.nginx:3.0 and abcdesktopio/oc.nginx:3.1 images

Please restart the nginx pod or deployment

kubectl rollout restart  deployment nginx-od -n abcdesktop

Let me know if this new image fix your issue

Thank you again for your message

best regards,

Alexandre

[drvgautam]

Hi Alexandre,

Thanks for looking into the issue. I do not see the previous error after restarting, but the nginx-pod restarts after every minute are so. The related pod's description and information is attached below. The nginx pod has restarted thrice over a period of 13 minutes.

memcached-od-78578c879-67t5v   1/1     Running   0               13m
mongodb-od-5b4dd4765f-dv8kl    1/1     Running   0               13m
nginx-od-788c97cdc9-jvt8d      1/1     Running   3 (4m44s ago)   13m
openldap-od-65759b74dc-5fgtg   1/1     Running   0               13m
pyos-od-7d5d9457cf-cdg9j       1/1     Running   0               13m
speedtest-od-c94b56c88-f7rtr   1/1     Running   0               13m

``
loginansatt01:~/abc-promo$ kubectl describe pod nginx-od-788c97cdc9-jvt8d -n abcdesktop
Name:             nginx-od-788c97cdc9-jvt8d
Namespace:        abcdesktop
Priority:         0
Service Account:  default
Node:             abc-promo-pr5do2wkkd56-node-0/10.0.0.145
Start Time:       Tue, 10 Oct 2023 14:55:42 +0200
Labels:           name=nginx-od
                  netpol/dns=true
                  netpol/memcached=true
                  netpol/ocuser=true
                  netpol/pyos=true
                  netpol/speedtest=true
                  pod-template-hash=788c97cdc9
                  run=nginx-od
                  type=frontend
Annotations:      cni.projectcalico.org/containerID: 8acd46f6080ec587237d30d7ba51cee3f176505cb966c8095f564fac9ce13e10
                  cni.projectcalico.org/podIP: 10.100.113.201/32
                  cni.projectcalico.org/podIPs: 10.100.113.201/32
                  kubernetes.io/psp: magnum.privileged
Status:           Running
IP:               10.100.113.201
IPs:
  IP:           10.100.113.201
Controlled By:  ReplicaSet/nginx-od-788c97cdc9
Containers:
  nginx:
    Container ID:   containerd://19908bf13a2b3960ae27fc48fd5e809ff03415d305909da5649c8aee34c5f615
    Image:          abcdesktopio/oc.nginx:3.0
    Image ID:       docker.io/abcdesktopio/oc.nginx@sha256:36615278cfdf45a278159b51876ecd4cdbbea73618d0ed6d80b87cb2e4346c4e
    Ports:          80/TCP, 443/TCP
    Host Ports:     0/TCP, 0/TCP
    State:          Running
      Started:      Tue, 10 Oct 2023 15:02:22 +0200
    Last State:     Terminated
      Reason:       Error
      Exit Code:    137
      Started:      Tue, 10 Oct 2023 14:59:46 +0200
      Finished:     Tue, 10 Oct 2023 15:02:13 +0200
    Ready:          True
    Restart Count:  2
    Limits:
      cpu:     2
      memory:  512Mi
    Requests:
      cpu:     250m
      memory:  64Mi
    Liveness:  http-get http://:80/ delay=0s timeout=1s period=10s #success=1 #failure=1
    Startup:   http-get http://:80/ delay=0s timeout=1s period=10s #success=1 #failure=5
    Environment:
      JWT_DESKTOP_PAYLOAD_PRIVATE_KEY:  /config.payload/abcdesktop_jwt_desktop_payload_private_key.pem
      JWT_DESKTOP_SIGNING_PUBLIC_KEY:   /config.signing/abcdesktop_jwt_desktop_signing_public_key.pem
      NODE_NAME:                         (v1:spec.nodeName)
      POD_NAME:                         nginx-od-788c97cdc9-jvt8d (v1:metadata.name)
      POD_NAMESPACE:                    abcdesktop (v1:metadata.namespace)
      POD_IP:                            (v1:status.podIP)
    Mounts:
      /config.payload from jwtpayloadkeys (ro)
      /config.signing from jwtsigningkeys (ro)
      /etc/nginx/sites-enabled/default from default-config (ro,path="default")
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-rpzmg (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  jwtsigningkeys:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  abcdesktopjwtdesktopsigning
    Optional:    false
  jwtpayloadkeys:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  abcdesktopjwtdesktoppayload
    Optional:    false
  default-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      nginx-config
    Optional:  false
  kube-api-access-rpzmg:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  8m31s                 default-scheduler  Successfully assigned abcdesktop/nginx-od-788c97cdc9-jvt8d to abc-promo-pr5do2wkkd56-node-0
  Normal   Pulled     8m29s                 kubelet            Successfully pulled image "abcdesktopio/oc.nginx:3.0" in 1.258776329s (1.258805854s including waiting)
  Normal   Pulled     4m28s                 kubelet            Successfully pulled image "abcdesktopio/oc.nginx:3.0" in 1.669156359s (1.669175252s including waiting)
  Normal   Pulling    2m (x3 over 8m30s)    kubelet            Pulling image "abcdesktopio/oc.nginx:3.0"
  Normal   Pulled     114s                  kubelet            Successfully pulled image "abcdesktopio/oc.nginx:3.0" in 6.096894509s (6.096910502s including waiting)
  Normal   Created    112s (x3 over 8m29s)  kubelet            Created container nginx
  Normal   Started    112s (x3 over 8m29s)  kubelet            Started container nginx
  Warning  Unhealthy  36s (x3 over 5m7s)    kubelet            Liveness probe failed: Get "http://10.100.113.201:80/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
  Normal   Killing    36s (x3 over 5m7s)    kubelet            Container nginx failed liveness probe, will be restarted

[drvgautam]

The following may or may not be a related issue, but the logs of pyos pod also show a PodSecurityPolicy error and prevent from login either anonymously or ladap-dummy-names.

A trailing end of the logs read is attached below.

loginansatt01:~/abc-promo$ kubectl logs pyos-od-7d5d9457cf-cdg9j -n abcdesktop
2023-10-10 13:15:01 rest [DEBUG  ] kubernetes.client.rest.request:ae2108e2-70e8-461c-a1d5-7000bae6ccef response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods \"anonymous-0634f\" is forbidden: PodSecurityPolicy: unable to admit pod: []","reason":"Forbidden","details":{"name":"anonymous-0634f","kind":"pods"},"code":403}

2023-10-10 13:15:01 orchestrator [ERROR  ] oc.od.orchestrator.ODOrchestratorKubernetes.createdesktop:ae2108e2-70e8-461c-a1d5-7000bae6ccef (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': 'bc76bd68-d508-4beb-8b83-ac3a51eb6e10', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '6379e6a8-db71-41bd-bc37-03cc81a549cf', 'X-Kubernetes-Pf-Prioritylevel-Uid': '20c71495-2702-473e-b00d-27b0e473a170', 'Date': 'Tue, 10 Oct 2023 13:15:01 GMT', 'Content-Length': '246'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods \"anonymous-0634f\" is forbidden: PodSecurityPolicy: unable to admit pod: []","reason":"Forbidden","details":{"name":"anonymous-0634f","kind":"pods"},"code":403}


2023-10-10 13:15:01 composer [DEBUG  ] oc.od.composer.on_desktoplaunchprogress_info:ae2108e2-70e8-461c-a1d5-7000bae6ccef 
2023-10-10 13:15:01 composer [ERROR  ] oc.od.composer.opendesktop:ae2108e2-70e8-461c-a1d5-7000bae6ccef Cannot create a new desktop return desktop=e.Create pod failed Forbidden {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods \"anonymous-0634f\" is forbidden: PodSecurityPolicy: unable to admit pod: []","reason":"Forbidden","details":{"name":"anonymous-0634f","kind":"pods"},"code":403}

2023-10-10 13:15:01 od [INFO   ] __main__.trace_response:ae2108e2-70e8-461c-a1d5-7000bae6ccef /composer/launchdesktop b'{"status": 500, "result": null, "error": "e.Create pod failed Forbidden {\\"kind\\":\\"Status\\",\\"apiVersion\\":\\"v1\\",\\"metadata\\":{},\\"status\\":\\"Failure\\",\\"message\\":\\"pods \\\\\\"anonymous-0634f\\\\\\" is forbidden: PodSecurityPolicy: unable to admit pod: []\\",\\"reason\\":\\"Forbidden\\",\\"details\\":{\\"name\\":\\"anonymous-0634f\\",\\"kind\\":\\"pods\\"},\\"code\\":403}\\n"}'
2023-10-10 13:15:04 od [INFO   ] __main__.trace_request:anonymous /healthz
2023-10-10 13:15:14 od [INFO   ] __main__.trace_request:anonymous /healthz
2023-10-10 13:15:24 od [INFO   ] __main__.trace_request:anonymous /healthz
2023-10-10 13:15:34 od [INFO   ] __main__.trace_request:anonymous /healthz
2023-10-10 13:15:44 od [INFO   ] __main__.trace_request:anonymous /healthz

[alexandredevely]

Hi Vinay,

Nginx works fine.
A pod Security Policy has denied create pod call.
Could you send me more details than PodSecurityPolicy: unable to admit pod: []","reason":"Forbidden" ?
In most case, the error come from hostPath usage.

Alexandre

[alexandredevely]

Hello Vinay,

Could you please update the od.config file to set the option desktop.homedirectorytype to None

# desktop.homedirectorytype: 'persistentVolumeClaim'
# desktop.homedirectorytype: 'hostPath'
# desktop.homedirectorytype: None
desktop.homedirectorytype: None

Set desktop.homedirectorytype to None to create volume as emptyDir

Then, please update the config map abcdesktop-config and restart pyos pod.

kubectl create -n abcdesktop configmap abcdesktop-config --from-file=od.config -o yaml --dry-run=client | kubectl replace -n abcdesktop -f -
kubectl delete pods -l run=pyos-od -n abcdesktop

I hope that it will pass the kubernetes's pod security admission.

See you

Alexandre