Missing note about authenticating to ghcr.io

[ulrikpedersen]

❤️ the new (feb 24) version of the tutorials! Working my way through the getting started, I found that deploying the first IOC locally fails:

(ec-venv) ukp@osloxf04:~/EPICS/bl01t$ ec deploy-local services/bl01t-ea-test-01
Deploy TEMPORARY version 2024.3.cb1f-b from /home/ukp/EPICS/bl01t/services/bl01t-ea-test-01 to the local docker instance
Are you sure ? [y/N]: y
Unable to find image 'ghcr.io/epics-containers/ioc-adsimdetector-linux-runtime:2024.2.1' locally
docker: Error response from daemon: Head "https://ghcr.io/v2/epics-containers/ioc-adsimdetector-linux-runtime/manifests/2024.2.1": denied: denied.
See 'docker run --help'.

Command Failed:
docker run -dit --net host --restart unless-stopped -l is_IOC=true -l version=2024.3.cb1f-b -v bl01t-ea-test-01_config:/epics/ioc/config/  --name bl01t-ea-test-01 ghcr.io/epics-containers/ioc-adsimdetector-linux-runtime:2024.2.1

It is because I haven't authenticated to ghcr.io (worked when I did login).

Instructions/links on how to authenticate to ghcr.io with docker on commandline, including information about what kind of github authentication token is required and what permissions are needed would be really helpful on https://epics-containers.github.io/main/tutorials/deploy_example.html

[gilesknap]

Welcome back!

I think that you should not need to be authenticated as that should be public. It is showing as public in the registry. I wonder if it something to do with your user ssh key autorisation settings?

[gilesknap]

But reading around it seems that SSH keys always give you full access and I don't think SSH keys are coming into play here anyway.

[gilesknap]

Everyone else who has tried this (using podman on RHEL8) has had no authentication issues.

This is me - at home using docker on ubuntu:

giles@ws1: ~/work/bl01t main
$ . ./environment.sh                                                                                                                                                                              [19:17:42]
Loading environment for beamline bl01t IOC Instances and Services ...
(venv)  
giles@ws1: ~/work/bl01t main
$ ec deploy-local services/bl01t-ea-test-01                                                                                                                                                       [19:17:45]
Deploy TEMPORARY version 2024.3.10f72-b from /home/giles/work/bl01t/services/bl01t-ea-test-01 to the local docker instance
Are you sure ? [y/N]: y 
2347ed50d8923d8bc54d78a006dc73c41ddeaa6a599c18aada586ce76545d40d
(venv)  
giles@ws1: ~/work/bl01t main

[gilesknap]

@ulrikpedersen one more thing. I have been getting intermittent errors from GitHub packages when trying to pull helm charts today. They look like auth errors. Repeating the same command then succeeds.

Perhaps it was just a transient server glitch? Can you try again and see if it works?

[ulrikpedersen]

That is very strange indeed. So last week I couldn't pull that image - I tried several times with the same error and it didnt' feel like a blip. Then I created a classic token on github and did a docker login to ghcr.io with that token - and then I could pull the image.

Now I have just retried: first I logged out of ghcr:

(ec-venv) ukp@osloxf04:~/EPICS/bl01t$ docker logout ghcr.io
Removing login credentials for ghcr.io

And now I could pull the image without authentication!

Thinking about what might have changed: I did have some old expired tokens on my github account, one of which I had previously used with ghcr.io (and its possible that it was still lingering around on my local machine - I don't even know how/where to check that). When creating the new token I just cleaned up by deleting these tokens.
Is it possible that having an old expired token around messes things up even when no authentication is required?

[gilesknap]

Thanks. Good to know.

[gilesknap]

UhOh. Today GHCR is showing the OP error all of the time.

[hgv27681@pc0116 bl20j]$ ec -v deploy-local services/nfsv2-tftp/
kubectl get namespace i20-1-iocs -o name
Deploy nfsv2-tftp TEMPORARY version 2024.4.d274-b from /scratch/hgv27681/work/bl20j/services/nfsv2-tftp to domain i20-1-iocs
Are you sure ? [y/N]: y
helm dependency update /scratch/hgv27681/work/bl20j/services/nfsv2-tftp/../../helm/shared

Command Failed:
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "gitlab" chart repository
Update Complete. ⎈Happy Helming!⎈
Saving 1 charts
Downloading ioc-instance from repo oci://ghcr.io/epics-containers
Save error occurred:  could not download oci://ghcr.io/epics-containers/ioc-instance: failed to authorize: failed to fetch oauth token: unexpected status from GET request to https://ghcr.io/token?scope=repository%3Aepics-containers%2Fioc-instance%3Apull&service=ghcr.io: 403 Forbidden

Error: could not download oci://ghcr.io/epics-containers/ioc-instance: failed to authorize: failed to fetch oauth token: unexpected status from GET request to https://ghcr.io/token?scope=repository%3Aepics-containers%2Fioc-instance%3Apull&service=ghcr.io: 403 Forbidden

This seems to imply that it is checking the list of tags that is failing:

[hgv27681@pc0116 bl20j]$ helm pull oci://ghcr.io/epics-containers/ioc-instance:3.4.0
Error: GET "https://ghcr.io/v2/epics-containers/ioc-instance/tags/list": GET "https://ghcr.io/token?scope=repository%3Aepics-containers%2Fioc-instance%3Apull&service=ghcr.io": unexpected status code 403: denied: denied

Perhaps the manifest summary is not public although the package is?

Never the less this is a bit of a showstopper.
It might just be a transient gh server issue, but this would be mighty inconvenient if epics-containers was live at the facility.
It might also be something to do with Harbour.

[gilesknap]

Actually this is not quite the same as the OP - I have never seen this with docker pull - I frequently see it intermittently with helm and only today see it all of the time.