Action fails with `sudo: cvmfs_config: command not found`

[andrewmogan]

Our nightly build uses the cvmfs-contrib action to check existing builds to prevent accidental duplication. Starting yesterday, this step started failing with

sudo: cvmfs_config: command not found

You can see the failed action here. This happens while running on ubuntu-latest, but not when switching to a self-hosted runner. I also notice that the ubuntu-latest build of this action failed with the same error. We've disabled the cvmfs-contrib action for now, but it would be nice to get this resolved soon.

[ocaisa]

You can fix it by removing the caches for the Action. There is an attempt to fix this in general in #48

[wdconinc]

This bug has been with us for a while but isn't easy to reproduce. Since we all (us too) got hit by it these recent days, can we identify what changed to cause it? @vvolkl anything on the packaging side?

[wdconinc]

One thing that may be related is we just got hit by a runner bug (in github.com/acts-project/acts) that seemed to indicate maybe the github-runner was upgraded (but no release in github.com/actions/runner).

[wdconinc]

So, we've long suspected this is due to stale caches. In this case, for scheduled 'canary' actions on this repo the cache was 7 to 9 months old and was installing cvmfs-release-latest_all.deb from back then, v2.5-1 (https://github.com/cvmfs-contrib/github-action-cvmfs/actions/runs/22031138405/job/63656396358#step:3:147). The other steps that follow should result in updates, but those aren't persisted. I think the underlying reason for the stale cache is that it does not get updated as intended unless permissions: actions: write is set on the pipeline, but even with those permissions the original cvmfs-release-latest_all.deb will not get updated. I still don't understand why this then results in a libuuid version issue, but ok...

So, I think these are the steps that I could imagine we should consider:

1. we need to check that cvmfs-release-latest_all.deb is the latest version (preferably without re-downloading it); maybe curl -o $file -z $file $url is what we want here in all cases
2. we need fail when apt-get install fails (likely with set -e in setup-cvmfs.sh`)
3. we need to detect and warn the user when permissions: actions: write is not set, and explain the reasons for this

[vvolkl]

This bug has been with us for a while but isn't easy to reproduce. Since we all (us too) got hit by it these recent days, can we identify what changed to cause it? @vvolkl anything on the packaging side?

No changes in CVMFS at all. How about #60 as a short-term fix though?

[fstagni]

Hello, the resolution of this issue is highly awaited, as many of our CIs are impacted, and blocking merging of PRs.

[vvolkl]

Hello, the resolution of this issue is highly awaited, as many of our CIs are impacted, and blocking merging of PRs.

Sorry about that. The problem can be fixed manually by going to https://github.com/ORG/REPO/actions/caches and deleting the "cvmfs-apt-cache*" caches, but that's obviously cumbersome if it concerns many repositories.

[wdconinc]

Tagged as v5.5, retagged v5. If you're on v5 with a failing pipeline, please try to restart a job without first clearing the cache to see if this adds resiliency, and report back your experience here.

[c-dilks]

please try to restart a job without first clearing the cache

I tested in coatjava using a 7-month old v5 cache, and it's successful:

JeffersonLab/coatjava#1110

(we worked around this issue by using @v5.4 rather than @v5, and the PR reverts to @v5; no caches were ever flushed)

[vvolkl]

please try to restart a job without first clearing the cache

I tested in coatjava using a 7-month old v5 cache, and it's successful:

JeffersonLab/coatjava#1110

(we worked around this issue by using @v5.4 rather than @v5, and the PR reverts to @v5; no caches were ever flushed)

Thanks for testing this! Closing the issue, do re-open or create a new one if you see this problem again.