rootless networking error on container startup since upgrade to F42

[averi]

Issue Description

At GNOME we run a set of gitlab-runners runners that support the Project's CI/CD needs, since the upgrade to Fedora 42 (this was working fine with Fedora 40) there seems to be a race condition that causes:

ERROR: Preparation failed: creating cache volume: set volume permissions: running permission container "c7309ea6d35f49a8d68ba660c62b676929b5744844bffd75151c664ea8fe1a28" for volume "runner-qy9jf-zq7-project-9004-concurrent-1-8c5c3832aa81ed63-cache-c33bcaa1fd2c77edfc3893b41966cea8": starting permission container: Error response from daemon: rootless netns: mount resolv.conf to "/run/user/991/containers/networks/rootless-netns/run/systemd/resolve/stub-resolv.conf": no such file or directory (linux_set.go:105:0s)

On top of this any container started from the podman user systemd session doesn't seem to create a persistent /run/user/$uid/containers/networks/rootless-netns directory structure, every container that is started as a build creates a brand new rootless-netns directory and associated files (i.e resolv.conf) which then gets deleted once the build has finished. This however sometimes leave the rootless-netns directory in an inconsistent state which then prevents the subsequent build to succeed due to the error above. Generally manually running "rm -rf /run/user/$uid/containers/networks/rootless-netns" temporarily fixes the issue.

Steps to reproduce the issue

Steps to reproduce the issue

1. Set up your system via https://github.com/gnome-infra/ansible/blob/master/roles/gitlab-runner/tasks/main.yml
2. Trigger new container creation via GitLab CI/CD
3. At some point the container creation process will fail with a similar error as "ERROR: Preparation failed: creating cache volume: set volume permissions: running permission container "c7309ea6d35f49a8d68ba660c62b676929b5744844bffd75151c664ea8fe1a28" for volume "runner-qy9jf-zq7-project-9004-concurrent-1-8c5c3832aa81ed63-cache-c33bcaa1fd2c77edfc3893b41966cea8": starting permission container: Error response from daemon: rootless netns: mount resolv.conf to "/run/user/991/containers/networks/rootless-netns/run/systemd/resolve/stub-resolv.conf": no such file or directory (linux_set.go:105:0s)"

Describe the results you received

Starting a new build fails with "ERROR: Preparation failed: creating cache volume: set volume permissions: running permission container "c7309ea6d35f49a8d68ba660c62b676929b5744844bffd75151c664ea8fe1a28" for volume "runner-qy9jf-zq7-project-9004-concurrent-1-8c5c3832aa81ed63-cache-c33bcaa1fd2c77edfc3893b41966cea8": starting permission container: Error response from daemon: rootless netns: mount resolv.conf to "/run/user/991/containers/networks/rootless-netns/run/systemd/resolve/stub-resolv.conf": no such file or directory (linux_set.go:105:0s)"

Describe the results you expected

A successful container creation

podman info output

host:
  arch: amd64
  buildahVersion: 1.39.4
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.13-1.fc42.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.13, commit: '
  cpuUtilization:
    idlePercent: 96.15
    systemPercent: 0.75
    userPercent: 3.11
  cpus: 20
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    version: "42"
  eventLogger: file
  freeLocks: 2042
  hostname: hetzner-1
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 991
      size: 1
    - container_id: 1
      host_id: 524288
      size: 1100000000
    uidmap:
    - container_id: 0
      host_id: 991
      size: 1
    - container_id: 1
      host_id: 524288
      size: 1100000000
  kernel: 6.14.5-300.fc42.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 22063742976
  memTotal: 67208761344
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.14.0-1.fc42.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.14.0
    package: netavark-1.14.1-1.fc42.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.14.1
  ociRuntime:
    name: crun
    package: crun-1.21-1.fc42.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.21
      commit: 10269840aa07fb7e6b7e1acff6198692d8ff5c88
      rundir: /run/user/991/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20250503.g587980c-1.fc42.x86_64
    version: ""
  remoteSocket:
    exists: true
    path: /run/user/991/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.3.1-2.fc42.x86_64
    version: |-
      slirp4netns version 1.3.1
      commit: e5e368c4f5db6ae75c2fce786e31eef9da6bf236
      libslirp: 4.8.0
      SLIRP_CONFIG_VERSION_MAX: 5
      libseccomp: 2.5.5
  swapFree: 8588943360
  swapTotal: 8589930496
  uptime: 70h 10m 48.00s (Approximately 2.92 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
store:
  configFile: /home/podman/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/podman/.local/share/containers/storage
  graphRootAllocated: 944913514496
  graphRootUsed: 779810328576
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 492
  runRoot: /run/user/991/containers
  transientStore: false
  volumePath: /home/podman/.local/share/containers/storage/volumes
version:
  APIVersion: 5.4.2
  BuildOrigin: Fedora Project
  Built: 1743552000
  BuiltTime: Wed Apr  2 00:00:00 2025
  GitCommit: be85287fcf4590961614ee37be65eeb315e5d9ff
  GoVersion: go1.24.1
  Os: linux
  OsArch: linux/amd64
  Version: 5.4.2

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

On premise bare metal podman installation, containers are initialized via the gitlab-runner service:

[podman@hetzner-1 ~]$ cat .config/systemd/user/default.target.wants/gitlab-runner.service
[Unit]
Description=GitLab Runner
Documentation=man:podman-generate-systemd(1)
Wants=network-online.target
After=network-online.target
RequiresMountsFor=/run/user/991/containers

[Service]
Environment=PODMAN_SYSTEMD_UNIT=%n
Restart=always
TimeoutStopSec=70
ExecStartPre=-rm -f /run/user/991/%n.ctr-id
ExecStart=/usr/bin/podman
--url unix:/run/user/991/podman/podman.sock container run
--cidfile=/run/user/991/%n.ctr-id
--cgroups=no-conmon
--rm
--attach STDOUT
--attach STDERR
--sdnotify=conmon
--replace
--name gitlab-runner
--volume /run/user/991/podman/podman.sock:/var/run/docker.sock:z
--volume /home/podman/gitlab-runner:/etc/gitlab-runner:Z
--label io.containers.autoupdate=image
--label org.gnome.pydocuum.skip=True registry.gitlab.com/gitlab-org/gitlab-runner:latest
ExecStop=/usr/bin/podman stop
--ignore -t 10
--cidfile=/run/user/991/%n.ctr-id
ExecStopPost=/usr/bin/podman rm
-f
--ignore -t 10
--cidfile=/run/user/991/%n.ctr-id
NotifyAccess=all

[Install]
WantedBy=default.target

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

[giuseppe]

@Luap99 PTAL

[Luap99]

Set up your system via https://github.com/gnome-infra/ansible/blob/master/roles/gitlab-runner/tasks/main.yml
Trigger new container creation via GitLab CI/CD

Sorry but this is not something we consider a reproducer, I have no time to do that or even the knowledge to debug whatever the CI pipelines would do. Can you create a reproducer with actual podman command or API queries that will trigger this?

[github-actions]

A reviewer has determined we need more information to understand the reported issue. A comment on what is missing should be provided. Be certain you:

• provide an exact reproducer where possible
• verify you have provided all relevant information - minimum is podman info
• answer any follow up questions

If no response to the needs-info is provided in 30 days, this issue may be closed by our stale bot.

For more information on reporting issues on this repository, consult our issue guide.

[averi]

@Luap99 if you have a system with podman running rootless with linger enabled and install the following systemd unit via /home/podman/.config/systemd/user/default.target.wants/gitlab-runner.service with gitlab-runner.service having the following content:

[Unit]
Description=GitLab Runner
Documentation=man:podman-generate-systemd(1)
Wants=network-online.target
After=network-online.target
RequiresMountsFor=/run/user/991/containers

[Service]
Environment=PODMAN_SYSTEMD_UNIT=%n
Restart=always
TimeoutStopSec=70
ExecStartPre=-rm -f /run/user/991/%n.ctr-id
ExecStart=/usr/bin/podman \
	--url unix:/run/user/991/podman/podman.sock container run \
	--cidfile=/run/user/991/%n.ctr-id \
	--cgroups=no-conmon \
	--rm \
	--attach STDOUT \
	--attach STDERR \
	--sdnotify=conmon \
	--replace \
	--name gitlab-runner \
	--volume /run/user/991/podman/podman.sock:/var/run/docker.sock:z \
	--volume /home/podman/gitlab-runner:/etc/gitlab-runner:Z \
	--label io.containers.autoupdate=image \
	--label org.gnome.pydocuum.skip=True registry.gitlab.com/gitlab-org/gitlab-runner:latest
ExecStop=/usr/bin/podman stop \
	--ignore -t 10 \
	--cidfile=/run/user/991/%n.ctr-id
ExecStopPost=/usr/bin/podman rm \
	-f \
	--ignore -t 10 \
	--cidfile=/run/user/991/%n.ctr-id
NotifyAccess=all

[Install]
WantedBy=default.target

From there:

systemctl --user start gitlab-runner

Then:

ls -al /run/user/991/containers/networks/rootless-netns/
total 0
drwx------. 2 podman podman  40 May 12 15:50 .
drwx------. 4 podman podman 120 May 12 15:50 ..

Any subsequent build will fail with:

ERROR: Preparation failed: creating cache volume: set volume permissions: running permission container "c7309ea6d35f49a8d68ba660c62b676929b5744844bffd75151c664ea8fe1a28" for volume "runner-qy9jf-zq7-project-9004-concurrent-1-8c5c3832aa81ed63-cache-c33bcaa1fd2c77edfc3893b41966cea8": starting permission container: Error response from daemon: rootless netns: mount resolv.conf to "/run/user/991/containers/networks/rootless-netns/run/systemd/resolve/stub-resolv.conf": no such file or directory (linux_set.go:105:0s)

If I run:

podman run --rm -it --entrypoint bash --network=bridge registry.access.redhat.com/ubi9/ubi:9.5-1731517889

This works as expected and I can see /run/user/991/containers/networks/rootless-netns being populated properly, I don't know off hand what API queries gitlab-runner uses behind the scenes, all we do is primarily mounting the podman socket to the gitlab-runner container via run/user/991/podman/podman.sock:/var/run/docker.sock:z. Do you have any idea on why something along the lines of a plain podman run recreates the rootless-netns directory and associated files correctly while whatever process gitlab-runner uses to trigger the container creation would not? This was working fine with Fedora 40.

[Luap99]

Do you have any idea on why something along the lines of a plain podman run recreates the rootless-netns directory and associated files correctly while whatever process gitlab-runner uses to trigger the container creation would not? This was working fine with Fedora 40.

Well the entire code has been most rewritten in the meantime so it is possible that something got broken by it.

Now an empty rootless-netns should totally be fine and as you mention a plain podman run command seem to work. CLI or API wise there should be no difference whatsoever in how the network code behaves.

https://github.com/containers/common/blob/54b90177af76122d69a6a08b6db1b305e39d4f3c/libnetwork/internal/rootlessnetns/netns_linux.go#L394-L502
The code will actually always call mkdir/open beforehand to create the resolv.conf parent dirs and file before calling mount so I really don't see how we would end up in a state where the file does not exit. By the time we mount resolv.conf we have mounted and created other directories there as well.

[Luap99]

The code will actually always call mkdir/open beforehand to create the resolv.conf parent dirs and file before calling mount so I really don't see how we would end up in a state where the file does not exit. By the time we mount resolv.conf we have mounted and created other directories there as well.

Or wait I guess it doesn't complain about the destination missing but rather the source missing.
That would make more sense since we init resolv.conf once when we start pasta and write the file there, if you remove that it will be missing. However if the dir is empty the netns file is also missing so we know we must start pasta and populate resolv.conf the first time so it still doesn't really make sense to me.

The fact that the API is behaving differently is concerning but code wise it really shouldn't. Is it possible that the service is running in a different namespace?

[averi]

No, we had that issue in the past where a specific systemd service file would start both as user root and user podman causing it to break but now we explicitly make sure that doesn't happen, what I'm seeing:

podman    682671  0.0  0.0 206440 17112 ?        Ss   16:41   0:01 /usr/bin/pasta --config-net --dns-forward 169.254.1.1 -t none -u none -T none -U none --no-map-gw --quiet --netns /run/user/991/netns/netns-43d8c7fd-e885-cdf7-4f63-23b51c0b6e41 --map-guest-addr 169.254.1.2
podman    683156  0.0  0.0 206440 17132 ?        Ss   16:41   0:00 /usr/bin/pasta --config-net --dns-forward 169.254.1.1 -t none -u none -T none -U none --no-map-gw --quiet --netns /run/user/991/netns/netns-e091af74-19c0-08b6-5b11-d1c15c086918 --map-guest-addr 169.254.1.2
podman    829234  0.0  0.0 206440 17012 ?        Ss   17:28   0:00 /usr/bin/pasta --config-net --pid /run/user/991/containers/networks/rootless-netns/rootless-netns-conn.pid --dns-forward 169.254.1.1 -t none -u none -T none -U none --no-map-gw --quiet --netns /run/user/991/containers/networks/rootless-netns/rootless-netns --map-guest-addr 169.254.1.2

[root@hetzner-1 ~]# ls -al /run/user/991/containers/networks/rootless-netns/
total 4
drwx------. 3 podman podman 100 May 12 17:37 .
drwx------. 4 podman podman 120 May 12 17:28 ..
-rw-------. 1 podman podman   0 May 12 17:28 rootless-netns
-rw-------. 1 podman podman   7 May 12 17:28 rootless-netns-conn.pid
drwx------. 4 podman podman  80 May 12 17:37 run

From here:

kill 829234 # pasta pid

Then:

podman run --rm -it --entrypoint bash --network=bridge registry.access.redhat.com/ubi9/ubi:9.5-1731517889
WARN[0000] rootless netns program no longer running, trying to start it again 

[root@hetzner-1 ~]# ls -al /run/user/991/containers/networks/rootless-netns/
total 16
drwx------. 3 podman podman 160 May 12 17:52 .
drwx------. 4 podman podman 120 May 12 17:52 ..
-rw-r--r--. 1 podman podman 121 May 12 17:52 info.json
-rw-------. 1 podman podman   1 May 12 17:53 ref-count
-rw-r--r--. 1 podman podman  85 May 12 17:52 resolv.conf
-rw-------. 1 podman podman   0 May 12 17:52 rootless-netns
-rw-------. 1 podman podman   7 May 12 17:52 rootless-netns-conn.pid
drwx------. 4 podman podman  80 May 12 17:52 run

Then:

[root@3d58977c7918 /]# 
exit

ls -al /run/user/991/containers/networks/rootless-netns/
no such file or directory

systemctl --user restart gitlab-runner # can really be any systemd unit file that starts a rootless container

ls -al /run/user/991/containers/networks/rootless-netns/
empty

At this point builds start to break until rmdir /run/user/991/containers/networks/rootless-netns/. Also noticed:

May 12 15:16:29 hetzner-1 systemd-coredump[632458]: Process 387634 (podman) of user 991 terminated abnormally with signal 11/SEGV, processing...
May 12 15:16:29 hetzner-1 systemd-coredump[632459]: Process 387634 (podman) of user 991 dumped core.
                                                    Module /usr/bin/podman from rpm podman-5.4.2-1.fc42.x86_64
                                                    #0  0x000055c66678f538 runtime.mallocgcSmallNoscan (/usr/bin/podman + 0x1b538)
                                                    #1  0x000055c6667eda59 runtime.mallocgc (/usr/bin/podman + 0x79a59)
                                                    #2  0x000055c6667f31a9 runtime.growslice (/usr/bin/podman + 0x7f1a9)
                                                    #3  0x000055c6667ea176 runtime.vgetrandomPutState (/usr/bin/podman + 0x76176)
                                                    #4  0x000055c6667be145 runtime.mexit (/usr/bin/podman + 0x4a145)
                                                    #5  0x000055c6667bddff runtime.mstart0 (/usr/bin/podman + 0x49dff)

Which may potentially explain why we're also seeing the podman systemd unit restarting every few hours:

May 12 15:16:29 hetzner-1 systemd-coredump[632458]: Process 387634 (podman) of user 991 terminated abnormally with signal 11/SEGV, processing...
May 12 15:16:29 hetzner-1 systemd-coredump[632459]: Process 387634 (podman) of user 991 dumped core.
May 12 15:21:03 hetzner-1 systemd-coredump[636457]: Process 632471 (podman) of user 991 terminated abnormally with signal 11/SEGV, processing...
May 12 15:21:03 hetzner-1 systemd-coredump[636458]: Process 632471 (podman) of user 991 dumped core.
May 12 17:28:10 hetzner-1 systemd-coredump[829235]: Process 682634 (podman) of user 991 terminated abnormally with signal 11/SEGV, processing...
May 12 17:28:11 hetzner-1 systemd-coredump[829236]: Process 682634 (podman) of user 991 dumped core.

Happy to provide any additional information you may need, thanks for your help!

[Luap99]

Ack yeah crashing could likely lead to a weird state, stack trace looks like golang/go#73141 (comment)

What is the exact podman rpm version? Your info shows GoVersion: go1.24.1 which likely means it is still the buggy version

[averi]

[podman@hetzner-1 ~]$ rpm -qa | grep podman
podman-5.4.2-1.fc42.x86_64
podman-docker-5.4.2-1.fc42.noarch

[Luap99]

yeah I think we need to rebuild podman with a newer go, but given we release v5.5.0 final tomorrow that likely doesn't make sense and we just build that one instead with the new golang then.

If you like to confirm you can install 5.5.0-rc2 right now https://bodhi.fedoraproject.org/updates/FEDORA-2025-fa52f863f1

---

And regardless of the go bug I still should understand the full failure sequence here so we can make the rootless-netns recover from any point as the process could be killed for plenty of reasons and we certainly don't want all future commands to run into this error.

[averi]

I've installed the rc2 on a couple of runners, will report back in a few hours/one day to see how it goes, available if you need any additional information if you want to dig the inconsistent behavior further.

[Luap99]

I don't have time to debug this fully, but I guess once I do I can just insert os.Exit() calls in the code paths there to find the ones where we are unable to recover from an early exit.
But overall it would still be nice to know why you can observe different behaviors between API and CLI.