feature: use Rosetta AOT Caching with CDI

[norio-nomura]

Description

This change introduces device configuration to enable Rosetta AOT Caching in Docker VMs.

• Modify Rosetta Caching Options from Abstract Socket to Unix Domain Socket:
  Unix Domain Socket can be mounted within a container using the Container Device Interface (CDI) mechanism.
  This requires merging the following pull request: Rosetta AOT caching options Code-Hex/vz#195.

• Register Rosettad AOT Caching Daemon as a service:

  • /etc/systemd/system/rosettad.service on systemd
  • /etc/init.d/rosettad on OpenRC

• Add "lima-vm.io/rosetta=cached" device specification to {~/.config,/etc}/cdi/rosetta.yaml
  see: https://github.com/cncf-tags/container-device-interface/blob/main/SPEC.md

• Add {~/.config,/etc}/docker/daemon.json to docker{,-rootful}.yaml

  • .features.cdi = true to enable CDI

• Add enable_cdi = true to {~/.config,/etc}/containerd/config.toml

To enable Rosetta AOT Caching in docker, use --device=lima-vm.io/rosetta=cached.
see: https://docs.docker.com/build/building/cdi/

Benchmark

1. Setup docker VM

$ _output/bin/limactl start --name docker-test template://docker --memory 16 --rosetta --tty=false --log-level warn
WARN[0000] vmType vz: ignoring [VMOpts]                 
3.50 GiB / 3.50 GiB [---------------------------------------] 100.00% 1.54 GiB/s
To run `docker` on the host (assumes docker-cli is installed), run the following commands:
------
docker context create lima-docker-test --docker "host=unix:///Users/norio/.lima/docker-test/sock/docker.sock"
docker context use lima-docker-test
docker run hello-world
------
$ docker context create lima-docker-test --docker "host=unix:///Users/norio/.lima/docker-test/sock/docker.sock"
lima-docker-test
Successfully created context "lima-docker-test"
$ docker context use lima-docker-test
lima-docker-test
Current context is now "lima-docker-test"

2. Confirm Rosetta AOT Caching is working (/var/cache/rosettad/*.aotcache are created)

$ limactl shell docker-test ls -la /var/cache/rosettad
total 12
drwxr-xr-x  3 root root 4096 Aug 18 18:57 .
drwxr-xr-x 15 root root 4096 Aug 18 18:57 ..
drwxr-xr-x  2 root root 4096 Aug 18 18:57 uds
$ docker run --rm --platform linux/amd64 --device=lima-vm.io/rosetta=cached ubuntu echo hello
Unable to find image 'ubuntu:latest' locally
latest: Pulling from library/ubuntu
b71466b94f26: Pull complete 
Digest: sha256:7c06e91f61fa88c08cc74f7e1b7c69ae24910d745357e0dfe1d2c0322aaf20f9
Status: Downloaded newer image for ubuntu:latest
hello
$ limactl shell docker-test ls -la /var/cache/rosettad
total 2336
drwxr-xr-x  3 root root    4096 Aug 18 18:58 .
drwxr-xr-x 15 root root    4096 Aug 18 18:57 ..
-rwxr-xr-x  1 root root  244608 Aug 18 18:58 4f961aefd1ecbc91b6de5980623aa389ca56e8bfb5f2a1d2a0b94b54b0fde894.aotcache
-rwxr-xr-x  1 root root   13228 Aug 18 18:58 75775b3c920d5c6686814bfb4c270e4d186c1155fb9136c9400581f9ef535835.aotcache
-rwxr-xr-x  1 root root 2113864 Aug 18 18:58 de259f5276c4a991f78bf87225d6b40e56edbffe0dcbc0ffca36ec7fe30f3f77.aotcache
drwxr-xr-x  2 root root    4096 Aug 18 18:57 uds

3. Benchmark with executing bash -c "time go run ./cmd/limactl -v 2>/dev/null" in golang container

$ docker pull --platform=linux/amd64 golang
Using default tag: latest
latest: Pulling from library/golang
80b7316254b3: Pull complete 
e70dc8762870: Pull complete 
36e4db86de6e: Pull complete 
8ea45766c644: Pull complete 
a9463fe86d3b: Pull complete 
4f4fb700ef54: Pull complete 
8286cb4ece30: Pull complete 
Digest: sha256:9e56f0d0f043a68bb8c47c819e47dc29f6e8f5129b8885bed9d43f058f7f3ed6
Status: Downloaded newer image for golang:latest
docker.io/library/golang:latest
$ docker run --rm --platform linux/amd64 -v$PWD:$PWD -w$PWD -eCGO_ENABLED=1 golang bash -c "time go run ./cmd/limactl -v 2>/dev/null"
limactl version <unknown>

real	1m31.342s
user	4m24.907s
sys	0m24.688s
$ docker run --rm --platform linux/amd64 -v$PWD:$PWD -w$PWD -eCGO_ENABLED=1 --device=lima-vm.io/rosetta=cached golang bash -c "time go run ./cmd/limactl -v 2>/dev/null"
limactl version <unknown>

real	0m48.412s
user	1m47.103s
sys	0m14.668s

edited: renamelima.io/rosetta=cached to lima-vm.io/rosetta=cached

[norio-nomura]

I also attempted to enable CDI in the containerd installed by the .container.user by making changes to the config.toml file:

  [plugins."io.containerd.grpc.v1.cri"]
    enable_cdi = true
    cdi_spec_dirs = ["/etc/cdi", "/var/run/cdi"]

However, I haven’t been successful yet.

[norio-nomura]

Rebased

[norio-nomura]

By creating symlink at ${LIMA_CIDATA_HOME}/.config/cdi/rosetta.yaml, now nerdctl recognizes CDI spec.

➜ ~13:01:29 13:03:18~ (use-rosetta-aot-caching-with-cdi) limactl start template://ubuntu-lts --name=default --tty=false --memory 16 --rosetta --log-level warn
WARN[0000] vmType vz: ignoring [VMOpts]                 
3.50 GiB / 3.50 GiB [---------------------------------------] 100.00% 1.22 GiB/s
➜ ~13:04:09 13:04:12~ (use-rosetta-aot-caching-with-cdi) nerdctl.lima pull --platform linux/amd64 golang
docker.io/library/golang:latest:                                                  resolved       |++++++++++++++++++++++++++++++++++++++| 
index-sha256:91e2cd436f7adbfad0a0cbb7bf8502fa863ed8461414ceebe36c6304731e0fd9:    done           |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:42a6459ee5ae70dd490c6b6164bef32b064a66b4f0a9744105969883b8b23840: done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:4ba81a847f6b20b63eec32f2ee16fce0385e8b06bf5789a66d4e51aae215e8cd:   done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:8ea45766c6449310ca2fc621a7e00bedb4b9b803a7fbfe2607efce6d2e07e435:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:e70dc87628705e52d31a6bee046711fe5dd4d9d73e7cfd02aaceec20b352ca48:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:80b7316254b3093eb3c7ac44bb6c34bde013f27947c1ed8d8afe456b957ebfdb:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:36e4db86de6eba33869491caa7946b80dd71c255f1940e96a9f755cc2b1f3829:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:8286cb4ece30afb97c398c2b5ac1f35e8f502f758d4ea2fc69e179efdf471ea2:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:a9463fe86d3b832fec17e98dd4dc2def946be2ce83848b242e45383fd2aba0d3:    done           |++++++++++++++++++++++++++++++++++++++| 
elapsed: 49.7s                                                                    total:  290.7  (5.8 MiB/s)                                       
➜ ~13:05:07 13:05:09~ (use-rosetta-aot-caching-with-cdi) nerdctl.lima run --rm --platform linux/amd64 -v$PWD:$PWD -w$PWD -eCGO_ENABLED=1 golang bash -c "time go run ./cmd/limactl -v 2>/dev/null"
limactl version <unknown>

real	1m43.245s
user	4m26.421s
sys	0m24.823s
➜ ~13:06:57 13:07:00~ (use-rosetta-aot-caching-with-cdi) nerdctl.lima run --rm --platform linux/amd64 -v$PWD:$PWD -w$PWD -eCGO_ENABLED=1 --device=lima-vm.io/rosetta=cached golang bash -c "time go run ./cmd/limactl -v 2>/dev/null"
limactl version <unknown>

real	0m53.677s
user	1m47.269s
sys	0m13.958s

edited: renamelima.io/rosetta=cached to lima-vm.io/rosetta=cached

[norio-nomura]

Renamed lima.io/rosetta=cached to lima-vm.io/rosetta=cached.
Also edited examples in the above comments.

[AkihiroSuda]

Let's mark this as a draft until Code-Hex/vz#195 is merged

[AkihiroSuda]

Needs rebase

[AkihiroSuda]

Also consider adding documentation