Adding support for Arm64

[gilesknap]

Intro

I have successfully got arm64 builds of epics-base to work.

However there are a couple of reasons not to merge this into main.

• DLS does not actually need arm64
• arm64 builds are prohibitively slow on Github Actions - see more details below.

Arm at DLS

Arm64 is useful for people using new macs for dev or running IOCs.

DLS's arm IOCs run on 32bit architectures armv7 and armv8 both of which have cross compilation in our current build system.

Even if we wanted to switch to doing native compile of these IOCs we could not as the 32 bit arm variants are lacking in package support and are not supported on GHA.

Hence these would be supported in containers as cross compile on linux:x86_64 just like RTEMS.

Arm64 Build Performance

I have tried two mechanisms for targeting arm64.

• use buildx and specify platform=linux/arm64,linux/amd64
• do a native MacOS build using the os matrix in GHA

In both cases, this snippet in the Dockerfile instructs the epics build system of the target os:

FROM base AS environment-amd64
ENV EPICS_HOST_ARCH=linux-x86_64

FROM base AS environment-arm64
ENV EPICS_HOST_ARCH=linux-arm

FROM environment-$TARGETARCH AS environment

The buildx approach uses QEMU to emulate arm64 for the compilation. This results in a build time of 1hr18m, see https://github.com/epics-containers/epics-base/actions/runs/5155081287

I had hoped that a native build on MacOS would improve things, but for whatever reason this is way worse at 3hr36m https://github.com/epics-containers/epics-base/actions/runs/5155705777.

[gilesknap]

@coretl @ulrikpedersen tagging you both for a discussion of how to proceed with this.

It is possible that my native arm64 compile was done wrong in some fashion as it seems a very bad result.

[gilesknap]

I was wondering if the cause of the slow native build is to do with target.
We are targetting linux:arm as that is what will be inside the ubuntu container running on a Mac.
But the native compiler for Mac would be darwin-aarch64.

Would this maybe cause the epics build system to use cross compilation? The answer should be no as the build itself is running inside a linux container. Looking at the logs is looks to be targeting linux:arm and using gcc - I don't see any cross compilation.

[gilesknap]

It occurs to me that even if GitHub has real Macs for GHA they are probably not M1 as that is less than 3 years old. We are probably running on a x86 mac and getting QEMU to arm64 emulate for us!

doh!

[gilesknap]

@ulrikpedersen I'm merging into main all the changes except with arm64 approaches commented out. I can confirm that M1 is coming to GHA and will enable the native compile when this happens.

See github/roadmap#528

The two approaches to building for arm64 are commented in .github/workflows/build.yml, see snippet below. The first approach is the fastest for now with build time 1h20m.

epics-base/.github/workflows/buiild.yml

Lines 7 to 41 in 96f1aba

	jobs:
	build-push-images:
	# pull requests are a duplicate of a branch push if within the same repo.
	if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name != github.repository
	permissions:
	contents: read
	packages: write
	env:
	CACHE: /tmp/.buildx-cache
	strategy:
	fail-fast: false
	matrix:
	include:
	# rtems is cross compiled on ubuntu-latest
	# linux is natively compiled on ubuntu-latest and macos-latest
	- architecture: rtems
	os: ubuntu-latest
	platform: linux/amd64
	- architecture: linux
	os: ubuntu-latest
	platform: linux/amd64
	# TO ENABLE MACOS: change the above platform to
	# platform: linux/amd64,linux/arm64
	# this causes buildx to use QEMU for arm64 emulation.
	# This will increase build time by 1 hour.
	# MACOS is removed as it takes 3 hours. This is because there are no
	# M1 macs in the github runners, so we still use QEMU to emulate
	# but on an x86 mac. NOTE: once GHA starts to use M1 macs, this
	# approach will be the FASTEST.
	# M1 Macs are coming!: https://github.com/github/roadmap/issues/528
	# - architecture: linux
	# os: macos-latest
	# platform: linux/arm64
	runs-on: ${{ matrix.os }}

[ulrikpedersen]

I can confirm that M1 is coming to GHA and will enable the native compile when this happens.

Very cool 👍
I was considering offering up my M1 Mini at home as a self-hosted runner. I even spent a couple of hours trying to figure out how - but I couldn't work out how to do it in a very safe way. It is not a dedicated machine for this task and has all my personal data on it.

In other news: I have bought a PC with Intel CPU and installed Ubuntu on it so that I can run both epics-containers and Tango containers without all the faff 😄
Looking forward to the day when GH releases Mac M1 runners though 🤞

[gilesknap]

Yeah self hosted runners are best done on isolated machines. We are planning to use machines in the IRIS cloud to do it.

FYI: as an esteemed customer of epics-containers I should warn you that Tom Gary and I have just sketched out a major refactor. The good news is that it should simplify the developer experience and align more closely with the standard devcontainer workflow. The bad news is its a bit of a rewrite of all the tutorials and I wont get started on that until mid Aug.

[ulrikpedersen]

Thanks for the heads-up. I don't think it will affect us in how we are currently using epics-containers because we haven't fully adopted the workflow outlined in the tutorials...

[gilesknap]

@ulrikpedersen now is a good time to adopt the full epics-containers workflow if you were considering it.

• Two major re-writes since you last used it and it is now looking stable.
• all the tutorials are up to date with the latest changes
• generic ioc and beamline repositories are created using a copier template so you can easily adopt any future changes
• I've just finished re-doing RTEMS and would be happy to take another look at arm64 if you still have an interest