Skip to content

Building on arm64 fails - arm64 BuildKit image ships x86-64 earth_debugger binary #394

@JulesClaussen

Description

@JulesClaussen

What went wrong?

Hello team,

Building for arm architecture fail, because it seems the earth_debugger using an x86 version. This is masked in MacOS, I believe because they have internal translater, but Linux arm64 indeed fails.

Minimal reproduction:

Earthfile:

test-arm64:
  FROM debian:bookworm-slim
  RUN echo "arch=$(uname -m)" && uname -m

Github action to reproduce (running on arm image):

name: Test ARM64 Build

on:
  push:
    branches:
      - feat/test-arm

env:
  # renovate: datasource=github-releases depName=earthbuild/earthbuild
  EARTHBUILD_VERSION: '0.8.17'

jobs:
  test-arm64:
    name: Minimal ARM64 build
    runs-on: runs-on=${{ github.run_id }}/runner=1cpu-linux-arm64
    timeout-minutes: 10

    steps:
      - name: Checkout
        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2

      - name: Install Earthbuild
        uses: earthbuild/actions-setup@e11e36ffea530484da8a4e5a4bd17d69de480786 # v2.0.0
        with:
          version: ${{ env.EARTHBUILD_VERSION }}
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

      - name: Diagnostics
        run: |
          echo "Host arch: $(uname -m)"
          echo "Earth binary: $(file $(which earth))"
          earth bootstrap
          echo "BuildKit container image:"
          docker inspect earth-buildkitd --format '{{.Config.Image}}'
          echo "earth_debugger binary inside BuildKit:"
          docker cp earth-buildkitd:/usr/bin/earth_debugger /tmp/earth_debugger
          file /tmp/earth_debugger

      - name: Build minimal arm64 target (without patch)
        id: build-no-patch
        continue-on-error: true
        run: earth --ci --platform=linux/arm64 +test-arm64

      - name: Apply earth_debugger patch
        if: steps.build-no-patch.outcome == 'failure'
        run: |
          echo "Build failed as expected. Patching earth_debugger..."
          docker exec earth-buildkitd sh -c \
            'printf "#!/bin/sh\nexec \"\$@\"\n" > /usr/bin/earth_debugger && chmod +x /usr/bin/earth_debugger'

      - name: Build minimal arm64 target (with patch)
        if: steps.build-no-patch.outcome == 'failure'
        run: earth --ci --platform=linux/arm64 +test-arm64

      - name: Summary
        if: always()
        run: |
          if [ "${{ steps.build-no-patch.outcome }}" = "failure" ]; then
            echo "### Confirmed: earth_debugger bug on arm64 :bug:" >> $GITHUB_STEP_SUMMARY
            echo "Unpatched build failed, patched build succeeded." >> $GITHUB_STEP_SUMMARY
          else
            echo "### Unpatched build succeeded :white_check_mark:" >> $GITHUB_STEP_SUMMARY
            echo "The earth_debugger bug may have been fixed." >> $GITHUB_STEP_SUMMARY
          fi

Output of the build without patch:

  Build 🔧
  Build 🔧 Build 🔧
  ————————————————————————————————————————————————————————————————————————————————
  
           +test-arm64 | --> FROM +base
  node:22-slim@sha256:dd9d21971ec4395903fa6143c2b9267d048ae01ca6d3ea96f16cb30df6187d94 | --> Load metadata node:22-slim@sha256:dd9d21971ec4395903fa6143c2b9267d048ae01ca6d3ea96f16cb30df6187d94 linux/arm64
  Warning: you are not logged into registry-1.docker.io, you may experience rate-limiting when pulling images
  debian:bookworm-slim | --> Load metadata debian:bookworm-slim linux/arm64
                 +base | --> FROM node:22-slim@sha256:dd9d21971ec4395903fa6143c2b9267d048ae01ca6d3ea96f16cb30df6187d94
                 +base | [----------] 100% FROM node:22-slim@sha256:dd9d21971ec4395903fa6143c2b9267d048ae01ca6d3ea96f16cb30df6187d94
  Warning: you are not logged into registry-1.docker.io, you may experience rate-limiting when pulling images
           +test-arm64 | --> FROM debian:bookworm-slim
           +test-arm64 | [----------] 100% FROM debian:bookworm-slim
           +test-arm64 | --> RUN echo "arch=$(uname -m)" && uname -m
           +test-arm64 | /bin/sh: 1: /usr/bin/earth_debugger: Exec format error
           +test-arm64 | ERROR Earthfile:345:2
           +test-arm64 |       The command
           +test-arm64 |           RUN echo "arch=$(uname -m)" && uname -m
           +test-arm64 |       did not complete successfully. Exit code 126
  
  ================================== ❌ FAILURE ===================================
  
           +test-arm64 *failed* | Repeating the failure error...
           +test-arm64 *failed* | --> RUN echo "arch=$(uname -m)" && uname -m
           +test-arm64 *failed* | /bin/sh: 1: /usr/bin/earth_debugger: Exec format error
           +test-arm64 *failed* | ERROR Earthfile:345:2
           +test-arm64 *failed* |       The command
           +test-arm64 *failed* |           RUN echo "arch=$(uname -m)" && uname -m
           +test-arm64 *failed* |       did not complete successfully. Exit code 126
  ::error 
  file=Earthfile,line=345,col=2,title=Error::The command RUN echo "arch=$(uname -m)" && uname -m did not complete successfully. Exit code 126               +base | WARN Earthfile:2:0: The command
                 +base |           FROM node:22-slim@sha256:dd9d21971ec4395903fa6143c2b9267d048ae01ca6d3ea96f16cb30df6187d94
                 +base | failed: context canceled

Output with patch

  Build 🔧
  Build 🔧 Build 🔧
  ————————————————————————————————————————————————————————————————————————————————
  
           +test-arm64 | --> FROM +base
  node:22-slim@sha256:dd9d21971ec4395903fa6143c2b9267d048ae01ca6d3ea96f16cb30df6187d94 | --> Load metadata node:22-slim@sha256:dd9d21971ec4395903fa6143c2b9267d048ae01ca6d3ea96f16cb30df6187d94 linux/arm64
  debian:bookworm-slim | --> Load metadata debian:bookworm-slim linux/arm64
                 +base | --> FROM node:22-slim@sha256:dd9d21971ec4395903fa6143c2b9267d048ae01ca6d3ea96f16cb30df6187d94
                 +base | [----------] 100% FROM node:22-slim@sha256:dd9d21971ec4395903fa6143c2b9267d048ae01ca6d3ea96f16cb30df6187d94
           +test-arm64 | --> FROM debian:bookworm-slim
           +test-arm64 | [----------] 100% FROM debian:bookworm-slim
           +test-arm64 | --> RUN echo "arch=$(uname -m)" && uname -m
           +test-arm64 | arch=aarch64
           +test-arm64 | aarch64
                 +base | --> WORKDIR /app
                output | --> exporting outputs
Build 🔧

What should have happened?

We should be able to build with arm64 arch

What earthly version?

0.8.17

Buildkit Logs

No response

Other Helpful Information

In all transparency, thank you Opus 4.6 for that finding 👀

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions