Proposal: TEST command

[kmannislands]

Copying this issue over from the original repo since I believe it is still relevant: earthly#4198

Use case

Problem statement

At the moment it is a bit difficult (as acknowledged in the docs) to use earthly to run tests.

Defining tests as commands that exits 0 if the test "passed" and non-zero otherwise, possibly creating outputs to be saved as artifacts (coverage, junit, tap etc). Generally, in CI it is desirable to continue execution after a single test has failed to get a sense of all tests impacted by a change "in one go". However, if any test failed, the entire "build" should be considered to have failed after output artifacts are written.

Implementing this behavior in user code at present requires significant work-arounds. These work-arounds, especially in monorepo setups, are not well-addressed in my opinion by current proposals like TRY/FINALLY. Specifically, TRY/FINALLY when used with SAVE ARTIFACT achieves the goal of continuing execution writing artifacts but leaves failing the overall build to the user. Present workarounds could include inspecting all test artifacts with aware tooling (checking all Junits with a tool that can tell if anything failed) or storing exit codes as artifacts to compare later. These approaches are boilerplate-ey and too prone to mistakes in my opinion.

Proposal

So, my proposal here is to add a new command to earthly's syntax to handle this: TEST

TEST should accept the same arguments as RUN and behave the same except:

• any command under TEST exits non-zero, earthly would "remember" the target, command and exit code but continue execution normally
• After the output phase, earthly would exit non-zero if any of the commands run under TEST failed, logging a report

Alternatively, a --test flag for RUN could also similarly suit.

Treating tests differently has good precedent in mature build systems like bazel and I think would unlock useful additional functionality in the future like identifying/retrying flaky tests.

Expected Behavior

For a hypothetical monorepo Earthfile with this hypothetical syntax:

VERSION --test 0.8

all:
  BUILD +lint
  BUILD +unit-tests
  BUILD +integration-tests

integration-tests:
  WITH DOCKER --load +front-end +server
    TEST docker run something > integration-test.log
  END
  
  SAVE ARTIFACT integration-test.log AS LOCAL integration-test.log

unit-tests:
  RUN mkdir test-results
  # Assume the +test target in each package uses `TEST` internally. Without `TEST`, how would exit codes be tracked?
  FOR package in "frontend ui-kit server common"
    COPY (./packages/${package}+test/junit.xml) ./test-results/${package}-unit.junit.xml
  END
  
  SAVE ARTIFACT test-results AS LOCAL test-results
  
front-end:
  FROM ./packages/front-end+build
  SAVE IMAGE frontend:latest
  
server:
  FROM ./packages/server+build
  SAVE IMAGE server:latest 
  
lint:
  FROM +deps
  COPY ./.eslintrc.json
  COPY ./packages ./packages/
  TEST npm run lint
  

All test result artifacts would be exported.

All test targets in the unit-tests FOR loop and the integration test would be run regardless of any failures.

If any of the command run under TEST failed (exited non-zero), earthly client should report failed tests and exit non-zero.

If earthly was run with -i, I think the behavior of TEST should be changed to drop into the interactive mode at the failure instead of continuing execution.