build system: unit test enhancements #25029
Conversation
For now, there is a flag to `zig build` called `--test-timeout-ms` which accepts a value in milliseconds. If the execution time of any individual unit test exceeds that number of milliseconds, the test is terminated and marked as timed out. In the future, we may want to increase the granularity of this feature by allowing timeouts to be specified per-step or even per-test. However, a global option is actually very useful. In particular, it can be used in CI scripts to ensure that no individual unit test exceeds some reasonable limit (e.g. 60 seconds) without having to assign limits to every individual test step in the build script. Also, individual unit test durations are now shown in the time report web interface -- this was fairly trivial to add since we're timing tests (to check for timeouts) anyway. This commit makes progress on ziglang#19821, but does not close it, because that proposal includes a more sophisticated mechanism for setting timeouts. Co-Authored-By: David Rubin <david@vortan.dev>
This is a major refactor to `Step.Run` which adds new functionality, primarily to the execution of Zig tests. * All tests are run, even if a test crashes. This happens through the same mechanism as timeouts where the test processes is repeatedly respawned as needed. * The build status output is more precise. For each unit test, it differentiates pass, skip, fail, crash, and timeout. Memory leaks are reported separately, as they do not indicate a test's "status", but are rather an additional property (a test with leaks may still pass!). * The number of memory leaks is tracked and reported, both per-test and for a whole `Run` step. * Reporting is made clearer when a step is failed solely due to error logs (`std.log.err`) where every unit test passed.
Recording the command in a separate field will give the build runner more freedom to choose how and when the command should be printed.
…-style` The new `--error-style` option decides how build failures are printed. The default mode "verbose" prints all context including the step graph fragment and the failed command (if any). The alternative mode "minimal" prints only the failed step itself, and does not print the failed command. There are also "verbose_clear" and "minimal_clear" modes, which have the distinction that the output is cleared (through ANSI escape codes) between updates, preventing different updates from being confused in the output. If `--error-style` is not specified, the environment variable `ZIG_BUILD_ERROR_STYLE` is checked before falling back to the default of "verbose"; this means the value can effectively be chosen system-wide since it is generally a personal preference. Also introduced is a `--multiline-errors` option which decides how to print errors which span multiple lines. By default, non-initial lines are indented to align with the first. Alternatively, a leading newline can be printed to align everyting on the first column, or no special treatment can be applied, resulting in misaligned output. Again, there is an environment variable (`ZIG_BUILD_MULTILINE_ERRORS`) to specify a preferred default if the option is not explicitly provided. Resolves: ziglang#23472
mlugg
added
the
ci-riscv64-linux
is this only global, eg is it planned to in the future to be able to set this in code at the test level? |
Yes, see #19821 (which this PR makes progress towards but does not close).
yep |
|
Okay, here's all the information I've gathered from the results of this first CI run. Detailed Analysis
TL;DR / SummaryMost of the failures are just caused by slow tests, particularly when running under QEMU, and will be solved by simplifying tests and/or increasing timeouts. However, there are two interesting problems which require further investigation:
On the whole, I'm quite happy with how this first run went -- it's caught both slow tests and bugs, exactly what we want from these build system enhancements! I'll spend some time hacking away at this branch over the next few days to hopefully get it into a better state. I'm actually pleasantly surprised with how few failures there were on this initial run. |
This branch started out as an attempt to work on #19821, but ended up including a whole host of enhancements to the build system and unit test evaluation specifically. In no particular order:
--test-timout-ms; if a test exceeds this, it is killed--multiline-errors)--prominent-compile-errorsreplaced with--error-style(orZIG_BUILD_ERROR_STYLEenvironment variable)--summary linemodeSome of these changes have more detailed explanations in the commit messages.
Here's a screenshot showing some of the new stuff:
The last commit here sets a 60 second unit test timeout in all of the CI scripts. I know that this is going to fail; I expect this PR to spend about a week in purgatory as I bump timeouts and split up tests. Consider it a CI stress-test!