build-system: add capability to execute scripts with custom executor

[miri64]

Contribution description

I sometimes find it more convenient to use pytest to execute and debug my more complex tests (it shows output in order of test execution, in color, you can filter tests and has flags to step in with pdb, ...). However, for that I either need to define all the RIOT environment in my shell, or touch makefile/tests/tests.mk to execute the test with pytest. Since I do not want to enforce pytest on everyone and the CI, this amends that file with two optional macros TEST_EXECUTOR and TEST_EXECUTOR_FLAGS, so the executor for the tests can be selected from the environment.

Testing procedure

Sadly, not all tests are able to be run this way: when the test... function has an argument, pytest thinks that argument is a fixture and gets confused (another reason why I made pytest optional). However, all tests that are based on unittest classes (so either unittests.TestCase or testrunner.unittest.PexpectTestCase (e.g. tests/turo can now be executed this way):

$ TEST_EXECUTOR=pytest TEST_EXECUTOR_FLAGS="-v -s" QUIETER=1 make -C tests/turo/ --no-print-directory -j flash test
Building application "tests_turo" for "native" with MCU "native".

/usr/bin/ld: warning: /home/mlenders/Repositories/RIOT-OS/RIOT/tests/turo/bin/native/tests_turo.elf has a LOAD segment with RWX permissions
   text	   data	    bss	    dec	    hex	filename
  39098	    704	  47800	  87602	  15632	/home/mlenders/Repositories/RIOT-OS/RIOT/tests/turo/bin/native/tests_turo.elf
============================= test session starts ==============================
platform linux -- Python 3.10.8, pytest-7.1.3, pluggy-1.0.0 -- /usr/bin/python
cachedir: .pytest_cache
rootdir: /home/mlenders/Repositories/RIOT-OS/RIOT/tests/turo
plugins: cov-4.0.0, anyio-3.6.1
collected 13 items                                                             

tests/01-run.py::TestTuro::test_test_multi_element_dict PASSED
tests/01-run.py::TestTuro::test_test_netif PASSED
tests/01-run.py::TestTuro::test_turo_simple_array_s32 PASSED
tests/01-run.py::TestTuro::test_turo_simple_array_s32_fail PASSED
tests/01-run.py::TestTuro::test_turo_simple_array_u8 PASSED
tests/01-run.py::TestTuro::test_turo_simple_array_u8_fail PASSED
tests/01-run.py::TestTuro::test_turo_simple_dict_s32 PASSED
tests/01-run.py::TestTuro::test_turo_simple_dict_s32_fail PASSED
tests/01-run.py::TestTuro::test_turo_simple_dict_string PASSED
tests/01-run.py::TestTuro::test_turo_simple_dict_string_fail PASSED
tests/01-run.py::TestTuro::test_turo_simple_exit_status PASSED
tests/01-run.py::TestTuro::test_turo_simple_s32 PASSED
tests/01-run.py::TestTuro::test_turo_simple_s32_fail PASSED

============================== 13 passed in 4.52s ==============================

Of course, the normal approach should also still work:

$ QUIETER=1 make -C tests/turo/ --no-print-directory -j flash test
Building application "tests_turo" for "native" with MCU "native".

/usr/bin/ld: warning: /home/mlenders/Repositories/RIOT-OS/RIOT/tests/turo/bin/native/tests_turo.elf has a LOAD segment with RWX permissions
   text	   data	    bss	    dec	    hex	filename
  39098	    704	  47800	  87602	  15632	/home/mlenders/Repositories/RIOT-OS/RIOT/tests/turo/bin/native/tests_turo.elf
.............
----------------------------------------------------------------------
Ran 13 tests in 4.453s

OK

Issues/PRs references

None, but got the idea while debugging #18769.

[riot-ci]

Murdock results

✔️ PASSED

14b7abd build-system: add capability to execute scripts with custom executor

Success	Failures	Total	Runtime
1991	0	1991	06m:23s

Artifacts

• Documentation preview

This only reflects a subset of all builds from https://ci-prod.riot-os.org. Please refer to https://ci.riot-os.org for a complete build for now.