Add a test runner that can run tests in parallel and in network namespaces with temporary directories.

[fruffy]

No description provided.

[jafingerhut]

I tried this with two test runs sequentially, while running top. Both tests passed, and I observed that it correctly stopped any tofino-model or bf_switchd process when the test run was done.

[jafingerhut]

I have not attempted to read the run-test.py script in detail, but it tests well for me, and also passes Ubuntu 22.04 CI test.

[jafingerhut]

I tried to run two tests in paralllel on my local system with these changes. One of the two tests had a very long delay before it started, and passed, but the other one failed. It was significantly faster to run the two tests sequentially.

You are welcome to spend N hours on getting tests to run in parallel, but I would strongly suggest considering being content with only running tests sequentially.

[fruffy]

I tried to run two tests in paralllel on my local system with these changes. One of the two tests had a very long delay before it started, and passed, but the other one failed. It was significantly faster to run the two tests sequentially.

You are welcome to spend N hours on getting tests to run in parallel, but I would strongly suggest considering being content with only running tests sequentially.

Curious to try this on CI. If you give me a list of tests I can automatically parallelize them.

[jafingerhut]

I tried to run two tests in paralllel on my local system with these changes. One of the two tests had a very long delay before it started, and passed, but the other one failed. It was significantly faster to run the two tests sequentially.
You are welcome to spend N hours on getting tests to run in parallel, but I would strongly suggest considering being content with only running tests sequentially.

Curious to try this on CI. If you give me a list of tests I can automatically parallelize them.

Here is a list of tests I just ran locally on my Ubuntu 22.04 VM built from source code as modified by PR #65, and using the version of run-test.py from this PR.

Note: I ran all of these tests sequentially. When they were run sequentially, each took at most 90 seconds, and most of them 30 sec or less. They all passed. There are more tests in this same category, but I wanted to get you a decent-length list of tests in this category sooner, while the rest are still running.

basic_switching
bri_handle
bri_with_pdfixed_thrift
chksum
default_entry
deparse_zero
fast_reconfig
ha
hash_driven
iterator
mirror_test
parser_error
parser_intr_md
perf_test
resubmit
tna_32q_2pipe
tna_32q_multiprogram
tna_action_profile
tna_alpm
tna_bridged_md
tna_checksum

[fruffy]

I tried to run two tests in paralllel on my local system with these changes. One of the two tests had a very long delay before it started, and passed, but the other one failed. It was significantly faster to run the two tests sequentially.
You are welcome to spend N hours on getting tests to run in parallel, but I would strongly suggest considering being content with only running tests sequentially.

Curious to try this on CI. If you give me a list of tests I can automatically parallelize them.

Here is a list of tests I just ran locally on my Ubuntu 22.04 VM built from source code as modified by PR #65, and using the version of run-test.py from this PR.

Note: I ran all of these tests sequentially. When they were run sequentially, each took at most 90 seconds, and most of them 30 sec or less. They all passed. There are more tests in this same category, but I wanted to get you a decent-length list of tests in this category sooner, while the rest are still running.

basic_switching
bri_handle
bri_with_pdfixed_thrift
chksum
default_entry
deparse_zero
fast_reconfig
ha
hash_driven
iterator
mirror_test
parser_error
parser_intr_md
perf_test
resubmit
tna_32q_2pipe
tna_32q_multiprogram
tna_action_profile
tna_alpm
tna_bridged_md
tna_checksum

Let's stick with sequential for now and parallize later. At least this script should work for that now.