Something I've thought about that might help speed up testing is consolidating tests. Right now we have a whole bunch of small, atomized tests, which is great for allowing parallel processing and narrowing down problems quickly. However, running a test which produces a pdf has a whole bunch of "fixed" costs from starting LuaTeX and loading the packages. If two (or more) tests are combined, then they can share those fixed costs. This means that the combined test should run faster than the individual tests can be run sequentially.
Given our number of tests, it's impossible to fully parallelize the testing; some tests will have to be run sequentially on the same processor. This suggests that there should be an optimal balance between multiple tests (for parallelization purposes) and combined tests (to make sequential operations more efficient). The absolute optimum will vary by the number of processors each user dedicates to running tests (either because of the number in their machine, or simply their need to have processing power available for other tasks while the tests run), but we might be able to get closer to optimum by making fewer larger tests, especially for older tests that we mostly keep around to prevent regressions.
What do others think about this idea?
Incidentally, combining tests would be an obvious solution for #369.
Something I've thought about that might help speed up testing is consolidating tests. Right now we have a whole bunch of small, atomized tests, which is great for allowing parallel processing and narrowing down problems quickly. However, running a test which produces a pdf has a whole bunch of "fixed" costs from starting LuaTeX and loading the packages. If two (or more) tests are combined, then they can share those fixed costs. This means that the combined test should run faster than the individual tests can be run sequentially.
Given our number of tests, it's impossible to fully parallelize the testing; some tests will have to be run sequentially on the same processor. This suggests that there should be an optimal balance between multiple tests (for parallelization purposes) and combined tests (to make sequential operations more efficient). The absolute optimum will vary by the number of processors each user dedicates to running tests (either because of the number in their machine, or simply their need to have processing power available for other tasks while the tests run), but we might be able to get closer to optimum by making fewer larger tests, especially for older tests that we mostly keep around to prevent regressions.
What do others think about this idea?
Incidentally, combining tests would be an obvious solution for #369.