CPU CI takes over an hour and seems to overlap with data downloading

[glwagner]

Currently the CPU CI takes a very long time to run, eg:

https://github.com/NumericalEarth/NumericalEarth.jl/actions/runs/25079084434/job/73485468142?pr=149

It also seems to test data downloading (?) which overlaps with the data downloading GHA jobs.

Do we have an idea why GPU tests are much faster? Usually GPU takes longer because of longer compile; however in this case the CPU for CPU tests and CPU for GPU tests are very different (since compile happens on CPU).

We could consider reducing the CPU testing while keeping GPU more complete. Or maybe there are other strategies to speed up the CPU tests.

[simone-silvestri]

I think one of the culprit of the CPU being slower is that we have 4 workers vs 8 workers on the GPU. It might be a good idea to remove some tests from the CPU given that GPU tests are typically more stringent. Maybe we can remove the slowest ones. Given these times

                                     │   Test   │   Init   │ Compile │ ──────────────── CPU ──────────────── │
Test                        (Worker) │ time (s) │ time (s) │   (%)   │ GC (s) │ GC % │ Alloc (MB) │ RSS (MB) │
test_diagnostics_2               (1) │        started at 2026-04-28T22:49:11.667
test_speedy_coupling             (3) │        started at 2026-04-28T22:49:12.781
test_sea_ice_ocean_heat_fluxes   (2) │        started at 2026-04-28T22:49:12.781
test_flux_and_restoring          (4) │        started at 2026-04-28T22:49:12.883
test_diagnostics_2               (1) │   130.97 │     4.59 │   93.71 │   4.93 │  3.8 │    6624.32 │  1813.54 │
test_ocean_only_model            (1) │        started at 2026-04-28T22:51:24.972
test_flux_and_restoring          (4) │   225.87 │     5.63 │   96.35 │   9.29 │  4.1 │   10039.14 │  1817.42 │
test_ecco4_en4                   (4) │        started at 2026-04-28T22:53:00.762
test_ecco4_en4                   (4) │     1.47 │     1.57 │   89.51 │   0.00 │  0.0 │      55.95 │  1817.42 │
test_surface_fluxes              (4) │        started at 2026-04-28T22:53:04.476
test_surface_fluxes              (4) │   223.46 │     1.09 │   97.09 │   4.58 │  2.1 │    9847.25 │  1850.53 │
test_ocean_sea_ice_model         (4) │        started at 2026-04-28T22:56:50.004
test_sea_ice_ocean_heat_fluxes   (2) │   502.24 │     5.54 │   96.18 │  13.90 │  2.8 │   20792.15 │  1828.25 │
test_diagnostics_1               (2) │        started at 2026-04-28T22:57:37.352
test_speedy_coupling             (3) │   513.78 │     5.52 │   69.31 │  17.47 │  3.4 │   18771.86 │  2003.42 │
test_veros                       (3) │        started at 2026-04-28T22:57:48.648
test_veros                       (3) │    39.64 │     1.67 │    failed at 2026-04-28T22:58:31.117
test_ecco_atmosphere             (3) │        started at 2026-04-28T22:58:31.118
test_ecco_atmosphere             (3) │   105.21 │     1.29 │   37.12 │   0.51 │  0.5 │    2294.90 │  2003.42 │
test_bathymetry                  (3) │        started at 2026-04-28T23:00:18.528
test_bathymetry                  (3) │   262.97 │     1.25 │   31.83 │  17.27 │  6.6 │   35721.45 │  5857.54 │
test_ecco2_monthly               (5) │        started at 2026-04-28T23:04:50.095
test_ecco2_monthly               (5) │    14.91 │     4.92 │   36.86 │   1.41 │  9.5 │     606.88 │  1923.92 │
test_ecco2_daily                 (5) │        started at 2026-04-28T23:05:06.773
test_ocean_only_model            (1) │  1454.85 │     1.52 │   94.00 │  62.78 │  4.3 │   45549.08 │  5486.90 │
test_woa                         (6) │        started at 2026-04-28T23:15:45.633
test_woa                         (6) │   642.73 │     3.88 │   13.44 │  11.63 │  1.8 │   13115.46 │  1923.92 │
test_metadata                    (6) │        started at 2026-04-28T23:26:30.174
test_metadata                    (6) │     4.27 │     1.46 │   95.42 │   0.00 │  0.0 │     248.16 │  1923.92 │
test_column_field                (6) │        started at 2026-04-28T23:26:36.676
test_column_field                (6) │    21.46 │     1.01 │   93.35 │   1.00 │  4.7 │    1732.98 │  1923.92 │
test_jra55                       (6) │        started at 2026-04-28T23:26:59.866
test_jra55                       (6) │   122.07 │     1.03 │   69.75 │   4.67 │  3.8 │   12297.83 │  1923.92 │
test_checkpointer                (6) │        started at 2026-04-28T23:29:03.821
test_ecco2_daily                 (5) │  1792.13 │     1.30 │   37.76 │  28.92 │  1.6 │   52153.43 │  3365.21 │
test_orca_grid                   (5) │        started at 2026-04-28T23:35:01.296
test_orca_grid                   (5) │    87.34 │     1.29 │   60.91 │  16.18 │ 18.5 │    3694.77 │  3365.21 │
test_breeze_coupling             (5) │        started at 2026-04-28T23:36:31.048
test_checkpointer                (6) │   466.03 │     1.10 │   96.59 │  13.00 │  2.8 │   21185.66 │  2566.55 │
test_breeze_coupling             (5) │    92.31 │     1.40 │   98.90 │   2.65 │  2.9 │    6012.09 │  3365.21 │
test_diagnostics_1               (2) │  3272.79 │     1.74 │    2.99 │   8.63 │  0.3 │   12496.58 │  5304.73 │
test_ocean_sea_ice_model         (4) │  3368.45 │     1.29 │   40.18 │  77.11 │  2.3 │   49659.31 │  5309.68 │

the main candidates to remove from CPU testing are test_ocean_only_model, test_ocean_sea_ice_model, test_diagnostics_1 and test_ecco2_daily

[simone-silvestri]

With regards to the veros test failing, I noticed that recently this test fails randomly. I can try to investigate a bit with claude's aid on what is happening to try to remove the flakyness

[simone-silvestri]

closed by #198