[Task]: Improve addons-server CI speed

[diox]

Description

addons-server's test_main (x) CI jobs are slow. Those jobs run our Python tests through pytest, and the following things are slowing them down:

• Starting full services. Those jobs start a full local development environment, but really we only need a few services. We should avoid starting:
  • addons-frontend (no frontend necessary)
  • worker (all celery tasks from unit tests should run in eager mode directly on the same container)
  • rabbitmq (see above)
  • elasticsearch (tests that need elasticsearch are already separate and not part of test_main` jobs)
  • nginx (no HTTP server necessary)
• Reinstalling prod dependencies. Installing dev dependencies on top of the production container is needed, but we should skip re-installing prod ones: they are already there.
• Recompiling locales. This is already done by building the web container. Furthermore, tests that need locales are already separate and not part of test_main jobs.
• Doing a full initialize of the web container. This is completely useless, those jobs won't use the main database.

The total cost for all of these things can add up to a couple minutes, sometimes more.

In addition, we can look into increasing the number of jobs, which has been hardcoded at 14 since mozilla/addons-server#22318 but hasn't been updated as the number of tests rose. We've got over 8k tests now, so we likely want to hardcode 15 or 16 jobs if we're not going to have automated logic around that.

Finally, there are a number of optimizations to make in tests themselves, but outside of easy quick wins, that should be a separate ticket.

Acceptance Criteria

• Our CI completes faster

┆Issue is synchronized with this Jira Task

[diox]

We should also look at whether increasing the number of pytest workers per job would help. The standard GitHub Actions runner has 2 cores, 4 with hyperthreading, and we run tests with -n auto, which results in 4 workers. But, I suspect most of the time spent in tests is waiting for I/O, so it's possible more than 4 workers would actually be beneficial. We can test that...

Edit: not faster, it even appears to hangs with -n 8.

[diox]

Another thing to consider: there is a lot of overhead per job: pytest typically finishes ~ 600 tests in 4 minutes, but there is an extra 5 minutes for... everything else, mostly docker setup.

600 tests in 4 minutes is obviously slow, that's mostly due to a bunch of tests relying on the database and not leveraging setUpTestData to avoid rolling back and redoing setup for each test, but still, 5 minutes of overhead is way too high.

[diox]

Tests from mozilla/addons-server#24493 (from a fork, which matters) are inconclusive. Bumping to 16 or even 20 does make the pytest tests execute faster (fewer tests to execute) but the overall time is not affected that much. We would likely gain more by optimizing the tests themselves or the setup.

[diox]

Another test I ran: we do a bunch of things in test_pre_setup creating a bunch of temporary directories. It's not that much faster without that though, and obviously it breaks a lot of tests.