ci: add max_auto_reruns to test-image job

[MikeMcC399]

• closes Transient CI HTTP 502 Bad Gateway errors #1168

Situation

The CircleCI pipeline based on circle.yml regularly exhibits flakiness due to network issues. The highest number of these issues are related to 502 Bad Gateway errors connecting to https://jsonplaceholder.cypress.io.

An example is https://app.circleci.com/pipelines/github/cypress-io/cypress-docker-images/2745/workflows/ed480f85-f0ba-4d64-b7f1-8a38620cb50e/jobs/86471 which issues cy.request('https://jsonplaceholder.cypress.io/comments') and fails with a 502 Bad Gateway error:

  Network Requests
[198:0924/142943.247804:ERROR:dbus/bus.cc:408] Failed to connect to the bus: Address does not contain a colon
[198:0924/142943.247835:ERROR:dbus/bus.cc:408] Failed to connect to the bus: Address does not contain a colon
[198:0924/142943.247845:ERROR:dbus/bus.cc:408] Failed to connect to the bus: Address does not contain a colon
    1) cy.request() - make an XHR request
    ✓ cy.request() - verify response using BDD syntax (227ms)
    ✓ cy.request() with query parameters (155ms)
    ✓ cy.request() - pass result to the second request (201ms)
    ✓ cy.request() - save response in the shared test context (175ms)
    ✓ cy.intercept() - route responses to matching requests (421ms)


  5 passing (3s)
  1 failing

  1) Network Requests
       cy.request() - make an XHR request:

      AssertionError: expected 502 to be one of [ 500, 501 ]
      + expected - actual

      -502
      +[ 500, 501 ]

      at Context.eval (webpack://test-project/./cypress/e2e/2-advanced-examples/network_requests.cy.js:17:64)
      at <unknown> (https://example.cypress.io/__cypress/runner/cypress_runner.js:121042:17)
      at tryCatcher (https://example.cypress.io/__cypress/runner/cypress_runner.js:1777:23)
      at Promise.attempt.Promise.try (https://example.cypress.io/__cypress/runner/cypress_runner.js:4285:29)
      at Context.shouldFnWithCallback (https://example.cypress.io/__cypress/runner/cypress_runner.js:121039:66)
      at Context.shouldFn (https://example.cypress.io/__cypress/runner/cypress_runner.js:121082:37)

https://jsonplaceholder.cypress.io/comments is hosted on cloudflare.

Change

Making use of the CircleCI Automatic reruns functionality, add max_auto_reruns and auto_rerun_delay to the test-image job, test step:

      - run:
          name: test
          command: |
            docker compose --progress plain build << parameters.test-target >>
            docker compose run --rm << parameters.test-target >>
          max_auto_reruns: 3
          auto_rerun_delay: 5m
          working_directory: factory/test-project

The 5 minute delay is an empirical choice, assuming that the Cloudflare sources are rate-limited and need time to reset. CircleCI allows a maximum of 10 minutes delay.

With knowledge of how https://jsonplaceholder.cypress.io is set up, it may be possible to choose a more optimal delay time. A related question posted to #1168 (comment) is still open.

[cypress-app-bot]

• Create a Draft Pull Request if your PR is not ready for review. Mark the PR as Ready for Review when you're ready for a Cypress team member to review the PR.

[MikeMcC399]

This PR has run successfully without triggering any re-runs.

The pipeline will need to be monitored to watch for 502 Gateway errors and to see if the re-run strategy is effective.

I suggest to merge, since the change is at least not causing any regression and may potentially be helpful.