Address flaky CI in networking steps

[nathanjmcdougall]

The CI seems to be flaky for this step:

https://github.com/pypest/pyemu/blob/5dd3c785991ec991974c53cd0b1058281a2f872f/.github/workflows/ci.yml#L118-L121}

It's failing with Error 504: Gateway Time-out for quite a few PRs

Details

Run get-pestpp --owner pestpp --repo pestpp-nightly-builds :home
Matplotlib is building the font cache; this may take a moment.
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\runneradmin\micromamba\envs\pyemu\Scripts\get-pestpp.exe\__main__.py", line 6, in <module>
    sys.exit(cli_main())
             ~~~~~~~~^^
  File "D:\a\pyemu\pyemu\pyemu\utils\get_pestpp.py", line 803, in cli_main
    run_main(**args, _is_cli=True)
    ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
  File "D:\a\pyemu\pyemu\pyemu\utils\get_pestpp.py", line 481, in run_main
    urllib.request.urlretrieve(download_url, download_pth)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\runneradmin\micromamba\envs\pyemu\Lib\urllib\request.py", line 214, in urlretrieve
    with contextlib.closing(urlopen(url, data)) as fp:
                            ~~~~~~~^^^^^^^^^^^
  File "C:\Users\runneradmin\micromamba\envs\pyemu\Lib\urllib\request.py", line 189, in urlopen
    return opener.open(url, data, timeout)
           ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\runneradmin\micromamba\envs\pyemu\Lib\urllib\request.py", line 495, in open
    response = meth(req, response)
  File "C:\Users\runneradmin\micromamba\envs\pyemu\Lib\urllib\request.py", line 604, in http_response
    response = self.parent.error(
        'http', request, response, code, msg, hdrs)
  File "C:\Users\runneradmin\micromamba\envs\pyemu\Lib\urllib\request.py", line 533, in error
    return self._call_chain(*args)
           ~~~~~~~~~~~~~~~~^^^^^^^
  File "C:\Users\runneradmin\micromamba\envs\pyemu\Lib\urllib\request.py", line 466, in _call_chain
    result = func(*args)
  File "C:\Users\runneradmin\micromamba\envs\pyemu\Lib\urllib\request.py", line 613, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 504: Gateway Time-out
auto-selecting option ':home' for 'bindir'

Fetched release '20251214' info from 'pestpp/pestpp-nightly-builds'.

Downloading 'https://github.com/pestpp/pestpp-nightly-builds/releases/download/20251214/pestpp-5.2.24dev20251214-win.zip' to 'C:\Users\runneradmin\Downloads\pestpp-20251214-pestpp-5.2.24dev20251214-win.zip'.
Error: Process completed with exit code 1.

I would start by adding a retry logic, e.g. with

 run: |
    for i in {1..5}; do
      get-pestpp --owner pestpp --repo pestpp-nightly-builds :home && break
      echo "Retry in 10 seconds..."
      sleep 10
    done

In the future, it might be worth considering a cache for the version downloads to reduce changes of getting API rate limited etc. But since they're nightly builds, it might not add as much value to do that.

[briochh]

wondering if these 504 errors are actually triggered at the GHA server end rather than the upstream repo as I also see them at other parts of the CI (e.g. micromamba install). May just be a symptom of hitting transfer requests with high frequency when a whole heap of CI runs get triggered at once. The true fix might be in staging CI runs better!

[briochh]

further note -- we occasionally hit the same 504 error when testing the get-pestpp functions within the tests. Maybe this is an evolution (or escalation) of the rate limiting errors we have seen before. @jdhughes-dev and/or @mwtoews might be able to comment.

[nathanjmcdougall]

Yeah I suspect likely to be related to GitHub API rate limiting. But yeah I think simple fix for now is probably just to take a look at configuring GitHub actions to try and force some queueing between runners, reduce the parallelism.

I still think it's probably good idea to add randomized exponential backoff to add a bit of resilience to things, but probably better to do that in the upstream tools like get-pestpp.

With micromamba install, I suspect setting up a cache would help reduce unnecessary package re-downloads.

[ynathaniel-source]

Hey! 👋

I looked into your flaky CI networking steps and identified the root cause - 504 Gateway Timeout errors when downloading dependencies.

What's Happening

Your CI is failing intermittently because:

1. The remote server (hosting pestpp-nightly-builds) isn't responding in time
2. Network requests are timing out before completion
3. No retry mechanism is in place to handle transient failures

How to Fix

Solution 1: Add Retry Logic with Exponential Backoff (Recommended)

# In .github/workflows/your-workflow.yml
- name: Download pestpp with retry
  run: |
    for i in {1..5}; do
      if get-pestpp --owner pestpp --repo pestpp-nightly-builds :home; then
        echo "✓ Download successful"
        break
      else
        if [ $i -lt 5 ]; then
          wait_time=$((2**i))
          echo "⚠ Attempt $i failed, retrying in ${wait_time}s..."
          sleep $wait_time
        else
          echo "✗ All retry attempts failed"
          exit 1
        fi
      fi
    done

Solution 2: Increase Timeout

- name: Download pestpp
  timeout-minutes: 10  # Increase from default
  run: get-pestpp --owner pestpp --repo pestpp-nightly-builds :home

Solution 3: Use GitHub Actions Cache

- name: Cache pestpp
  uses: actions/cache@v3
  with:
    path: ~/pestpp
    key: pestpp-${{ runner.os }}-${{ hashFiles('**/pestpp-version.txt') }}
    
- name: Download pestpp if not cached
  if: steps.cache.outputs.cache-hit != 'true'
  run: get-pestpp --owner pestpp --repo pestpp-nightly-builds :home

Why This Works

• Retry logic handles transient network issues automatically
• Exponential backoff prevents overwhelming the server
• Caching reduces dependency on external servers

Prevention Tips

• Always add retry logic for external network calls in CI
• Use caching for large dependencies
• Monitor external service status pages
• Consider mirroring critical dependencies

Estimated Fix Time: 10-15 minutes
Impact: High - Flaky CI wastes developer time and slows down development

Let me know if you need help implementing any of these solutions! You can also automatically diagnose this with https://github.com/marketplace/actions/whydiditfail