Test suite requires active network connection

[freakboy3742]

Describe the bug

The Briefcase test suite requires external network access to run. If the network

Steps to reproduce

1. Turn off your wifi/network connection
2. tox -m test

Test suite will fail ~23 tests, mostly related to downloading tools.

Expected behavior

Test suite should pass without network access being required.

Screenshots

No response

Environment

• Operating System:All
• Python version: All
• Software versions:
  • Briefcase: 0.3.24

Logs

As of July 22, the test failures are:

FAILED tests/integrations/android_sdk/AndroidSDK/test_properties.py::test_cmdline_tools_url[Darwin-arm64-mac] - httpx.ConnectError: [Errno 8] nodename nor servname provided, or not known
FAILED tests/integrations/android_sdk/AndroidSDK/test_properties.py::test_cmdline_tools_url[Darwin-x86_64-mac] - httpx.ConnectError: [Errno 8] nodename nor servname provided, or not known
FAILED tests/integrations/android_sdk/AndroidSDK/test_properties.py::test_cmdline_tools_url[Linux-x86_64-linux] - httpx.ConnectError: [Errno 8] nodename nor servname provided, or not known
FAILED tests/integrations/android_sdk/AndroidSDK/test_properties.py::test_cmdline_tools_url[Windows-AMD64-win] - httpx.ConnectError: [Errno 8] nodename nor servname provided, or not known
FAILED tests/integrations/java/test_JDK__verify.py::test_successful_jdk_download[Darwin-x86_64-https://github.com/adoptium/temurin17-binaries/releases/download/jdk-17.0.15+6/OpenJDK17U-jdk_x64_mac_hotspot_17.0.15_6.tar.gz-java17/Contents/Home-False] - httpx.ConnectError: [Errno 8] nodename nor servname provided, or not known
FAILED tests/integrations/java/test_JDK__verify.py::test_successful_jdk_download[Darwin-arm64-https://github.com/adoptium/temurin17-binaries/releases/download/jdk-17.0.15+6/OpenJDK17U-jdk_aarch64_mac_hotspot_17.0.15_6.tar.gz-java17/Contents/Home-False] - httpx.ConnectError: [Errno 8] nodename nor servname provided, or not known
FAILED tests/integrations/java/test_JDK__verify.py::test_successful_jdk_download[Linux-x86_64-https://github.com/adoptium/temurin17-binaries/releases/download/jdk-17.0.15+6/OpenJDK17U-jdk_x64_linux_hotspot_17.0.15_6.tar.gz-java17-False] - httpx.ConnectError: [Errno 8] nodename nor servname provided, or not known
FAILED tests/integrations/java/test_JDK__verify.py::test_successful_jdk_download[Linux-aarch64-https://github.com/adoptium/temurin17-binaries/releases/download/jdk-17.0.15+6/OpenJDK17U-jdk_aarch64_linux_hotspot_17.0.15_6.tar.gz-java17-False] - httpx.ConnectError: [Errno 8] nodename nor servname provided, or not known
FAILED tests/integrations/java/test_JDK__verify.py::test_successful_jdk_download[Linux-aarch64-https://github.com/adoptium/temurin17-binaries/releases/download/jdk-17.0.15+6/OpenJDK17U-jdk_arm_linux_hotspot_17.0.15_6.tar.gz-java17-True] - httpx.ConnectError: [Errno 8] nodename nor servname provided, or not known
FAILED tests/integrations/java/test_JDK__verify.py::test_successful_jdk_download[Linux-armv7l-https://github.com/adoptium/temurin17-binaries/releases/download/jdk-17.0.15+6/OpenJDK17U-jdk_arm_linux_hotspot_17.0.15_6.tar.gz-java17-False] - httpx.ConnectError: [Errno 8] nodename nor servname provided, or not known
FAILED tests/integrations/java/test_JDK__verify.py::test_successful_jdk_download[Linux-armv8l-https://github.com/adoptium/temurin17-binaries/releases/download/jdk-17.0.15+6/OpenJDK17U-jdk_arm_linux_hotspot_17.0.15_6.tar.gz-java17-False] - httpx.ConnectError: [Errno 8] nodename nor servname provided, or not known
FAILED tests/integrations/java/test_JDK__verify.py::test_successful_jdk_download[Windows-AMD64-https://github.com/adoptium/temurin17-binaries/releases/download/jdk-17.0.15+6/OpenJDK17U-jdk_x64_windows_hotspot_17.0.15_6.zip-java17-False] - httpx.ConnectError: [Errno 8] nodename nor servname provided, or not known
FAILED tests/integrations/linuxdeploy/test_LinuxDeployGtkPlugin__properties.py::test_download_url - httpx.ConnectError: [Errno 8] nodename nor servname provided, or not known
FAILED tests/integrations/linuxdeploy/test_LinuxDeployQtPlugin__properties.py::test_download_url[Linux-x86_64-x86_64] - httpx.ConnectError: [Errno 8] nodename nor servname provided, or not known
FAILED tests/integrations/linuxdeploy/test_LinuxDeployQtPlugin__properties.py::test_download_url[Linux-i686-i386] - httpx.ConnectError: [Errno 8] nodename nor servname provided, or not known
FAILED tests/integrations/linuxdeploy/test_LinuxDeployQtPlugin__properties.py::test_download_url[Darwin-x86_64-x86_64] - httpx.ConnectError: [Errno 8] nodename nor servname provided, or not known
FAILED tests/integrations/linuxdeploy/test_LinuxDeploy__properties.py::test_download_url[Linux-x86_64-x86_64] - httpx.ConnectError: [Errno 8] nodename nor servname provided, or not known
FAILED tests/integrations/linuxdeploy/test_LinuxDeploy__properties.py::test_download_url[Linux-i686-i386] - httpx.ConnectError: [Errno 8] nodename nor servname provided, or not known
FAILED tests/integrations/linuxdeploy/test_LinuxDeploy__properties.py::test_download_url[Darwin-x86_64-x86_64] - httpx.ConnectError: [Errno 8] nodename nor servname provided, or not known
FAILED tests/integrations/rcedit/test_RCEdit__properties.py::test_rcedit_url - httpx.ConnectError: [Errno 8] nodename nor servname provided, or not known
FAILED tests/integrations/wix/test_WiX__verify.py::test_download_missing - httpx.ConnectError: [Errno 8] nodename nor servname provided, or not known
==================================================================== 21 failed, 3352 passed, 41 skipped in 82.22s (0:01:22) ====================================================================

Additional context

The fix for these tests should all be very similar - mocking the underlying httpx requests so that network access isn't required.

The network requirement access isn't normally a problem - but if the CI box (or the user) has an unstable network connection, it can manifest as test failures (see this example). These tests are also the slowest tests in the suite, so mocking will have the added benefit of speeding up the test suite.

[rmartin16]

We added these tests to explicitly ensure the URLs were resolvable on the internet. So, I don't think we want to mock out the network connection. We may as well just remove the tests at that point. Alternatively, it may be reasonable to add a flag to allow the test suite to pass while xfailing/skipping these perhaps.

[freakboy3742]

Hrm... I forgot that those tests were deliberate. We definitely need to retain some sort of check that the URLs remain valid - but I think the build app tests almost completely meet that requirement.

Looking at the logs from recent builds, Windows, Linux and macOS Toga builds all download the Android SDK; and Windows builds download the RCEdit and WiX binary. There's no tests of the Linuxdeploy downloads... but we're almost at the point of formally deprecating AppImage support anyway. Those are all "live" tests of the download, so we're not just doing a HEAD request - we're checking that the final binary is actually usable.

That only leaves the Java download - CI uses the JDK on the CI machine via JAVA_HOME. However, we don't really want to add the JDK download to CI, because it is redundant, and the JDK download isn't small, so the test slowdown would be noticeable; and guaranteeing isolation from the CI environment would be difficult.

As for skipping/xfailing - the problem is that the place we're most likely to see this problem is in CI (because Github Actions networking is so unreliable) - but that's the one place where we can't enable a soft skip, because it's the canary for a missing/changed download.

So - maybe in the grand scheme of things, a couple of slow (and very occasionally unreliable) tests isn't that bad. Closing on that basis.