Add a healthcheck to the dockercompose for crackq to detect when GPU devices have disappeared

[hkelley]

In an attempt to address the NVIDIA GPU flukiness (the crackq container sometimes loses the devices - NVIDIA/nvidia-container-toolkit#48), I'm experimenting with:

1. Adding a healthcheck to the crackq service in docker-compose to detect when the GPUs go missing

    crackq:
        build:
            context: ./build
            dockerfile: Dockerfile
        image: "nvidia-ubuntu"
        ports:
            - "127.0.0.1:8080:8080"
        depends_on:
            - redis
        healthcheck:
          test: hashcat -I | grep 'Backend Device'
          interval: 5m
          retries: 1
          start_period: 60s
          timeout: 30s
        networks:
            - crackq_net

2. Once I'm confident the healthcheck is reliable, adding a service for https://hub.docker.com/r/willfarrell/autoheal/ to the docker-compose. This should be able to restart the crackq container. https://stackoverflow.com/questions/47088261/restarting-an-unhealthy-docker-container-based-on-healthcheck

I will update this issue as I make progress.

[hkelley]

Step one complete. Docker healthcheck correctly marked the container as unhealthy:

	#  sudo docker inspect --format='{{json .State.Health}}' crackq
	{
		"Status": "unhealthy",
		"FailingStreak": 38,
		"Log": [
			{
				"Start": "2024-08-23T14:03:59.238735966Z",
				"End": "2024-08-23T14:03:59.339752004Z",
				"ExitCode": 1,
				"Output": "\u001B[31mcuInit(): no CUDA-capable device is detected\u001B[0m\n\n\u001B[31mclGetPlatformIDs(): CL_PLATFORM_NOT_FOUND_KHR\u001B[0m\n\n\u001B[31mATTENTION! No OpenCL-compatible or CUDA-compatible platform found.\u001B[0m\n\n"
			},

We had previously been detecting this via log monitoring for the following:

SystemError: <method 'hashcat_session_execute' of 'pyhashcat.hashcat' objects> returned a result with an error set

On to step two, autoheal.

[hkelley]

Key snippets from the yml file below. This configuration has both a healthcheck in crackq container and a separate "restarter" container to bounce crackq/hashcat if it loses the GPUs.

    crackq:
...
        depends_on:
            - redis
        healthcheck:
          test: [  "CMD-SHELL"
                 , "echo \"Expecting $$GPU_INSTALLED GPUs.\";
                   gpu_c=$$(hashcat -I | grep -c \"Backend Device\");
                   echo \"Found $$gpu_c GPUs exposed to hashcat\";
                   if [ $$gpu_c = $$GPU_INSTALLED ]; then exit 0; else exit 1; fi"  ]
          interval: 5m
          retries: 1
          start_period: 10s
          timeout: 30s
...
        environment:
...
                GPU_INSTALLED: ${GPU_INSTALLED:?The GPU_INSTALLED variable must be set so that healthchecks can run}
....
    unhealthy_container_restarter:
        image: docker:cli
        hostname: "unhealthy_container_restarter"
        network_mode: none
        cap_drop:
           - ALL
        volumes: [ "/var/run/docker.sock:/var/run/docker.sock" ]
        command: [ "/bin/sh", "-c", "while true; do sleep 60; docker ps -q -f health=unhealthy | xargs -t -r docker restart; done" ]
        restart: unless-stopped

The details of the healthcheck are reported here:

 sudo docker inspect --format "{{json .State.Health }}"  crackq

{
	"Status": "healthy",
	"FailingStreak": 0,
	"Log": [
		{
			"Start": "2024-09-27T20:57:55.597041742Z",
			"End": "2024-09-27T20:58:08.20713869Z",
			"ExitCode": 0,
			"Output": "Expecting 4 GPUs.\n\u001B[31mclGetPlatformIDs(): CL_PLATFORM_NOT_FOUND_KHR\u001B[0m\n\nFound 4 GPUs exposed to hashcat\n"
		},
		{
			"Start": "2024-09-27T21:03:08.213678798Z",
			"End": "2024-09-27T21:03:20.870452953Z",
			"ExitCode": 0,
			"Output": "Expecting 4 GPUs.\n\u001B[31mclGetPlatformIDs(): CL_PLATFORM_NOT_FOUND_KHR\u001B[0m\n\nFound 4 GPUs exposed to hashcat\n"
		}
	]
}