GPU not working on prod

[homework36]

Updated 1/26:

Clarify. GPU jobs (excluding training) CAN be submitted on production, but we cannot use the full capacity of GPUs, so it takes a very long time. Please use staging during this time until further notice. There's nothing we can do from our end at this point.

---

Well, it's working, but also not working... Probably related to the Arbutus outage

root@prod-rodan-vgpu:/srv/webapps/Rodan# make gpu-celery_log
[2026-01-19 19:24:37,467: INFO/MainProcess] Connected to amqp://someadmin:**@rabbitmq:5672//
[2026-01-19 19:24:37,659: INFO/MainProcess] mingle: searching for neighbors
[2026-01-19 19:24:39,375: INFO/MainProcess] mingle: all alone
[2026-01-19 19:24:40,428: INFO/MainProcess] celery@GPU ready.
[2026-01-19 19:31:07,251: INFO/MainProcess] Received task: Fast Pixelwise Analysis of Music Document, Classifying[0a08daed-c17f-42ef-92e6-849bfcd0755c]
[2026-01-19 19:31:07,331: INFO/ForkPoolWorker-1] started running the task!
[2026-01-19 19:37:30,073: WARNING/ForkPoolWorker-1] Fast Pixelwise Analysis of Music Document, Classifying[0a08daed-c17f-42ef-92e6-849bfcd0755c]: 0 / 1535

[stuck here forever]

root@prod-rodan-vgpu:/srv/webapps/Rodan# nvidia-smi
Tue Jan 20 00:43:30 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.54.15              Driver Version: 550.54.15      CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  GRID V100D-16C                 On  |   00000000:00:05.0 Off |                    0 |
| N/A   N/A    P0             N/A /  N/A  |   14470MiB /  16384MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A    254044      C   /usr/bin/python3.7                          14470MiB |
+-----------------------------------------------------------------------------------------+

[lizima]

[2026-01-23 11:37:24,582: INFO/MainProcess] Received task: Fast Pixelwise Analysis of Music Document, Classifying[691da11d-b504-4e86-a643-a85f784200e8]  
[2026-01-23 11:37:24,659: INFO/ForkPoolWorker-7] started running the task!
[2026-01-23 11:37:25,081: WARNING/ForkPoolWorker-7] The my_error_information method is not implemented properly (or not implemented at all). Exception: 
[2026-01-23 11:37:25,082: WARNING/ForkPoolWorker-7] TypeError: 'NoneType' object is not subscriptable
[2026-01-23 11:37:25,082: WARNING/ForkPoolWorker-7] Using default sources for error information.
[2026-01-23 11:37:25,981: ERROR/ForkPoolWorker-7] Task Fast Pixelwise Analysis of Music Document, Classifying[691da11d-b504-4e86-a643-a85f784200e8] raised unexpected: InternalError()
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/celery/app/trace.py", line 412, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/celery/app/trace.py", line 704, in __protected_call__
    return self.run(*args, **kwargs)
  File "/code/Rodan/rodan/jobs/base.py", line 791, in run
    retval = self.run_my_task(inputs, settings, arg_outputs)
  File "/code/Rodan/rodan/jobs/Paco_classifier/fast_paco_classifier.py", line 156, in run_my_task
    analyses = recognition.process_image_msae(image, model_paths, height, width, mode = mode)
  File "/usr/local/lib/python3.7/dist-packages/Paco_classifier/recognition_engine.py", line 54, in process_image_msae
    sae_models.append(load_model(model_paths[id_label]))
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/saving/save.py", line 202, in load_model
    compile)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/saving/hdf5_format.py", line 181, in load_model_from_hdf5
    custom_objects=custom_objects)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/saving/model_config.py", line 52, in model_from_config
    return deserialize(config, custom_objects=custom_objects)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/layers/serialization.py", line 163, in deserialize
    printable_module_name='layer')
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/utils/generic_utils.py", line 672, in deserialize_keras_object
    list(custom_objects.items())))
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/functional.py", line 669, in from_config
    config, custom_objects)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/functional.py", line 1289, in reconstruct_from_config
    process_node(layer, node_data)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/functional.py", line 1237, in process_node
    output_tensors = layer(input_tensors, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/base_layer.py", line 970, in __call__
    input_list)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/base_layer.py", line 1108, in _functional_construction_call
    inputs, input_masks, args, kwargs)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/base_layer.py", line 840, in _keras_tensor_symbolic_call
    return self._infer_output_signature(inputs, args, kwargs, input_masks)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/base_layer.py", line 878, in _infer_output_signature
    self._maybe_build(inputs)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/base_layer.py", line 2625, in _maybe_build
    self.build(input_shapes)  # pylint:disable=not-callable
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/layers/convolutional.py", line 204, in build
    dtype=self.dtype)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/base_layer.py", line 655, in add_weight
    caching_device=caching_device)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/training/tracking/base.py", line 815, in _add_variable_with_custom_getter
    **kwargs_for_getter)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/base_layer_utils.py", line 139, in make_variable
    shape=variable_shape if variable_shape else None)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/ops/variables.py", line 260, in __call__
    return cls._variable_v1_call(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/ops/variables.py", line 221, in _variable_v1_call
    shape=shape)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/ops/variables.py", line 199, in <lambda>
    previous_getter = lambda **kwargs: default_variable_creator(None, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/ops/variable_scope.py", line 2626, in default_variable_creator
    shape=shape)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/ops/variables.py", line 264, in __call__
    return super(VariableMetaclass, cls).__call__(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/ops/resource_variable_ops.py", line 1595, in __init__
    distribute_strategy=distribute_strategy)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/ops/resource_variable_ops.py", line 1722, in _init_from_args
    initial_value = initial_value()
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/initializers/initializers_v2.py", line 517, in __call__
    return self._random_generator.truncated_normal(shape, 0.0, stddev, dtype)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/initializers/initializers_v2.py", line 988, in truncated_normal
    shape=shape, mean=mean, stddev=stddev, dtype=dtype, seed=self.seed)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/util/dispatch.py", line 206, in wrapper
    return target(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/ops/random_ops.py", line 199, in truncated_normal
    shape_tensor = tensor_util.shape_tensor(shape)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/framework/tensor_util.py", line 1080, in shape_tensor
    return ops.convert_to_tensor(shape, dtype=dtype, name="shape")
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/profiler/trace.py", line 163, in wrapped
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/framework/ops.py", line 1566, in convert_to_tensor
    ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/framework/constant_op.py", line 339, in _constant_tensor_conversion_function
    return constant(v, dtype=dtype, name=name)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/framework/constant_op.py", line 265, in constant
    allow_broadcast=True)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/framework/constant_op.py", line 276, in _constant_impl
    return _constant_eager_impl(ctx, value, dtype, shape, verify_shape)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/framework/constant_op.py", line 301, in _constant_eager_impl
    t = convert_to_eager_tensor(value, ctx, dtype)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/framework/constant_op.py", line 97, in convert_to_eager_tensor
    ctx.ensure_initialized()
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/context.py", line 525, in ensure_initialized
    context_handle = pywrap_tfe.TFE_NewContext(opts)
tensorflow.python.framework.errors_impl.InternalError: CUDA runtime implicit initialization on GPU:0 failed. Status: all CUDA-capable devices are busy or unavailable

Just found that error

[lizima]

We tried restarting gpu docker container, but didn't work.
We also restarted Rodan and Docker, but didn't work.
We also checked nvidia-smi output, it looks fine on the instance and inside the gpu container.

[homework36]

Disclaimer:
It was very late for me so I dumped notes, logs, and findings into ChatGPT and asked it to format them into something readable. Logs and findings are real, but the language and formatting are completely AI-generated below this section.
TL;DR:
Issue found. This is an infrastructure-level NVIDIA vGPU licensing problem (not application-side). Evidence below. Will contact Compute Canada.

GPU CUDA Initialization Issue - vGPU License Problem

What happened

GPU jobs started failing on Jan 23 with this error:

tensorflow.python.framework.errors_impl.InternalError: CUDA runtime implicit initialization on GPU:0 failed. Status: all CUDA-capable devices are busy or unavailable

The weird thing was that nvidia-smi looked totally fine and showed the GPU available with 0% utilization.

What we tried

1. Restarted the gpu-celery container - didn't help

2. Checked if GPU was visible inside container - yes it was:

   docker exec -it <container> python -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"
   

   This worked and showed the GPU

3. Found stale Celery workers from Jan 23 that might have corrupted CUDA context - killed them and restarted. Still didn't work.

4. Tried changing Celery pool to --pool=solo to avoid fork issues with CUDA - modified production.yml and redeployed. Still same error.

5. Tested actual GPU computation (not just listing devices):

   with tf.device('/GPU:0'):
       a = tf.constant([[1.0, 2.0], [3.0, 4.0]])
       c = tf.matmul(a, a)

   This also failed with the same CUDA error - so it wasn't a Celery forking issue

6. Tried nvidia-smi --gpu-reset - failed because it's a vGPU:

   GPU Reset couldn't run because GPU 00000000:00:05.0 is in vGPU mode.
   

7. Rebooted the entire VM - after this, GPU operations started working but were EXTREMELY slow (~4 minutes just to initialize CUDA)

Root cause found

Checked the vGPU license status:

nvidia-smi -q | grep -i license

Output:

vGPU Software Licensed Product
    Product Name                      : NVIDIA Virtual Compute Server
    License Status                    : Unlicensed (Restricted)

The nvidia-gridd service is running and trying to acquire a license, but keeps failing:

Jan 26 03:44:45 prod-rodan-vgpu nvidia-gridd[605]: Acquiring license. (Info: api.cls.licensing.nvidia.com; NVIDIA Virtual Compute Server)
Jan 26 03:44:46 prod-rodan-vgpu nvidia-gridd[605]: Failed to acquire license from api.cls.licensing.nvidia.com

The license token exists at /etc/nvidia/ClientConfigToken/clientconfig.tok (created March 2024) and hasn't expired (exp date is year 3000 lol), and we can reach the license server fine:

curl -v https://api.cls.licensing.nvidia.com  # returns 404 but connection works

So the network is fine, token is valid, but license acquisition still fails.

Timeline from logs

When was license last acquired successfully:

$ journalctl -u nvidia-gridd --no-pager | grep -i "acquired"
Jan 03 09:50:05 prod-rodan-vgpu nvidia-gridd[12723]: License acquired successfully. (Info: api.cls.licensing.nvidia.com, NVIDIA Virtual Compute Server; Expiry: 2025-1-4 9:50:5 GMT)
Apr 24 15:22:48 prod-rodan-vgpu nvidia-gridd[800]: License acquired successfully. (Info: api.cls.licensing.nvidia.com, NVIDIA Virtual Compute Server; Expiry: 2025-4-25 15:22:48 GMT)
May 09 19:13:17 prod-rodan-vgpu nvidia-gridd[602]: License acquired successfully. (Info: api.cls.licensing.nvidia.com, NVIDIA Virtual Compute Server; Expiry: 2025-5-10 19:13:17 GMT)

Only 3 successful acquisitions in the logs - the last one was May 9, 2025. (The VM was only rebooted a few times, and the license is acquired on each boot)

When did "license allotment disabled" error first appear:

$ journalctl -u nvidia-gridd --no-pager | grep -i "allotment" | head -5
Jan 19 17:25:47 prod-rodan-vgpu nvidia-gridd[718]: Failed to acquire license from api.cls.licensing.nvidia.com (Info: NVIDIA Virtual Compute Server - Error: License allotment is disabled.)
Jan 19 17:26:15 prod-rodan-vgpu nvidia-gridd[718]: Failed to acquire license from api.cls.licensing.nvidia.com (Info: NVIDIA Virtual Compute Server - Error: License allotment is disabled.)
Jan 19 17:26:46 prod-rodan-vgpu nvidia-gridd[718]: Failed to acquire license from api.cls.licensing.nvidia.com (Info: NVIDIA Virtual Compute Server - Error: License allotment is disabled.)
Jan 19 17:27:31 prod-rodan-vgpu nvidia-gridd[718]: Failed to acquire license from api.cls.licensing.nvidia.com (Info: NVIDIA Virtual Compute Server - Error: License allotment is disabled.)
Jan 19 17:28:14 prod-rodan-vgpu nvidia-gridd[718]: Failed to acquire license from api.cls.licensing.nvidia.com (Info: NVIDIA Virtual Compute Server - Error: License allotment is disabled.)

So the "License allotment is disabled" error first appeared on Jan 19, 2026. 4 days later on Jan 23, GPU jobs started failing with CUDA errors.

Tried manually restarting the license service:

$ sudo systemctl restart nvidia-gridd
$ journalctl -u nvidia-gridd --no-pager | tail -10
Jan 26 03:58:40 prod-rodan-vgpu nvidia-gridd[27370]: Valid GRID license not found. GPU features and performance are restricted.

Still can't acquire license - confirms the issue is on NVIDIA's side.

"License allotment is disabled" means the NVIDIA licensing server is actively rejecting the request. This isn't a network issue or token issue - something changed on NVIDIA's side around Jan 19. Either:

• The license subscription expired
• Someone disabled the allotment in the NVIDIA licensing portal
• The license pool ran out of available seats

Current state

• GPU technically works after reboot, but in "Unlicensed (Restricted)" mode
• CUDA initialization takes ~4 minutes instead of seconds
• Once initialized, computations do run but everything is slow
• Before the reboot, it was completely broken (hard CUDA errors)

What needs to be done

This is an NVIDIA licensing portal issue, not something we can fix locally. Need to:

1. Check the NVIDIA Enterprise Licensing Portal for the account status
2. See if the license subscription expired or was disabled around Jan 19, 2026
3. Contact Compute Canada if they manage the NVIDIA licensing for this vGPU
4. Renew/re-enable the "NVIDIA Virtual Compute Server" license allotment

The vGPU is a GRID V100D-16C which requires a valid "NVIDIA Virtual Compute Server" license to run at full speed. Without it, everything still works but is extremely slow (restricted mode).

[fujinaga]

This may be due to the power outage. Please inquire.

[homework36]

Copying response from Compute Canada below:

Last July you should have received this email:
===================================
You are receiving this message because you are a user of Arbutus cloud.

Please note that the current vGPU license on Arbutus cloud is set to expire on July 31, 2025. After this date, instances with vGPUs will continue running, but GPU accelerator functionality will not be available until the new Arbutus cloud resources are in production. Due to delays in hardware delivery, we anticipate a production launch of the new equipment in early September.

We recognize that this may impact your workflows and GPU-based workloads on Arbutus. We will notify you as soon as more information on GPU availability for Arbutus is available.

No action is required at this time.

The status of Arbutus cloud can be checked on link for updates. You can also visit the dedicated Infrastructure Renewal Wiki page where National Host Sites continue to publish planned activities and anticipated impacts including outages and system reductions.

===================================

So the GPUs on the legacy Arbutus are no longer licensed. At some point after the renewal Arbutus comes online GPUs will again be made available. We have no specific date for that.