DCGM_FI_DEV_GPU_UTIL missing

[okyspace]

I have used nvidia-dcgm-exporter to expose the nvidia metrics via port 9400.
I managed to find some metrics like DCGM_FI_DEV_GPU_TEMP, etc but I am not able to find metrics for GPU utilisation like DCGM_FI_DEV_GPU_UTIL.

Any advice how to get it to work? I need to display GPU utilisation on Grafana.

[sagar-shrestha24]

I'm facing the same issue on the V100-32C GPU node—DCGM_FI_DEV_GPU_TEMP and power usage metrics are not being collected.

[frittentheke]

I did rework the dashboard a while back in NVIDIA/dcgm-exporter#355 ... unfortunately without any response from NVIDIA.

[github-actions]

This issue is stale because it has been open 90 days with no activity. This issue will be closed in 30 days unless new comments are made or the stale label is removed. To skip these checks, apply the "lifecycle/frozen" label.

[frittentheke]

unstale

[shivamerla]

@frittentheke @okyspace Could you open an issue in the dcgm-exporter repository instead? The operator configures dcgm-exporter using the default DCP metrics configuration found here. This configuration should already include the metrics you mentioned, so if they aren’t being collected for certain GPU configurations, it’s likely an issue in the exporter itself."

[empovit]

There are two DCGM metrics for GPU utilization, depending on the GPU model/architecture:

1. DCGM_FI_PROF_GR_ENGINE_ACTIVE seems to be the "modern" one, better reflects GPU utilization, and works with MIG. It does not work on pre-Volta GPUs.
2. DCGM_FI_DEV_GPU_UTIL is the "old" metrics, works on older GPUs but does not seem to work with MIG.

So I would start by checking the GPU model and DCGM documentation. There are other metrics that are unsupported on some GPUs, e.g. the entire Profiling group (Group ID 8, DCGM_FI_PROF_* metrics) is not supported on pre-Volta GPUs.

[cdesiniotis]

I am closing this issue in an effort to keep our issue tracker clean. As suggested in #614 (comment) please file an issue against the dcgm-exporter repository if this is still relevant. I did find NVIDIA/dcgm-exporter#19 (comment) which provides some context about this specific metric.