[bitnami/kube-prometheus] bug: container_network_transmit_bytes_total and container_network_receive_bytes_total not work correctly.

[DannyRavi]

Name and Version

bitnami/kube-prometheus 11.1.9

What architecture are you using?

None

What steps will reproduce the bug?

I am working with the Bitnami kube-prometheus Helm chart.
In a previous version of Kubernetes, 1.22.0, I was able to see pod and container network metrics. However, after installing a new Kubernetes cluster using version 1.32.0 and deploying the Bitnami kube-prometheus chart, the pod and container network usage metrics no longer appear as expected.

To investigate, I tested the helm/bitnami/kube-prometheus chart using Minikube in a GitHub Codespace. I found that when using Kubernetes version 1.22.0, the metrics container_network_transmit_bytes_total and container_network_receive_bytes_total work properly and return a full response, including labels like container and interface.

| container_network_transmit_bytes_total{container="POD", endpoint="https-metrics", id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08a3871e1baa241b73e5af01a6d01393.slice/docker-ac920d3bab5f317a452dd3f40f391a0cef0c8bba295211f38ccc3acaef08de97.scope", image="k8s.gcr.io/pause:3.5", instance="192.168.49.2:10250", interface="eth0", job="kubelet", name="k8s_POD_etcd-minikube_kube-system_08a3871e1baa241b73e5af01a6d01393_0", namespace="kube-system", node="minikube", pod="etcd-minikube", service="my-kube-prometheus-kubelet"}                                                                                   | 5842783 |
| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- |
| container_network_transmit_bytes_total{container="POD", endpoint="https-metrics", id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bc62e37_d1ad_4305_a4e8_c126e18541c7.slice/docker-11ce2b3ac872187eec7158d4f54b965b212969780e0a149462469d57af4d511f.scope", image="k8s.gcr.io/pause:3.5", instance="192.168.49.2:10250", interface="eth0", job="kubelet", name="k8s_POD_alertmanager-my-kube-prometheus-alertmanager-0_default_3bc62e37-d1ad-4305-a4e8-c126e18541c7_0", namespace="default", node="minikube", pod="alertmanager-my-kube-prometheus-alertmanager-0", service="my-kube-prometheus-kubelet"}                 | 138614  |

But when I use Kubernetes version 1.32.0, the response for the same metric is incomplete—it does not include important labels such as container and interface. The response looks something like this:

container_network_transmit_bytes_total{endpoint="https-metrics", id="/", instance="192.168.49.2:10250", interface="eth0", job="kubelet", node="minikube", service="my-kube-prometheus-stack-kubelet"}

For more detail, I use these commands to install minikube and run bitnami/kube-prometheus chart:

kubernetes version 1.32.0 by minikube

minikube start --kubernetes-version=v1.32.0
helm repo add bitnami https://charts.bitnami.com/bitnami
helm install my-kube-prometheus bitnami/kube-prometheus --version 11.1.9
minikube kubectl -- port-forward svc/my-kube-prometheus-stack-prometheus 9091:9090 --address 0.0.0.0

all pods and services are running.

So, in k8s version 1.32.0 there are no interface and container labels.

kubernetes version 1.22.0 by minikube

minikube stop
minikube delete
minikube start --kubernetes-version=v1.22.0
helm repo add bitnami https://charts.bitnami.com/bitnami
helm install my-kube-prometheus bitnami/kube-prometheus --version 11.1.9
minikube kubectl -- port-forward svc/my-kube-prometheus-stack-prometheus 9091:9090 --address 0.0.0.0

kubernetes version:

Finally, in Kubernetes version 1.22.0, these labels (interface, container) exist.

| container_network_transmit_bytes_total{endpoint="https-metrics", id="/", instance="192.168.49.2:10250", interface="eth0", job="kubelet", node="minikube", service="my-kube-prometheus-kubelet"}                                                                                                                                                                                                                                                                                                                                                                                                                                               | 5886733 |
| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- |
| container_network_transmit_bytes_total{container="POD", endpoint="https-metrics", id="/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod22219353_f19a_4350_bac2_b52ee5153154.slice/docker-61e51f7a68ae5a0a132eb847070a7c2a42a09552e9f718842857ba2a9547d0da.scope", image="k8s.gcr.io/pause:3.5", instance="192.168.49.2:10250", interface="eth0", job="kubelet", name="k8s_POD_storage-provisioner_kube-system_22219353-f19a-4350-bac2-b52ee5153154_0", namespace="kube-system", node="minikube", pod="storage-provisioner", service="my-kube-prometheus-kubelet"}                                                             | 5886733 |
| container_network_transmit_bytes_total{container="POD", endpoint="https-metrics", id="/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod339729cf_6a27_4b9b_8c42_728820b75cf4.slice/docker-ad7d6a0a0604ff06b9b3f64285bca5ea67ca89d85dc64031c2908fcf923ef858.scope", image="k8s.gcr.io/pause:3.5", instance="192.168.49.2:10250", interface="eth0", job="kubelet", name="k8s_POD_kube-proxy-78dcm_kube-system_339729cf-6a27-4b9b-8c42-728820b75cf4_0", namespace="kube-system", node="minikube", pod="kube-proxy-78dcm", service="my-kube-prometheus-kubelet"}                                                                   | 5886733 |
| container_network_transmit_bytes_total{container="POD", endpoint="https-metrics", id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08a3871e1baa241b73e5af01a6d01393.slice/docker-ac920d3bab5f317a452dd3f40f391a0cef0c8bba295211f38ccc3acaef08de97.scope", image="k8s.gcr.io/pause:3.5", instance="192.168.49.2:10250", interface="eth0", job="kubelet", name="k8s_POD_etcd-minikube_kube-system_08a3871e1baa241b73e5af01a6d01393_0", namespace="kube-system", node="minikube", pod="etcd-minikube", service="my-kube-prometheus-kubelet"}                                                                                   | 5842783 |
| container_network_transmit_bytes_total{container="POD", endpoint="https-metrics", id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bc62e37_d1ad_4305_a4e8_c126e18541c7.slice/docker-11ce2b3ac872187eec7158d4f54b965b212969780e0a149462469d57af4d511f.scope", image="k8s.gcr.io/pause:3.5", instance="192.168.49.2:10250", interface="eth0", job="kubelet", name="k8s_POD_alertmanager-my-kube-prometheus-alertmanager-0_default_3bc62e37-d1ad-4305-a4e8-c126e18541c7_0", namespace="default", node="minikube", pod="alertmanager-my-kube-prometheus-alertmanager-0", service="my-kube-prometheus-kubelet"}                 | 138614  |
| container_network_transmit_bytes_total{container="POD", endpoint="https-metrics", id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d079dac_8d97_4f74_b1e5_1136457a356d.slice/docker-5210e44eae3ae96326d16958406b3e5804bef7b6afb8fab66335053a53bbc396.scope", image="k8s.gcr.io/pause:3.5", instance="192.168.49.2:10250", interface="eth0", job="kubelet", name="k8s_POD_coredns-78fcd69978-s8rtx_kube-system_4d079dac-8d97-4f74-b1e5-1136457a356d_0", namespace="kube-system", node="minikube", pod="coredns-78fcd69978-s8rtx", service="my-kube-prometheus-kubelet"}                                                     | 98852   |

Are you using any custom parameters or values?

helm install my-kube-prometheus bitnami/kube-prometheus --version 11.1.9

What is the expected behavior?

show container and interface labels in container_network_transmit_bytes_total metrics like:

container_network_transmit_bytes_total{container="POD", endpoint="https-metrics", id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61f80e2a_1530_4cbf_b615_f7e365c00784.slice/docker-61813e8c7227a759aa9d54d50d32a1c528ca710e4e5e7957561c8a4ad37f5bea.scope", image="k8s.gcr.io/pause:3.5", instance="192.168.49.2:10250", interface="eth0", job="kubelet", name="k8s_POD_my-kube-prometheus-blackbox-exporter-75894d7676-csv77_default_61f80e2a-1530-4cbf-b615-f7e365c00784_0", namespace="default", node="minikube", pod="my-kube-prometheus-blackbox-exporter-75894d7676-csv77", service="my-kube-prometheus-kubelet"}

What do you see instead?

There are no container and interface labels in container_network_transmit_bytes_total metrics like:

container_network_transmit_bytes_total{endpoint="https-metrics", id="/", instance="192.168.49.2:10250", interface="bridge", job="kubelet", node="minikube", service="my-kube-prometheus-stack-kubelet"}