Avoid hard-coded values for API server address #288

brandond · 2025-10-03T20:19:42Z

You cannot assume that the in-cluster apiserver port is the same as the actual port on the host. For example, on both K3s and RKE2, the port for the in-cluster kubernetes service is 443, and that's what you'll see here if you look at the client config - but the apiserver is actually listening on 6443 on the server nodes.

You need to understand the difference between service ports, and service endpoints. Bootstrap charts need to access the local apiserver endpoint directly, as kube-proxy and the rest of the CNI cannot be relied on to be functional yet.

root@systemd-node-001:/# kubectl get service kubernetes -o wide NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR kubernetes ClusterIP 10.43.0.1 <none> 443/TCP 21h <none> root@systemd-node-001:/# kubectl get endpointslice kubernetes -o wide NAME ADDRESSTYPE PORTS ENDPOINTS AGE kubernetes IPv4 6443 172.17.0.4 21h

Yes, I understood that.

I tried the patch with minikube and k3s by using helm-controller outside the cluster. In that case it worked since the kube-config file is set with direct api endpoint and not the in-cluster service.

I forgot to try the in-cluster image. I suppose in that case the code will use the in-cluster service and fail.

I will try and report back.

Yes, doing this the "right way" such that it will work in all possible environments is non-trivial, which is why I opted to just fix it the easy way in K3s.

I couldn’t find a use-case where the helm controller can connect to the API but not the helm bootstrap chart pod.

If the helm controller is using the in-cluster api and has access to the custom resources, then the pod running the bootstrap chart will be able to connect as well to the in-cluster API.

If the helm controller is configured to connect to the in-cluster API and the in-cluster API is not available, then the helm controller will never see the custom resource and will never start any pod anyways.

So using the in-cluster API for bootstrap charts when helm controller is using in-cluster API shouldn’t be a problem?

helm-controller runs embedded in the K3s and RKE2 supervisor process, and always talks to the local apiserver endpoint, not the in-cluster service endpoint.

Users can also deploy it as a daemonset that runs with host network on control-plane nodes, and point it to the local apiserver endpoint. This would be necessary if it was to be used for bootstrap charts on other Kubernetes distributions.

Running it against the in-cluster service endpoint will never work for bootstrap charts, as you cannot rely on the cluster network being up yet.

helm-controller runs embedded in the K3s and RKE2 supervisor process, and always talks to the local apiserver endpoint, not the in-cluster service endpoint.

That use-case is not related to the PR, as host and port will be given by the supervisor process.

Users can also deploy it as a daemonset that runs with host network on control-plane nodes, and point it to the local apiserver endpoint. This would be necessary if it was to be used for bootstrap charts on other Kubernetes distributions.

In that case, the PR works since bootstrap charts will re-use host and port of local api server endpoint given by the user.

Running it against the in-cluster service endpoint will never work for bootstrap charts, as you cannot rely on the cluster network being up yet.

The only case in which the PR will use in-cluster service endpoint for bootstrap charts is if helm-controller is using in-cluster service endpoint to get the custom resource.

Either network is up, helm-controller can get custom resources, and bootstrap charts will work using in-cluster api.

Or network is down, helm-controller can’t get custer resources, and no bootstrap charts will ever be started.

I’ll make more tests and report logs, but I still don’t see what I’ve missed.

I couldn’t find a failing use-case. See below for configuration I tested and logs.

Could you provide a way to make the PR fail? I have no problem updating the PR to use hard-coded values in specific situations, but I’d rather limit using 127.0.0.1 to situations where re-using host and port from the controller actually fail.

Test 1: embedded in k3s

curl -sfL https://get.k3s.io | sh -s - server --cluster-cidr=2001:db8:42::/56 --service-cidr=2001:db8:43::/112 --flannel-backend=none sudo systemctl stop k3s sudo cp bin/k3s /usr/local/bin/ sudo systemctl start k3s sudo kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml apply -f - <<EOF apiVersion: helm.cattle.io/v1 kind: HelmChart metadata: name: apache namespace: kube-system spec: repo: https://charts.bitnami.com/bitnami chart: apache targetNamespace: kube-system bootstrap: true valuesContent: |- service: type: ClusterIP ingress: enabled: true hostname: www.example.com metrics: enabled: true EOF sudo kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml describe pod -n kube-system helm-install-apache | grep KUBERNETES_SERVICE KUBERNETES_SERVICE_HOST: ::1 KUBERNETES_SERVICE_PORT: 6444 sudo kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml logs -n kube-system -l helmcharts.helm.cattle.io/chart=apache --tail=50 | grep STATUS: STATUS: deployed

Test 2: out of cluster

curl -sfL https://get.k3s.io | sh -s - server --cluster-cidr=2001:db8:42::/56 --service-cidr=2001:db8:43::/112 --flannel-backend=none --disable-helm-controller sudo bin/helm-controller --kubeconfig /etc/rancher/k3s/k3s.yaml sudo kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml apply -f - <<EOF apiVersion: helm.cattle.io/v1 kind: HelmChart metadata: name: apache namespace: kube-system spec: repo: https://charts.bitnami.com/bitnami chart: apache targetNamespace: kube-system bootstrap: true valuesContent: |- service: type: ClusterIP ingress: enabled: true hostname: www.example.com metrics: enabled: true EOF sudo kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml describe pod -n kube-system helm-install-apache | grep KUBERNETES_SERVICE KUBERNETES_SERVICE_HOST: ::1 KUBERNETES_SERVICE_PORT: 6443 sudo kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml logs -n kube-system -l helmcharts.helm.cattle.io/chart=apache --tail=50 | grep STATUS: STATUS: deployed

Test 3: in-cluster helm-controller using local api server endpoint

curl -sfL https://get.k3s.io | sh -s - server --cluster-cidr=2001:db8:42::/56 --service-cidr=2001:db8:43::/112 --flannel-backend=none --disable-helm-controller sudo kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml apply -f - <<EOF apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: helm-controller rules: - apiGroups: - '*' resources: - '*' verbs: - '*' --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: helm-controller roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: helm-controller subjects: - kind: ServiceAccount name: helm-controller namespace: kube-system --- apiVersion: v1 kind: ServiceAccount metadata: name: helm-controller namespace: kube-system --- apiVersion: apps/v1 kind: Deployment metadata: name: helm-controller namespace: kube-system labels: app: helm-controller spec: replicas: 1 selector: matchLabels: app: helm-controller template: metadata: labels: app: helm-controller spec: hostNetwork: true containers: - name: helm-controller image: codeberg.org/jip149/helm-controller:dev command: ["helm-controller"] env: - name: KUBECONFIG value: /kubeconfig - name: NODE_NAME valueFrom: fieldRef: fieldPath: spec.nodeName - name: JOB_CLUSTER_ROLE valueFrom: fieldRef: fieldPath: spec.serviceAccountName volumeMounts: - mountPath: /kubeconfig name: kubeconfig volumes: - name: kubeconfig hostPath: path: /etc/rancher/k3s/k3s.yaml type: File tolerations: - effect: NoSchedule key: node.kubernetes.io/not-ready - effect: NoSchedule key: node.kubernetes.io/network-unavailable - effect: NoSchedule key: node.cloudprovider.kubernetes.io/uninitialized operator: Equal value: "true" - key: CriticalAddonsOnly operator: Exists - effect: NoExecute key: node-role.kubernetes.io/etcd operator: Exists - effect: NoSchedule key: node-role.kubernetes.io/control-plane operator: Exists serviceAccountName: helm-controller EOF sudo kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml apply -f - <<EOF apiVersion: helm.cattle.io/v1 kind: HelmChart metadata: name: apache namespace: kube-system spec: repo: https://charts.bitnami.com/bitnami chart: apache targetNamespace: kube-system bootstrap: true valuesContent: |- service: type: ClusterIP ingress: enabled: true hostname: www.example.com metrics: enabled: true EOF sudo kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml describe pod -n kube-system helm-install-apache | grep KUBERNETES_SERVICE KUBERNETES_SERVICE_HOST: ::1 KUBERNETES_SERVICE_PORT: 6443 sudo kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml logs -n kube-system -l helmcharts.helm.cattle.io/chart=apache --tail=50 | grep STATUS: STATUS: deployed

Test 4: in-cluster helm-controller using in-cluster api service

curl -sfL https://get.k3s.io | sh -s - server --cluster-cidr=2001:db8:42::/56 --service-cidr=2001:db8:43::/112 --flannel-backend=none --disable-helm-controller sudo kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml apply -f - <<EOF apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: helm-controller rules: - apiGroups: - '*' resources: - '*' verbs: - '*' --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: helm-controller roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: helm-controller subjects: - kind: ServiceAccount name: helm-controller namespace: kube-system --- apiVersion: v1 kind: ServiceAccount metadata: name: helm-controller namespace: kube-system --- apiVersion: apps/v1 kind: Deployment metadata: name: helm-controller namespace: kube-system labels: app: helm-controller spec: replicas: 1 selector: matchLabels: app: helm-controller template: metadata: labels: app: helm-controller spec: hostNetwork: true containers: - name: helm-controller image: codeberg.org/jip149/helm-controller:dev command: ["helm-controller"] env: - name: NODE_NAME valueFrom: fieldRef: fieldPath: spec.nodeName - name: JOB_CLUSTER_ROLE valueFrom: fieldRef: fieldPath: spec.serviceAccountName tolerations: - effect: NoSchedule key: node.kubernetes.io/not-ready - effect: NoSchedule key: node.kubernetes.io/network-unavailable - effect: NoSchedule key: node.cloudprovider.kubernetes.io/uninitialized operator: Equal value: "true" - key: CriticalAddonsOnly operator: Exists - effect: NoExecute key: node-role.kubernetes.io/etcd operator: Exists - effect: NoSchedule key: node-role.kubernetes.io/control-plane operator: Exists serviceAccountName: helm-controller EOF sudo kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml apply -f - <<EOF apiVersion: helm.cattle.io/v1 kind: HelmChart metadata: name: apache namespace: kube-system spec: repo: https://charts.bitnami.com/bitnami chart: apache targetNamespace: kube-system bootstrap: true valuesContent: |- service: type: ClusterIP ingress: enabled: true hostname: www.example.com metrics: enabled: true EOF sudo kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml describe pod -n kube-system helm-install-apache | grep KUBERNETES_SERVICE KUBERNETES_SERVICE_HOST: 2001:db8:43::1 KUBERNETES_SERVICE_PORT: 6443 sudo kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml logs -n kube-system -l helmcharts.helm.cattle.io/chart=apache --tail=50 | grep STATUS: STATUS: deployed

I made more tests, this time by disabling both flannel and kube-proxy.

This is now the behavior I described above. In-cluster api service cannot be reached, but the problem is not with the bootstrap chart, it’s just that helm-controller does not start (see logs below).

I still can’t find a situation where using a hard-coded value is better than connecting to host and port used by helm-controller.

Test 5: in-cluster helm-controller using in-cluster api service without kube-proxy

curl -sfL https://get.k3s.io | sh -s - server --cluster-cidr=2001:db8:42::/56 --service-cidr=2001:db8:43::/112 --flannel-backend=none --disable-helm-controller --disable-kube-proxy sudo kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml apply -f - <<EOF apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: helm-controller rules: - apiGroups: - '*' resources: - '*' verbs: - '*' --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: helm-controller roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: helm-controller subjects: - kind: ServiceAccount name: helm-controller namespace: kube-system --- apiVersion: v1 kind: ServiceAccount metadata: name: helm-controller namespace: kube-system --- apiVersion: apps/v1 kind: Deployment metadata: name: helm-controller namespace: kube-system labels: app: helm-controller spec: replicas: 1 selector: matchLabels: app: helm-controller template: metadata: labels: app: helm-controller spec: hostNetwork: true containers: - name: helm-controller image: codeberg.org/jip149/helm-controller:dev command: ["helm-controller"] env: - name: NODE_NAME valueFrom: fieldRef: fieldPath: spec.nodeName - name: JOB_CLUSTER_ROLE valueFrom: fieldRef: fieldPath: spec.serviceAccountName tolerations: - effect: NoSchedule key: node.kubernetes.io/not-ready - effect: NoSchedule key: node.kubernetes.io/network-unavailable - effect: NoSchedule key: node.cloudprovider.kubernetes.io/uninitialized operator: Equal value: "true" - key: CriticalAddonsOnly operator: Exists - effect: NoExecute key: node-role.kubernetes.io/etcd operator: Exists - effect: NoSchedule key: node-role.kubernetes.io/control-plane operator: Exists serviceAccountName: helm-controller EOF sudo kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml logs -n kube-system -l app=helm-controller time="2025-10-18T05:46:40Z" level=warning msg="unable to list existing CRDs: failed to list crds: Get \"https://[2001:db8:43::1]:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions\": dial tcp [2001:db8:43::1]:443: connect: network is unreachable" time="2025-10-18T05:46:40Z" level=info msg="Creating embedded CRD helmcharts.helm.cattle.io" time="2025-10-18T05:46:40Z" level=fatal msg="failed to create crd 'helmcharts.helm.cattle.io': Post \"https://[2001:db8:43::1]:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions\": dial tcp [2001:db8:43::1]:443: connect: network is unreachable"

Test 6: in-cluster helm-controller using local api server endpoint without kube-proxy

curl -sfL https://get.k3s.io | sh -s - server --cluster-cidr=2001:db8:42::/56 --service-cidr=2001:db8:43::/112 --flannel-backend=none --disable-helm-controller --disable-kube-proxy sudo kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml apply -f - <<EOF apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: helm-controller rules: - apiGroups: - '*' resources: - '*' verbs: - '*' --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: helm-controller roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: helm-controller subjects: - kind: ServiceAccount name: helm-controller namespace: kube-system --- apiVersion: v1 kind: ServiceAccount metadata: name: helm-controller namespace: kube-system --- apiVersion: apps/v1 kind: Deployment metadata: name: helm-controller namespace: kube-system labels: app: helm-controller spec: replicas: 1 selector: matchLabels: app: helm-controller template: metadata: labels: app: helm-controller spec: hostNetwork: true containers: - name: helm-controller image: codeberg.org/jip149/helm-controller:dev command: ["helm-controller"] env: - name: KUBECONFIG value: /kubeconfig - name: NODE_NAME valueFrom: fieldRef: fieldPath: spec.nodeName - name: JOB_CLUSTER_ROLE valueFrom: fieldRef: fieldPath: spec.serviceAccountName volumeMounts: - mountPath: /kubeconfig name: kubeconfig volumes: - name: kubeconfig hostPath: path: /etc/rancher/k3s/k3s.yaml type: File tolerations: - effect: NoSchedule key: node.kubernetes.io/not-ready - effect: NoSchedule key: node.kubernetes.io/network-unavailable - effect: NoSchedule key: node.cloudprovider.kubernetes.io/uninitialized operator: Equal value: "true" - key: CriticalAddonsOnly operator: Exists - effect: NoExecute key: node-role.kubernetes.io/etcd operator: Exists - effect: NoSchedule key: node-role.kubernetes.io/control-plane operator: Exists serviceAccountName: helm-controller EOF sudo kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml apply -f - <<EOF apiVersion: helm.cattle.io/v1 kind: HelmChart metadata: name: apache namespace: kube-system spec: repo: https://charts.bitnami.com/bitnami chart: apache targetNamespace: kube-system bootstrap: true valuesContent: |- service: type: ClusterIP ingress: enabled: true hostname: www.example.com metrics: enabled: true EOF sudo kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml describe pod -n kube-system helm-install-apache | grep KUBERNETES_SERVICE KUBERNETES_SERVICE_HOST: ::1 KUBERNETES_SERVICE_PORT: 6443 sudo kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml logs -n kube-system -l helmcharts.helm.cattle.io/chart=apache --tail=50 | grep STATUS: STATUS: deployed

-Original file line number
+Diff line change
@@ Expand Up / @@ -92,6 +92,7 @@ type Controller struct { @@
     	secretCache     corecontroller.SecretCache
     	apply           apply.Apply
     	recorder        record.EventRecorder
+    	apiServerHost   string
     	apiServerPort   string
     }
@@ Expand All / @@ -100,6 +101,7 @@ func Register( @@
     	systemNamespace,
     	managedBy,
     	jobClusterRole string,
+    	apiServerHost string,
     	apiServerPort string,
     	k8s kubernetes.Interface,
     	apply apply.Apply,
@@ Expand Down Expand Up / @@ -128,6 +130,7 @@ func Register( @@
     		jobCache:        jobCache,
     		secretCache:     sCache,
     		recorder:        recorder,
+    		apiServerHost:   apiServerHost,
     		apiServerPort:   apiServerPort,
     	}
@@ Expand Down Expand Up @@
     	}
     	// get the default job and configmaps
-    	job, valuesSecret, contentConfigMap := job(chart, c.apiServerPort)
+    	job, valuesSecret, contentConfigMap := job(chart, c.apiServerHost, c.apiServerPort)
     	objects := []metav1.Object{contentConfigMap, valuesSecret}
     	// make sure that changes to HelmChart ValuesSecrets triger change to hash
@@ Expand Down Expand Up @@
     	return keys.UnsortedList(), nil
     }
-    func job(chart *v1.HelmChart, apiServerPort string) (*batch.Job, *corev1.Secret, *corev1.ConfigMap) {
+    func job(chart *v1.HelmChart, apiServerHost string, apiServerPort string) (*batch.Job, *corev1.Secret, *corev1.ConfigMap) {
     	jobImage := strings.TrimSpace(chart.Spec.JobImage)
     	if jobImage == "" {
     		jobImage = DefaultJobImage
@@ Expand Down Expand Up @@
     		job.Spec.Template.Spec.Containers[0].Env = append(job.Spec.Template.Spec.Containers[0].Env, []corev1.EnvVar{
     			{
     				Name:  "KUBERNETES_SERVICE_HOST",
-    				Value: "127.0.0.1"},
+    				Value: apiServerHost},
     			{
     				Name:  "KUBERNETES_SERVICE_PORT",
     				Value: apiServerPort},
@@ Expand Down @@

-Original file line number
+Diff line change
@@ Expand Up / @@ -3,6 +3,7 @@ package controllers @@
     import (
     	"context"
     	"time"
+    	"net/url"
     	"github.com/k3s-io/helm-controller/pkg/controllers/chart"
     	"github.com/k3s-io/helm-controller/pkg/controllers/common"
@@ Expand Down Expand Up @@
     		chart.DefaultJobImage = opts.DefaultJobImage
     	}
+    	host, port, err := getHostPortFromClientConfig(cfg);
+    	if err != nil {
+    		return err
+    	}
     	chart.Register(ctx,
     		systemNamespace,
     		controllerName,
     		opts.JobClusterRole,
-    		"6443",
+    		host,
+    		port,
     		appCtx.K8s,
     		appCtx.Apply,
     		recorder,
@@ Expand Down Expand Up @@
     		},
     	}, nil
     }
+    func getHostPortFromClientConfig(cfg clientcmd.ClientConfig) (string, string, error) {
+    	client, err := cfg.ClientConfig()
+    	if err != nil {
+    		return "", "", err
+    	}
+    	u, err := url.ParseRequestURI(client.Host)
+    	if err != nil {
+    		return "", "", err
+    	}
+    	return u.Hostname(), u.Port(), nil
+    }

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Avoid hard-coded values for API server address #288

Uh oh!

Diff view

Diff view

There are no files selected for viewing

brandond Oct 3, 2025

Uh oh!

jip149 Oct 3, 2025

Uh oh!

brandond Oct 3, 2025 •

edited

Loading

Uh oh!

jip149 Oct 5, 2025 •

edited

Loading

Uh oh!

brandond Oct 16, 2025 •

edited

Loading

Uh oh!

jip149 Oct 16, 2025

Uh oh!

jip149 Oct 17, 2025

Uh oh!

jip149 Oct 18, 2025 •

edited

Loading

Uh oh!

Avoid hard-coded values for API server address #288

Are you sure you want to change the base?

Uh oh!

Avoid hard-coded values for API server address #288

Uh oh!

Uh oh!

Diff view

Diff view

There are no files selected for viewing

brandond Oct 3, 2025

Choose a reason for hiding this comment

Uh oh!

jip149 Oct 3, 2025

Choose a reason for hiding this comment

Uh oh!

brandond Oct 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jip149 Oct 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

brandond Oct 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jip149 Oct 16, 2025

Choose a reason for hiding this comment

Uh oh!

jip149 Oct 17, 2025

Choose a reason for hiding this comment

Uh oh!

jip149 Oct 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

brandond Oct 3, 2025 •

edited

Loading

jip149 Oct 5, 2025 •

edited

Loading

brandond Oct 16, 2025 •

edited

Loading

jip149 Oct 18, 2025 •

edited

Loading