K8s-Cloud-Tagger

Kubernetes cloud tagger watches cluster resources and applies labels in your cloud provider.

Develop

nix develop gives you a shell with all the dependencies.

• Nix
• Rust
• Docker Desktop

Configure nix

You need to enable two experimental features for nix to work.

mkdir -p ~/.config/nix/
echo "extra-experimental-features = nix-command flakes" >> ~/.config/nix/nix.conf

Build

nix develop
nix build .#image-dev
docker load < result
docker tag quay.io/upgrades/k8s-cloud-tagger-dev:dev \
  quay.io/upgrades/k8s-cloud-tagger-dev:"${TAG:-}"
docker push quay.io/upgrades/k8s-cloud-tagger-dev:"${TAG:-}"

Test

Unit tests

cargo test

Run all CI checks locally:

nix build

Integration tests

Note to Mac users: The e2e runs on Mac, but the Dockerimage is built by Docker instead of Nix, due to compatibility issues.

Linux users can opt in to use Docker build with USE_DOCKER_BUILD=true, but that is mainly for troubleshooting since it's generally slower than Nix.

nix develop
KEEP_CLUSTER=true nix run .#kind-test

• builds an image using Nix
• creates a Kind cluster
• deploys your image using Helm
• runs the app in test mode
• creates a PVC and listens for an Event

You can also specify the image:

nix develop
IMAGE=quay.io/upgrades/k8s-cloud-tagger-dev:sha-6f4cbfe nix run .#kind-test

KEEP_CLUSTER=true prints a message saying how to use kubectl in case you want to inspect the cluster. Otherwise, the cluster is deleted after the test.

To get the raw Kubernetes manifests:

nix develop
helm template k8s-cloud-tagger helm/k8s-cloud-tagger/ --set serviceMonitor.enabled=true

Label sanitisation

GCP

Kubernetes label keys and values can contain characters that are not valid in GCP labels. GCP labels only allow lowercase letters, digits, hyphens, and underscores ([a-z0-9_-]), with keys limited to 63 characters and required to start with a lowercase letter. To bridge this gap, k8s-cloud-tagger sanitises labels before applying them to cloud resources: all characters are lowercased, and any character outside the allowed set is replaced with a hyphen. This follows the conventions used by Google's own resource labels (such as the goog-gke-* labels applied by GKE), where hyphens are the standard word separator. For more detail on GCP label requirements, see the Google Cloud labeling best practices.

Kubernetes label	GCP label
app.kubernetes.io/name: frontend	app-kubernetes-io-name: frontend
helm.sh/chart: myapp-1.2.0	helm-sh-chart: myapp-1-2-0
env: production	env: production
upgrades.dev/managed-by: k8s-cloud-tagger	upgrades-dev-managed-by: k8s-cloud-tagger
Team: Platform	team: platform

Azure

Azure resource tag keys may contain any Unicode character except <, >, %, &, \, ?, and /. Keys are limited to 512 characters and values to 256 characters. k8s-cloud-tagger replaces each disallowed character in a key with a hyphen, and truncates keys and values to their respective limits. Unlike GCP, Azure tags are not lowercased — tag names are case-insensitive in Azure but case is preserved as supplied, and tag values are case-sensitive. For more detail on Azure tag requirements, see the Azure tag limitations.

Kubernetes label	Azure tag
app.kubernetes.io/name: frontend	app.kubernetes.io-name: frontend
helm.sh/chart: myapp-1.2.0	helm.sh-chart: myapp-1.2.0
env: production	env: production
upgrades.dev/managed-by: k8s-cloud-tagger	upgrades.dev-managed-by: k8s-cloud-tagger
Team: Platform	Team: Platform

AWS

AWS resource tag keys may contain any UTF-8 character, with keys limited to 128 characters and values to 256 characters. The following characters are not allowed in tag keys or values: aws: (as a prefix), and the characters <, >, %, \, &, ?, /. k8s-cloud-tagger replaces each disallowed character in a key with a hyphen, and truncates keys and values to their respective limits. Like Azure, AWS tags are not lowercased — case is preserved as supplied for both keys and values. For more detail on AWS tag requirements, see the AWS tagging best practices.

Kubernetes label	AWS tag
app.kubernetes.io/name: frontend	app.kubernetes.io-name: frontend
helm.sh/chart: myapp-1.2.0	helm.sh-chart: myapp-1.2.0
env: production	env: production
upgrades.dev/managed-by: k8s-cloud-tagger	upgrades.dev-managed-by: k8s-cloud-tagger
Team: Platform	Team: Platform

Note: AWS reserves the aws: key prefix for its own use. Any Kubernetes label key that begins with aws: after sanitisation will be rejected and not applied to the resource.

Release

1. Check out a new branch
2. Update CHANGELOG.md following the Keep a Changelog format and SemVer conventions.
3. Run cargo xtask release <version>. This updates and commits the relevant files to git.
4. Raise PR, merge PR
5. When the PR is merged, a job will run that adds a git tag, and builds and pushes the release Docker image