aerOS Gitlab internal repository guidelines

This repository contains guidelines to create, manage and use a common repository in Gitlab to host already packaged software (container images, Helm charts, Docker Compose files, K8s manifests files, documents, ...) by leveraging Gitlab's Packages and Registries.

In the aerOS project, this guidelines have been used to create a common and internal repository for all the software developed (and previously packaged) in the project. This repository has allowed to stablish a common packaging and deployment procedure for all the components of the aerOS Meta-OS, and it's described in section 3.1.2 of Deliverable D5.2.

🔑 Authentication

To interact with the registries of the Gitlab repository, users can authenticate via:

Private user credentials (username/access token): only for users with access to the repository
Common and shared credentials from a project access token: valid for anyone who has the username (e.g. project_25_bot_3722ac54dafeb1892d06042e948bb5e6) and access token associated with the previously created project access token. A project access token can be created in the menu of the repository (Settings -> Access token).

🐳 Container images in the Container Registry

In Gitlab, the container images are stored in the Container Registry of a code repository. You can find more information in the Gitlab's official documentation. This section only provides instructions for Docker, but you are free to use any existing containerization tool (e.g. Podman).

🔑 Authentication

Use the credentials provided below to authenticate with the Container Registry:

docker login <gitlab-container-registry-url> -u <username> -p <access-token>

If you have been already authenticated using the old credentials, please run again the command to update them.

⬆️ Push a container image

Tag the images using a common naming convention:

<gitlab-container-registry-url>/<group>/<repository>/<component-name>/<subcomponent-name>:<tag>

For instance:

 <gitlab-container-registry-url>/aeros-public/common-deployments/<component-name>/<subcomponent-name>:<tag>

Using the Docker CLI, build

docker build -t <gitlab-container-registry-url>/aeros-public/common-deployments/<component-name>/<subcomponent-name>:<tag> <path-to-Dockerfile>

or tag

docker tag <existing-image>:<tag> <gitlab-container-registry-url>/aeros-public/common-deployments/<component-name>/<subcomponent-name>:<tag>

the container images, and then push them to the Container Registry

docker push <gitlab-container-registry-url>/aeros-public/common-deployments/<component-name>/<subcomponent-name>:<tag>

Examples:

docker build -t <gitlab-container-registry-url>/aeros-public/common-deployments/orion-ld:1.1.1 .

docker tag aeros/self-awareness-hardware_info:1.4.4 <gitlab-container-registry-url>/aeros-public/common-deployments/self-awareness/hardware_info:1.4.4

docker push <gitlab-container-registry-url>/aeros-public/common-deployments/orion-ld:1.1.1

Multi-arch images

Currently, there are some issues in the Gitlab Container Registry related to multi-architecture images (see #393202, #388865). These images are displayed with an invalid image tag warning message, but it's just a visualization problem in the Gitlab UI which haven't been solved yet. Despite this warning, the images can be used without any problems.

To avoid this problem, use the --provenance=false option when building the images using docker buildx.

docker buildx build --no-cache --platform <target_architectures> -t <image_tag> --push --provenance=false .

For instance:

docker buildx build --no-cache --platform linux/amd64,linux/arm64 -t <gitlab-container-registry-url>/aeros-public/common-deployments/self-awareness/hardware_info:1.4.4 --push --provenance=false .

The other issue is that it's not possible to use the project access token to push multi-architecture container images, so you must first authenticate with the Container Registry using your personal user and token.

💻 Use a container image

Using the Docker CLI:

docker pull <gitlab-container-registry-url>/aeros-public/common-deployments/self-awareness/hardware_info:1.4.4

docker run <gitlab-container-registry-url>/aeros-public/common-deployments/self-awareness/hardware_info:1.4.4

🐳 Docker Compose files in the Package Registry

In Gitlab, the Docker Compose files (YAML files) are stored in the Package Registry of a code repository as generic files. You can find more information in the Gitlab's official documentation.

🔑 Authentication

To use container images from the Container Registry of the common repository, you must use the credentials provided below to authenticate with the Container Registry:

docker login <gitlab-container-registry-url> -u <username> -p <access-token>

If you have been already authenticated using the old credentials, please run again the command to update them.

⬆️ Upload a Docker Compose file

Upload the files using a common naming convention:

<component_name>/<component_version>/<file>

Send an HTTP request to the common repository's API using the provided token:

curl --header "PRIVATE-TOKEN: <access-token>" --upload-file <your_docker_compose.yaml> "https://<gitlab-url>/api/v4/projects/<project-id>/packages/generic/<component_name>/<component_version>/<your_docker_compose.yaml>"

For example:

curl --header "PRIVATE-TOKEN: <access-token>" --upload-file self-orchestrator_docker-compose.yml "https://<gitlab-url>/api/v4/projects/65/packages/generic/self-orchestrator/1.1.0/self-orchestrator_docker-compose.yml"

💻 Download and deploy a Docker compose file

Send an HTTP request to the common repository's API using the provided token to download the selected file:

wget --header "PRIVATE-TOKEN: <access-token>" https://<gitlab-url>/api/v4/projects/<project-id>/packages/generic/<component_name>/<component_version>/<your_docker_compose.yaml>

For instance:

wget --header "PRIVATE-TOKEN: <access-token>" https://<gitlab-url>/api/v4/projects/65/packages/generic/self-orchestrator/1.1.0/self-orchestrator_docker-compose.yml

Or directly deploy the Docker Compose file in the machine:

curl --header "PRIVATE-TOKEN: <access-token>" https://<gitlab-url>/api/v4/projects/<project-id>/packages/generic/<component_name>/<component_version>/<your_docker_compose.yaml> | docker compose -f /dev/stdin up -d

For instance:

curl --header "PRIVATE-TOKEN: <access-token>" https://<gitlab-url>/api/v4/projects/65/packages/generic/self-orchestrator/1.1.0/self-orchestrator_docker-compose.yml | docker compose -f /dev/stdin up -d

☸️ Kubernetes manifests in the Package Registry

In Gitlab, the Kubernetes manifests (YAML files) are stored in the Package Registry of a code repository as generic files. You can find more information in the Gitlab's official documentation.

NOTE
The secret and the Helm chart repository have already been created in the K8s clusters of CloudFerro and NCSRD.

🔑 Authentication

To use container images from the Container Registry of the common repository, you must create a K8s secret to authenticate with the Container Registry:

kubectl create secret docker-registry <secret-name> --docker-server=<gitlab-registry-url> --docker-username=<username> --docker-password=<access-token>

For instance:

kubectl create secret docker-registry aeros-common-deployments --docker-server=<gitlab-registry-url> --docker-username=<username> --docker-password=<access-token>

Then, ensure that you are using this secret in the K8s manifest files (it must be included inside the imagePullSecrets entry), for instance:

imagePullSecrets:
  - name: aeros-common-deployments

For more information, you can check the official K8s documentation.

For instance:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: self-awareness-hardware-info-ds
spec:
  selector:
    matchLabels:
      name: self-awareness-hardware-info-pod
  template:
    metadata:
      labels:
        name: self-awareness-hardware-info-pod
    spec:
      # Include here the secret
      imagePullSecrets:
        - name: aeros-common-deployments
      #
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
        - name: self-awareness-hardware-info-ctr
          image: <gitlab-registry-url>/aeros-public/common-deployments/self-awareness/hardware_info:1.3.0
          imagePullPolicy: IfNotPresent
          securityContext:
            privileged: true
          env:
            - name: AEROS_KRAKEND_URL
              value: "krakend-service:8000"
            - name: AEROS_IE_ID
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            - name: AEROS_IE_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP

⬆️ Upload a K8s manifest

Send an HTTP request to the API of the common repository using the provided token:

curl --header "PRIVATE-TOKEN: <access-token>" --upload-file <your_manifest.yaml> "https://<gitlab-url>/api/v4/projects/<project-id>/packages/generic/<component_name>/<component_version>/<your_manifest.yaml>"

For example:

curl --header "PRIVATE-TOKEN: <access-token>" --upload-file self-awareness-hardware-info_daemonset.yaml "https://<gitlab-url>/api/v4/projects/65/packages/generic/self-awareness/1.3.0/self-awareness-hardware-info_daemonset.yaml"

💻 Download and deploy a K8s manifest

Send an HTTP request to the API of the common repository using the provided token to download the selected file:

wget --header "PRIVATE-TOKEN: <access-token>" https://<gitlab-url>/api/v4/projects/<project-id>/packages/generic/<component_name>/<component_version>/<your_manifest.yaml>

For instance:

wget --header "PRIVATE-TOKEN: <access-token>" https://<gitlab-url>/api/v4/projects/65/packages/generic/self-awareness/1.3.0/self-awareness-hardware-info_daemonset.yaml

Or directly deploy the manifests in a K8s cluster:

curl --header "PRIVATE-TOKEN: <access-token>" https://<gitlab-url>/api/v4/projects/<project-id>/packages/generic/<component_name>/<component_version>/<your_manifest.yaml> | kubectl apply -f -

For instance:

curl --header "PRIVATE-TOKEN: <access-token>" https://<gitlab-url>/api/v4/projects/65/packages/generic/self-awareness/1.3.0/self-awareness-hardware-info_daemonset.yaml | kubectl apply -f -

☸️ Helm charts in the Package Registry

In Gitlab, the Helm charts are stored in the Package Registry of a code repository. You can find more information in the Gitlab's official documentation.

🔑 Authentication

If the charts include container images from the Container Registry of the common repository with the proper imagePullSecrets configuration in their manifests, you must create a K8s secret to authenticate with the Container Registry:

kubectl create secret docker-registry aeros-common-deployments --docker-server=<gitlab-registry-url> --docker-username=<username> --docker-password=<access-token>

📦 Add the Helm chart repository

You must add the Helm chart repository to a K8s cluster using a valid username and access token in order to use the charts of the common repository:

helm repo add --username <username> --password <access-token> <helm-chart-repository-name> https://<gitlab-url>/api/v4/projects/<project-id>/packages/helm/stable

For instance

helm repo add --username <username> --password <access-token> aeros-common https://<gitlab-url>/api/v4/projects/65/packages/helm/stable

Finally, you can check if the addition of this Helm chart repository has been performed properly by listing all the available charts:

helm search repo aeros-common

Additionally, to obtain all the versions of all the charts:

helm search repo aeros-common -l

⬆️ Upload a Helm chart

⚠️Warning
Remember to include the previously created secret in the K8s manifest files of your Helm charts to use the container images of the common repository.

Package the chart:
```
helm package <path_to_chart_folder>
```
Upload the chart to the repository:

Using the helm cm-push plugin:

helm cm-push <path_to_chart_folder>/<chart.tgz> aeros-common

Example:

helm cm-push /home/aeros/orionld-helm-chart/orion-ld-1.1.0.tgz aeros-common

Making an HTTP POST request:

curl \
  --request POST \
  --user <username> \
  --form 'chart=@<path_to_chart_folder>/<chart.tgz>' \
  https://<gitlab-url>/api/v4/projects/<project-id>/packages/helm/api/stable/charts

💻 Install a Helm chart

⚠️Warning
Ensure that you have previously created the secret in the K8s cluster if the Helm charts use container images of the common repository.

⚠️Warning
It is recommended to always update the Helm chart repository before installing or upgrading Helm charts
 helm repo update

Install a Helm chart from the repository. If the version is not specified, it is installed the latest one:

helm install <chart-release> <helm-chart-repository>/<chart-name> --version <chart-version>

Examples:

helm install orion-ld aeros-common/orion-ld --version 1.0.0

helm install orion-ld aeros-common/orion-ld

To update an already installed Helm chart, follow the same process:

helm upgrade <chart-release> <helm-chart-repository>/<chart-name> --version <chart-version> --set <chart-value-key>=<chart-value-value>

or if you have created a values file in YAML:

helm upgrade <chart-release> <helm-chart-repository>/<chart-name> --version <chart-version> -f <values-file.yaml>

Examples:

helm upgrade orion-ld aeros-common/orion-ld --version 1.0.0 --set broker.args.brokerId=UPV

helm upgrade orion-ld aeros-common/orion-ld -f orion-ld-custom-values.yaml

Name		Name	Last commit message	Last commit date
Latest commit History 1 Commit
images		images
LICENSE		LICENSE
README.md		README.md

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Repository files navigation

aerOS Gitlab internal repository guidelines

🔑 Authentication

🐳 Container images in the Container Registry

🔑 Authentication

⬆️ Push a container image

Multi-arch images

💻 Use a container image

🐳 Docker Compose files in the Package Registry

🔑 Authentication

⬆️ Upload a Docker Compose file

💻 Download and deploy a Docker compose file

☸️ Kubernetes manifests in the Package Registry

🔑 Authentication

⬆️ Upload a K8s manifest

💻 Download and deploy a K8s manifest

☸️ Helm charts in the Package Registry

🔑 Authentication

📦 Add the Helm chart repository

⬆️ Upload a Helm chart

💻 Install a Helm chart

About

Uh oh!

Releases

Packages

License

aerOS-Project/Gitlab-common-repository-guidelines

Folders and files

Latest commit

History

Repository files navigation

aerOS Gitlab internal repository guidelines

🔑 Authentication

🐳 Container images in the Container Registry

🔑 Authentication

⬆️ Push a container image

Multi-arch images

💻 Use a container image

🐳 Docker Compose files in the Package Registry

🔑 Authentication

⬆️ Upload a Docker Compose file

💻 Download and deploy a Docker compose file

☸️ Kubernetes manifests in the Package Registry

🔑 Authentication

⬆️ Upload a K8s manifest

💻 Download and deploy a K8s manifest

☸️ Helm charts in the Package Registry

🔑 Authentication

📦 Add the Helm chart repository

⬆️ Upload a Helm chart

💻 Install a Helm chart

About

Topics

Resources

License

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Packages