Skip to content

gitops

Jump to bottom
Jesús Daniel Colmenares Oviedo edited this page Sep 22, 2025 · 2 revisions

Using GitOps with Overlord

GitOps is a modern operational framework that uses Git as the single source of truth. It is often mandatory to use a tool that emphasizes a declarative approach, where you define the desired state and the tool does the hard work. Or, in other words, an "everything is code" philosophy.

More than just an add-on, implementing a continuous deployment is a requirement, and the tool (in our case, Overlord) must reconcile the differences between the old and new deployments.

Requirements

In this article, we will use the following tools to make this possible:

  • Gitea: As our hosting for private git repositories. We'll configure a webhook that will run after submitting any changes to the repository.

    Note: This article assumes that you already have a repository in an existing Gitea instance.

  • Webhook: A lightweight service that executes a command when the webhook is triggered.

  • Pipelight: A lightweight CLI tool for creating our pipeline.

  • SOPS: It is not a good idea to put secrets in the repository without using a specialized tool to encrypt them. We will combine this with rage.

  • ntfy: We need to know whether a deployment has been successful or has failed. However, it should be noted that this does not mean that we know whether the service has been deployed correctly or not, what we mean here is that a deployment has been successfully applied.

  • Scripts: Simple scripts that update the environment file, pull changes, notify whether a deployment was successful or not, and apply a deployment.

Preface

Before starting, we need to generate two rage keys: the first key is our personal key, and the second is the key that will be on the remote machine where our pipeline will be deployed.

Local machine:

$ mkdir -p ~/.config/sops/age
$ rage-keygen -o ~/.config/sops/age/keys.txt
Public key: age12...

Remote machine:

# mkdir -p /var/appjail-volumes/sops
# rage-keygen -o /var/appjail-volumes/sops/keys.txt
Public key: age13...
# chown -Rf 1001:1001 /var/appjail-volumes/sops

The Tree

The tree structure of our repository is as follows:

.sops.yaml:

Here we'll define a list of recipients to encrypt our environment file.

creation_rules:
  - age: >-
      age12...,
      age13...

pipelight.yaml:

Pipelight configuration file that defines our process.

pipelines:
  - name: Overlord Deployments
    steps:
      - name: update environment
        commands:
          - ./update-env.sh
      - name: deploy
        commands:
          - ./deploy.sh
    on_success:
      - name: notify
        commands:
          - ntfy publish -t "Pipeline Status" -m "Pipeline finished successfully!" -T "heavy_check_mark" overlord-pipeline
    on_failure:
      - name: notify
        commands:
          - ntfy publish -t "Pipeline Status" -m "Oh no! Pipeline has failed!" -T "rotating_light" overlord-pipeline

.gitignore:

We do not want to push confidential files such as the .env file that we will create later, or unnecessary files such as .timestamp, created by the scripts, or .pipelight, created by pipelight.

/.env
/.timestamp
/.pipelight

update-env.sh:

Responsible for retrieving new changes and updating the unencrypted environment file using the encrypted environment file.

#!/bin/sh

unset SSL_CERT_DIR
unset SSL_CERT_FILE

set -xe
set -o pipefail

git pull

sops decrypt .enc.env > .env

deploy.sh:

This is the heart of our operations. This script uses the .timestamp file containing the latest commit hash, so we can use it to obtain a list of files based on the difference.

In order for this script to deploy a file, we need to define a subdirectory containing the deployment files. In this subdirectory, in addition to the deployment file (e.g.: app.yml), we need an index.txt file containing an ordered list of the files to be applied each time this script detects a change in any file in this subdirectory.

#!/bin/sh

unset SSL_CERT_DIR
unset SSL_CERT_FILE

BASEDIR=`dirname -- "$0"` || exit $?
BASEDIR=`realpath -- "${BASEDIR}"` || exit $?

LAST_COMMIT_FILE="${BASEDIR}/.timestamp"

ntfy_func()
{
    if which -s ntfy; then
        ntfy "$@" >&2 || return $?
    fi

    return 0
}

ntfy_publish()
{
    ntfy_func publish "$@" overlord-pipeline
}

get_changed_files()
{
    git diff --name-only "${LAST_COMMIT_HASH}" | while IFS= read -r file; do
        if [ ! -f "${file}" ]; then
            continue
        fi

        basedir=`dirname -- "${file}"` || exit $?

        if [ "${basedir}" = "." ]; then
            continue
        fi

        index_file="${basedir}/index.txt"

        if [ ! -f "${index_file}" ]; then
            continue
        fi

        cat -- "${index_file}" | xargs -I % echo "${basedir}/%"
    done

    return $?
}

set -ex
set -o pipefail

if [ ! -f "${LAST_COMMIT_FILE}" ]; then
    LAST_COMMIT_HASH=`git rev-parse HEAD`

    echo "${LAST_COMMIT_HASH}" > "${LAST_COMMIT_FILE}"
else
    LAST_COMMIT_HASH=`head -1 -- "${LAST_COMMIT_FILE}"`
fi

get_changed_files | while IFS= read -r deployment_file; do
    ntfy_publish \
        -t "New deployment triggered!" \
        -m "Deploying '${deployment_file}' ..." \
        -T "warning"

    overlord apply -f "${deployment_file}"
done

LAST_COMMIT_HASH=`git rev-parse HEAD`

echo "${LAST_COMMIT_HASH}" > "${LAST_COMMIT_FILE}"

info.yml:

This deployment file is only for testing purposes and to obtain general information about our projects.

kind: readOnly
datacenters:
  main:
    entrypoint: !ENV '${ENTRYPOINT}'
    access_token: !ENV '${TOKEN}'
deployIn:
  labels:
    - all

pipeline/app.yml:

This deployment file defines the jail that will have webhook, pipelight, etc. We need to mount the sops volume containing the keys to decrypt the encrypted environment file.

Attentive readers will notice that we are not exposing any ports. In my case, I'm using NGINX on that machine, so I will create an entry to proxy HTTP requests to the webhook program. If you don't want to do this, simply expose the webhook port.

kind: directorProject
datacenters:
  main:
    entrypoint: !ENV '${ENTRYPOINT}'
    access_token: !ENV '${TOKEN}'
deployIn:
  labels:
    - r2
projectName: pipeline
projectFile: |
  options:
    - virtualnet: ':<random> default'
    - nat:
  services:
    pipeline:
      makejail: !ENV '${OVERLORD_METADATA}/pipeline.makejail'
      options:
        - label: 'overlord.skydns:1'
        - label: 'overlord.skydns.group:pipeline'
        - label: 'overlord.skydns.interface:tailscale0'
        - label: 'appjail.dns.alt-name:pipeline'
      volumes:
        - sops-keys: /pipeline/.config/sops/age
  volumes:
    sops-keys:
      device: /var/appjail-volumes/sops
      options: ro

pipeline/metadata.yml:

The metadata containing the Makejail and other configuration files used by other processes.

kind: metadata
datacenters:
  main:
    entrypoint: !ENV '${ENTRYPOINT}'
    access_token: !ENV '${TOKEN}'
deployIn:
  labels:
    - r2
metadata:
  pipeline.makejail: |
    OPTION start
    OPTION overwrite=force

    CMD pw useradd -n pipeline -d /pipeline
    CMD mkdir -p /pipeline
    CMD chown pipeline:pipeline /pipeline

    CMD mkdir -p /usr/local/etc/pkg/repos
    COPY ${OVERLORD_METADATA}/pipeline.pkg.conf /usr/local/etc/pkg/repos/Latest.conf

    PKG webhook sops git-tiny py311-supervisor py311-pipx go-ntfy pipelight

    CMD git clone --depth 1 https://github.com/DtxdF/overlord.git /overlord
    CMD (cd /overlord; make NOEDITABLE=yes)
    CMD rm -rf /overlord

    USER pipeline

    RUN git clone http://gitea.overlord.lan/DtxdF/git-deployments.git /pipeline/deployments

    CMD mkdir -p /pipeline/.config/ntfy
    COPY ${OVERLORD_METADATA}/pipeline.ntfy.yml /pipeline/.config/ntfy/client.yml

    COPY ${OVERLORD_METADATA}/pipeline.webhook.yaml /usr/local/etc/webhook.yaml

    COPY ${OVERLORD_METADATA}/pipeline.supervisord.conf /usr/local/etc/supervisord.conf

    SYSRC supervisord_enable=YES
    SERVICE supervisord start
  pipeline.ntfy.yml: |
    default-host: http://ntfy.overlord.lan
  pipeline.webhook.yaml: !ENV |
    - id: pipelight
      execute-command: /usr/local/bin/pipelight
      pass-arguments-to-command:
        - source: string
          name: 'run'
        - source: string
          name: 'Overlord Deployments'
      command-working-directory: /pipeline/deployments
      trigger-rule:
        and:
          - match:
              type: payload-hmac-sha256
              secret: '${PIPELINE_SECRET}'
              parameter:
                source: header
                name: X-Hub-Signature-256
          - match:
              type: value
              value: 'refs/heads/main'
              parameter:
                source: payload
                name: ref
  pipeline.supervisord.conf: |
    [unix_http_server]
    file=/var/run/supervisor/supervisor.sock
    [supervisord]
    logfile=/var/log/supervisord.log
    logfile_maxbytes=50MB
    logfile_backups=10
    loglevel=info
    pidfile=/var/run/supervisor/supervisord.pid
    nodaemon=false
    silent=false
    minfds=1024
    minprocs=200
    [rpcinterface:supervisor]
    supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
    [supervisorctl]
    serverurl=unix:///var/run/supervisor/supervisor.sock
    [program:webhook]
    command=/usr/local/sbin/webhook -verbose -hooks /usr/local/etc/webhook.yaml
    user=pipeline
    environment=PATH="%(ENV_PATH)s:/usr/local/bin",HOME=/pipeline,USER=pipeline
    autorestart=false
    redirect_stderr=true
    stdout_logfile=/var/log/%(program_name)s.log
  pipeline.pkg.conf: |
    FreeBSD: {
      url: "pkg+https://pkg.FreeBSD.org/${ABI}/latest",
      mirror_type: "srv",
      signature_type: "fingerprints",
      fingerprints: "/usr/share/keys/pkg",
      enabled: yes
    }

    FreeBSD-kmods: {
      enabled: no
    }

The environment

Let's create the environment file in order to deploy our pipeline.

$ sops edit .enc.env
$ sops decrypt .enc.env | tee .env
...
$ overlord get-info -f info.yml -t projects
...

At a minimum, define three env vars in the .env file: ENTRYPOINT (root chain of our Overlord cluster), TOKEN (access token to the root chain), and PIPELINE_SECRET (a secret that must match the webhook configured in the Gitea repository).

Profit!

$ git commit -m 'Profit!'
...
$ git push
...
$ overlord apply -f pipeline/metadata.yml
$ overlord apply -f pipeline/app.yml
$ overlord get-info -f pipeline/app.yml -t projects --filter-per-project
datacenter: http://controller.namespace.lan:8888
  entrypoint: main
  chain: r2
  labels:
    - all
    - r2
    - services
    - dc-air
  projects:
    pipeline:
      state: DONE
      last_log: 2025-09-02_18h50m32s
      locked: False
      services:
        - {'name': 'pipeline', 'status': 0, 'jail': '4d409f7740'}
      up:
        operation: COMPLETED
        output:
         rc: 0
         stdout: {'errlevel': 0, 'message': None, 'failed': []}
        last_update: 2 minutes and 36.39 seconds
        job_id: 31
        restarted: False
        labels:
         error: False
         message: None
         load-balancer:
           services:
             pipeline:
               error: False
               message: None
         skydns:
           services:
             pipeline:
               error: False
               message: (project:pipeline, service:pipeline, records:[address:True,ptr:None,srv:None] records has been updated.

After the webhook program is up and running, create a webhook at http://gitea.overlord.lan/*user*/*repository*/settings/hooks/gitea/new and configure it.

Now you can deploy using Git!

hello/app.yml:

kind: directorProject
datacenters:
  main:
    entrypoint: !ENV '${ENTRYPOINT}'
    access_token: !ENV '${TOKEN}'
deployIn:
  labels:
    - centralita
projectName: hello-http
projectFile: |
  options:
    - virtualnet: ':<random> default'
    - nat:
  services:
    hello-http:
      makejail: gh+DtxdF/hello-http-makejail
      options:
        - expose: '8412:80 ext_if:tailscale0 on_if:tailscale0'
      arguments:
        - darkhttpd_tag: 14.3

hello/index.txt:

app.yml

console:

$ git commit -m 'Add hello'
[main 45bcd90] Add hello
 2 files changed, 21 insertions(+)
 create mode 100644 hello/app.yml
 create mode 100644 hello/index.txt
$ git push
Enumerating objects: 6, done.
Counting objects: 100% (6/6), done.
Delta compression using up to 4 threads
Compressing objects: 100% (4/4), done.
Writing objects: 100% (5/5), 658 bytes | 658.00 KiB/s, done.
Total 5 (delta 1), reused 0 (delta 0), pack-reused 0 (from 0)
remote: . Processing 1 references
remote: Processed 1 references in total
To ssh://gitea.overlord.lan:2022/DtxdF/git-deployments.git
   de383a4..45bcd90  main -> main
$ overlord get-info -f hello/app.yml -t projects --filter-per-project
datacenter: http://controller.namespace.lan:8888
  entrypoint: main
  chain: centralita
  labels:
    - all
    - centralita
    - services
    - dc-earth
  projects:
    hello-http:
      state: None
      last_log: None
      locked: None
      services:
      up:
        operation: RUNNING
        last_update: 2.64 seconds
        job_id: 7
$ ssh -o LogLevel=ERROR -t control-r2 \
    appjail cmd jexec 4d409f7740 -U pipeline sh -c "'cd deployments; pipelight logs'"
● Succeeded - Tue, 2 Sep 2025 22:27:40 -0400
branch: main
action: manual
commit: de383a45a6dad72b6afed1f21e151cde90cfbd7d
pipeline: Overlord Deployments (23s998ms)
│
├─on_failure
│
╰─on_success

$ overlord get-info -f hello/app.yml -t projects --filter-per-project
datacenter: http://controller.namespace.lan:8888
  entrypoint: main
  chain: centralita
  labels:
    - all
    - centralita
    - services
    - dc-earth
  projects:
    hello-http:
      state: DONE
      last_log: 2025-09-02_22h27m57s
      locked: False
      services:
        - {'name': 'hello-http', 'status': 0, 'jail': 'eb03b178e1'}
      up:
        operation: COMPLETED
        output:
         rc: 0
         stdout: {'errlevel': 0, 'message': None, 'failed': []}
        last_update: 4 minutes and 42.22 seconds
        job_id: 7
        restarted: False
        labels:
         error: False
         message: None
$ curl http://centralita.namespace.lan:8412
Hello, world!
UUID: dae9263e-4d84-4d0c-9b7e-9aa2c619ae38
$ overlord destroy -Ff hello/app.yml

Clone this wiki locally