A. GitLab Backup to S3 with Retention

This project provides a simple solution for backing up GitLab projects to S3-compatible storage (e.g., MinIO, Arvan S3) and retaining only the latest 3 backups.

It consists of two scripts:

• group-backup.sh → Exports all GitLab projects in a given group and compresses them.
• main.sh → Uploads the backup archive to S3 and enforces retention (keeps only the latest 3 backups).

---

📦 How it Works

1. group-backup.sh — Backup GitLab Projects

• Fetches all project IDs from a GitLab group.
• Exports each project using GitLab’s.
• Waits until each export finishes.
• Downloads the .tar.gz archive and renames it based on the project’s namespace path.
• Stores all project backups under a directory named:

2. main.sh — Upload to S3 & Enforce Retention

• Runs group-backup.sh to generate the backup archive.
• Configures the mc CLI with your S3 endpoint and credentials.
• Uploads the backup archive to the configured S3 bucket/directory.
• Keeps only the latest 3 backup archives in S3 (removes older ones).
• Cleans up local backup files.

---

⚙️ Requirements (for local running only without Docker-Compose)

• GitLab API Access
• A Personal Access Token with read_api and api scope.
• Environment variables for GitLab source:

  export SOURCE_GITLAB_URL="https://gitlab.example.com"
  export SOURCE_GITLAB_TOKEN="your_token"
  export SOURCE_GITLAB_GROUP_ID="12345"

• S3 Access
• Any S3-compatible object storage (AWS S3, MinIO, Arvan S3, etc.).
• Install mc (MinIO Client).
• Environment variables for S3:

  export S3_ENDPOINT="https://s3.example.com"
  export S3_ACCESS_KEY="your_access_key"
  export S3_SECRET_KEY="your_secret_key"
  export S3_BUCKET="your-bucket-name"
  export S3_DIRECTORY="gitlab-backups"

---

🚀 Usage

Backup GitLab Projects with Docker-Compose (Recommended)

docker compose up -d --build gitlab-backup

Backup GitLab Projects Locally

bash ./main.sh

B. GitLab Backup Restore from S3

This guide explains how to restore GitLab projects from the S3 backups created using the backup scripts.

The restore process is handled by:

• group-restore.sh → Imports .tar.gz project exports into the target GitLab group.
• main.sh → Fetches the latest backup from S3, extracts it, and calls group-restore.sh.

---

🔄 How it Works

1. main.sh

• Configures the mc S3 client.
• Finds the latest backup archive in your S3 bucket.
• Downloads and extracts the archive.
• Calls group-restore.sh to import each project into the target GitLab group.
• Cleans up downloaded files after restore.

2. group-restore.sh

• Iterates through all .tar.gz files in the extracted backup folder.
• Uses the GitLab Project Import API.
• Creates a new project for each archive inside the target GitLab group.

---

⚙️ Requirements (for local running only without Docker-Compose)

• Target GitLab Access

  • A Personal Access Token with api scope.
  • A Target Group ID where projects should be restored.
  • Environment variables:

    export TARGET_GITLAB_URL="https://gitlab.example.com"
    export TARGET_GITLAB_TOKEN="your_target_token"
    export TARGET_GITLAB_GROUP_ID="12345"

• S3 Access

  • Same environment variables used during backup:

    export S3_ENDPOINT="https://s3.example.com"
    export S3_ACCESS_KEY="your_access_key"
    export S3_SECRET_KEY="your_secret_key"
    export S3_BUCKET="your-bucket-name"
    export S3_DIRECTORY="gitlab-backups"

• Tools

  • mc CLI
  • tar
  • curl
  • jq (optional for debugging)

---

🚀 Usage

restore GitLab Projects with Docker-Compose (Recommended)

docker compose up -d --build gitlab-restore

restore GitLab Projects Locally

bash ./main.sh