This repository provides reproducible S3 migration testing environments for Godwit Sync. It contains hands-on labs for simulating real-world S3 migrations locally using Docker, so you can rehearse, validate, and troubleshoot a migration end-to-end without any cloud credentials. Each lab is a self-contained Docker Compose stack that provisions source and target object storage, runs a real S3 migration workflow, and exposes the verification or observability tooling used to validate it.
- Platform and DevOps engineers migrating data between S3, MinIO, RustFS, Garage, Moto, or other S3-compatible object stores.
- SREs adopting Prometheus and Grafana observability for long-running data transfers.
- Backup, compliance, and data engineers validating object counts, checksums, version history, and Object Lock state across a migration.
Five Docker-based labs covering the core surfaces of an S3 migration: transfer direction, version history, multi-bucket orchestration, real-time monitoring, and post-migration verification. Each lab pairs with a long-form S3 migration guide and is designed to run to completion on a developer laptop.
- S3 Migration Guide: Upload, Transfer, and Download S3 Data — Run the three canonical migration directions (
fs → s3,s3 → s3,s3 → fs) across MinIO and Garage. Demonstrates basic sync, plan-first transfers with--plan-onlyand--resume, and checksum verification. - S3 Version History Migration — Migrate versioned S3 objects with mixed storage classes (STANDARD, GLACIER) and replicate Object Lock metadata using the
--version-modeflag. Covers full history, point-in-time filtering, and Object Lock-protected buckets. - Multi-Bucket Migration Orchestration — Migrate four S3 buckets in parallel between two RustFS clusters using a YAML runbook, a Python orchestrator, and a single Grafana dashboard that shows all pairs. One pair is pre-configured to fail so you can practice the retry workflow.
- Real-Time Migration Monitoring with Prometheus and Grafana — Scrape Godwit Sync's built-in Prometheus metrics every 5 seconds and render them in a pre-provisioned 52-panel Grafana dashboard. Tracks throughput, latency, multipart uploads, retries, and verification across a multi-round, three-hop migration.
- Verifying S3 Migrations — Walk through the
godwit plansubcommands to list migration runs, inspect object counts and storage class distribution, surface failed or case-conflicting objects, and generate a compliance audit report from the state database.
Each lab ships with a docker-compose.yml that provisions the object storage endpoints (MinIO, RustFS, Garage, or Moto), any supporting services (Prometheus, Grafana), and a seed step that populates the source buckets. A migration script or YAML runbook then invokes Godwit Sync against the local stack and writes results to a state database. All state lives in Docker volumes, so a lab can be reset to a clean slate and re-run.
The labs exercise S3-compatible object stores that run locally under Docker:
- MinIO — primary test target with full S3 API coverage.
- RustFS — used for multi-cluster and multi-bucket scenarios.
- Garage — used as a cross-provider transfer target.
- Moto — used to emulate AWS S3 storage classes and Object Lock behavior.
These labs are provided as-is for educational and evaluation purposes. Use them at your own risk. The authors and Godwit.io assume no liability for any damage or data loss resulting from running these labs.
The scripts, configuration files, and documentation in this repository are free to use, modify, and distribute under the MIT License.
Note: Godwit Sync itself is proprietary licensed software. These labs demonstrate its usage but do not grant any license to the Godwit Sync application. A valid Godwit Sync license is required for production use.