From 87b78f691abcbd1bc0da180d97256afb354c9271 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?St=C3=A9phane=20Wirtel?= Date: Wed, 4 Feb 2026 11:39:07 +0100 Subject: [PATCH 1/2] feat: add Docker branch isolation for parallel development Implement automatic environment isolation based on git branch names, allowing developers to work on multiple branches simultaneously without conflicts between databases, containers, or Docker volumes. Key changes: - Auto-detect git branch and normalize name (replace special chars) - Tag Docker images by branch (python.ie/website:{branch}) - Isolate PostgreSQL databases per branch (pythonie_{branch}) - Create separate Docker volumes per branch - Add configurable ports for parallel development (WEB_PORT, PG_PORT) - Upgrade Redis from 6.2 to 7-alpine with persistence - Add healthchecks for PostgreSQL and Redis - Remove unused minio and mc services New Taskfile commands: - task branch:info - Show current branch environment - task branch:volumes - List all Docker volumes by branch - task branch:clean - Remove volumes for current branch - task env:write - Generate .env file for docker-compose Documentation: - Add DOCKER-BRANCHES.md with usage examples and troubleshooting - Add .env.example template for custom port overrides - Update .gitignore to exclude generated .env file This enables developers to switch branches or run multiple instances in parallel without losing data or experiencing port conflicts. Co-Authored-By: Claude Sonnet 4.5 --- .env.example | 16 ++++ .gitignore | 1 + DOCKER-BRANCHES.md | 223 +++++++++++++++++++++++++++++++++++++++++++++ Taskfile.yaml | 69 +++++++++++++- docker-compose.yml | 74 ++++++++------- 5 files changed, 344 insertions(+), 39 deletions(-) create mode 100644 .env.example create mode 100644 DOCKER-BRANCHES.md diff --git a/.env.example b/.env.example new file mode 100644 index 0000000..a0635f1 --- /dev/null +++ b/.env.example @@ -0,0 +1,16 @@ +# Docker Compose Environment Variables +# Copy this file to .env and customize as needed + +# Override ports for parallel development (optional) +# Useful when running multiple branches simultaneously +# WEB_PORT=8001 +# PG_PORT=5433 + +# Override git branch detection (optional - normally auto-detected) +# GIT_BRANCH=my-custom-branch + +# Override Docker image name (optional) +# DOCKER_IMAGE=python.ie/website:custom-tag + +# Override database name (optional) +# PGDATABASE=pythonie_custom diff --git a/.gitignore b/.gitignore index d96d0d2..56ecc1b 100644 --- a/.gitignore +++ b/.gitignore @@ -66,6 +66,7 @@ project.db # Environment files with sensitive credentials .envrc +.env development.env production.env diff --git a/DOCKER-BRANCHES.md b/DOCKER-BRANCHES.md new file mode 100644 index 0000000..4ff58b0 --- /dev/null +++ b/DOCKER-BRANCHES.md @@ -0,0 +1,223 @@ +# Docker Multi-Branch Development + +This setup allows you to run isolated Docker environments for each git branch, preventing conflicts when switching branches or working on multiple features simultaneously. + +## How It Works + +### Automatic Branch Detection + +The system automatically: +- Detects your current git branch (`git rev-parse --abbrev-ref HEAD`) +- Normalizes the branch name (e.g., `feature/add-auth` → `feature-add-auth`) +- Uses this normalized name for: + - Docker image tags: `python.ie/website:feature-add-auth` + - Database names: `pythonie_feature-add-auth` + - Container names: `pythonie-web-feature-add-auth` + - Docker volumes: `postgres-data-feature-add-auth` + +### Branch Isolation + +Each branch gets: +- ✅ Its own Docker image +- ✅ Its own PostgreSQL database (isolated data) +- ✅ Its own Redis instance (isolated cache) +- ✅ Its own Docker volumes (persistent storage) +- ✅ Unique container names (no conflicts) + +## Usage + +### Basic Commands + +```bash +# Check your current branch environment +task branch:info + +# Build image for current branch +task docker:build + +# Run development server (uses current branch) +task run + +# Run tests (uses current branch database) +task tests + +# Django shell for current branch +task django:shell-plus +``` + +### Working on Multiple Branches + +**Scenario**: You want to work on `feature/auth` while keeping `main` running. + +```bash +# Terminal 1: Main branch +git checkout main +task run +# → Uses: python.ie/website:main, pythonie_main database, port 8000 + +# Terminal 2: Feature branch (in another terminal) +git checkout feature/auth +task run +# → ERROR: Port 8000 already in use! +``` + +**Solution**: Use custom ports for parallel instances: + +```bash +# Terminal 1: Main branch +git checkout main +task run + +# Terminal 2: Feature branch with custom port +git checkout feature/auth +WEB_PORT=8001 PG_PORT=5433 task run +# → Uses: port 8001, separate database +``` + +### Managing Volumes + +```bash +# List all volumes (all branches) +task branch:volumes + +# Example output: +# pythonie-postgres-data-master +# pythonie-postgres-data-feature-add-auth +# pythonie-redis-data-master +# pythonie-redis-data-feature-add-auth + +# Clean volumes for current branch (DESTRUCTIVE!) +task branch:clean +# → Deletes all data for current branch +``` + +### Switching Branches + +When you switch branches, the system automatically uses the correct environment: + +```bash +# On main branch +git checkout main +task run # Uses main database + +# Switch to feature branch +git checkout feature/add-auth +task run # Automatically uses feature-add-auth database +``` + +**Important**: You must rebuild the image after switching branches if dependencies changed: + +```bash +git checkout feature/new-dependencies +task docker:build # Rebuild image with new dependencies +task run +``` + +## Advanced: Custom Ports for Parallel Development + +Create a `.env` file (git-ignored) to override default ports: + +```bash +# .env +WEB_PORT=8001 +PG_PORT=5433 +``` + +Or pass them inline: + +```bash +WEB_PORT=8001 PG_PORT=5433 task run +``` + +## Troubleshooting + +### Port Already in Use + +``` +Error: port 8000 already in use +``` + +**Solution**: Stop other instances or use custom ports: +```bash +task down # Stop all containers for current branch +# OR +WEB_PORT=8001 PG_PORT=5433 task run +``` + +### Database Not Found + +``` +FATAL: database "pythonie_feature-xyz" does not exist +``` + +**Solution**: Run migrations to create the database: +```bash +task django:migrate +``` + +### Wrong Database Being Used + +```bash +# Check which environment is active +task branch:info + +# Ensure you're on the correct git branch +git branch + +# Rebuild if needed +task docker:build +``` + +### Cleaning Up Old Branches + +After deleting a git branch, clean up its Docker resources: + +```bash +# Remove unused images +docker image prune -a + +# Remove volumes for deleted branches +# (replace 'old-branch-name' with actual normalized branch name) +docker volume rm pythonie-postgres-data-old-branch-name +docker volume rm pythonie-redis-data-old-branch-name + +# Or use the helper (must be on that branch) +git checkout old-branch-name +task branch:clean +``` + +## Environment Variables Reference + +| Variable | Default | Description | +|----------|---------|-------------| +| `GIT_BRANCH` | Auto-detected | Current git branch (normalized) | +| `DOCKER_IMAGE` | `python.ie/website:{branch}` | Docker image name | +| `PGDATABASE` | `pythonie_{branch}` | PostgreSQL database name | +| `WEB_PORT` | `8000` | Web server port | +| `PG_PORT` | `5432` | PostgreSQL port | + +## Tips + +1. **Branch Naming**: Use descriptive branch names. They'll appear in container names and logs. + +2. **Disk Space**: Each branch creates separate volumes. Monitor disk usage: + ```bash + docker system df + ``` + +3. **Production**: This system is for **development only**. Production uses fixed names and ports. + +4. **Database Migrations**: Each branch has independent migrations. Remember to run `task django:migrate` after switching branches with schema changes. + +5. **Shared Data**: If you need to copy data between branches: + ```bash + # Export from main + git checkout main + task run:postgres # Start just postgres + pg_dump -U postgres pythonie_main > /tmp/main.sql + + # Import to feature branch + git checkout feature/xyz + task run:postgres + psql -U postgres pythonie_feature-xyz < /tmp/main.sql + ``` diff --git a/Taskfile.yaml b/Taskfile.yaml index be24ecb..f5cc66b 100644 --- a/Taskfile.yaml +++ b/Taskfile.yaml @@ -3,13 +3,26 @@ version: '3' +vars: + # Detect current git branch and normalize the name + # (replace / with - and remove special characters) + GIT_BRANCH: + sh: git rev-parse --abbrev-ref HEAD 2>/dev/null | sed 's/[^a-zA-Z0-9._-]/-/g' | tr '[:upper:]' '[:lower:]' || echo "dev" + env: - DOCKER_IMAGE: python.ie/website-dev - PGDATABASE: pythonie + # Pass branch name to docker-compose + GIT_BRANCH: "{{.GIT_BRANCH}}" + # Docker image tagged by branch + DOCKER_IMAGE: "python.ie/website:{{.GIT_BRANCH}}" + # Database isolated by branch + PGDATABASE: "pythonie_{{.GIT_BRANCH}}" PGPASSWORD: pythonie PGUSER: postgres PGHOST: 127.0.0.1 HEROKU_APP: pythonie + # Dynamic ports (optional - can stay fixed if running only one instance at a time) + WEB_PORT: 8000 + PG_PORT: 5432 dotenv: ['production.env'] @@ -100,48 +113,68 @@ tasks: cmds: - heroku maintenance:off + env:write: + desc: Write environment variables to .env file for docker-compose + cmds: + - | + cat > .env </dev/null || echo "Postgres volume not found" + - docker volume rm pythonie-redis-data-{{.GIT_BRANCH}} 2>/dev/null || echo "Redis volume not found" + - echo "Volumes cleaned for branch {{.GIT_BRANCH}}" diff --git a/docker-compose.yml b/docker-compose.yml index 702d522..d3c282d 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -1,68 +1,72 @@ services: web: - image: python.ie/website-dev + build: . + image: "python.ie/website:${GIT_BRANCH:-dev}" + container_name: "pythonie-web-${GIT_BRANCH:-dev}" ports: - - "8000:8000" + - "${WEB_PORT:-8000}:8000" command: /usr/bin/fish depends_on: - - postgres - - minio - - redis + postgres: + condition: service_healthy + redis: + condition: service_healthy env_file: - development.env environment: -# DJANGO_SETTINGS_MODULE: pythonie.settings.dev - PGDATABASE: pythonie + PGDATABASE: "pythonie_${GIT_BRANCH:-dev}" PGUSER: postgres PGPASSWORD: pythonie + GIT_BRANCH: "${GIT_BRANCH:-dev}" volumes: - .:/app working_dir: /app - postgres: image: postgres:17 - ports: - - "5432:5432" + container_name: "pythonie-postgres-${GIT_BRANCH:-dev}" + # ports: + # - "${PG_PORT:-5432}:5432" environment: POSTGRES_PASSWORD: pythonie + POSTGRES_DB: "pythonie_${GIT_BRANCH:-dev}" + healthcheck: + test: ["CMD-SHELL", "pg_isready -U postgres"] + interval: 10s + timeout: 5s + retries: 5 + restart: unless-stopped + volumes: + - postgres-data:/var/lib/postgresql/data redis: - image: redis:6.2 - - minio: - image: quay.io/minio/minio - ports: - - "9000:9000" - - "9001:9001" - command: server --console-address ":9001" /data - environment: - MINIO_ROOT_USER: pythonie - MINIO_ROOT_PASSWORD: pythonie + restart: unless-stopped + image: redis:7-alpine + container_name: "pythonie-redis-${GIT_BRANCH:-dev}" volumes: - - minio-data:/data + - redis-data:/data + command: redis-server --appendonly yes healthcheck: - test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"] - interval: 30s - timeout: 20s + test: ["CMD", "redis-cli", "ping"] + interval: 10s + timeout: 3s retries: 3 - mc: - image: quay.io/minio/mc - links: - - minio - volumes: - - ./mc:/root/.mc - - ../s3:/s3 - test: - image: python.ie/website-dev + image: "python.ie/website:${GIT_BRANCH:-dev}" environment: DJANGO_SETTINGS_MODULE: pythonie.settings.tests + PGDATABASE: "pythonie_${GIT_BRANCH:-dev}" + depends_on: + postgres: + condition: service_healthy volumes: - .:/app working_dir: /app command: python pythonie/manage.py test pythonie --verbosity=3 volumes: - minio-data: + postgres-data: + name: "pythonie-postgres-data-${GIT_BRANCH:-dev}" + redis-data: + name: "pythonie-redis-data-${GIT_BRANCH:-dev}" From 3bab347f342064517fde461e0936f290b0f7a911 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?St=C3=A9phane=20Wirtel?= Date: Wed, 4 Feb 2026 12:09:23 +0100 Subject: [PATCH 2/2] feat: add SQLite database isolation by branch and documentation Extend branch isolation to SQLite databases for local development and add comprehensive documentation for the branch isolation feature. Changes to SQLite isolation: - Modify dev.py to use branch-specific SQLite files (db-{branch}.sqlite3) - Add *.sqlite3 to .gitignore to exclude database files - Remove debug print statement from dev.py - Each branch now has isolated SQLite data without requiring worktrees Documentation improvements: - Add complete "Docker Branch Isolation" section to README.md - Document all branch:* commands (info, verify, volumes, clean) - Add examples for multi-branch development and branch switching - Add troubleshooting section for database configuration issues - Update DOCKER-BRANCHES.md to mention SQLite isolation - Add note in local setup about GIT_BRANCH export New Taskfile command: - task branch:verify - Comprehensive verification of branch configuration Shows git branch, expected database, .env file, and PostgreSQL databases This ensures complete data isolation between branches whether using Docker (PostgreSQL) or local development (SQLite), preventing data loss when switching branches. Co-Authored-By: Claude Sonnet 4.5 --- .gitignore | 1 + DOCKER-BRANCHES.md | 1 + README.md | 85 ++++++++++++++++++++++++++++++- Taskfile.yaml | 13 +++++ pythonie/pythonie/settings/dev.py | 4 +- 5 files changed, 102 insertions(+), 2 deletions(-) diff --git a/.gitignore b/.gitignore index 56ecc1b..a86116c 100644 --- a/.gitignore +++ b/.gitignore @@ -73,6 +73,7 @@ production.env # Database dumps and local databases *.dump *.duckdb +*.sqlite3 # Local settings pythonie/pythonie/settings/pgdev.py diff --git a/DOCKER-BRANCHES.md b/DOCKER-BRANCHES.md index 4ff58b0..cef9158 100644 --- a/DOCKER-BRANCHES.md +++ b/DOCKER-BRANCHES.md @@ -20,6 +20,7 @@ The system automatically: Each branch gets: - ✅ Its own Docker image - ✅ Its own PostgreSQL database (isolated data) +- ✅ Its own SQLite database for local dev: `pythonie/db-{branch}.sqlite3` - ✅ Its own Redis instance (isolated cache) - ✅ Its own Docker volumes (persistent storage) - ✅ Unique container names (no conflicts) diff --git a/README.md b/README.md index 0ed9a13..95eecc9 100644 --- a/README.md +++ b/README.md @@ -12,6 +12,8 @@ Website for Python Ireland (python.ie / pycon.ie) community, built with Django 6 ## Quick Start (Docker - Recommended) +**Note**: Each git branch automatically gets its own isolated database and Docker environment. See [Docker Branch Isolation](#docker-branch-isolation) below. + 1. Build the Docker image: ```bash task docker:build @@ -20,7 +22,7 @@ Website for Python Ireland (python.ie / pycon.ie) community, built with Django 6 2. Start supporting services: ```bash - docker compose up -d postgres redis minio + docker compose up -d postgres redis ``` 3. Run database migrations: @@ -48,10 +50,79 @@ Website for Python Ireland (python.ie / pycon.ie) community, built with Django 6 7. Visit http://127.0.0.1:8000/ to see the site with sample content 8. Access Wagtail admin at http://127.0.0.1:8000/admin/ +## Docker Branch Isolation + +The project automatically isolates Docker environments by git branch, allowing you to work on multiple branches simultaneously without conflicts. + +### How It Works + +Each git branch gets: +- **Separate Docker image**: `python.ie/website:{branch-name}` +- **Isolated PostgreSQL database**: `pythonie_{branch-name}` (for docker-compose with pgdev settings) +- **Isolated SQLite database**: `pythonie/db-{branch-name}.sqlite3` (for local dev) +- **Dedicated volumes**: `pythonie-postgres-data-{branch}`, `pythonie-redis-data-{branch}` +- **Unique containers**: `pythonie-web-{branch}`, `pythonie-postgres-{branch}`, `pythonie-redis-{branch}` + +### Branch Commands + +```bash +# Check your current branch environment +task branch:info + +# Verify database configuration +task branch:verify + +# List all Docker volumes (all branches) +task branch:volumes + +# Clean up volumes for current branch (DESTRUCTIVE!) +task branch:clean +``` + +### Working on Multiple Branches + +**Scenario**: Work on `main` and `feature/new-auth` simultaneously. + +```bash +# Terminal 1: Main branch on default ports +git checkout main +task run # Uses pythonie_main database on port 8000 + +# Terminal 2: Feature branch on custom ports +git checkout feature/new-auth +WEB_PORT=8001 PG_PORT=5433 task run # Uses pythonie_feature-new-auth on port 8001 +``` + +### Switching Branches + +When you switch branches, the system automatically uses the correct environment: + +```bash +git checkout main +task run # Automatically uses main database + +git checkout feature/xyz +task run # Automatically uses feature-xyz database (isolated from main) +``` + +**Note**: Rebuild the Docker image after switching branches if dependencies changed: +```bash +git checkout feature/new-dependencies +task docker:build +task run +``` + +For detailed documentation, see [DOCKER-BRANCHES.md](DOCKER-BRANCHES.md). + ## Local Setup (Without Docker) If you prefer to develop without Docker: +**Note**: When developing locally, each git branch automatically uses its own SQLite database (`db-{branch}.sqlite3`). Export `GIT_BRANCH` for automatic isolation: +```bash +export GIT_BRANCH=$(git rev-parse --abbrev-ref HEAD | sed 's/[^a-zA-Z0-9._-]/-/g' | tr '[:upper:]' '[:lower:]') +``` + 1. Fork the repository into your own personal GitHub account 2. Clone your fork: `git clone git@github.com:YourGitHubName/website.git` 3. Ensure you are running Python 3.13: `python -V` should output `Python 3.13.x` @@ -134,6 +205,12 @@ task heroku:releases # Show deployment history task heroku:rollback # Rollback to previous release task heroku:maintenance:on # Enable maintenance mode task heroku:maintenance:off # Disable maintenance mode + +# Branch Isolation (Docker) +task branch:info # Show current branch and environment info +task branch:verify # Verify database configuration +task branch:volumes # List all Docker volumes (all branches) +task branch:clean # Remove volumes for current branch (CAUTION!) ``` ### Direct Django Commands @@ -257,6 +334,12 @@ This project uses several tools to streamline development: - Rebuild Docker image: `task docker:build` - Reinstall dependencies: `pip install -r requirements.txt` +### Wrong Database Being Used +- Check current branch: `git branch` +- Verify environment: `task branch:info` +- Check database configuration: `task branch:verify` +- Ensure .env file is up to date: `task env:write` + ## Contributing 1. Fork the repository into your own GitHub account diff --git a/Taskfile.yaml b/Taskfile.yaml index f5cc66b..c76b966 100644 --- a/Taskfile.yaml +++ b/Taskfile.yaml @@ -302,3 +302,16 @@ tasks: - docker volume rm pythonie-postgres-data-{{.GIT_BRANCH}} 2>/dev/null || echo "Postgres volume not found" - docker volume rm pythonie-redis-data-{{.GIT_BRANCH}} 2>/dev/null || echo "Redis volume not found" - echo "Volumes cleaned for branch {{.GIT_BRANCH}}" + + branch:verify: + desc: Verify database configuration for current branch + cmds: + - echo "=== Branch Configuration ===" + - echo "Git branch → {{.GIT_BRANCH}}" + - echo "Expected database → pythonie_{{.GIT_BRANCH}}" + - echo "" + - echo "=== Docker Environment ===" + - cat .env 2>/dev/null || echo ".env file not found (run any docker compose task to generate)" + - echo "" + - echo "=== PostgreSQL Databases ===" + - docker exec pythonie-postgres-{{.GIT_BRANCH}} psql -U postgres -c "\l" 2>/dev/null | grep pythonie || echo "PostgreSQL container not running" diff --git a/pythonie/pythonie/settings/dev.py b/pythonie/pythonie/settings/dev.py index 9e09010..800a24b 100644 --- a/pythonie/pythonie/settings/dev.py +++ b/pythonie/pythonie/settings/dev.py @@ -16,10 +16,12 @@ WAGTAILADMIN_BASE_URL = "http://localhost:8000" # SQLite (simplest install) +# Database name includes git branch for isolation between branches +GIT_BRANCH = os.environ.get("GIT_BRANCH", "dev") DATABASES = { "default": { "ENGINE": "django.db.backends.sqlite3", - "NAME": join(PROJECT_ROOT, "db.sqlite3"), + "NAME": join(PROJECT_ROOT, f"db-{GIT_BRANCH}.sqlite3"), } }