Deploy OPSA to Homelab#71
Conversation
… homelab deployment
- fix type definition in ChannelView component
…ose for new services - Introduced .env.prod for production environment variables with encrypted values. - Updated .gitignore to exclude .env.keys. - Modified docker-compose.yml to use images for excretor, digester, and garnisher services.
- Added comprehensive comments outlining prerequisites and workflow steps for deploying OPSA to a homelab server. - Implemented error handling for missing .env.prod file and validation of required environment variables. - Included optional Slack archive download functionality and checks for successful processing. - Improved service startup sequence and health checks for better deployment reliability.
- Changed the npm install command for dotenvx to use sudo for proper permissions during installation.
- Updated encrypted credentials in .env.prod for database and Slack configuration. - Enabled SQLX offline mode in Dockerfile for improved build performance.
- Introduced new SQL query files for user and message data retrieval in PostgreSQL. - Updated Makefile to include a migration step for running SQL migrations during the tummy-dev startup process.
- Updated the migration path in tummy.rs from "../migrations" to "./migrations" for proper migration execution.
…ainContentProps - Modified handleSearch to accept an object with additional parameters: channelId, userId, before, and after. - Changed messageListRef type to allow null values for better type safety.
- Included an EOF marker to properly terminate the Slack archive download command in the workflow script.
- Changed the EOF marker to a tab-indented version for proper command termination in the SSH script.
- Replaced EOF marker with a direct command string for improved readability and execution in the deploy-homelab workflow.
- Added `test-deployment.sh` script to simulate production deployment locally. - Created `TEST_DEPLOYMENT.md` to outline testing procedures and environment setup. - Updated `docker-compose.yml` to include health checks and environment variables for services. - Modified `.gitignore` to exclude `test-deployment/` directory. - Introduced entrypoint script for garnisher service to handle environment variable substitution in nginx configuration.
- Added `migrations.Dockerfile` for a lightweight image using PostgreSQL to run migrations. - Created `run-migrations.sh` script to handle migration execution and tracking. - Updated `TEST_DEPLOYMENT.md` to document the new migrations service and its benefits. - Modified `test-deployment.sh` and GitHub Actions workflow to build and utilize the new migrations image.
…andling - Revised `TEST_DEPLOYMENT.md` to clarify the roles of Docker images, specifically noting that the `opsa-excretor` handles migrations on startup. - Modified `test-deployment.sh` to clean up conflicting migration records and updated logging messages for clarity. - Adjusted GitHub Actions workflow to reflect that the migrations image is not built in the main workflow, as migrations are now managed by the excretor service.
- Deleted `migrations.Dockerfile` and `run-migrations.sh` as the migration handling is now managed by the `opsa-excretor` service. - Updated `TEST_DEPLOYMENT.md` to reflect the removal of the migrations service. - Modified `test-deployment.sh` to remove the migration image build step.
- Deleted the command to remove `slack-archive.zip` from the deploy-homelab workflow as it is no longer necessary.
There was a problem hiding this comment.
Pull Request Overview
This pull request introduces comprehensive deployment automation for the OPSA application, enabling automated deployment to a homelab server with enhanced security and improved developer experience. The changes focus on containerizing all services, securing sensitive configuration through encryption, and streamlining both production deployment and local development workflows.
- Adds GitHub Actions workflow for automated homelab deployment with Docker image builds, secure file transfers, database setup, and health checks
- Implements encrypted production environment configuration using dotenvx for secure secret management
- Enhances developer experience by automating database migrations in local development workflow
Reviewed Changes
Copilot reviewed 14 out of 19 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
.github/workflows/deploy-homelab.yaml |
Comprehensive deployment workflow with Docker builds, SSH transfers, and health checks |
.env.prod |
Encrypted production environment variables using dotenvx encryption |
test-deployment.sh |
Local deployment testing script simulating production workflow |
TEST_DEPLOYMENT.md |
Documentation for testing deployment process locally |
Makefile |
Added automatic database migration execution for local development |
docker-compose.yml |
Added digester and garnisher services with proper image references |
garnisher/Dockerfile |
Multi-stage Docker build for React frontend with nginx |
garnisher/nginx.conf |
Nginx configuration with API proxying and compression |
garnisher/docker-entrypoint.sh |
Dynamic nginx configuration using environment variables |
excretor/Dockerfile |
Updated Rust backend Docker build with migration support |
excretor/src/db/tummy.rs |
Fixed migration path for containerized deployment |
digester/Dockerfile |
Updated Go service Docker build configuration |
garnisher/src/components/ChannelView.tsx |
TypeScript ref type correction |
garnisher/src/App.tsx |
Enhanced search interface and ref type updates |
Files not reviewed (4)
- excretor/.sqlx/query-067912972667664bc8447a7141da5235b25b7775fffdc81d88a8fbba646e3fb0.json: Language not supported
- excretor/.sqlx/query-8e7f2692c4b4f6af366a6ad08e2924c5aff080e19fdf9c413bf7f4b41e1b94d3.json: Language not supported
- excretor/.sqlx/query-98bfa0cfc5ac46089a081ef3a47e924da544a6c83040398b98bd692fdfd96ff7.json: Language not supported
- excretor/.sqlx/query-ed857b1be34cfa6e714159bbb011114e39e6ff42e181a0664d6364366c1208e0.json: Language not supported
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
| cleanup() { | ||
| log_info "Cleaning up..." | ||
| rm -rf "$DEPLOYMENT_DIR" | ||
| docker compose down --remove-orphans 2>/dev/null || true |
There was a problem hiding this comment.
The cleanup function runs docker compose down without specifying a project directory. This could affect other Docker Compose projects running on the same system. Consider using cd "$DEPLOYMENT_DIR" && docker compose down or using the -f flag to specify the compose file explicitly.
| docker compose down --remove-orphans 2>/dev/null || true | |
| if [ -d "$DEPLOYMENT_DIR" ]; then | |
| (cd "$DEPLOYMENT_DIR" && docker compose down --remove-orphans 2>/dev/null) || true | |
| rm -rf "$DEPLOYMENT_DIR" | |
| fi |
| log_info "Cleaning up..." | ||
| rm -rf "$DEPLOYMENT_DIR" | ||
| docker compose down --remove-orphans 2>/dev/null || true | ||
| docker system prune -f --volumes 2>/dev/null || true |
There was a problem hiding this comment.
Running docker system prune -f --volumes in cleanup is overly aggressive and will remove all unused Docker resources system-wide, not just those related to this deployment. This could delete unrelated Docker volumes and images. Consider removing only the specific resources created by this deployment.
| docker system prune -f --volumes 2>/dev/null || true | |
| docker compose down --volumes --remove-orphans 2>/dev/null || true | |
| # Removed overly aggressive docker system prune -f --volumes |
| echo "ERROR: .env.prod file not found. Please ensure it exists in the repository." | ||
| exit 1 | ||
| fi | ||
| DOTENV_PRIVATE_KEY_PROD="${{ secrets.DOTENV_PRIVATE_KEY_PROD }}" dotenvx decrypt -f .env.prod --stdout > .env |
There was a problem hiding this comment.
The private key is exposed in the command line, which could be visible in process lists or logs. Consider using environment variables or a more secure method to pass the key to dotenvx, such as using a temporary file with restricted permissions.
| DOTENV_PRIVATE_KEY_PROD="${{ secrets.DOTENV_PRIVATE_KEY_PROD }}" dotenvx decrypt -f .env.prod --stdout > .env | |
| dotenvx decrypt -f .env.prod --stdout > .env |
| #!/bin/sh | ||
|
|
||
| # Substitute environment variables in nginx configuration | ||
| envsubst '${EXCRETOR_HOST} ${EXCRETOR_PORT}' < /etc/nginx/conf.d/default.conf.template > /etc/nginx/conf.d/default.conf |
There was a problem hiding this comment.
The script references /etc/nginx/conf.d/default.conf.template but the Dockerfile copies nginx.conf to this location as default.conf.template. The source file should be /etc/nginx/conf.d/default.conf.template which matches what's copied in the Dockerfile.
|
|
||
| # Stop any existing services and clean volumes for fresh start | ||
| log_info "Stopping existing services and cleaning volumes..." | ||
| docker compose down --remove-orphans -v 2>/dev/null || true |
There was a problem hiding this comment.
The command is executed in the wrong directory context. At this point, the script is in the deployment directory, but it should ensure it's in the correct directory before running docker compose commands. Consider adding cd "$DEPLOYMENT_DIR" before this command or using the -f flag to specify the compose file path.
|
Very epic, feels like this kind of deployment should be made into a template or something. Will be very useful for deploying further projects. |
harshkhandeparkar
left a comment
harshkhandeparkar
left a comment
There was a problem hiding this comment.
This is beyond me, but I trust it.
Deployment URL: https://opsa.home.karthikeyay.com/
This pull request introduces a GitHub Actions workflow for deploying the OPSA application to a homelab server, adds a production environment file with encrypted secrets, and improves the local development process by ensuring database migrations are run automatically. The main focus is on automating and securing the deployment process for the "rewrite" branch.
Deployment automation and configuration:
.github/workflows/deploy-homelab.yamlto automate deployment to a homelab server. This workflow builds Docker images, securely transfers files, handles environment decryption, sets up and initializes the database, processes Slack archives, starts all services, performs health checks, and sends deployment notifications..env.prodwith encrypted environment variables and a public key for secure decryption during deployment. This file includes all necessary secrets for database, application, and Slack integration configuration.Developer experience improvements:
Makefileto automatically run database migrations (cargo sqlx migrate run) after starting the development database, ensuring the schema is up-to-date before running the digester during local development.test-deployment.sh.Future Scope: