Production Deployment Implementation #36
Description
Production Deployment Implementation
Overview
Implement production deployment for Holistix Forge on Ubuntu VPS. This issue tracks the complete production setup based on our Production Deployment Guide.
Status: 📋 Ready to start after Issue #30
Priority: High
Related Documentation: PRODUCTION_DEPLOYMENT.md
Context
Our development environment is designed with production parity in mind, achieving 85% reusability of existing scripts and 90% architecture reuse. The main work involves:
- ❌ Removing dev container layer (direct install on Ubuntu VPS)
⚠️ Implementing new DNS architecture (after Issue feat(infra): simplify DNS infrastructure by replacing PowerDNS with CoreDNS file plugin #30)- ✏️ Replacing mkcert with Let's Encrypt SSL
- ➕ Adding systemd service management
- ➕ Implementing security hardening
- ➕ Setting up operational monitoring and backups
Dependencies
- Issue feat(infra): simplify DNS infrastructure by replacing PowerDNS with CoreDNS file plugin #30 - DNS architecture simplification (CoreDNS file plugin - BLOCKING)
Implementation Tasks
Phase 1: Core Infrastructure
- Provision Ubuntu 24.04 VPS (4 vCPU, 8GB RAM, 100GB SSD minimum)
- Configure SSH hardening and firewall (ufw)
- Configure DNS at domain registrar (NS records)
- Install core dependencies (Node.js, PostgreSQL, Nginx, Docker)
- Setup Let's Encrypt SSL with wildcard certificate
Phase 2: Script Adaptation
- Wait for Issue feat(infra): simplify DNS infrastructure by replacing PowerDNS with CoreDNS file plugin #30 completion
- Create
scripts/production/setup-production.sh - Create production DNS setup script (based on feat(infra): simplify DNS infrastructure by replacing PowerDNS with CoreDNS file plugin #30 outcome)
- Create systemd service files for all services
- Adapt
create-env.shfor production (Let's Encrypt paths, strong passwords) - Create
scripts/production/deploy.shfor deployment automation - Create backup and restore scripts
- Document production procedures
Phase 3: Deployment & Testing
- Build application artifacts (
npx nx run-many --target=build --all --configuration=production) - Run production setup script
- Create production environment
- Deploy artifacts to VPS
- Start and verify all services
- Test full stack functionality (DNS, HTTPS, API, frontend, gateways, containers)
- Perform security audit
Phase 4: Operations Setup
- Configure Grafana alerts (pool exhaustion, disk usage, memory, SSL expiry, API errors)
- Setup external uptime monitoring
- Test and verify backup/restore procedures
- Create operational runbooks
- Perform load testing
- Document disaster recovery plan
- Setup CI/CD integration (optional)
Success Criteria
- Production deployment works end-to-end
- All services auto-start on boot
- Monitoring and alerts configured
- Backups tested and working
- Security audit passed
- Load testing completed
- Operational documentation complete
VPS Requirements
- Ubuntu 24.04 LTS
- 4 vCPU, 8GB RAM, 100GB SSD (minimum)
- Static public IP
- Owned domain name with DNS registrar access
- DNS provider API access for Let's Encrypt
Security Checklist
- Firewall configured (ufw)
- SSH hardened (no root, key-only)
- Strong database passwords (32+ characters)
- Limited database user privileges
- SSL/TLS enforced (Let's Encrypt)
- Rate limiting configured (Nginx)
- Security headers added (X-Frame-Options, CSP, etc.)
- Automated security updates enabled
Monitoring Checklist
- System metrics (CPU, RAM, disk, network)
- Application metrics (API requests, response times)
- Database metrics (connections, queries)
- Gateway pool status
- Container health
- Log aggregation (Loki)
- Distributed tracing (Tempo)
- External uptime monitoring
Backup Strategy
What to backup: PostgreSQL databases, organization data (org-data/), Nginx configs, environment files, SSL certificates
Retention: Daily (7 days), Weekly (4 weeks), Monthly (12 months)
References
- Production Deployment Guide
- Local Development Guide
- DNS Complete Guide
- Gateway Architecture
- System Architecture
- GitHub Issue #30 - DNS Simplification (BLOCKING)
Notes
- 85% of development scripts are reusable with minor modifications
- 90% of architecture transfers directly to production
- Main differences are simplifications (no dev container, simpler DNS after feat(infra): simplify DNS infrastructure by replacing PowerDNS with CoreDNS file plugin #30)
- Production will be simpler than dev after DNS changes
- Development environment designed with production parity from the start
Labels: enhancement, production, deployment, infrastructure
Milestone: Production Ready
Assignees: Core team
Blocked by: #30