feat: [#14] Phase 1 - 12-Factor App configuration management system

.github/prompts/run-integration-testing-guide.prompt.md

@@ -0,0 +1,153 @@
+---
+mode: agent
+---
+
+# Integration Testing Guide Execution Instructions
+
+As an expert system administrator, you will execute the **complete integration testing process** following the [Integration Testing Guide](../../docs/guides/integration-testing-guide.md).
+
+## 📋 Overview
+
+This guide performs a **full end-to-end integration test** that includes:
+
+1. **Clean existing state** (VM, application data, certificates)
+2. **Deploy fresh infrastructure** (VM with Ubuntu 24.04)
+3. **Wait for cloud-init completion** (system provisioning)
+4. **Run comprehensive integration tests** (services, connectivity, functionality)
+5. **Perform smoke testing** (external validation with official client tools)
+6. **Clean up resources** (return to clean state)
+
+**Expected Duration**: ~8-12 minutes total
+**Prerequisites**: Must have completed initial setup (`make test-prereq`)
+
+## 🎯 Execution Requirements
+
+### CRITICAL Rules to Follow:
+
+1. **Sequential Execution**: Follow steps in exact order - do NOT skip or reorder
+2. **No Command Modifications**: Execute commands exactly as written in the guide
+3. **Working Directory**: Always run from project root directory
+4. **Error Handling**: Document any failures or deviations immediately
+5. **Complete Process**: Execute the entire guide from start to finish
+
+### What Gets Cleaned (Destructive Operations):
+
+- **Virtual Machine**: Complete VM destruction and recreation
+- **Application Storage**: Database, SSL certificates, configuration files
+- **OpenTofu State**: Infrastructure state reset
+- **libvirt Resources**: VM disks, cloud-init ISOs, network configurations
+
+## 📝 Step-by-Step Instructions
+
+### Phase 1: Preparation and Cleanup
+
+- **Step 1.1-1.8**: Clean existing infrastructure and application state
+- **Critical**: Step 1.8 (Clean Application Storage) is destructive but recommended
+- **Outcome**: Clean slate for fresh deployment
+
+### Phase 2: Infrastructure Deployment
+
+- **Step 2.1-2.4**: Deploy VM with OpenTofu/Terraform
+- **Critical**: Wait for cloud-init completion (Step 3)
+- **Outcome**: Provisioned VM with Torrust Tracker ready
+
+### Phase 3: Integration Testing
+
+- **Step 4**: Run comprehensive integration tests
+- **Step 5**: Optional manual verification
+- **Step 6**: Optional performance testing
+- **Outcome**: Validated working system
+
+### Phase 4: External Validation
+
+- **Step 7**: External smoke testing with official client tools
+- **Reference**: Use [Smoke Testing Guide](../../docs/guides/smoke-testing-guide.md) for details
+- **Outcome**: Black-box validation of tracker functionality
+
+### Phase 5: Cleanup
+
+- **Step 8**: Clean up all resources
+- **Step 9**: Review insights and best practices
+- **Outcome**: Return to clean state
+
+## 🚨 Important Notes
+
+### SSH Key Configuration
+
+- **Required**: Must configure SSH keys before deployment
+- **Location**: `infrastructure/terraform/local.tfvars`
+- **Template**: Available in `infrastructure/terraform/terraform.tfvars.example`
+
+### Cloud-Init Wait Time
+
+- **Critical**: DO NOT skip Step 3 (cloud-init completion)
+- **Duration**: 2-3 minutes typically
+- **Failure Mode**: SSH connection failures if rushed
+
+### Error Documentation
+
+- **Immediate**: Document any command failures or unexpected outputs
+- **Location**: Add issues directly to the integration testing guide
+- **Format**: Include error messages, context, and resolution steps
+
+### Non-Standard Commands
+
+- **Approval Required**: Only execute commands not in the guide if absolutely necessary
+- **Documentation**: Clearly indicate when deviating from guide
+- **Justification**: Explain why the deviation was needed
+
+## 🔧 Troubleshooting Guidance
+
+### Common Issues and Solutions:
+
+1. **"Command not found"**: Verify you're in project root directory
+2. **SSH connection failures**: Ensure cloud-init has completed
+3. **libvirt permission errors**: Check user is in libvirt group
+4. **VM deployment timeouts**: Normal during cloud-init, wait longer
+5. **Storage volume conflicts**: Run manual cleanup steps from guide
+
+### When to Deviate from Guide:
+
+- **System-specific issues**: Different Linux distributions may need adjustments
+- **Network configuration**: Firewall or DNS issues requiring resolution
+- **Permission problems**: User/group configuration fixes
+- **Always document**: Any deviations with full explanation
+
+## 📊 Success Criteria
+
+### Integration Test Success Indicators:
+
+- ✅ All services start successfully (Docker Compose)
+- ✅ Tracker responds to UDP/HTTP requests
+- ✅ API endpoints return expected data
+- ✅ Grafana dashboards display metrics
+- ✅ MySQL database is accessible and functional
+
+### Smoke Test Success Indicators:
+
+- ✅ UDP tracker clients receive responses
+- ✅ HTTP tracker clients receive responses
+- ✅ API health checks return "Ok"
+- ✅ Statistics endpoints return valid data
+- ✅ Metrics endpoints return Prometheus data
+
+## 🎯 Final Deliverables
+
+Upon completion, you should have:
+
+1. **Executed Complete Guide**: All steps from 1.1 through 9
+2. **Documented Issues**: Any problems encountered and how they were resolved
+3. **Validated Functionality**: Both integration and smoke tests passed
+4. **Clean State**: All resources cleaned up and ready for next test
+5. **Updated Documentation**: Any guide improvements or corrections needed
+
+## 📖 Additional Resources
+
+- **Integration Testing Guide**: [docs/guides/integration-testing-guide.md](../../docs/guides/integration-testing-guide.md)
+- **Smoke Testing Guide**: [docs/guides/smoke-testing-guide.md](../../docs/guides/smoke-testing-guide.md)
+- **Quick Start Guide**: [docs/infrastructure/quick-start.md](../../docs/infrastructure/quick-start.md)
+- **Troubleshooting**: See infrastructure documentation for libvirt and OpenTofu issues
+
+---
+
+**Remember**: This is a comprehensive test that validates the entire deployment pipeline. Take your time, follow each step carefully, and document everything for future improvements.

.yamllint-ci.yml

@@ -1,5 +1,8 @@
 extends: default
 
+ignore: |
+  application/storage/
+
 rules:
   line-length:
     max: 120 # More reasonable for infrastructure code

Makefile

@@ -1,5 +1,5 @@
 # Makefile for Torrust Tracker Local Testing Infrastructure
-.PHONY: help init plan apply destroy test clean status refresh-state ssh install-deps console vm-console lint lint-yaml lint-shell lint-markdown
+.PHONY: help init plan apply destroy test clean status refresh-state ssh install-deps console vm-console lint lint-yaml lint-shell lint-markdown configure-local configure-production validate-config validate-config-production deploy-local deploy-production start-services stop-services
 
 # Default variables
 VM_NAME ?= torrust-tracker-demo
@@ -189,7 +189,23 @@ clean-and-fix: ## Clean up all VMs and fix libvirt permissions
 	@cd $(TERRAFORM_DIR) && rm -f terraform.tfstate terraform.tfstate.backup .terraform.lock.hcl 2>/dev/null || true
 	@echo "3. Cleaning libvirt images:"
 	@sudo rm -f /var/lib/libvirt/images/torrust-tracker-demo* /var/lib/libvirt/images/ubuntu-24.04-base.qcow2 2>/dev/null || true
-	@echo "4. Fixing libvirt setup:"
+	@echo "4. Cleaning application storage (generated configuration files):"
+	@if [ -d "application/storage" ]; then \
+		echo "   WARNING: This will delete all generated configuration files in application/storage/"; \
+		echo "   This includes nginx configs, tracker configs, and any cached data."; \
+		echo "   These files will be regenerated when you run 'make configure-local'."; \
+		read -p "   Do you want to delete application/storage? (y/N): " confirm; \
+		if [ "$$confirm" = "y" ] || [ "$$confirm" = "Y" ]; then \
+			echo "   Removing application/storage..."; \
+			rm -rf application/storage; \
+			echo "   ✓ Application storage cleaned"; \
+		else \
+			echo "   Skipping application/storage cleanup"; \
+		fi; \
+	else \
+		echo "   No application/storage directory found"; \
+	fi
+	@echo "5. Fixing libvirt setup:"
 	@$(MAKE) fix-libvirt
 	@echo "✓ Clean up complete. You can now run 'make apply' safely."
 
@@ -342,3 +358,61 @@ vm-console: ## Access VM graphical console (GUI)
 		echo "virt-viewer not found. Please install it:"; \
 		echo "  sudo apt install virt-viewer"; \
 	fi
+
+# Configuration Management Targets
+configure-local: ## Generate local environment configuration
+	@echo "Generating local environment configuration..."
+	@infrastructure/scripts/configure-env.sh local
+
+configure-production: ## Generate production environment configuration (requires secrets)
+	@echo "Generating production environment configuration..."
+	@infrastructure/scripts/configure-env.sh production
+
+validate-config: ## Validate generated configuration files
+	@echo "Validating configuration files..."
+	@infrastructure/scripts/validate-config.sh local
+
+validate-config-production: ## Validate production configuration files
+	@echo "Validating production configuration files..."
+	@infrastructure/scripts/validate-config.sh production
+
+# Deployment workflow targets
+deploy-local: configure-local ## Deploy VM and configure for local environment
+	@echo "Deploying local environment..."
+	@$(MAKE) apply
+	@echo "Waiting for VM to be ready..."
+	@sleep 30
+	@echo "Starting application services..."
+	@$(MAKE) start-services
+
+deploy-production: configure-production ## Deploy and configure for production environment (requires secrets)
+	@echo "Deploying production environment..."
+	@$(MAKE) apply
+	@echo "Waiting for VM to be ready..."
+	@sleep 30
+	@echo "Starting application services..."
+	@$(MAKE) start-services
+
+start-services: ## Start Docker Compose services in the VM
+	@echo "Starting Docker Compose services..."
+	@VM_IP=$$(cd $(TERRAFORM_DIR) && tofu output -raw vm_ip 2>/dev/null) || \
+	 VM_IP=$$(virsh domifaddr $(VM_NAME) | grep ipv4 | awk '{print $$4}' | cut -d'/' -f1); \
+	if [ -n "$$VM_IP" ]; then \
+		echo "Starting services on $$VM_IP..."; \
+		ssh -o StrictHostKeyChecking=no torrust@$$VM_IP 'cd /home/torrust/github/torrust/torrust-tracker-demo/application && docker compose up -d'; \
+	else \
+		echo "Could not get VM IP. Is the VM deployed?"; \
+		exit 1; \
+	fi
+
+stop-services: ## Stop Docker Compose services in the VM
+	@echo "Stopping Docker Compose services..."
+	@VM_IP=$$(cd $(TERRAFORM_DIR) && tofu output -raw vm_ip 2>/dev/null) || \
+	 VM_IP=$$(virsh domifaddr $(VM_NAME) | grep ipv4 | awk '{print $$4}' | cut -d'/' -f1); \
+	if [ -n "$$VM_IP" ]; then \
+		echo "Stopping services on $$VM_IP..."; \
+		ssh -o StrictHostKeyChecking=no torrust@$$VM_IP 'cd /home/torrust/github/torrust/torrust-tracker-demo/application && docker compose down'; \
+	else \
+		echo "Could not get VM IP. Is the VM deployed?"; \
+		exit 1; \
+	fi

application/compose.yaml

@@ -108,11 +108,12 @@ services:
     tty: true
     restart: unless-stopped
     environment:
-      - USER_ID=${USER_ID}
-      - TORRUST_TRACKER_DATABASE=${TORRUST_TRACKER_DATABASE:-mysql}
-      - TORRUST_TRACKER_CONFIG_OVERRIDE_CORE__DATABASE__DRIVER=${TORRUST_TRACKER_CONFIG_OVERRIDE_CORE__DATABASE__DRIVER:-mysql}
-      - TORRUST_TRACKER_CONFIG_OVERRIDE_CORE__DATABASE__PATH=${TORRUST_TRACKER_CONFIG_OVERRIDE_CORE__DATABASE__PATH:-mysql://torrust:password@mysql:3306/torrust_tracker}
-      - TORRUST_TRACKER_CONFIG_OVERRIDE_HTTP_API__ACCESS_TOKENS__ADMIN=${TORRUST_TRACKER_CONFIG_OVERRIDE_HTTP_API__ACCESS_TOKENS__ADMIN:-MyAccessToken}
+      - USER_ID=${USER_ID:-1000}
+      # Database connection for tracker (using Figment override pattern)
+      - TORRUST_TRACKER_CONFIG_OVERRIDE_CORE__DATABASE__DRIVER=mysql
+      - TORRUST_TRACKER_CONFIG_OVERRIDE_CORE__DATABASE__PATH=mysql://${MYSQL_USER}:${MYSQL_PASSWORD}@mysql:3306/${MYSQL_DATABASE}
+      # Admin API token for tracker (using Figment override pattern)
+      - TORRUST_TRACKER_CONFIG_OVERRIDE_HTTP_API__ACCESS_TOKENS__ADMIN=${TRACKER_ADMIN_TOKEN}
     networks:
       - backend_network
     ports:

application/share/bin/install.sh

@@ -1,21 +1,42 @@
 #!/bin/bash
 
+# Torrust Tracker Demo Installation Script
+# This script creates the required directory structure for the application.
+# Following 12-factor principles, it expects .env to be provided by the infrastructure layer.
+
 if ! [ -f "./.env" ]; then
-	echo "Creating compose .env './.env'"
-	cp .env.production .env
+	echo "ERROR: Environment file './.env' not found!"
+	echo "The .env file must be provided by the infrastructure configuration system."
+	echo "Expected location: $(pwd)/.env"
+	echo ""
+	echo "To generate the .env file, run:"
+	echo "  make configure-local    # For local development"
+	echo "  make configure-production # For production deployment"
+	exit 1
 fi
 
+echo "Found environment file: ./.env"
+
 ## Proxy
 
 mkdir -p ./storage/proxy/etc/nginx-conf
 mkdir -p ./storage/proxy/webroot
 mkdir -p ./storage/dhparam
 
+# Verify nginx configuration exists (should be provided by infrastructure)
 if ! [ -f "./storage/proxy/etc/nginx-conf/nginx.conf" ]; then
-	echo "Creating proxy config file: './storage/proxy/etc/nginx-conf/nginx.conf'"
-	cp ./share/container/default/config/nginx.conf ./storage/proxy/etc/nginx-conf/nginx.conf
+	echo "ERROR: Nginx configuration file './storage/proxy/etc/nginx-conf/nginx.conf' not found!"
+	echo "This file should be generated by the infrastructure configuration system."
+	echo "Expected location: $(pwd)/storage/proxy/etc/nginx-conf/nginx.conf"
+	echo ""
+	echo "To generate the configuration file, run:"
+	echo "  make configure-local    # For local development"
+	echo "  make configure-production # For production deployment"
+	exit 1
 fi
 
+echo "Found nginx configuration: ./storage/proxy/etc/nginx-conf/nginx.conf"
+
 ## Certbot
 
 mkdir -p ./storage/certbot/etc
@@ -33,16 +54,37 @@ fi
 
 mkdir -p ./storage/tracker/etc
 
-if ! [ -f "./storage/tracker/etc/tracker.prod.container.sqlite3.toml" ]; then
-	echo "Creating tracker configuration: './storage/tracker/etc/tracker.toml'"
-	cp ./share/container/default/config/tracker.prod.container.sqlite3.toml ./storage/tracker/etc/tracker.toml
+# Verify tracker configuration exists (should be provided by infrastructure)
+if ! [ -f "./storage/tracker/etc/tracker.toml" ]; then
+	echo "ERROR: Tracker configuration file './storage/tracker/etc/tracker.toml' not found!"
+	echo "This file should be generated by the infrastructure configuration system."
+	echo "Expected location: $(pwd)/storage/tracker/etc/tracker.toml"
+	echo ""
+	echo "To generate the configuration file, run:"
+	echo "  make configure-local    # For local development"
+	echo "  make configure-production # For production deployment"
+	exit 1
 fi
 
+echo "Found tracker configuration: ./storage/tracker/etc/tracker.toml"
+
 ## Prometheus
 
 mkdir -p ./storage/prometheus/etc
 
+# Verify prometheus configuration exists (should be provided by infrastructure)
 if ! [ -f "./storage/prometheus/etc/prometheus.yml" ]; then
-	echo "Creating prometheus config file: './storage/prometheus/etc/prometheus.yml'"
-	cp ./share/container/default/config/prometheus.yml ./storage/prometheus/etc/prometheus.yml
+	echo "ERROR: Prometheus configuration file './storage/prometheus/etc/prometheus.yml' not found!"
+	echo "This file should be generated by the infrastructure configuration system."
+	echo "Expected location: $(pwd)/storage/prometheus/etc/prometheus.yml"
+	echo ""
+	echo "To generate the configuration file, run:"
+	echo "  make configure-local    # For local development"
+	echo "  make configure-production # For production deployment"
+	exit 1
 fi
+
+echo "Found prometheus configuration: ./storage/prometheus/etc/prometheus.yml"
+
+echo "Installation completed successfully!"
+echo "All required directories created and configuration files verified."