Project Redesign: From PoC to Production-Ready Deployment System

.github/copilot-instructions.md

@@ -49,23 +49,37 @@
 
 ## 📋 Document Maintenance
 
-**Torrust Tracker Demo** is the complete production deployment configuration for running a live [Torrust Tracker](https://github.com/torrust/torrust-tracker) instance. This repository provides:
+> ⚠️ **REPOSITORY STATUS: This repository is now FROZEN as a historical Proof of Concept (PoC).**
+>
+> - No new features or major refactors will be implemented here.
+> - Active engineering has moved to a **greenfield redesign initiative** documented under
+>   `docs/redesign/` ([Issue #31](https://github.com/torrust/torrust-tracker-demo/issues/31)).
+> - Only documentation, requirements, and architecture specification updates are accepted in
+>   this repo.
+>
+> If you are evaluating how Torrust Tracker _will_ be deployed going forward, start with: `docs/redesign/README.md`.
+
+**Torrust Tracker Demo** is a historical Proof of Concept that demonstrates a complete production deployment configuration for running a live [Torrust Tracker](https://github.com/torrust/torrust-tracker) instance. This repository provides:
 
 - **Production deployment** configurations for Hetzner cloud infrastructure
 - **Local testing environment** using KVM/libvirt virtualization
 - **Infrastructure as Code** approach using OpenTofu/Terraform and cloud-init
 - **Monitoring setup** with Grafana dashboards and Prometheus metrics
 - **Automated deployment** scripts and Docker Compose configurations
 
+This PoC still demonstrates a full twelve-factor style deployment (infrastructure provisioning + application lifecycle) and remains a reference for baseline behaviors. Its documentation is being actively curated to extract reusable requirements for the next-generation implementation.
+
 ### Current Major Initiative
 
-We are migrating the tracker to a new infrastructure on Hetzner, involving:
+**Legacy Context (Superseded)**: We were migrating the tracker to a new infrastructure on Hetzner, involving:
 
 - Running the tracker binary directly on the host for performance
 - Using Docker for supporting services (Nginx, Prometheus, Grafana, MySQL)
 - Migrating the database from SQLite to MySQL
 - Implementing Infrastructure as Code for reproducible deployments
 
+**Current Focus**: Active engineering has moved to a **greenfield redesign initiative** documented under `docs/redesign/` ([Issue #31](https://github.com/torrust/torrust-tracker-demo/issues/31)). This repository is now frozen and serves as a historical reference.
+
 ## 🏗️ Twelve-Factor Architecture
 
 This project implements a complete twelve-factor app architecture with clear separation between infrastructure provisioning and application deployment:
@@ -692,7 +706,9 @@ When providing assistance:
 - Prioritize security and best practices
 - Test infrastructure changes locally before suggesting them
 - Provide clear explanations and documentation
-- Consider the migration to Hetzner infrastructure in suggestions
+- **CRITICAL**: Understand this repository's frozen status - focus on documentation, requirements extraction, and architecture specification only
+- **NEW FEATURES**: Direct users to the redesign initiative in `docs/redesign/` for new functionality discussions
+- **INFRASTRUCTURE CHANGES**: Legacy infrastructure should only be modified for documentation purposes or critical fixes
 - **CRITICAL**: Respect the three-layer testing architecture (see Testing Requirements above)
 
 #### Testing Layer Separation (CRITICAL)

README.md

@@ -1,16 +1,28 @@
 [![Testing](https://github.com/torrust/torrust-tracker-demo/actions/workflows/testing.yml/badge.svg)](https://github.com/torrust/torrust-tracker-demo/actions/workflows/testing.yml)
 
-# Torrust Tracker Demo
+# Torrust Tracker Demo (Frozen Proof of Concept)
 
-This repo contains all the configuration needed to run the live Torrust Tracker demo.
+> ⚠️ REPOSITORY STATUS: **This repository is now FROZEN as a historical Proof of Concept (PoC).**
+>
+> - No new features or major refactors will be implemented here.
+> - Active engineering has moved to a **greenfield redesign initiative** documented under
+>   `docs/redesign/` ([Issue #31](https://github.com/torrust/torrust-tracker-demo/issues/31)).
+> - Only documentation, requirements, and architecture specification updates are accepted in
+>   this repo.
+>
+> If you are evaluating how Torrust Tracker _will_ be deployed going forward, start with: `docs/redesign/README.md`.
+
+This PoC still demonstrates a full twelve-factor style deployment (infrastructure provisioning +
+application lifecycle) and remains a reference for baseline behaviors. Its documentation is
+being actively curated to extract reusable requirements for the next-generation implementation.
 
-It's also used to track issues in production.
+Historic description (legacy context retained below for reference):
+
+This repo contains all the configuration needed to run the live Torrust Tracker demo.
 
-> IMPORTANT: We are in the process of [splitting the Torrust Demo repo into
-> two repos](https://github.com/torrust/torrust-demo/issues/79). This will
-> allow us to deploy both services independently and it would make easier for
-> users who only want to setup the tracker to re-use this setup. The content
-> of this repo may change drastically in the future.
+> (Legacy notice) We were in the process of
+> [splitting the Torrust Demo repo into two repos](https://github.com/torrust/torrust-demo/issues/79).
+> That plan has been superseded by the broader redesign captured in Issue #31.
 
 ## 🏗️ Repository Structure
 

docs/redesign/README.md

@@ -0,0 +1,103 @@
+# Redesign Docs (Freeze Mode)
+
+> **STATUS: DESIGN FREEZE** – Code here stays as-is. We are only improving the redesign
+> docs until the new implementation repo is created.
+
+These docs (Issue **#31**) explain where we are going and why. Keep it light, clear and
+useful for future contributors.
+
+## 🔄 Repository Transition Strategy
+
+This repository serves as the **specification and design phase** for the new production  
+system. Here's the complete transition plan:
+
+### 1. **Current Repository (`torrust-tracker-demo`)**
+
+- **Final Status**: Will be archived as `torrust-tracker-demo-poc`
+- **Purpose**: Historical reference and complete specification for new system
+- **Contains**: Complete documentation through Phase 4 (Goals → Requirements → Analysis → Design → Planning)
+- **Role**: Blueprint and specification source for the new implementation
+
+### 2. **New Repository (`torrust-tracker-installer`)**
+
+- **Purpose**: Production-grade deployment system implementation
+- **Foundation**: Copy of this redesign documentation as starting point
+- **Implementation Phases**:
+  - **Phase 5**: Implementation 🔨
+  - **Phase 6**: Testing & Validation 🧪
+  - **Phase 7**: Migration & Deployment 🚀
+
+### 3. **Documentation Handover**
+
+- **What Transfers**: All `docs/redesign/` content copied to new repository
+- **What Stays**: PoC code and configuration (for historical reference)
+- **Timing**: After Phase 4 (Planning) completion in this repository
+
+This approach separates **specification** (this repo) from **implementation** (new repo),  
+ensuring clean separation of concerns and clear project boundaries.
+
+## What You Can Do Now
+
+| You want to…                                      | Allowed? | How                                             |
+| ------------------------------------------------- | -------- | ----------------------------------------------- |
+| Improve existing redesign docs                    | ✅       | Edit files under `docs/redesign/`               |
+| Add a missing focused requirement (e.g. firewall) | ✅       | New file + link it here + reference #31         |
+| Add an ADR                                        | ✅       | Follow ADR format; keep it short & scoped       |
+| Change PoC code / refactor                        | ❌       | Archived; will rebuild clean later              |
+| Bump dependencies / tooling                       | ❌       | Unless a security issue (then open issue first) |
+| Modify scripts or tests                           | ❌       | Only if a doc would be incorrect otherwise      |
+
+If something outside the list is really needed (security/legal), open an issue and we’ll list it below.
+
+## Current Focus
+
+Closing **Phase 1 (Requirements)**. Last gap just filled:
+
+- Dynamic firewall / network exposure management → [`phase1-requirements/firewall-dynamic-handling.md`](./phase1-requirements/firewall-dynamic-handling.md)
+
+After a quick review we move to Phase 2 (measure current behaviour: performance, state, operational toil).
+
+## Folder Map
+
+| Folder                 | Purpose                                    |
+| ---------------------- | ------------------------------------------ |
+| `phase0-goals/`        | Project goals & scope                      |
+| `phase1-requirements/` | Agreed requirements & technical details    |
+| `phase2-analysis/`     | (Next) What the PoC actually does / limits |
+| `phase3-design/`       | Future architecture sketches & decisions   |
+| `phase4-planning/`     | Milestones & rollout plan                  |
+| `community-input/`     | Collected suggestions & feedback           |
+
+## Simple 8-Phase Flow
+
+**Specification & Design Phases** (in this repository):
+
+0. Goals & scope (what we're trying to achieve)
+1. Requirements (what matters technically)
+2. Analyse current PoC (truth vs assumptions)
+3. Design new solution
+4. Plan build & migration
+
+**Implementation Phases** (in new `torrust-tracker-installer` repository):
+
+1. Implementation (build the new system)
+2. Testing & validation (comprehensive testing)
+3. Migration & deployment (production rollout)
+
+## Next Up (Short List)
+
+| Item                                                | Phase | Status         |
+| --------------------------------------------------- | ----- | -------------- |
+| Runtime state & persistence inventory               | 2     | Planned        |
+| Performance baseline (throughput/latency/resources) | 2     | Planned        |
+| Dynamic firewall ADR (pick final approach)          | 1     | Pending review |
+| Deployment topology options                         | 3     | Drafting       |
+| Build graph / incremental strategy                  | 3     | Backlog        |
+
+## Related
+
+- Master issue: [#31 – Redesign](https://github.com/torrust/torrust-tracker-demo/issues/31)
+
+## Exceptional Changes (none)
+
+_Empty. Add entries here only if an approved out‑of‑scope change happens._

docs/redesign/phase0-goals/project-goals-and-scope.md

@@ -0,0 +1,205 @@
+# Project Goals and Scope
+
+**Category**: Product Vision and Scope  
+**Priority**: Critical  
+**Status**: Draft
+
+## Primary Goal
+
+**Enable system administrators to provision a virtual machine and set up the Torrust tracker in an
+almost fully automated way (90% automation), providing excellent user experience and lowering
+the barrier to tracker adoption.**
+
+### Success Criteria
+
+- 90% automation of the installation process
+- Clear, intuitive user experience for system administrators
+- Significantly reduced time-to-deployment compared to manual installation
+- Comprehensive documentation that guides users through the entire process
+- Minimal technical expertise required beyond basic system administration
+
+## Secondary Goals
+
+### Documentation and Knowledge Transfer
+
+**Comprehensive documentation of tracker installation requirements including:**
+
+- System dependencies and prerequisites
+- Host system configuration best practices
+- Firewall configuration and security requirements
+- Performance tuning recommendations
+- Troubleshooting guides and common issues
+
+### Benefits
+
+- Reduces support burden through self-service documentation
+- Establishes best practices for tracker deployment
+- Enables community contribution to installation knowledge
+- Provides reference for manual installations when automation isn't sufficient
+
+## Long-Term Goals
+
+### Multi-Provider Support
+
+**Provide support for multiple cloud hosting providers to maximize deployment flexibility.**
+
+#### Planned Providers
+
+- Local virtualization (libvirt/KVM) - _Currently implemented_
+- Cloud providers (AWS, DigitalOcean, Hetzner, etc.) - _Future roadmap_
+
+#### Benefits
+
+- User choice and flexibility in hosting platform
+- Reduced vendor lock-in
+- Market expansion to different cloud ecosystems
+- Resilience against provider-specific limitations
+
+## Explicit Out-of-Scope
+
+### Server Maintenance
+
+**Rationale**: This is a one-execution installer focused on initial deployment.
+
+- **Not included**: Post-installation system updates
+- **Not included**: Application updates and patching
+- **Not included**: Ongoing maintenance automation
+- **Alternative**: Users handle maintenance through standard system administration practices
+
+### Dynamic Scaling and High Availability
+
+**Rationale**: The installer is intentionally focused on a single-node deployment
+for two primary reasons:
+
+1. **Application Architecture**: The Torrust tracker application itself does not
+   natively support horizontal scaling. Peer data is managed in memory on a
+   single instance, meaning that true high availability or load balancing would
+   require significant changes to the core tracker application, which is beyond
+   the scope of this installer project.
+2. **Target Audience**: The primary users are often hobbyists or small groups
+   who require a simple, cost-effective, single-server deployment. The current
+   architecture meets this need directly.
+
+- **Not included**: Auto-scaling based on load.
+- **Not included**: Multi-instance load balancing or high-availability clusters.
+- **Not included**: Automatic migration to larger servers.
+- **Alternative**: Users can manually migrate to a more powerful server by
+  provisioning new infrastructure and transferring their data.
+
+### Migration Between Providers
+
+**Rationale**: Complex cross-provider migration is beyond project scope.
+
+- **Not included**: Automated provider-to-provider migration
+- **Not included**: Data migration tooling
+- **Not included**: Cross-provider compatibility layers
+- **Alternative**: Fresh deployment on new provider with manual data migration
+
+### 100% Automation
+
+**Rationale**: Perfect automation has diminishing returns for a typically one-time installation.
+
+- **Acceptable**: 10% manual steps for complex or rarely-automated tasks
+- **Acceptable**: Manual verification steps for security-critical operations
+- **Acceptable**: Provider-specific manual configuration where APIs are insufficient
+- **Focus**: Automate the 90% that provides the most value
+
+### Provider Account Resource Isolation
+
+**Rationale**: Provider-level resource isolation requires complex provider-specific
+implementation that varies significantly across cloud providers.
+
+### Multi-User Deployment Management
+
+**Rationale**: The project is designed for a single system administrator to perform a one-time
+deployment. It is not intended to be a multi-user platform for managing different
+environments.
+
+- **Not included**: Remote state management for team collaboration (e.g., Terraform Cloud, S3 backend)
+- **Not included**: Role-based access control for infrastructure changes
+- **Not included**: Environment management for multiple users
+- **Alternative**: The system uses local state files, which is sufficient for the
+  single-administrator use case. Disaster recovery relies on data and configuration backups,
+  not on collaborative state management.
+
+### Generic Infrastructure Abstraction Layer
+
+**Rationale**: Building a custom abstraction layer to normalize infrastructure resources across
+different cloud providers (e.g., creating a generic "server" or "network" concept) is a
+significant engineering effort that replicates the core functionality of tools like OpenTofu
+and Terraform. The project's goal is to leverage these existing IaC tools, not to reinvent
+them.
+
+- **Not included**: A custom, intermediate API or schema for defining infrastructure.
+- **Alternative**: Directly use provider-specific configurations within OpenTofu, mapping
+  project needs to the native capabilities of each provider. This approach is more maintainable
+  and aligns with industry best practices.
+
+- **Not included**: Resource name prefixes for environment isolation
+- **Not included**: Private network creation for environment separation
+- **Not included**: Provider-specific isolation mechanisms (VPCs, resource groups, etc.)
+- **Not included**: Automatic project/account boundary management
+
+**Implication**: Multiple environments deployed to the same provider account will
+create independent resources (VMs, storage, networking) but these resources remain
+visible and potentially accessible to each other within the provider account scope.
+
+**Provider-Specific Workarounds**: Some providers offer account-level isolation:
+
+- **Hetzner Cloud**: Use separate projects with project-specific API tokens for true isolation
+- **AWS**: Use separate accounts or strict IAM policies per environment
+- **Application Perspective**: The installer treats each provider profile (token/credentials)
+  as a completely isolated infrastructure boundary, regardless of actual provider-level separation
+
+**Alternative**: Manual provider account management and project separation by users who
+require strict environment isolation.
+
+## Target Audience
+
+### Primary Users
+
+- **System Administrators**: Setting up tracker infrastructure
+- **DevOps Engineers**: Integrating tracker deployment into existing workflows
+- **Self-hosters**: Individuals running personal tracker instances
+
+### User Characteristics
+
+- Basic understanding of Linux system administration
+- Familiarity with command-line interfaces
+- Understanding of networking concepts (DNS, firewalls, etc.)
+- May or may not have cloud provider experience
+
+## Value Proposition
+
+### For Users
+
+- **Reduced Complexity**: Streamlined installation process
+- **Time Savings**: Hours reduced to minutes for deployment
+- **Reliability**: Tested, repeatable deployment process
+- **Flexibility**: Choice of hosting providers and configurations
+
+### For Torrust Ecosystem
+
+- **Adoption**: Lower barriers increase user base
+- **Quality**: Standardized deployments reduce support issues
+- **Community**: Enables focus on tracker features rather than deployment
+
+## Measurement Criteria
+
+### Quantitative Metrics
+
+- **Deployment Time**: From start to working tracker (target: < 30 minutes)
+- **Automation Percentage**: Automated steps vs total steps (target: 90%)
+- **Success Rate**: Successful deployments vs attempted deployments
+- **Documentation Coverage**: Percentage of installation scenarios documented
+
+### Qualitative Metrics
+
+- **User Feedback**: Ease of use and clarity of process
+- **Community Adoption**: Usage in community deployments
+- **Support Reduction**: Fewer installation-related support requests
+
+---
+
+**Note**: This scope definition emerged from lessons learned during the proof of concept phase
+and community feedback about deployment complexity.