diff --git a/docs/README.md b/docs/README.md index 8e7f9994..2b3e5d87 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,26 +1,37 @@ -# Nix Store, storage space and filesystem considerations +# Architecture -The tracker requires significant storage space to run properly. In particular, -there are two different areas you'll need storage for, the database and the Nix -store. +The Nixpkgs vulnerability tracker consists of +- a server process for handling HTTP requests +- worker processes for recurring data ingestion -## Database +![Service architecture diagram](./architecture.mermaid) -In order to store three diffent Nixpkgs releases, you'll need around 80GB of -space for the Postgresql database. This is the case right now for storing -24.05, 24.11 and 25.05. +## External services -As the number of CVEs and packages increases, the storage space requirement -will most likely increase as well. +The tracker needs to communicate with third party services, namely: -## Nix Store and filesystem +- The GitHub API for user authentication, team permissions, creating issues, etc. +- GitHub repositories: + - https://github.com/nixos/nixpkgs to pull the latest changes from Nixpkgs + - https://github.com/CVEProject/cvelistV5 to pull CVE data +- https://prometheus.nixos.org/ to get information about the latest channels -Since the tracker builds derivations for all packages, it creates a lot of -small files in the filesystem. You need to make sure you have both enough space -for this and enough inodes on your filesystem. We suggest you [optimise the nix -store](https://wiki.nixos.org/wiki/Storage_optimization) and turn on automatic +## Storage space considerations + +The tracker requires significant storage space to run properly. +In particular, there are two different areas you'll need storage for, the database and the Nix store. + +### Database + +In order to store three diffent Nixpkgs releases, you'll need around 80GB of space for the Postgresql database. + +As the number of CVEs and packages increases, the storage space requirement will most likely increase as well. + +### Nix store and filesystem + +Since the tracker instantiates derivations for all packages, it creates a lot of small files in the filesystem. +You need to make sure you have both enough space for this and enough inodes on your filesystem. +We suggest you [optimise the nix store](https://wiki.nixos.org/wiki/Storage_optimization) and turn on automatic optimisation. -If you're using ext4, read this [Nix issue](https://github.com/NixOS/nix/issues/1522) -as you'll need to enable support for `large_dir`in your filesystem for -optimisation to work. +If you're using ext4, read this [Nix issue](https://github.com/NixOS/nix/issues/1522) as you'll need to enable support for `large_dir`in your filesystem for optimisation to work. diff --git a/docs/architecture.mermaid b/docs/architecture.mermaid new file mode 100644 index 00000000..31d30f75 --- /dev/null +++ b/docs/architecture.mermaid @@ -0,0 +1,67 @@ +graph TB + Users["**👥 Users**"] + + subgraph External["**External Services**"] + GitHub["**GitHub API**"] + GitHubNixos["**GitHub Repository**
*nixos/nixpkgs*"] + GitHubCVEs["**GitHub Repository**
*CVEProject/cvelistV5*"] + NixMonitoring["**NixOS Monitoring**
*Channel Status*"] + end + + subgraph SecurityTracker ["**Security Tracker Host**"] + subgraph Web["**Web**"] + Nginx["**Nginx HTTP**"] + WSGI["**WSGI Django**
*Django Views*"] + end + + subgraph ManageCommands["**Management Commands**"] + FetchAllChannels["**fetch_all_channels**"] + IngestCVEs["**ingest_delta_cve**"] + end + + subgraph Background["**Background Tasks**"] + SystemdTimerChannels["**Systemd Timer Fetch Channels**"] + SystemdTimerCVEs["**Systemd Timer Ingest CVEs**"] + NixEval["**Evaluate Nix**"] + DjangoWorker["**Django worker**"] + end + + subgraph Storage["**Storage**"] + PostgreSQL["**PostgreSQL**
*CVE Records
Channels
Users
Issues*"] + LocalGitCheckout["**Local Git Repo**
*nixpkgs clone /var/lib/web-security-tracker/nixpkgs-repo*"] + NixStore["**Nix store**"] + end + + end + + %% User interactions + Users -->|HTTP Request| Nginx -->|Forward| WSGI -->|Queries| PostgreSQL + WSGI --> GitHub + + %% Timers + SystemdTimerChannels -.->|**Triggers Daily**| FetchAllChannels + FetchAllChannels -->|1 Fetch Channels| NixMonitoring + FetchAllChannels -->|2 Git pull| GitHubNixos + FetchAllChannels -->|3 Update Repo| LocalGitCheckout + FetchAllChannels -->|4 Evaluate Nix| NixEval --> NixStore + + SystemdTimerCVEs -.->|**Triggers Daily**| IngestCVEs + IngestCVEs -->|1 Fetch CVEs| GitHubCVEs + IngestCVEs -->|2 Update Database| PostgreSQL + IngestCVEs -->|3 PgTrigger Suggestions| DjangoWorker + + classDef userClass fill:#e1f5fe,stroke:#01579b,stroke-width:3px,color:#000 + classDef externalClass fill:#f3e5f5,stroke:#4a148c,stroke-width:3px,color:#000 + classDef webClass fill:#e8f5e8,stroke:#2e7d32,stroke-width:3px,color:#000 + classDef commandClass fill:#fff3e0,stroke:#e65100,stroke-width:3px,color:#000 + classDef backgroundClass fill:#fce4ec,stroke:#c2185b,stroke-width:3px,color:#000 + classDef storageClass fill:#f1f8e9,stroke:#33691e,stroke-width:3px,color:#000 + classDef subgraphClass fill:#fafafa,stroke:#424242,stroke-width:3px + + class Users userClass + class GitHub,GitHubNixos,GitHubCVEs,NixMonitoring externalClass + class Nginx,WSGI webClass + class FetchAllChannels,IngestCVEs commandClass + class SystemdTimerChannels,SystemdTimerCVEs,NixEval,DjangoWorker backgroundClass + class PostgreSQL,LocalGitCheckout,NixStore storageClass + class Storage,Background,ManageCommands,Web subgraphClass