From 5afef78f2a68c697523db04a0fe42de18e630ce7 Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Thu, 17 Jul 2025 17:01:29 +0200 Subject: [PATCH 1/2] docs: formatting --- docs/README.md | 27 ++++++++++----------------- 1 file changed, 10 insertions(+), 17 deletions(-) diff --git a/docs/README.md b/docs/README.md index 8e7f9994..9f846d32 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,26 +1,19 @@ -# Nix Store, storage space and filesystem considerations +# Storage space considerations -The tracker requires significant storage space to run properly. In particular, -there are two different areas you'll need storage for, the database and the Nix -store. +The tracker requires significant storage space to run properly. +In particular, there are two different areas you'll need storage for, the database and the Nix store. ## Database -In order to store three diffent Nixpkgs releases, you'll need around 80GB of -space for the Postgresql database. This is the case right now for storing -24.05, 24.11 and 25.05. +In order to store three diffent Nixpkgs releases, you'll need around 80GB of space for the Postgresql database. -As the number of CVEs and packages increases, the storage space requirement -will most likely increase as well. +As the number of CVEs and packages increases, the storage space requirement will most likely increase as well. -## Nix Store and filesystem +## Nix store and filesystem -Since the tracker builds derivations for all packages, it creates a lot of -small files in the filesystem. You need to make sure you have both enough space -for this and enough inodes on your filesystem. We suggest you [optimise the nix -store](https://wiki.nixos.org/wiki/Storage_optimization) and turn on automatic +Since the tracker instantiates derivations for all packages, it creates a lot of small files in the filesystem. +You need to make sure you have both enough space for this and enough inodes on your filesystem. +We suggest you [optimise the nix store](https://wiki.nixos.org/wiki/Storage_optimization) and turn on automatic optimisation. -If you're using ext4, read this [Nix issue](https://github.com/NixOS/nix/issues/1522) -as you'll need to enable support for `large_dir`in your filesystem for -optimisation to work. +If you're using ext4, read this [Nix issue](https://github.com/NixOS/nix/issues/1522) as you'll need to enable support for `large_dir`in your filesystem for optimisation to work. From 31ee16d15da27773b902212c81c98f70f2338be3 Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Thu, 17 Jul 2025 17:01:57 +0200 Subject: [PATCH 2/2] docs: service architecture diagram --- docs/README.md | 24 ++++++++++++-- docs/architecture.mermaid | 67 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 88 insertions(+), 3 deletions(-) create mode 100644 docs/architecture.mermaid diff --git a/docs/README.md b/docs/README.md index 9f846d32..2b3e5d87 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,15 +1,33 @@ -# Storage space considerations +# Architecture + +The Nixpkgs vulnerability tracker consists of +- a server process for handling HTTP requests +- worker processes for recurring data ingestion + +![Service architecture diagram](./architecture.mermaid) + +## External services + +The tracker needs to communicate with third party services, namely: + +- The GitHub API for user authentication, team permissions, creating issues, etc. +- GitHub repositories: + - https://github.com/nixos/nixpkgs to pull the latest changes from Nixpkgs + - https://github.com/CVEProject/cvelistV5 to pull CVE data +- https://prometheus.nixos.org/ to get information about the latest channels + +## Storage space considerations The tracker requires significant storage space to run properly. In particular, there are two different areas you'll need storage for, the database and the Nix store. -## Database +### Database In order to store three diffent Nixpkgs releases, you'll need around 80GB of space for the Postgresql database. As the number of CVEs and packages increases, the storage space requirement will most likely increase as well. -## Nix store and filesystem +### Nix store and filesystem Since the tracker instantiates derivations for all packages, it creates a lot of small files in the filesystem. You need to make sure you have both enough space for this and enough inodes on your filesystem. diff --git a/docs/architecture.mermaid b/docs/architecture.mermaid new file mode 100644 index 00000000..31d30f75 --- /dev/null +++ b/docs/architecture.mermaid @@ -0,0 +1,67 @@ +graph TB + Users["**👥 Users**"] + + subgraph External["**External Services**"] + GitHub["**GitHub API**"] + GitHubNixos["**GitHub Repository**
*nixos/nixpkgs*"] + GitHubCVEs["**GitHub Repository**
*CVEProject/cvelistV5*"] + NixMonitoring["**NixOS Monitoring**
*Channel Status*"] + end + + subgraph SecurityTracker ["**Security Tracker Host**"] + subgraph Web["**Web**"] + Nginx["**Nginx HTTP**"] + WSGI["**WSGI Django**
*Django Views*"] + end + + subgraph ManageCommands["**Management Commands**"] + FetchAllChannels["**fetch_all_channels**"] + IngestCVEs["**ingest_delta_cve**"] + end + + subgraph Background["**Background Tasks**"] + SystemdTimerChannels["**Systemd Timer Fetch Channels**"] + SystemdTimerCVEs["**Systemd Timer Ingest CVEs**"] + NixEval["**Evaluate Nix**"] + DjangoWorker["**Django worker**"] + end + + subgraph Storage["**Storage**"] + PostgreSQL["**PostgreSQL**
*CVE Records
Channels
Users
Issues*"] + LocalGitCheckout["**Local Git Repo**
*nixpkgs clone /var/lib/web-security-tracker/nixpkgs-repo*"] + NixStore["**Nix store**"] + end + + end + + %% User interactions + Users -->|HTTP Request| Nginx -->|Forward| WSGI -->|Queries| PostgreSQL + WSGI --> GitHub + + %% Timers + SystemdTimerChannels -.->|**Triggers Daily**| FetchAllChannels + FetchAllChannels -->|1 Fetch Channels| NixMonitoring + FetchAllChannels -->|2 Git pull| GitHubNixos + FetchAllChannels -->|3 Update Repo| LocalGitCheckout + FetchAllChannels -->|4 Evaluate Nix| NixEval --> NixStore + + SystemdTimerCVEs -.->|**Triggers Daily**| IngestCVEs + IngestCVEs -->|1 Fetch CVEs| GitHubCVEs + IngestCVEs -->|2 Update Database| PostgreSQL + IngestCVEs -->|3 PgTrigger Suggestions| DjangoWorker + + classDef userClass fill:#e1f5fe,stroke:#01579b,stroke-width:3px,color:#000 + classDef externalClass fill:#f3e5f5,stroke:#4a148c,stroke-width:3px,color:#000 + classDef webClass fill:#e8f5e8,stroke:#2e7d32,stroke-width:3px,color:#000 + classDef commandClass fill:#fff3e0,stroke:#e65100,stroke-width:3px,color:#000 + classDef backgroundClass fill:#fce4ec,stroke:#c2185b,stroke-width:3px,color:#000 + classDef storageClass fill:#f1f8e9,stroke:#33691e,stroke-width:3px,color:#000 + classDef subgraphClass fill:#fafafa,stroke:#424242,stroke-width:3px + + class Users userClass + class GitHub,GitHubNixos,GitHubCVEs,NixMonitoring externalClass + class Nginx,WSGI webClass + class FetchAllChannels,IngestCVEs commandClass + class SystemdTimerChannels,SystemdTimerCVEs,NixEval,DjangoWorker backgroundClass + class PostgreSQL,LocalGitCheckout,NixStore storageClass + class Storage,Background,ManageCommands,Web subgraphClass