This repository is designed to generate Software Bill of Materials (SBOMs) using a comprehensive benchmark across a wide variety of tools on defined targets across multiple programming languages. The goal is to provide a consistent and standardized method for evaluating and comparing the effectiveness and accuracy of various SBOM generation tools, helping users to identify the best tool for their specific needs.
The list of tools used is pulled from our SBOM resources page that includes a comprehensive list of SBOM tools.
- Multi-Tool Support: Run benchmarks across a diverse set of SBOM generation tools including Trivy, Syft, and sbomify.
- Cross-Language Compatibility: Supports multiple programming languages (Python, JavaScript, Java, Go, Rust) and container images.
- Automated Workflow: Easily set up and execute benchmarks with minimal manual intervention.
- Detailed Reports: Generate detailed comparisons and summaries of the SBOMs produced by different tools, highlighting strengths and weaknesses.
- Quality Scoring: Each SBOM is scored using sbomqs to measure SBOM quality.
- Extensibility: Add support for new tools or languages with minimal configuration changes.
| Tool | Description |
|---|---|
| Trivy | Comprehensive security scanner with SBOM generation |
| Syft | CLI tool and library for generating SBOMs |
| sbomify | SBOM generation with enrichment from package registries |
| cyclonedx-python | Native Python SBOM generator (Python benchmarks only) |
| sbom4python | Python-specific SBOM generator (Python benchmarks only) |
| Target | Language/Type | Project | Description |
|---|---|---|---|
| Python | Python | Django | Python web framework dependencies |
| JavaScript | JavaScript/TypeScript | workers-sdk | Cloudflare's Wrangler CLI monorepo |
| Java | Java/Maven | Keycloak | Enterprise IAM with complex Maven dependencies |
| Go | Go | OSV Scanner | Go modules-based security tool |
| Rust | Rust | quiche | Cloudflare's QUIC/HTTP3 implementation |
| Docker | Container | nginx + vim | Container image with added packages |
Each benchmark runs automatically on push to master and produces:
- SBOMs in both CycloneDX and SPDX formats (where supported)
- Quality scores from sbomqs
- Comparison tables in the GitHub Actions summary
Click on any badge above to see the latest benchmark results.
- Python Benchmark - Django requirements.txt
- JavaScript Benchmark - Cloudflare workers-sdk (pnpm-lock.yaml)
- Java Benchmark - Keycloak (Maven/pom.xml)
- Go Benchmark - OSV Scanner (go.mod)
- Rust Benchmark - Cloudflare quiche (Cargo.lock)
- Docker Benchmark - nginx container with vim installed
Current tool versions used in benchmarks:
| Tool | Version |
|---|---|
| Trivy | 0.68.2 |
| Syft | 1.39.0 |
| sbomqs | 2.0.2 |
| cyclonedx-bom | 7.2.1 |
| sbom4python | 0.12.4 |