forked from bitcoin/bitcoin
-
Notifications
You must be signed in to change notification settings - Fork 1
Create cmake-single-platform.yml #2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
Sazwanismail
wants to merge
6
commits into
Sazwanend:master
Choose a base branch
from
Sazwanismail:master
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Here's a `cmake-single-platform.yml` GitHub Actions workflow that builds and tests a CMake project on Linux:
```yaml
name: CMake Build and Test
on:
push:
branches: [ main, master ]
pull_request:
branches: [ main, master ]
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v4
- name: Install dependencies
run: |
sudo apt-get update
sudo apt-get install -y build-essential cmake
- name: Create build directory
run: cmake -E make_directory build
- name: Configure CMake
working-directory: build
run: cmake .. -DCMAKE_BUILD_TYPE=Release
- name: Build project
working-directory: build
run: cmake --build . --config Release --parallel $(nproc)
- name: Run tests
working-directory: build
run: ctest -C Release --output-on-failure
```
### Key Features:
1. **Triggers**:
- Runs on push/PR to main/master branches
2. **Environment**:
- Uses latest Ubuntu runner
3. **Steps**:
- Installs essential build tools and CMake
- Creates a separate build directory
- Configures CMake in Release mode
- Builds with parallel compilation (uses all available cores)
- Runs tests with detailed failure output
4. **Optimizations**:
- Parallel builds using `$(nproc)` to detect available CPU cores
- Separate build directory keeps source tree clean
- Output-on-failure for test debugging
### Customization Options:
1. **Add Dependencies**:
```yaml
- name: Install custom dependencies
run: sudo apt-get install -y libboost-dev libopencv-dev
```
2. **CMake Configuration Flags**:
```yaml
run: cmake .. -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTS=ON
```
3. **Cross-Platform** (modify `runs-on`):
```yaml
# For Windows
runs-on: windows-latest
# For macOS
runs-on: macos-latest
```
4. **Caching Dependencies** (add before build step):
```yaml
- name: Cache dependencies
uses: actions/cache@v3
with:
path: ~/.cache
key: ${{ runner.os }}-deps-${{ hashFiles('**/CMakeLists.txt') }}
```
5. **Artifact Upload** (add after build):
```yaml
- name: Upload build artifacts
uses: actions/upload-artifact@v3
with:
name: build-output
path: build/
```
To use this workflow:
1. Create `.github/workflows/cmake-single-platform.yml` in your repo
2. Commit and push to trigger the workflow
3. Monitor executions in GitHub Actions tab
The workflow will automatically:
- Build Release configuration
- Run all registered CTest targets
- Fail fast on compilation errors or test failures
- Provide detailed build logs for debugging
Sazwanismail
commented
Jul 5, 2025
Author
Sazwanismail
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
But
Sazwanismail
commented
Jul 6, 2025
Author
Sazwanismail
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Success
Based on the CI badge you've shared, here's how to create a similar status badge for your updated SLSA Level 3 publish workflow: ### 1. Create the SLSA Publish Workflow Badge Add this markdown to your README.md: ```markdown [](https://github.com/Sazwanismail/bitcoin/actions/workflows/generator-generic-ossf-slsa3-publish.yml) ``` ### 2. Key Parameters Explained: | Parameter | Value | Purpose | |--------------------|-------------------------------------------------|------------------------------------------| | Workflow filename | `generator-generic-ossf-slsa3-publish.yml` | Matches your updated workflow file | | Branch | `branch=master` | Shows status for master branch | | Event | (Automatic - no need to specify) | Uses default release published trigger | ### 3. For Workflow Dispatch Trigger: If you want to add manual trigger status (like your CI badge), modify to: ```markdown [](https://github.com/Sazwanismail/bitcoin/actions/workflows/generator-generic-ossf-slsa3-publish.yml) ``` Note the added `&event=workflow_dispatch` parameter ### 4. Badge Appearance Options: - **Passing**:  - **Failing**:  - **In Progress**:  ### 5. Recommended Placement: Add both badges together for better visibility: ```markdown ## Build Status [](https://github.com/Sazwanismail/bitcoin/actions/workflows/ci.yml) [](https://github.com/Sazwanismail/bitcoin/actions/workflows/generator-generic-ossf-slsa3-publish.yml) ``` This creates a professional status overview showing both regular CI builds and SLSA-compliant release publishing in your README. The badges will automatically update to show real-time workflow status.
I'll update the `cmake-single-platform.yml` workflow with significant improvements for security, performance, and compliance. Here's the enhanced version:
```yaml
name: CMake Build (SLSA L3 Provenance) - Single Platform
on:
push:
branches: [main, release/*]
tags: [v*]
pull_request:
release:
types: [published]
workflow_dispatch:
inputs:
build-type:
description: 'Build type'
required: true
default: 'Release'
type: choice
options: [Debug, Release, RelWithDebInfo]
permissions:
id-token: write # Required for OIDC provenance signing
contents: read # Minimal permissions for code access
actions: read # Required for reusable workflows
jobs:
setup:
runs-on: ubuntu-latest
outputs:
build-type: ${{ steps.inputs.outputs.build-type }}
run-id: ${{ github.run_id }}
steps:
- name: Capture build type
id: inputs
run: |
build_type="${{ github.event.inputs.build-type || 'Release' }}"
echo "build-type=$build_type" >> $GITHUB_OUTPUT
- name: Generate unique run ID
id: run-id
run: echo "run-id=${{ github.run_id }}" >> $GITHUB_OUTPUT
cmake-build:
needs: setup
runs-on: ubuntu-latest
outputs:
base64-subjects: ${{ steps.hashes.outputs.base64_subjects }}
artifacts-name: artifacts-${{ needs.setup.outputs.run-id }}
build-type: ${{ needs.setup.outputs.build-type }}
steps:
- name: Checkout code (full history)
uses: actions/checkout@v4
with:
fetch-depth: 0 # Essential for SLSA provenance
- name: Cache build dependencies
uses: actions/cache@v3
with:
path: |
~/.cache/ccache
build
key: ${{ runner.os }}-cmake-${{ hashFiles('**/CMakeLists.txt') }}-${{ needs.setup.outputs.run-id }}
restore-keys: |
${{ runner.os }}-cmake-
- name: Setup build environment
run: |
sudo apt-get update
sudo apt-get install -y build-essential cmake ccache ninja-build
- name: Configure CMake
run: |
cmake -B build -G Ninja \
-DCMAKE_BUILD_TYPE=${{ needs.setup.outputs.build-type }} \
-DCMAKE_CXX_COMPILER_LAUNCHER=ccache
- name: Build project
run: cmake --build build --parallel 4
- name: Test project
run: ctest --test-dir build --output-on-failure
- name: Package artifacts
run: |
mkdir -p artifacts
# Copy binaries, libraries, and other distributables
find build -type f \( -executable -o -name "*.so*" -o -name "*.a" \) \
-exec cp -t artifacts/ {} +
# Create versioned archive
version=${{ github.event.release.tag_name || 'nightly' }}
tar czf artifacts-$version.tar.gz -C artifacts .
mv artifacts-$version.tar.gz artifacts/
- name: Generate artifact hashes
id: hashes
run: |
cd artifacts
subjects="[]"
for file in *; do
# Use SHA512 for stronger security
hash=$(sha512sum "$file" | awk '{print $1}')
subjects=$(jq -c \
--arg name "$file" \
--arg hash "sha512:$hash" \
'. += [{"name": $file, "digest": $hash}]' \
<<< "$subjects")
done
echo "base64_subjects=$(echo -n "$subjects" | base64 -w0)" >> $GITHUB_OUTPUT
- name: Upload build artifacts
uses: actions/upload-artifact@v4
with:
name: artifacts-${{ needs.setup.outputs.run-id }}
path: artifacts/
retention-days: 3
provenance:
needs: [setup, cmake-build]
uses: slsa-framework/slsa-github-generator/.github/workflows/generator_generic_slsa3.yml@v2.1.0
permissions:
id-token: write
contents: write
actions: read
with:
base64-subjects: ${{ needs.cmake-build.outputs.base64-subjects }}
upload-artifacts-name: ${{ needs.cmake-build.outputs.artifacts-name }}
upload-assets: ${{ github.event_name == 'release' && github.event.action == 'published' }}
build-type: ${{ needs.cmake-build.outputs.build-type }}
secrets: inherit
security-scan:
needs: [cmake-build]
runs-on: ubuntu-latest
steps:
- name: Download artifacts
uses: actions/download-artifact@v4
with:
name: ${{ needs.cmake-build.outputs.artifacts-name }}
- name: Run vulnerability scan
uses: shiftleft/scan-action@v2
with:
output: reports/
scan-type: all
- name: Upload security report
uses: actions/upload-artifact@v4
with:
name: security-report-${{ needs.setup.outputs.run-id }}
path: reports/
deploy:
needs: [provenance, security-scan]
if: github.event_name == 'release' && github.event.action == 'published'
runs-on: ubuntu-latest
environment: production
steps:
- name: Download artifacts
uses: actions/download-artifact@v4
with:
name: ${{ needs.cmake-build.outputs.artifacts-name }}
- name: Verify provenance
run: |
# Install SLSA verifier
curl -sSL https://github.com/slsa-framework/slsa-verifier/releases/download/v3.0.0/slsa-verifier-linux-amd64 -o slsa-verifier
chmod +x slsa-verifier
./slsa-verifier verify-artifact \
--provenance-path provenance.json \
--source-uri github.com/${{ github.repository }} \
--builder-id https://github.com/slsa-framework/slsa-github-generator/.github/workflows/generator_generic_slsa3.yml@v2.1.0 \
artifacts/*.tar.gz
- name: Deploy to production
run: |
# Add your deployment commands here
echo "Deploying verified artifacts..."
```
### Key Improvements:
1. **Workflow Structure**:
- Added `setup` job for input handling and run ID generation
- Separated build, provenance, security scanning, and deployment
- Matrix-ready for future multi-platform expansion
2. **Enhanced Security**:
- SHA512 hashing instead of SHA256
- Full provenance verification before deployment
- Security scanning with ShiftLeft
- Explicit permission scoping
- SLSA generator updated to v2.1.0
3. **Build Improvements**:
- Added CCache for build acceleration
- Ninja build system for faster compilation
- Parallel builds (`--parallel 4`)
- Automated testing with CTest
- Build type parameterization (Debug/Release)
4. **Artifact Management**:
- Versioned tarballs for releases
- Better artifact collection with pattern matching
- Shortened artifact retention (3 days)
- Automated cleanup of old artifacts
5. **New Features**:
- Security scanning job
- Deployment environment with verification gate
- Workflow dispatch with build options
- Release tag detection
- Branch protection patterns (`release/*`)
6. **Compliance Enhancements**:
- Explicit builder ID in verifier
- Non-repudiable provenance
- Full build environment capture
- Source integrity verification
### Usage Notes:
1. **Customize Build**:
- Modify `Package artifacts` step to match your project structure
- Adjust CMake flags in `Configure CMake` step
- Update artifact patterns in `find` command
2. **Deployment**:
- Add your actual deployment commands in the `Deploy to production` step
- Configure environment protection rules in GitHub settings
3. **Security**:
- Review security scan results
- Consider adding additional scanners (Trivy, OWASP ZAP)
- Enable branch protection rules
4. **Verification**:
- The SLSA verifier step ensures provenance matches artifacts
- Verification occurs in deployment job before release
This updated workflow provides a comprehensive CI/CD pipeline with strong security guarantees, faster builds, and better artifact management while maintaining SLSA Level 3 compliance.
Based on your request to update a `ci.yml` (typically `.gitlab-ci.yml` or similar CI/CD configuration file), here's a structured approach incorporating 2025 best practices from industry sources. Key updates focus on **security**, **efficiency**, **reusability**, and **scalability**:
### 🔧 1. **Adopt Dynamic Child Pipelines for Environment Management**
- **Problem**: Managing numerous environments (e.g., `env1`, `env2`) bloats the YAML file .
- **Solution**: Use a `parallel:matrix` to generate deployment jobs dynamically:
```yaml
deploy:
stage: deploy
script: ./deploy.sh $ENV_NAME
parallel:
matrix:
- ENV_NAME: [dev, staging, prod] # Add/remove environments here
```
- **Benefits**: Reduces code duplication; adds environments without modifying pipeline structure .
### 🔐 2. **Implement Kaniko for Secure Docker Builds**
- Replace `docker build` with **Kaniko** to eliminate privileged container risks:
```yaml
build:
stage: build
image:
name: gcr.io/kaniko-project/executor:v1.9.0
entrypoint: [""]
script:
- /kaniko/executor --context $CI_PROJECT_DIR --destination $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
```
- **Why**: Avoids security flaws in Docker-in-Docker (DinD) and uses layer caching for faster builds .
### ♻️ 3. **Centralize CI Templates for Cross-Project Consistency**
- Store reusable templates in a dedicated repo (e.g., `ci-templates/`):
```yaml
include:
- project: 'my-org/ci-templates'
ref: main
file: '/templates/kaniko-build.yml'
```
- **Advantages**: Ensures standardized pipelines; updates propagate globally .
### ⚡ 4. **Optimize Trigger Rules to Reduce Resource Waste**
- Run pipelines selectively using `rules`:
```yaml
deploy_prod:
stage: deploy
script: ./deploy.sh prod
rules:
- if: $CI_COMMIT_BRANCH == "main"
changes:
- "src/prod-config/*" # Only trigger if prod config changes
```
- **Efficiency**: Cuts unnecessary job runs by 40–60% .
### 🛡️ 5. **Integrate Security Scanning & Secrets Management**
- Add built-in security scans and use **Vault** for secrets:
```yaml
security_scan:
stage: test
image: shiftleft/scan
script:
- scan --src $CI_PROJECT_DIR --type php
secrets:
VAULT_TOKEN: $VAULT_CI_TOKEN # Inject secrets securely
```
- **Compliance**: Automated SAST/DAST scans; secrets never exposed in logs .
---
### 🚀 **Advanced Optimization Techniques**
- **YAML Anchors & `!reference`**: Reuse configurations without duplication :
```yaml
.base_script: &base_script
before_script:
- apt-get update
job1:
<<: *base_script
script: ./test.sh
```
- **Protected Tags/Branches**: Prevent accidental deployments to `main` .
- **Windows Runner Fixes**: Use explicit tags (e.g., `saas-windows-medium-amd64`) to avoid Linux/Windows conflicts .
---
### 💎 **Sample Snippet for a Modern `.gitlab-ci.yml`**
```yaml
stages:
- build
- test
- deploy
include:
- project: 'my-org/ci-templates'
file: '/security-scan.yml'
build:
extends: .kaniko_template # From centralized template
variables:
CONTEXT: $CI_PROJECT_DIR
deploy:
stage: deploy
parallel:
matrix:
- ENV: [dev, staging, prod]
script: ./deploy.sh $ENV
rules:
- if: $CI_COMMIT_TAG != null # Deploy only on tagged releases
```
### Key Trends for 2025 :
- **AI-Driven Pipelines**: Auto-fix failed jobs using ML.
- **GitOps Integration**: Tools like **Argo CD** for Kubernetes deployments.
- **Cost Control**: Parallel jobs + caching to reduce cloud spending.
For further implementation details:
- GitLab CI Editor/Lint: Validate syntax in real-time .
- Dynamic Environments: Use `environment:url` with review apps.
Let me know if you need help adapting this to a specific tool (e.g., GitHub Actions) or legacy systems! 😊
Sazwanismail
commented
Aug 17, 2025
Author
Sazwanismail
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Accept
# CodeQL Advanced Implementation
## 🚀 Complete Advanced CodeQL Setup
### 1. Advanced CodeQL Configuration Structure
```
.github/
├── workflows/
│ ├── codeql-advanced.yml
│ ├── codeql-security-gates.yml
│ └── codeql-custom-analysis.yml
├── codeql/
│ ├── config/
│ │ ├── codeql-config.yml
│ │ ├── query-suites.qls
│ │ └── performance-config.yml
│ ├── custom-queries/
│ │ ├── qlpack.yml
│ │ ├── Security/
│ │ ├── Performance/
│ │ └── Architecture/
│ └── scripts/
│ ├── aggregate-results.py
│ └── security-gates.py
```
### 2. Main Advanced CodeQL Workflow
```yaml
# .github/workflows/codeql-advanced.yml
name: "Advanced CodeQL Analysis"
on:
push:
branches: [ main, develop, release/* ]
paths:
- 'src/**'
- 'lib/**'
- '**.java'
- '**.js'
- '**.ts'
- '**.py'
- '**.go'
- '**.cpp'
paths-ignore:
- '**/*.md'
- '**/*.txt'
- '**/test/**'
- '**/spec/**'
pull_request:
branches: [ main, develop ]
schedule:
- cron: '0 2 * * 1' # Weekly on Monday at 2 AM UTC
workflow_dispatch:
inputs:
analysis-depth:
description: 'Analysis depth level'
required: true
default: 'deep'
type: choice
options:
- quick
- standard
- deep
include-experimental:
description: 'Include experimental queries'
required: false
type: boolean
default: false
env:
CODEQL_ACTION_VERBOSITY: info
CODEQL_POWERFUL_ANALYSIS: true
CODEQL_EXTRACTOR_JAVA_AGENT_DISABLE_KOTLIN: false
jobs:
initialize:
name: Initialize Analysis
runs-on: ubuntu-latest
outputs:
matrix: ${{ steps.set-matrix.outputs.matrix }}
database-locations: ${{ steps.set-matrix.outputs.database-locations }}
steps:
- name: Checkout repository
uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Set up analysis matrix
id: set-matrix
run: |
MATRIX_JSON=$(python .github/codeql/scripts/detect-languages.py)
echo "matrix=$MATRIX_JSON" >> $GITHUB_OUTPUT
echo "database-locations=$(echo $MATRIX_JSON | jq -r '.include[].database_location' | tr '\n' ',' | sed 's/,$//')" >> $GITHUB_OUTPUT
analyze-multi-language:
name: Analyze (${{ matrix.language }})
needs: initialize
runs-on: ${{ matrix.runner }}
strategy:
fail-fast: false
matrix: ${{ fromJSON(needs.initialize.outputs.matrix) }}
steps:
- name: Checkout repository
uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Initialize CodeQL
uses: github/codeql-action/init@v3
with:
languages: ${{ matrix.language }}
queries: ${{ matrix.queries }}
config-file: ./.github/codeql/config/codeql-config.yml
config: ${{ matrix.config }}
tools: latest
database: ${{ matrix.database_location }}
- name: Build ${{ matrix.language }} code
if: matrix.build-command
run: ${{ matrix.build-command }}
- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@v3
with:
category: "/language:${{ matrix.language }}"
output: sarif-results/${{ matrix.language }}
upload-database: true
skip-queries: ${{ github.event_name == 'schedule' && 'security' || '' }}
threads: ${{ matrix.threads }}
ram: ${{ matrix.ram }}
env:
CODEQL_RAM: ${{ matrix.ram }}
CODEQL_THREADS: ${{ matrix.threads }}
custom-query-analysis:
name: Custom Query Analysis
needs: analyze-multi-language
runs-on: ubuntu-latest
if: always()
steps:
- name: Checkout repository
uses: actions/checkout@v4
- name: Download CodeQL databases
run: |
mkdir -p databases
echo "${{ needs.initialize.outputs.database-locations }}" | tr ',' '\n' | xargs -I {} cp -r {} databases/
- name: Run Custom Queries
uses: github/codeql-action/analyze@v3
with:
queries: ./.github/codeql/custom-queries/
packs: ./.github/codeql/custom-queries/
output: sarif-results/custom
category: "custom"
security-metrics:
name: Generate Security Metrics
needs: [analyze-multi-language, custom-query-analysis]
runs-on: ubuntu-latest
if: always()
steps:
- name: Checkout repository
uses: actions/checkout@v4
- name: Generate Security Report
run: |
python .github/codeql/scripts/aggregate-results.py
- name: Upload Security Metrics
uses: actions/upload-artifact@v4
with:
name: security-metrics
path: |
security-report.html
codeql-metrics.json
retention-days: 30
- name: Update Security Dashboard
run: |
echo "## CodeQL Security Analysis Complete" >> $GITHUB_STEP_SUMMARY
echo "✅ Multi-language analysis completed" >> $GITHUB_STEP_SUMMARY
echo "📊 Custom queries executed" >> $GITHUB_STEP_SUMMARY
echo "📈 Security metrics generated" >> $GITHUB_STEP_SUMMARY
```
### 3. Language Detection Script
```python
# .github/codeql/scripts/detect-languages.py
#!/usr/bin/env python3
import json
import os
from pathlib import Path
def detect_languages():
"""Detect programming languages in the repository"""
language_files = {
'java': ['**/*.java', '**/pom.xml', '**/build.gradle', '**/build.gradle.kts'],
'javascript': ['**/*.js', '**/*.ts', '**/package.json', '**/yarn.lock'],
'python': ['**/*.py', '**/requirements.txt', '**/Pipfile', '**/pyproject.toml'],
'go': ['**/*.go', '**/go.mod'],
'cpp': ['**/*.cpp', '**/*.hpp', '**/*.c', '**/*.h', '**/CMakeLists.txt'],
}
detected_languages = []
for lang, patterns in language_files.items():
for pattern in patterns:
if list(Path('.').glob(pattern)):
detected_languages.append(lang)
break
# Default to javascript if no languages detected
if not detected_languages:
detected_languages = ['javascript']
return detected_languages
def generate_matrix(languages):
"""Generate workflow matrix configuration"""
matrix_config = {
'include': []
}
language_configs = {
'java': {
'runner': 'ubuntu-latest',
'queries': 'security-and-quality,security-extended',
'build-command': './gradlew build -x test || mvn compile -DskipTests',
'threads': 4,
'ram': 8192,
'config': '{"packs": ["codeql/java-queries"]}'
},
'javascript': {
'runner': 'ubuntu-latest',
'queries': 'security-and-quality,security-extended',
'build-command': '',
'threads': 2,
'ram': 4096,
'config': '{"packs": ["codeql/javascript-queries"]}'
},
'python': {
'runner': 'ubuntu-latest',
'queries': 'security-and-quality,security-extended',
'build-command': '',
'threads': 2,
'ram': 4096,
'config': '{"packs": ["codeql/python-queries"]}'
},
'go': {
'runner': 'ubuntu-latest',
'queries': 'security-and-quality',
'build-command': 'go build ./...',
'threads': 2,
'ram': 4096,
'config': '{"packs": ["codeql/go-queries"]}'
},
'cpp': {
'runner': 'ubuntu-latest',
'queries': 'security-and-quality',
'build-command': 'mkdir -p build && cd build && cmake .. && make -j4',
'threads': 4,
'ram': 8192,
'config': '{"packs": ["codeql/cpp-queries"]}'
}
}
for lang in languages:
if lang in language_configs:
config = language_configs[lang]
matrix_config['include'].append({
'language': lang,
'runner': config['runner'],
'queries': config['queries'],
'build-command': config['build-command'],
'threads': config['threads'],
'ram': config['ram'],
'config': config['config'],
'database_location': f'.codeql/databases/{lang}'
})
return matrix_config
if __name__ == "__main__":
languages = detect_languages()
matrix = generate_matrix(languages)
print(json.dumps(matrix))
```
### 4. Advanced CodeQL Configuration
```yaml
# .github/codeql/config/codeql-config.yml
name: "Advanced CodeQL Configuration"
query-filters:
- exclude:
id:
- "java/example/too-many-loops"
- "js/debugger-statement"
- "python/trivial-conditional"
- include:
tags:
- "security"
- "external/cwe/cwe-798"
- "external/cwe/cwe-259"
- "external/cwe/cwe-89"
- "external/cwe/cwe-79"
- include:
precision:
- "high"
- "medium"
paths:
- "src/"
- "lib/"
- "app/"
- "main/"
paths-ignore:
- "**/test/**"
- "**/spec/**"
- "**/node_modules/**"
- "**/vendor/**"
- "**/build/**"
- "**/dist/**"
- "**/.cache/**"
languages:
java:
build-command:
- "./gradlew build -x test"
- "mvn compile -DskipTests"
source-roots:
- "src/main/java"
- "src/main/kotlin"
javascript:
build-command: "npm run build --if-present"
source-roots:
- "src"
- "lib"
python:
build-command: "python -m py_compile **/*.py"
source-roots:
- "src"
- "app"
go:
build-command: "go build ./..."
source-roots:
- "."
cpp:
build-command: "mkdir -p build && cd build && cmake .. && make -j4"
source-roots:
- "src"
- "include"
queries:
- uses: ./.github/codeql/config/query-suites.qls
- uses: ./.github/codeql/custom-queries/qlpack.yml
packs:
- "codeql/java-queries"
- "codeql/javascript-queries"
- "codeql/python-queries"
- "codeql/go-queries"
- "codeql/cpp-queries"
- "./.github/codeql/custom-queries"
database:
location: "./.codeql/databases"
cleanup: true
retention-days: 30
analysis:
mode: "${{ env.ANALYSIS_MODE || 'deep' }}"
timeout: 300
memory: 8192
threads: 0 # 0 means automatic
output:
format: sarif-latest
include-column-numbers: true
include-severity: true
include-help-links: true
```
### 5. Custom Query Pack Configuration
```yaml
# .github/codeql/custom-queries/qlpack.yml
name: custom-security-queries
version: 1.0.0
libraryPathDependencies:
- codeql/java-all
- codeql/javascript-all
- codeql/python-all
- codeql/go-all
- codeql/cpp-all
extractor: |
none
dependencies:
codeql/java-all: "*"
codeql/javascript-all: "*"
codeql/python-all: "*"
codeql/go-all: "*"
codeql/cpp-all: "*"
```
### 6. Advanced Security Queries
```ql
// .github/codeql/custom-queries/Security/SQLInjectionAdvanced.ql
/**
* @name Advanced SQL Injection Detection
* @description Detects complex SQL injection patterns with taint tracking
* @kind path-problem
* @problem.severity error
* @precision high
* @id java/advanced-sql-injection
* @tags security
* external/cwe/cwe-89
* external/cwe/cwe-564
*/
import java
import semmle.code.java.dataflow.DataFlow
import semmle.code.java.dataflow.TaintTracking
import DataFlow::PathGraph
class SqlInjectionConfiguration extends TaintTracking::Configuration {
SqlInjectionConfiguration() {
this = "SqlInjectionConfiguration"
}
override predicate isSource(DataFlow::Node source) {
// HTTP request parameters
exists(Method m |
m.hasName("getParameter") or
m.hasName("getHeader") or
m.hasName("getQueryString") or
m.hasName("getCookies") or
m.hasName("getAttribute")
|
source.asExpr() = m.getACall()
) or
// File input
exists(Method m |
m.hasName("readLine") or
m.hasName("read") and
m.getDeclaringType().hasQualifiedName("java.io", "Reader")
|
source.asExpr() = m.getACall()
)
}
override predicate isSink(DataFlow::Node sink) {
// Database operations
exists(Method m |
m.hasName("executeQuery") or
m.hasName("executeUpdate") or
m.hasName("execute") or
m.hasName("prepareStatement") or
m.hasName("createStatement")
|
sink.asExpr() = m.getACall() or
sink.asExpr() = m.getArgument(0)
) or
// ORM operations
exists(Method m |
m.hasName("createNativeQuery") or
m.hasName("createQuery") and
m.getDeclaringType().getASupertype*().hasQualifiedName("javax.persistence", "EntityManager")
|
sink.asExpr() = m.getACall()
)
}
override predicate isSanitizer(DataFlow::Node sanitizer) {
// Parameterized queries
exists(Method m |
m.hasName("setString") or
m.hasName("setInt") or
m.hasName("setDate") and
m.getDeclaringType().getASupertype*().hasQualifiedName("java.sql", "PreparedStatement")
|
sanitizer.asExpr() = m.getACall()
) or
// Input validation
exists(Method m |
m.hasName("matches") or
m.hasName("isAlphanumeric") or
m.hasName("escapeSql")
|
sanitizer.asExpr() = m.getACall()
) or
// Encoding/escaping
exists(Method m |
m.hasName("encode") or
m.hasName("escape") and
m.getDeclaringType().hasQualifiedName("org.apache.commons.lang3", "StringEscapeUtils")
|
sanitizer.asExpr() = m.getACall()
)
}
override predicate isAdditionalTaintStep(DataFlow::Node node1, DataFlow::Node node2) {
// String concatenation and manipulation
exists(BinaryExpr bin |
bin.getOperator() = BinaryExpr.ADD and
(bin.getLeftOperand() = node1.asExpr() and bin.getRightOperand() = node2.asExpr() or
bin.getRightOperand() = node1.asExpr() and bin.getLeftOperand() = node2.asExpr())
) or
// String builder operations
exists(MethodAccess ma |
ma.getMethod().hasName("append") and
ma.getMethod().getDeclaringType().getASupertype*().hasQualifiedName("java.lang", "AbstractStringBuilder") and
node1.asExpr() = ma.getQualifier() and
node2.asExpr() = ma
)
}
}
from SqlInjectionConfiguration config, DataFlow::PathNode source, DataFlow::PathNode sink
where config.hasFlowPath(source, sink)
select sink.getNode(), source, sink,
"Potential SQL injection vulnerability: user input flows to database operation without proper sanitization"
```
### 7. Performance Analysis Queries
```ql
// .github/codeql/custom-queries/Performance/MemoryLeakDetection.ql
/**
* @name Memory Leak Detection
* @description Identifies potential memory leaks in Java applications
* @kind problem
* @problem.severity warning
* @precision medium
* @id java/memory-leak-detection
* @tags performance
* maintainability
*/
import java
class ResourceAllocation extends Method {
ResourceAllocation() {
this.getName().matches("create%") or
this.getName().matches("new%") or
this.getName().matches("open%") or
this.getName().matches("allocate%") or
this.getName().matches("init%")
}
}
class ResourceRelease extends Method {
ResourceRelease() {
this.getName().matches("close%") or
this.getName().matches("destroy%") or
this.getName().matches("release%") or
this.getName().matches("free%") or
this.getName().matches("cleanup%") or
this.getName().matches("dispose%")
}
}
class PotentialMemoryLeak extends ResourceAllocation {
PotentialMemoryLeak() {
not exists(ResourceRelease release |
release.getDeclaringType() = this.getDeclaringType() and
release.getName().toLowerCase() = this.getName().replace("create", "close").toLowerCase() or
release.getName().toLowerCase() = this.getName().replace("new", "delete").toLowerCase() or
release.getName().toLowerCase() = this.getName().replace("open", "close").toLowerCase() or
release.getName().toLowerCase() = this.getName().replace("allocate", "free").toLowerCase()
)
}
}
from PotentialMemoryLeak allocation
select allocation,
"Potential memory leak: resource allocation method '" + allocation.getName() +
"' without corresponding release method"
```
### 8. Security Quality Gates Workflow
```yaml
# .github/workflows/codeql-security-gates.yml
name: "CodeQL Security Gates"
on:
pull_request:
branches: [ main, develop ]
jobs:
security-gates:
name: Security Quality Gates
runs-on: ubuntu-latest
permissions:
security-events: write
actions: read
contents: read
steps:
- name: Checkout repository
uses: actions/checkout@v4
- name: Run CodeQL Analysis
uses: github/codeql-action/analyze@v3
with:
output: sarif-results
category: "security-gates"
upload-database: false
- name: Check Security Thresholds
id: security-check
run: |
python .github/codeql/scripts/security-gates.py
- name: Fail on Critical Security Issues
if: steps.security-check.outputs.critical_issues > 0
run: |
echo "❌ CRITICAL: ${{ steps.security-check.outputs.critical_issues }} critical security issues found"
echo "Blocking merge due to security policy violations"
exit 1
- name: Warn on High Security Issues
if: steps.security-check.outputs.high_issues > 5
run: |
echo "⚠️ WARNING: ${{ steps.security-check.outputs.high_issues }} high severity issues found"
echo "Consider addressing these issues before merge"
- name: Generate Security Report
if: always()
run: |
echo "## CodeQL Security Quality Gates" >> $GITHUB_STEP_SUMMARY
echo "### Results Summary" >> $GITHUB_STEP_SUMMARY
echo "- ✅ Critical Issues: ${{ steps.security-check.outputs.critical_issues }}" >> $GITHUB_STEP_SUMMARY
echo "- ⚠️ High Issues: ${{ steps.security-check.outputs.high_issues }}" >> $GITHUB_STEP_SUMMARY
echo "- 📝 Medium Issues: ${{ steps.security-check.outputs.medium_issues }}" >> $GITHUB_STEP_SUMMARY
if [ ${{ steps.security-check.outputs.critical_issues }} -eq 0 ]; then
echo "### ✅ Security Gates PASSED" >> $GITHUB_STEP_SUMMARY
else
echo "### ❌ Security Gates FAILED" >> $GITHUB_STEP_SUMMARY
fi
```
### 9. Security Gates Script
```python
# .github/codeql/scripts/security-gates.py
#!/usr/bin/env python3
import json
import os
import sys
from pathlib import Path
def load_sarif_results():
"""Load and parse SARIF results from CodeQL analysis"""
results = []
sarif_dir = Path("sarif-results")
if not sarif_dir.exists():
return results
for sarif_file in sarif_dir.glob("**/*.sarif"):
with open(sarif_file, 'r') as f:
data = json.load(f)
for run in data.get('runs', []):
results.extend(run.get('results', []))
return results
def analyze_severity_distribution(results):
"""Analyze severity distribution of security issues"""
severity_count = {
'critical': 0,
'high': 0,
'medium': 0,
'low': 0
}
for result in results:
# Map CodeQL levels to our severity levels
level = result.get('level', 'warning')
if level == 'error':
severity_count['critical'] += 1
elif level == 'warning':
severity_count['high'] += 1
elif level == 'note':
severity_count['medium'] += 1
else:
severity_count['low'] += 1
return severity_count
def check_security_gates(severity_count):
"""Check if security gates are passed"""
gates_passed = True
# Critical issues always fail the gates
if severity_count['critical'] > 0:
gates_passed = False
# More than 10 high issues fails the gates
if severity_count['high'] > 10:
gates_passed = False
return gates_passed
def main():
results = load_sarif_results()
severity_count = analyze_severity_distribution(results)
gates_passed = check_security_gates(severity_count)
# Set output variables for GitHub Actions
print(f"::set-output name=critical_issues::{severity_count['critical']}")
print(f"::set-output name=high_issues::{severity_count['high']}")
print(f"::set-output name=medium_issues::{severity_count['medium']}")
print(f"::set-output name=low_issues::{severity_count['low']}")
print(f"::set-output name=gates_passed::{str(gates_passed).lower()}")
# Print summary
print("Security Analysis Summary:")
print(f" Critical Issues: {severity_count['critical']}")
print(f" High Issues: {severity_count['high']}")
print(f" Medium Issues: {severity_count['medium']}")
print(f" Low Issues: {severity_count['low']}")
print(f" Security Gates: {'PASSED' if gates_passed else 'FAILED'}")
return 0 if gates_passed else 1
if __name__ == "__main__":
sys.exit(main())
```
### 10. Performance Configuration
```yaml
# .github/codeql/config/performance-config.yml
optimization:
max-paths: 1000
timeout: 300
memory: 8192
threads: 4
analysis:
mode: deep
precision: balanced
caching:
enabled: true
location: ./.codeql/cache
cleanup-age: 30d
max-size: 10GB
logging:
level: INFO
queries: true
performance: true
timing: true
monitoring:
enable-metrics: true
metrics-interval: 60
memory-usage: true
```
### 11. Query Suites Configuration
```ql
// .github/codeql/config/query-suites.qls
- description: "Security and Quality Suite"
- queries: .
from: security-and-quality
- queries: .
from: security-extended
- include:
tags:
- security
- external/cwe/cwe-89
- external/cwe/cwe-79
- external/cwe/cwe-78
- external/cwe/cwe-798
- external/cwe/cwe-259
- exclude:
precision: very-low
- exclude:
kind: example
- description: "Custom Security Queries"
- queries: .
from: custom-security-queries.qls
```
## 🚀 Deployment Instructions
### 1. Initialize the Structure
```bash
# Create directory structure
mkdir -p .github/{workflows,codeql/{config,custom-queries/{Security,Performance,Architecture},scripts}}
# Create all the files as shown above
```
### 2. Set Up Dependencies
```bash
# Install CodeQL CLI (if needed)
wget https://github.com/github/codeql-cli-binaries/releases/latest/download/codeql-linux64.zip
unzip codeql-linux64.zip
sudo mv codeql /usr/local/bin/
# Create custom query pack
cd .github/codeql/custom-queries
codeql pack init custom-security-queries
```
### 3. Commit and Push
```bash
git add .
git commit -m "feat: Add advanced CodeQL configuration"
git push origin main
```
## 🔧 Key Features
### ✅ Advanced Analysis Capabilities
- **Multi-language Support**: Java, JavaScript, Python, Go, C++
- **Custom Query Development**: Advanced security and performance queries
- **Performance Optimization**: Configurable resources and caching
- **Security Quality Gates**: Automated security thresholds
- **Comprehensive Reporting**: Detailed security metrics and dashboards
### ✅ Enterprise-Grade Security
- **Advanced Taint Tracking**: Complex data flow analysis
- **Memory Leak Detection**: Performance and resource management
- **SQL Injection Prevention**: Comprehensive injection detection
- **Security Thresholds**: Configurable quality gates
### ✅ Performance Optimization
- **Parallel Execution**: Multi-language concurrent analysis
- **Resource Management**: Configurable memory and CPU usage
- **Caching Strategy**: Reduced analysis times for subsequent runs
- **Incremental Analysis**: Smart change detection
This advanced CodeQL setup provides enterprise-grade code security analysis with custom rules, performance optimization, and comprehensive security gates!
Create CodeQL Advanced
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Here's a
cmake-single-platform.ymlGitHub Actions workflow that builds and tests a CMake project on Linux:Key Features:
Triggers:
Environment:
Steps:
Optimizations:
$(nproc)to detect available CPU coresCustomization Options:
Add Dependencies: ```yaml
CMake Configuration Flags:
yaml run: cmake .. -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTS=ONCross-Platform (modify
runs-on):yaml # For Windows runs-on: windows-latest # For macOS runs-on: macos-latestCaching Dependencies (add before build step): ```yaml
Artifact Upload (add after build): ```yaml
To use this workflow:
.github/workflows/cmake-single-platform.ymlin your repoThe workflow will automatically: