From b767b702f7041487ae7425a396d3b7a6fe47ea9e Mon Sep 17 00:00:00 2001 From: Muhamad Sazwan Bin Ismail Date: Sat, 5 Jul 2025 18:40:19 +0800 Subject: [PATCH 1/5] Create cmake-single-platform.yml Here's a `cmake-single-platform.yml` GitHub Actions workflow that builds and tests a CMake project on Linux: ```yaml name: CMake Build and Test on: push: branches: [ main, master ] pull_request: branches: [ main, master ] jobs: build: runs-on: ubuntu-latest steps: - name: Checkout repository uses: actions/checkout@v4 - name: Install dependencies run: | sudo apt-get update sudo apt-get install -y build-essential cmake - name: Create build directory run: cmake -E make_directory build - name: Configure CMake working-directory: build run: cmake .. -DCMAKE_BUILD_TYPE=Release - name: Build project working-directory: build run: cmake --build . --config Release --parallel $(nproc) - name: Run tests working-directory: build run: ctest -C Release --output-on-failure ``` ### Key Features: 1. **Triggers**: - Runs on push/PR to main/master branches 2. **Environment**: - Uses latest Ubuntu runner 3. **Steps**: - Installs essential build tools and CMake - Creates a separate build directory - Configures CMake in Release mode - Builds with parallel compilation (uses all available cores) - Runs tests with detailed failure output 4. **Optimizations**: - Parallel builds using `$(nproc)` to detect available CPU cores - Separate build directory keeps source tree clean - Output-on-failure for test debugging ### Customization Options: 1. **Add Dependencies**: ```yaml - name: Install custom dependencies run: sudo apt-get install -y libboost-dev libopencv-dev ``` 2. **CMake Configuration Flags**: ```yaml run: cmake .. -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTS=ON ``` 3. **Cross-Platform** (modify `runs-on`): ```yaml # For Windows runs-on: windows-latest # For macOS runs-on: macos-latest ``` 4. **Caching Dependencies** (add before build step): ```yaml - name: Cache dependencies uses: actions/cache@v3 with: path: ~/.cache key: ${{ runner.os }}-deps-${{ hashFiles('**/CMakeLists.txt') }} ``` 5. **Artifact Upload** (add after build): ```yaml - name: Upload build artifacts uses: actions/upload-artifact@v3 with: name: build-output path: build/ ``` To use this workflow: 1. Create `.github/workflows/cmake-single-platform.yml` in your repo 2. Commit and push to trigger the workflow 3. Monitor executions in GitHub Actions tab The workflow will automatically: - Build Release configuration - Run all registered CTest targets - Fail fast on compilation errors or test failures - Provide detailed build logs for debugging --- .github/workflows/cmake-single-platform.yml | 39 +++++++++++++++++++++ 1 file changed, 39 insertions(+) create mode 100644 .github/workflows/cmake-single-platform.yml diff --git a/.github/workflows/cmake-single-platform.yml b/.github/workflows/cmake-single-platform.yml new file mode 100644 index 0000000000000..4bd1c79b4868d --- /dev/null +++ b/.github/workflows/cmake-single-platform.yml @@ -0,0 +1,39 @@ +# This starter workflow is for a CMake project running on a single platform. There is a different starter workflow if you need cross-platform coverage. +# See: https://github.com/actions/starter-workflows/blob/main/ci/cmake-multi-platform.yml +name: CMake on a single platform + +on: + push: + branches: [ "master" ] + pull_request: + branches: [ "master" ] + +env: + # Customize the CMake build type here (Release, Debug, RelWithDebInfo, etc.) + BUILD_TYPE: Release + +jobs: + build: + # The CMake configure and build commands are platform agnostic and should work equally well on Windows or Mac. + # You can convert this to a matrix build if you need cross-platform coverage. + # See: https://docs.github.com/en/free-pro-team@latest/actions/learn-github-actions/managing-complex-workflows#using-a-build-matrix + runs-on: ubuntu-latest + + steps: + - uses: actions/checkout@v4 + + - name: Configure CMake + # Configure CMake in a 'build' subdirectory. `CMAKE_BUILD_TYPE` is only required if you are using a single-configuration generator such as make. + # See https://cmake.org/cmake/help/latest/variable/CMAKE_BUILD_TYPE.html?highlight=cmake_build_type + run: cmake -B ${{github.workspace}}/build -DCMAKE_BUILD_TYPE=${{env.BUILD_TYPE}} + + - name: Build + # Build your program with the given configuration + run: cmake --build ${{github.workspace}}/build --config ${{env.BUILD_TYPE}} + + - name: Test + working-directory: ${{github.workspace}}/build + # Execute tests defined by the CMake configuration. + # See https://cmake.org/cmake/help/latest/manual/ctest.1.html for more detail + run: ctest -C ${{env.BUILD_TYPE}} + From 80a087da9997cfc62b7b5ad814ce150fdd2b5bf6 Mon Sep 17 00:00:00 2001 From: Muhamad Sazwan Bin Ismail Date: Fri, 11 Jul 2025 20:50:13 +0800 Subject: [PATCH 2/5] Update ci.yml Based on the CI badge you've shared, here's how to create a similar status badge for your updated SLSA Level 3 publish workflow: ### 1. Create the SLSA Publish Workflow Badge Add this markdown to your README.md: ```markdown [![SLSA Publish](https://github.com/Sazwanismail/bitcoin/actions/workflows/generator-generic-ossf-slsa3-publish.yml/badge.svg?branch=master)](https://github.com/Sazwanismail/bitcoin/actions/workflows/generator-generic-ossf-slsa3-publish.yml) ``` ### 2. Key Parameters Explained: | Parameter | Value | Purpose | |--------------------|-------------------------------------------------|------------------------------------------| | Workflow filename | `generator-generic-ossf-slsa3-publish.yml` | Matches your updated workflow file | | Branch | `branch=master` | Shows status for master branch | | Event | (Automatic - no need to specify) | Uses default release published trigger | ### 3. For Workflow Dispatch Trigger: If you want to add manual trigger status (like your CI badge), modify to: ```markdown [![SLSA Publish](https://github.com/Sazwanismail/bitcoin/actions/workflows/generator-generic-ossf-slsa3-publish.yml/badge.svg?branch=master&event=workflow_dispatch)](https://github.com/Sazwanismail/bitcoin/actions/workflows/generator-generic-ossf-slsa3-publish.yml) ``` Note the added `&event=workflow_dispatch` parameter ### 4. Badge Appearance Options: - **Passing**: ![](https://img.shields.io/badge/SLSA-passing-brightgreen) - **Failing**: ![](https://img.shields.io/badge/SLSA-failing-red) - **In Progress**: ![](https://img.shields.io/badge/SLSA-pending-yellow) ### 5. Recommended Placement: Add both badges together for better visibility: ```markdown ## Build Status [![CI](https://github.com/Sazwanismail/bitcoin/actions/workflows/ci.yml/badge.svg?branch=master&event=workflow_dispatch)](https://github.com/Sazwanismail/bitcoin/actions/workflows/ci.yml) [![SLSA Level 3](https://github.com/Sazwanismail/bitcoin/actions/workflows/generator-generic-ossf-slsa3-publish.yml/badge.svg?branch=master)](https://github.com/Sazwanismail/bitcoin/actions/workflows/generator-generic-ossf-slsa3-publish.yml) ``` This creates a professional status overview showing both regular CI builds and SLSA-compliant release publishing in your README. The badges will automatically update to show real-time workflow status. --- .github/workflows/ci.yml | 1 - 1 file changed, 1 deletion(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 1dff87ad238d1..3fb3f72e3ab8a 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -15,7 +15,6 @@ on: concurrency: group: ${{ github.event_name != 'pull_request' && github.run_id || github.ref }} - cancel-in-progress: true env: CI_FAILFAST_TEST_LEAVE_DANGLING: 1 # GHA does not care about dangling processes and setting this variable avoids killing the CI script itself on error From bdf1dd91273b38b994c85d680faa8824db2efc4b Mon Sep 17 00:00:00 2001 From: Muhamad Sazwan Bin Ismail Date: Fri, 11 Jul 2025 21:06:00 +0800 Subject: [PATCH 3/5] Update cmake-single-platform.yml I'll update the `cmake-single-platform.yml` workflow with significant improvements for security, performance, and compliance. Here's the enhanced version: ```yaml name: CMake Build (SLSA L3 Provenance) - Single Platform on: push: branches: [main, release/*] tags: [v*] pull_request: release: types: [published] workflow_dispatch: inputs: build-type: description: 'Build type' required: true default: 'Release' type: choice options: [Debug, Release, RelWithDebInfo] permissions: id-token: write # Required for OIDC provenance signing contents: read # Minimal permissions for code access actions: read # Required for reusable workflows jobs: setup: runs-on: ubuntu-latest outputs: build-type: ${{ steps.inputs.outputs.build-type }} run-id: ${{ github.run_id }} steps: - name: Capture build type id: inputs run: | build_type="${{ github.event.inputs.build-type || 'Release' }}" echo "build-type=$build_type" >> $GITHUB_OUTPUT - name: Generate unique run ID id: run-id run: echo "run-id=${{ github.run_id }}" >> $GITHUB_OUTPUT cmake-build: needs: setup runs-on: ubuntu-latest outputs: base64-subjects: ${{ steps.hashes.outputs.base64_subjects }} artifacts-name: artifacts-${{ needs.setup.outputs.run-id }} build-type: ${{ needs.setup.outputs.build-type }} steps: - name: Checkout code (full history) uses: actions/checkout@v4 with: fetch-depth: 0 # Essential for SLSA provenance - name: Cache build dependencies uses: actions/cache@v3 with: path: | ~/.cache/ccache build key: ${{ runner.os }}-cmake-${{ hashFiles('**/CMakeLists.txt') }}-${{ needs.setup.outputs.run-id }} restore-keys: | ${{ runner.os }}-cmake- - name: Setup build environment run: | sudo apt-get update sudo apt-get install -y build-essential cmake ccache ninja-build - name: Configure CMake run: | cmake -B build -G Ninja \ -DCMAKE_BUILD_TYPE=${{ needs.setup.outputs.build-type }} \ -DCMAKE_CXX_COMPILER_LAUNCHER=ccache - name: Build project run: cmake --build build --parallel 4 - name: Test project run: ctest --test-dir build --output-on-failure - name: Package artifacts run: | mkdir -p artifacts # Copy binaries, libraries, and other distributables find build -type f \( -executable -o -name "*.so*" -o -name "*.a" \) \ -exec cp -t artifacts/ {} + # Create versioned archive version=${{ github.event.release.tag_name || 'nightly' }} tar czf artifacts-$version.tar.gz -C artifacts . mv artifacts-$version.tar.gz artifacts/ - name: Generate artifact hashes id: hashes run: | cd artifacts subjects="[]" for file in *; do # Use SHA512 for stronger security hash=$(sha512sum "$file" | awk '{print $1}') subjects=$(jq -c \ --arg name "$file" \ --arg hash "sha512:$hash" \ '. += [{"name": $file, "digest": $hash}]' \ <<< "$subjects") done echo "base64_subjects=$(echo -n "$subjects" | base64 -w0)" >> $GITHUB_OUTPUT - name: Upload build artifacts uses: actions/upload-artifact@v4 with: name: artifacts-${{ needs.setup.outputs.run-id }} path: artifacts/ retention-days: 3 provenance: needs: [setup, cmake-build] uses: slsa-framework/slsa-github-generator/.github/workflows/generator_generic_slsa3.yml@v2.1.0 permissions: id-token: write contents: write actions: read with: base64-subjects: ${{ needs.cmake-build.outputs.base64-subjects }} upload-artifacts-name: ${{ needs.cmake-build.outputs.artifacts-name }} upload-assets: ${{ github.event_name == 'release' && github.event.action == 'published' }} build-type: ${{ needs.cmake-build.outputs.build-type }} secrets: inherit security-scan: needs: [cmake-build] runs-on: ubuntu-latest steps: - name: Download artifacts uses: actions/download-artifact@v4 with: name: ${{ needs.cmake-build.outputs.artifacts-name }} - name: Run vulnerability scan uses: shiftleft/scan-action@v2 with: output: reports/ scan-type: all - name: Upload security report uses: actions/upload-artifact@v4 with: name: security-report-${{ needs.setup.outputs.run-id }} path: reports/ deploy: needs: [provenance, security-scan] if: github.event_name == 'release' && github.event.action == 'published' runs-on: ubuntu-latest environment: production steps: - name: Download artifacts uses: actions/download-artifact@v4 with: name: ${{ needs.cmake-build.outputs.artifacts-name }} - name: Verify provenance run: | # Install SLSA verifier curl -sSL https://github.com/slsa-framework/slsa-verifier/releases/download/v3.0.0/slsa-verifier-linux-amd64 -o slsa-verifier chmod +x slsa-verifier ./slsa-verifier verify-artifact \ --provenance-path provenance.json \ --source-uri github.com/${{ github.repository }} \ --builder-id https://github.com/slsa-framework/slsa-github-generator/.github/workflows/generator_generic_slsa3.yml@v2.1.0 \ artifacts/*.tar.gz - name: Deploy to production run: | # Add your deployment commands here echo "Deploying verified artifacts..." ``` ### Key Improvements: 1. **Workflow Structure**: - Added `setup` job for input handling and run ID generation - Separated build, provenance, security scanning, and deployment - Matrix-ready for future multi-platform expansion 2. **Enhanced Security**: - SHA512 hashing instead of SHA256 - Full provenance verification before deployment - Security scanning with ShiftLeft - Explicit permission scoping - SLSA generator updated to v2.1.0 3. **Build Improvements**: - Added CCache for build acceleration - Ninja build system for faster compilation - Parallel builds (`--parallel 4`) - Automated testing with CTest - Build type parameterization (Debug/Release) 4. **Artifact Management**: - Versioned tarballs for releases - Better artifact collection with pattern matching - Shortened artifact retention (3 days) - Automated cleanup of old artifacts 5. **New Features**: - Security scanning job - Deployment environment with verification gate - Workflow dispatch with build options - Release tag detection - Branch protection patterns (`release/*`) 6. **Compliance Enhancements**: - Explicit builder ID in verifier - Non-repudiable provenance - Full build environment capture - Source integrity verification ### Usage Notes: 1. **Customize Build**: - Modify `Package artifacts` step to match your project structure - Adjust CMake flags in `Configure CMake` step - Update artifact patterns in `find` command 2. **Deployment**: - Add your actual deployment commands in the `Deploy to production` step - Configure environment protection rules in GitHub settings 3. **Security**: - Review security scan results - Consider adding additional scanners (Trivy, OWASP ZAP) - Enable branch protection rules 4. **Verification**: - The SLSA verifier step ensures provenance matches artifacts - Verification occurs in deployment job before release This updated workflow provides a comprehensive CI/CD pipeline with strong security guarantees, faster builds, and better artifact management while maintaining SLSA Level 3 compliance. --- .github/workflows/cmake-single-platform.yml | 158 ++++++++++++++++---- 1 file changed, 133 insertions(+), 25 deletions(-) diff --git a/.github/workflows/cmake-single-platform.yml b/.github/workflows/cmake-single-platform.yml index 4bd1c79b4868d..a0cb60102e276 100644 --- a/.github/workflows/cmake-single-platform.yml +++ b/.github/workflows/cmake-single-platform.yml @@ -1,39 +1,147 @@ -# This starter workflow is for a CMake project running on a single platform. There is a different starter workflow if you need cross-platform coverage. -# See: https://github.com/actions/starter-workflows/blob/main/ci/cmake-multi-platform.yml -name: CMake on a single platform +Here's an updated `cmake-single-platform.yml` workflow that builds your CMake project with SLSA Level 3 provenance and enhanced security: + +```yaml +name: CMake Build (Single Platform) with SLSA L3 on: push: - branches: [ "master" ] + branches: [main] pull_request: - branches: [ "master" ] + release: + types: [published] + workflow_dispatch: -env: - # Customize the CMake build type here (Release, Debug, RelWithDebInfo, etc.) - BUILD_TYPE: Release +permissions: + id-token: write # OIDC token for Sigstore signing + contents: read # Minimal read-only access + packages: write # Only needed if publishing packages jobs: - build: - # The CMake configure and build commands are platform agnostic and should work equally well on Windows or Mac. - # You can convert this to a matrix build if you need cross-platform coverage. - # See: https://docs.github.com/en/free-pro-team@latest/actions/learn-github-actions/managing-complex-workflows#using-a-build-matrix + cmake-build: runs-on: ubuntu-latest + outputs: + base64-subjects: ${{ steps.hashes.outputs.base64_subjects }} + artifacts-name: artifacts-${{ github.run_id }} steps: - - uses: actions/checkout@v4 + - name: Checkout code + uses: actions/checkout@v4 + with: + fetch-depth: 0 # Required for full commit history in provenance + + - name: Install dependencies + run: sudo apt-get update && sudo apt-get install -y build-essential cmake + + - name: Configure CMake + run: cmake -B build -DCMAKE_BUILD_TYPE=Release + + - name: Build project + run: cmake --build build --config Release --parallel 4 + + - name: Create artifacts directory + run: mkdir -p artifacts + + - name: Collect binaries + run: | + find build -type f -executable -exec cp {} artifacts/ \; + # Add other artifacts as needed (libraries, config files, etc.) + + - name: Generate artifact hashes + id: hashes + run: | + cd artifacts + subjects="[]" + for file in *; do + sha=$(sha256sum "$file" | awk '{print $1}') + subjects=$(jq -c \ + --arg name "$file" \ + --arg sha "sha256:$sha" \ + '. += [{"name": $name, "digest": $sha}]' \ + <<< "$subjects") + done + echo "base64_subjects=$(echo -n "$subjects" | base64 -w0)" >> $GITHUB_OUTPUT + + - name: Upload artifacts + uses: actions/upload-artifact@v4 + with: + name: ${{ steps.hashes.outputs.artifacts-name }} + path: artifacts/ + retention-days: 5 # Auto-clean old artifacts + + provenance: + needs: [cmake-build] + uses: slsa-framework/slsa-github-generator/.github/workflows/generator_generic_slsa3.yml@v1.9.0 + permissions: + id-token: write # For Sigstore signing + contents: write # For release assets + actions: read # For reusable workflow + with: + base64-subjects: ${{ needs.cmake-build.outputs.base64-subjects }} + upload-artifacts-name: ${{ needs.cmake-build.outputs.artifacts-name }} + upload-assets: ${{ github.event_name == 'release' && github.event.action == 'published' }} + secrets: inherit + + # Optional: Add package publishing step here if needed + # publish: + # needs: [provenance] + # runs-on: ubuntu-latest + # steps: + # - name: Download artifacts + # uses: actions/download-artifact@v4 + # with: + # name: ${{ needs.cmake-build.outputs.artifacts-name }} + # + # # Add your package publishing commands here +``` + +### Key Features: + +1. **Secure CMake Build**: + - Minimal dependencies installation + - Release-mode builds by default + - Parallel compilation (`--parallel 4`) + - Explicit artifact collection + +2. **SLSA L3 Provenance**: + - Uses official SLSA generator v1.9.0 + - Full non-falsifiable build attestations + - Automatic signature via Sigstore + - Includes all build parameters and environment details + +3. **Artifact Security**: + - Unique artifact names using `run_id` to prevent collisions + - SHA256 hashing of all binaries + - 5-day auto-cleanup of artifacts + - Base64-encoded subject manifest + +4. **Release Integration**: + - Automatic asset upload only for published releases + - Prevents accidental publishing during PRs + - Manual trigger support (`workflow_dispatch`) + +5. **Minimal Permissions**: + - `id-token: write` only for provenance job + - `contents: read` for most jobs + - Explicit package write permission - - name: Configure CMake - # Configure CMake in a 'build' subdirectory. `CMAKE_BUILD_TYPE` is only required if you are using a single-configuration generator such as make. - # See https://cmake.org/cmake/help/latest/variable/CMAKE_BUILD_TYPE.html?highlight=cmake_build_type - run: cmake -B ${{github.workspace}}/build -DCMAKE_BUILD_TYPE=${{env.BUILD_TYPE}} +### How to Use: +1. Place this file in `.github/workflows/cmake-single-platform.yml` +2. Adjust these sections as needed: + - **Dependencies**: Add any required packages in `Install dependencies` + - **CMake Flags**: Modify `Configure CMake` step with your flags + - **Artifacts**: Update `Collect binaries` to match your output files + - **Publishing**: Uncomment and configure the publish job if needed - - name: Build - # Build your program with the given configuration - run: cmake --build ${{github.workspace}}/build --config ${{env.BUILD_TYPE}} +3. For multi-platform support, duplicate the `cmake-build` job with different `runs-on` values and matrix strategy - - name: Test - working-directory: ${{github.workspace}}/build - # Execute tests defined by the CMake configuration. - # See https://cmake.org/cmake/help/latest/manual/ctest.1.html for more detail - run: ctest -C ${{env.BUILD_TYPE}} +### Verification: +After a release, verify provenance with: +```bash +slsa-verifier verify-artifact \ + --provenance-path provenance.json \ + --source-uri github.com/$YOUR_REPO \ + --builder-id https://github.com/slsa-framework/slsa-github-generator/.github/workflows/generator_generic_slsa3.yml@v1.9.0 \ + YOUR_BINARY +``` +This workflow provides cryptographic guarantees of build integrity while maintaining build performance and flexibility. From 1411208f79d39cda75320252737d9b27e59b98b4 Mon Sep 17 00:00:00 2001 From: Muhamad Sazwan Bin Ismail Date: Sun, 17 Aug 2025 18:53:16 +0800 Subject: [PATCH 4/5] New Update ci.yml MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Based on your request to update a `ci.yml` (typically `.gitlab-ci.yml` or similar CI/CD configuration file), here's a structured approach incorporating 2025 best practices from industry sources. Key updates focus on **security**, **efficiency**, **reusability**, and **scalability**: ### 🔧 1. **Adopt Dynamic Child Pipelines for Environment Management** - **Problem**: Managing numerous environments (e.g., `env1`, `env2`) bloats the YAML file . - **Solution**: Use a `parallel:matrix` to generate deployment jobs dynamically: ```yaml deploy: stage: deploy script: ./deploy.sh $ENV_NAME parallel: matrix: - ENV_NAME: [dev, staging, prod] # Add/remove environments here ``` - **Benefits**: Reduces code duplication; adds environments without modifying pipeline structure . ### 🔐 2. **Implement Kaniko for Secure Docker Builds** - Replace `docker build` with **Kaniko** to eliminate privileged container risks: ```yaml build: stage: build image: name: gcr.io/kaniko-project/executor:v1.9.0 entrypoint: [""] script: - /kaniko/executor --context $CI_PROJECT_DIR --destination $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA ``` - **Why**: Avoids security flaws in Docker-in-Docker (DinD) and uses layer caching for faster builds . ### ♻️ 3. **Centralize CI Templates for Cross-Project Consistency** - Store reusable templates in a dedicated repo (e.g., `ci-templates/`): ```yaml include: - project: 'my-org/ci-templates' ref: main file: '/templates/kaniko-build.yml' ``` - **Advantages**: Ensures standardized pipelines; updates propagate globally . ### ⚡ 4. **Optimize Trigger Rules to Reduce Resource Waste** - Run pipelines selectively using `rules`: ```yaml deploy_prod: stage: deploy script: ./deploy.sh prod rules: - if: $CI_COMMIT_BRANCH == "main" changes: - "src/prod-config/*" # Only trigger if prod config changes ``` - **Efficiency**: Cuts unnecessary job runs by 40–60% . ### 🛡️ 5. **Integrate Security Scanning & Secrets Management** - Add built-in security scans and use **Vault** for secrets: ```yaml security_scan: stage: test image: shiftleft/scan script: - scan --src $CI_PROJECT_DIR --type php secrets: VAULT_TOKEN: $VAULT_CI_TOKEN # Inject secrets securely ``` - **Compliance**: Automated SAST/DAST scans; secrets never exposed in logs . --- ### 🚀 **Advanced Optimization Techniques** - **YAML Anchors & `!reference`**: Reuse configurations without duplication : ```yaml .base_script: &base_script before_script: - apt-get update job1: <<: *base_script script: ./test.sh ``` - **Protected Tags/Branches**: Prevent accidental deployments to `main` . - **Windows Runner Fixes**: Use explicit tags (e.g., `saas-windows-medium-amd64`) to avoid Linux/Windows conflicts . --- ### 💎 **Sample Snippet for a Modern `.gitlab-ci.yml`** ```yaml stages: - build - test - deploy include: - project: 'my-org/ci-templates' file: '/security-scan.yml' build: extends: .kaniko_template # From centralized template variables: CONTEXT: $CI_PROJECT_DIR deploy: stage: deploy parallel: matrix: - ENV: [dev, staging, prod] script: ./deploy.sh $ENV rules: - if: $CI_COMMIT_TAG != null # Deploy only on tagged releases ``` ### Key Trends for 2025 : - **AI-Driven Pipelines**: Auto-fix failed jobs using ML. - **GitOps Integration**: Tools like **Argo CD** for Kubernetes deployments. - **Cost Control**: Parallel jobs + caching to reduce cloud spending. For further implementation details: - GitLab CI Editor/Lint: Validate syntax in real-time . - Dynamic Environments: Use `environment:url` with review apps. Let me know if you need help adapting this to a specific tool (e.g., GitHub Actions) or legacy systems! 😊 From 993bd3d2dbcd5c19fcbe36e15a99613609a5574f Mon Sep 17 00:00:00 2001 From: Muhamad Sazwan Bin Ismail Date: Wed, 12 Nov 2025 02:11:12 +0800 Subject: [PATCH 5/5] Create CodeQL Advanced MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit # CodeQL Advanced Implementation ## 🚀 Complete Advanced CodeQL Setup ### 1. Advanced CodeQL Configuration Structure ``` .github/ ├── workflows/ │ ├── codeql-advanced.yml │ ├── codeql-security-gates.yml │ └── codeql-custom-analysis.yml ├── codeql/ │ ├── config/ │ │ ├── codeql-config.yml │ │ ├── query-suites.qls │ │ └── performance-config.yml │ ├── custom-queries/ │ │ ├── qlpack.yml │ │ ├── Security/ │ │ ├── Performance/ │ │ └── Architecture/ │ └── scripts/ │ ├── aggregate-results.py │ └── security-gates.py ``` ### 2. Main Advanced CodeQL Workflow ```yaml # .github/workflows/codeql-advanced.yml name: "Advanced CodeQL Analysis" on: push: branches: [ main, develop, release/* ] paths: - 'src/**' - 'lib/**' - '**.java' - '**.js' - '**.ts' - '**.py' - '**.go' - '**.cpp' paths-ignore: - '**/*.md' - '**/*.txt' - '**/test/**' - '**/spec/**' pull_request: branches: [ main, develop ] schedule: - cron: '0 2 * * 1' # Weekly on Monday at 2 AM UTC workflow_dispatch: inputs: analysis-depth: description: 'Analysis depth level' required: true default: 'deep' type: choice options: - quick - standard - deep include-experimental: description: 'Include experimental queries' required: false type: boolean default: false env: CODEQL_ACTION_VERBOSITY: info CODEQL_POWERFUL_ANALYSIS: true CODEQL_EXTRACTOR_JAVA_AGENT_DISABLE_KOTLIN: false jobs: initialize: name: Initialize Analysis runs-on: ubuntu-latest outputs: matrix: ${{ steps.set-matrix.outputs.matrix }} database-locations: ${{ steps.set-matrix.outputs.database-locations }} steps: - name: Checkout repository uses: actions/checkout@v4 with: fetch-depth: 0 - name: Set up analysis matrix id: set-matrix run: | MATRIX_JSON=$(python .github/codeql/scripts/detect-languages.py) echo "matrix=$MATRIX_JSON" >> $GITHUB_OUTPUT echo "database-locations=$(echo $MATRIX_JSON | jq -r '.include[].database_location' | tr '\n' ',' | sed 's/,$//')" >> $GITHUB_OUTPUT analyze-multi-language: name: Analyze (${{ matrix.language }}) needs: initialize runs-on: ${{ matrix.runner }} strategy: fail-fast: false matrix: ${{ fromJSON(needs.initialize.outputs.matrix) }} steps: - name: Checkout repository uses: actions/checkout@v4 with: fetch-depth: 0 - name: Initialize CodeQL uses: github/codeql-action/init@v3 with: languages: ${{ matrix.language }} queries: ${{ matrix.queries }} config-file: ./.github/codeql/config/codeql-config.yml config: ${{ matrix.config }} tools: latest database: ${{ matrix.database_location }} - name: Build ${{ matrix.language }} code if: matrix.build-command run: ${{ matrix.build-command }} - name: Perform CodeQL Analysis uses: github/codeql-action/analyze@v3 with: category: "/language:${{ matrix.language }}" output: sarif-results/${{ matrix.language }} upload-database: true skip-queries: ${{ github.event_name == 'schedule' && 'security' || '' }} threads: ${{ matrix.threads }} ram: ${{ matrix.ram }} env: CODEQL_RAM: ${{ matrix.ram }} CODEQL_THREADS: ${{ matrix.threads }} custom-query-analysis: name: Custom Query Analysis needs: analyze-multi-language runs-on: ubuntu-latest if: always() steps: - name: Checkout repository uses: actions/checkout@v4 - name: Download CodeQL databases run: | mkdir -p databases echo "${{ needs.initialize.outputs.database-locations }}" | tr ',' '\n' | xargs -I {} cp -r {} databases/ - name: Run Custom Queries uses: github/codeql-action/analyze@v3 with: queries: ./.github/codeql/custom-queries/ packs: ./.github/codeql/custom-queries/ output: sarif-results/custom category: "custom" security-metrics: name: Generate Security Metrics needs: [analyze-multi-language, custom-query-analysis] runs-on: ubuntu-latest if: always() steps: - name: Checkout repository uses: actions/checkout@v4 - name: Generate Security Report run: | python .github/codeql/scripts/aggregate-results.py - name: Upload Security Metrics uses: actions/upload-artifact@v4 with: name: security-metrics path: | security-report.html codeql-metrics.json retention-days: 30 - name: Update Security Dashboard run: | echo "## CodeQL Security Analysis Complete" >> $GITHUB_STEP_SUMMARY echo "✅ Multi-language analysis completed" >> $GITHUB_STEP_SUMMARY echo "📊 Custom queries executed" >> $GITHUB_STEP_SUMMARY echo "📈 Security metrics generated" >> $GITHUB_STEP_SUMMARY ``` ### 3. Language Detection Script ```python # .github/codeql/scripts/detect-languages.py #!/usr/bin/env python3 import json import os from pathlib import Path def detect_languages(): """Detect programming languages in the repository""" language_files = { 'java': ['**/*.java', '**/pom.xml', '**/build.gradle', '**/build.gradle.kts'], 'javascript': ['**/*.js', '**/*.ts', '**/package.json', '**/yarn.lock'], 'python': ['**/*.py', '**/requirements.txt', '**/Pipfile', '**/pyproject.toml'], 'go': ['**/*.go', '**/go.mod'], 'cpp': ['**/*.cpp', '**/*.hpp', '**/*.c', '**/*.h', '**/CMakeLists.txt'], } detected_languages = [] for lang, patterns in language_files.items(): for pattern in patterns: if list(Path('.').glob(pattern)): detected_languages.append(lang) break # Default to javascript if no languages detected if not detected_languages: detected_languages = ['javascript'] return detected_languages def generate_matrix(languages): """Generate workflow matrix configuration""" matrix_config = { 'include': [] } language_configs = { 'java': { 'runner': 'ubuntu-latest', 'queries': 'security-and-quality,security-extended', 'build-command': './gradlew build -x test || mvn compile -DskipTests', 'threads': 4, 'ram': 8192, 'config': '{"packs": ["codeql/java-queries"]}' }, 'javascript': { 'runner': 'ubuntu-latest', 'queries': 'security-and-quality,security-extended', 'build-command': '', 'threads': 2, 'ram': 4096, 'config': '{"packs": ["codeql/javascript-queries"]}' }, 'python': { 'runner': 'ubuntu-latest', 'queries': 'security-and-quality,security-extended', 'build-command': '', 'threads': 2, 'ram': 4096, 'config': '{"packs": ["codeql/python-queries"]}' }, 'go': { 'runner': 'ubuntu-latest', 'queries': 'security-and-quality', 'build-command': 'go build ./...', 'threads': 2, 'ram': 4096, 'config': '{"packs": ["codeql/go-queries"]}' }, 'cpp': { 'runner': 'ubuntu-latest', 'queries': 'security-and-quality', 'build-command': 'mkdir -p build && cd build && cmake .. && make -j4', 'threads': 4, 'ram': 8192, 'config': '{"packs": ["codeql/cpp-queries"]}' } } for lang in languages: if lang in language_configs: config = language_configs[lang] matrix_config['include'].append({ 'language': lang, 'runner': config['runner'], 'queries': config['queries'], 'build-command': config['build-command'], 'threads': config['threads'], 'ram': config['ram'], 'config': config['config'], 'database_location': f'.codeql/databases/{lang}' }) return matrix_config if __name__ == "__main__": languages = detect_languages() matrix = generate_matrix(languages) print(json.dumps(matrix)) ``` ### 4. Advanced CodeQL Configuration ```yaml # .github/codeql/config/codeql-config.yml name: "Advanced CodeQL Configuration" query-filters: - exclude: id: - "java/example/too-many-loops" - "js/debugger-statement" - "python/trivial-conditional" - include: tags: - "security" - "external/cwe/cwe-798" - "external/cwe/cwe-259" - "external/cwe/cwe-89" - "external/cwe/cwe-79" - include: precision: - "high" - "medium" paths: - "src/" - "lib/" - "app/" - "main/" paths-ignore: - "**/test/**" - "**/spec/**" - "**/node_modules/**" - "**/vendor/**" - "**/build/**" - "**/dist/**" - "**/.cache/**" languages: java: build-command: - "./gradlew build -x test" - "mvn compile -DskipTests" source-roots: - "src/main/java" - "src/main/kotlin" javascript: build-command: "npm run build --if-present" source-roots: - "src" - "lib" python: build-command: "python -m py_compile **/*.py" source-roots: - "src" - "app" go: build-command: "go build ./..." source-roots: - "." cpp: build-command: "mkdir -p build && cd build && cmake .. && make -j4" source-roots: - "src" - "include" queries: - uses: ./.github/codeql/config/query-suites.qls - uses: ./.github/codeql/custom-queries/qlpack.yml packs: - "codeql/java-queries" - "codeql/javascript-queries" - "codeql/python-queries" - "codeql/go-queries" - "codeql/cpp-queries" - "./.github/codeql/custom-queries" database: location: "./.codeql/databases" cleanup: true retention-days: 30 analysis: mode: "${{ env.ANALYSIS_MODE || 'deep' }}" timeout: 300 memory: 8192 threads: 0 # 0 means automatic output: format: sarif-latest include-column-numbers: true include-severity: true include-help-links: true ``` ### 5. Custom Query Pack Configuration ```yaml # .github/codeql/custom-queries/qlpack.yml name: custom-security-queries version: 1.0.0 libraryPathDependencies: - codeql/java-all - codeql/javascript-all - codeql/python-all - codeql/go-all - codeql/cpp-all extractor: | none dependencies: codeql/java-all: "*" codeql/javascript-all: "*" codeql/python-all: "*" codeql/go-all: "*" codeql/cpp-all: "*" ``` ### 6. Advanced Security Queries ```ql // .github/codeql/custom-queries/Security/SQLInjectionAdvanced.ql /** * @name Advanced SQL Injection Detection * @description Detects complex SQL injection patterns with taint tracking * @kind path-problem * @problem.severity error * @precision high * @id java/advanced-sql-injection * @tags security * external/cwe/cwe-89 * external/cwe/cwe-564 */ import java import semmle.code.java.dataflow.DataFlow import semmle.code.java.dataflow.TaintTracking import DataFlow::PathGraph class SqlInjectionConfiguration extends TaintTracking::Configuration { SqlInjectionConfiguration() { this = "SqlInjectionConfiguration" } override predicate isSource(DataFlow::Node source) { // HTTP request parameters exists(Method m | m.hasName("getParameter") or m.hasName("getHeader") or m.hasName("getQueryString") or m.hasName("getCookies") or m.hasName("getAttribute") | source.asExpr() = m.getACall() ) or // File input exists(Method m | m.hasName("readLine") or m.hasName("read") and m.getDeclaringType().hasQualifiedName("java.io", "Reader") | source.asExpr() = m.getACall() ) } override predicate isSink(DataFlow::Node sink) { // Database operations exists(Method m | m.hasName("executeQuery") or m.hasName("executeUpdate") or m.hasName("execute") or m.hasName("prepareStatement") or m.hasName("createStatement") | sink.asExpr() = m.getACall() or sink.asExpr() = m.getArgument(0) ) or // ORM operations exists(Method m | m.hasName("createNativeQuery") or m.hasName("createQuery") and m.getDeclaringType().getASupertype*().hasQualifiedName("javax.persistence", "EntityManager") | sink.asExpr() = m.getACall() ) } override predicate isSanitizer(DataFlow::Node sanitizer) { // Parameterized queries exists(Method m | m.hasName("setString") or m.hasName("setInt") or m.hasName("setDate") and m.getDeclaringType().getASupertype*().hasQualifiedName("java.sql", "PreparedStatement") | sanitizer.asExpr() = m.getACall() ) or // Input validation exists(Method m | m.hasName("matches") or m.hasName("isAlphanumeric") or m.hasName("escapeSql") | sanitizer.asExpr() = m.getACall() ) or // Encoding/escaping exists(Method m | m.hasName("encode") or m.hasName("escape") and m.getDeclaringType().hasQualifiedName("org.apache.commons.lang3", "StringEscapeUtils") | sanitizer.asExpr() = m.getACall() ) } override predicate isAdditionalTaintStep(DataFlow::Node node1, DataFlow::Node node2) { // String concatenation and manipulation exists(BinaryExpr bin | bin.getOperator() = BinaryExpr.ADD and (bin.getLeftOperand() = node1.asExpr() and bin.getRightOperand() = node2.asExpr() or bin.getRightOperand() = node1.asExpr() and bin.getLeftOperand() = node2.asExpr()) ) or // String builder operations exists(MethodAccess ma | ma.getMethod().hasName("append") and ma.getMethod().getDeclaringType().getASupertype*().hasQualifiedName("java.lang", "AbstractStringBuilder") and node1.asExpr() = ma.getQualifier() and node2.asExpr() = ma ) } } from SqlInjectionConfiguration config, DataFlow::PathNode source, DataFlow::PathNode sink where config.hasFlowPath(source, sink) select sink.getNode(), source, sink, "Potential SQL injection vulnerability: user input flows to database operation without proper sanitization" ``` ### 7. Performance Analysis Queries ```ql // .github/codeql/custom-queries/Performance/MemoryLeakDetection.ql /** * @name Memory Leak Detection * @description Identifies potential memory leaks in Java applications * @kind problem * @problem.severity warning * @precision medium * @id java/memory-leak-detection * @tags performance * maintainability */ import java class ResourceAllocation extends Method { ResourceAllocation() { this.getName().matches("create%") or this.getName().matches("new%") or this.getName().matches("open%") or this.getName().matches("allocate%") or this.getName().matches("init%") } } class ResourceRelease extends Method { ResourceRelease() { this.getName().matches("close%") or this.getName().matches("destroy%") or this.getName().matches("release%") or this.getName().matches("free%") or this.getName().matches("cleanup%") or this.getName().matches("dispose%") } } class PotentialMemoryLeak extends ResourceAllocation { PotentialMemoryLeak() { not exists(ResourceRelease release | release.getDeclaringType() = this.getDeclaringType() and release.getName().toLowerCase() = this.getName().replace("create", "close").toLowerCase() or release.getName().toLowerCase() = this.getName().replace("new", "delete").toLowerCase() or release.getName().toLowerCase() = this.getName().replace("open", "close").toLowerCase() or release.getName().toLowerCase() = this.getName().replace("allocate", "free").toLowerCase() ) } } from PotentialMemoryLeak allocation select allocation, "Potential memory leak: resource allocation method '" + allocation.getName() + "' without corresponding release method" ``` ### 8. Security Quality Gates Workflow ```yaml # .github/workflows/codeql-security-gates.yml name: "CodeQL Security Gates" on: pull_request: branches: [ main, develop ] jobs: security-gates: name: Security Quality Gates runs-on: ubuntu-latest permissions: security-events: write actions: read contents: read steps: - name: Checkout repository uses: actions/checkout@v4 - name: Run CodeQL Analysis uses: github/codeql-action/analyze@v3 with: output: sarif-results category: "security-gates" upload-database: false - name: Check Security Thresholds id: security-check run: | python .github/codeql/scripts/security-gates.py - name: Fail on Critical Security Issues if: steps.security-check.outputs.critical_issues > 0 run: | echo "❌ CRITICAL: ${{ steps.security-check.outputs.critical_issues }} critical security issues found" echo "Blocking merge due to security policy violations" exit 1 - name: Warn on High Security Issues if: steps.security-check.outputs.high_issues > 5 run: | echo "⚠️ WARNING: ${{ steps.security-check.outputs.high_issues }} high severity issues found" echo "Consider addressing these issues before merge" - name: Generate Security Report if: always() run: | echo "## CodeQL Security Quality Gates" >> $GITHUB_STEP_SUMMARY echo "### Results Summary" >> $GITHUB_STEP_SUMMARY echo "- ✅ Critical Issues: ${{ steps.security-check.outputs.critical_issues }}" >> $GITHUB_STEP_SUMMARY echo "- ⚠️ High Issues: ${{ steps.security-check.outputs.high_issues }}" >> $GITHUB_STEP_SUMMARY echo "- 📝 Medium Issues: ${{ steps.security-check.outputs.medium_issues }}" >> $GITHUB_STEP_SUMMARY if [ ${{ steps.security-check.outputs.critical_issues }} -eq 0 ]; then echo "### ✅ Security Gates PASSED" >> $GITHUB_STEP_SUMMARY else echo "### ❌ Security Gates FAILED" >> $GITHUB_STEP_SUMMARY fi ``` ### 9. Security Gates Script ```python # .github/codeql/scripts/security-gates.py #!/usr/bin/env python3 import json import os import sys from pathlib import Path def load_sarif_results(): """Load and parse SARIF results from CodeQL analysis""" results = [] sarif_dir = Path("sarif-results") if not sarif_dir.exists(): return results for sarif_file in sarif_dir.glob("**/*.sarif"): with open(sarif_file, 'r') as f: data = json.load(f) for run in data.get('runs', []): results.extend(run.get('results', [])) return results def analyze_severity_distribution(results): """Analyze severity distribution of security issues""" severity_count = { 'critical': 0, 'high': 0, 'medium': 0, 'low': 0 } for result in results: # Map CodeQL levels to our severity levels level = result.get('level', 'warning') if level == 'error': severity_count['critical'] += 1 elif level == 'warning': severity_count['high'] += 1 elif level == 'note': severity_count['medium'] += 1 else: severity_count['low'] += 1 return severity_count def check_security_gates(severity_count): """Check if security gates are passed""" gates_passed = True # Critical issues always fail the gates if severity_count['critical'] > 0: gates_passed = False # More than 10 high issues fails the gates if severity_count['high'] > 10: gates_passed = False return gates_passed def main(): results = load_sarif_results() severity_count = analyze_severity_distribution(results) gates_passed = check_security_gates(severity_count) # Set output variables for GitHub Actions print(f"::set-output name=critical_issues::{severity_count['critical']}") print(f"::set-output name=high_issues::{severity_count['high']}") print(f"::set-output name=medium_issues::{severity_count['medium']}") print(f"::set-output name=low_issues::{severity_count['low']}") print(f"::set-output name=gates_passed::{str(gates_passed).lower()}") # Print summary print("Security Analysis Summary:") print(f" Critical Issues: {severity_count['critical']}") print(f" High Issues: {severity_count['high']}") print(f" Medium Issues: {severity_count['medium']}") print(f" Low Issues: {severity_count['low']}") print(f" Security Gates: {'PASSED' if gates_passed else 'FAILED'}") return 0 if gates_passed else 1 if __name__ == "__main__": sys.exit(main()) ``` ### 10. Performance Configuration ```yaml # .github/codeql/config/performance-config.yml optimization: max-paths: 1000 timeout: 300 memory: 8192 threads: 4 analysis: mode: deep precision: balanced caching: enabled: true location: ./.codeql/cache cleanup-age: 30d max-size: 10GB logging: level: INFO queries: true performance: true timing: true monitoring: enable-metrics: true metrics-interval: 60 memory-usage: true ``` ### 11. Query Suites Configuration ```ql // .github/codeql/config/query-suites.qls - description: "Security and Quality Suite" - queries: . from: security-and-quality - queries: . from: security-extended - include: tags: - security - external/cwe/cwe-89 - external/cwe/cwe-79 - external/cwe/cwe-78 - external/cwe/cwe-798 - external/cwe/cwe-259 - exclude: precision: very-low - exclude: kind: example - description: "Custom Security Queries" - queries: . from: custom-security-queries.qls ``` ## 🚀 Deployment Instructions ### 1. Initialize the Structure ```bash # Create directory structure mkdir -p .github/{workflows,codeql/{config,custom-queries/{Security,Performance,Architecture},scripts}} # Create all the files as shown above ``` ### 2. Set Up Dependencies ```bash # Install CodeQL CLI (if needed) wget https://github.com/github/codeql-cli-binaries/releases/latest/download/codeql-linux64.zip unzip codeql-linux64.zip sudo mv codeql /usr/local/bin/ # Create custom query pack cd .github/codeql/custom-queries codeql pack init custom-security-queries ``` ### 3. Commit and Push ```bash git add . git commit -m "feat: Add advanced CodeQL configuration" git push origin main ``` ## 🔧 Key Features ### ✅ Advanced Analysis Capabilities - **Multi-language Support**: Java, JavaScript, Python, Go, C++ - **Custom Query Development**: Advanced security and performance queries - **Performance Optimization**: Configurable resources and caching - **Security Quality Gates**: Automated security thresholds - **Comprehensive Reporting**: Detailed security metrics and dashboards ### ✅ Enterprise-Grade Security - **Advanced Taint Tracking**: Complex data flow analysis - **Memory Leak Detection**: Performance and resource management - **SQL Injection Prevention**: Comprehensive injection detection - **Security Thresholds**: Configurable quality gates ### ✅ Performance Optimization - **Parallel Execution**: Multi-language concurrent analysis - **Resource Management**: Configurable memory and CPU usage - **Caching Strategy**: Reduced analysis times for subsequent runs - **Incremental Analysis**: Smart change detection This advanced CodeQL setup provides enterprise-grade code security analysis with custom rules, performance optimization, and comprehensive security gates! --- CodeQL Advanced | 714 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 714 insertions(+) create mode 100644 CodeQL Advanced diff --git a/CodeQL Advanced b/CodeQL Advanced new file mode 100644 index 0000000000000..686c32a140b64 --- /dev/null +++ b/CodeQL Advanced @@ -0,0 +1,714 @@ +Configuring CodeQL in advanced setup gives you granular control to customize scanning for your project's specific needs, going far beyond what default setup offers. You can fine-tune everything from scan triggers and operating systems to creating your own security queries. + +Here's a comparison to help you understand the core differences: + +| Feature | Default Setup | Advanced Setup | +| :--- | :--- | :--- | +| **Control** | Automated, minimal input | Full manual control | +| **Configuration** | Simple UI-based | Editing YAML workflow files | +| **Customization** | Limited | Extensive (triggers, OS, queries, etc.) | +| **Workflow Location** | Automatically managed | `.github/workflows/` directory | +| **Best For** | Quick start, standard needs | Complex projects, specific requirements | + +### ⚙️ Configure the Advanced Workflow + +The advanced setup revolves around modifying a CodeQL workflow file (typically `codeql-analysis.yml`) located in your repository's `.github/workflows` directory. + +**Set Scanning Triggers and Frequency** +You can configure the workflow to run on specific events, on a schedule, or both. This is defined using the `on` key in the YAML file. + +- **Scan on Push and Pull Requests**: This is the standard practice for catching vulnerabilities as they are introduced. You can specify which branches to monitor. +- **Avoid Unnecessary Scans**: Use `paths-ignore` to skip scanning PRs that only change certain files (like documentation), saving resources. +- **Schedule Scans**: Run periodic scans (e.g., weekly) to catch new vulnerabilities from updated CodeQL queries, even during inactive periods. + +```yaml +on: + push: + branches: [ main, protected ] + pull_request: + branches: [ main ] + paths-ignore: + - '**/*.md' + - '**/*.txt' + schedule: + - cron: '20 14 * * 1' # Every Monday at 14:20 UTC +``` + +**Specify the Operating System and Languages** +The `runs-on` key determines the OS of the runner. CodeQL supports `ubuntu-latest`, `windows-latest`, and `macos-latest`. + +- **For Swift code**, CodeQL uses macOS runners by default, which are more expensive. You can configure it to use a different runner for all steps except the build. +- Use a **matrix strategy** to analyze multiple languages efficiently in parallel jobs. For compiled languages like Java, you may need to include custom build steps. + +```yaml +jobs: + analyze: + name: Analyze + runs-on: ubuntu-latest + strategy: + fail-fast: false + matrix: + language: [ 'javascript', 'python', 'java' ] + steps: + - name: Checkout repository + uses: actions/checkout@v4 + - name: Initialize CodeQL + uses: github/codeql-action/init@v4 + with: + languages: ${{ matrix.language }} +``` + +### 🔍 Develop Custom Queries + +Advanced setup allows you to extend analysis by running additional queries, including those you write yourself. + +**Create a Query Pack** +CodeQL queries are organized into "packs". To start writing your own, create a new pack using the CodeQL CLI, which generates a `qlpack.yml` file. +```bash +codeql pack init / +``` +You then need to add dependencies for the standard library of the language you want to analyze. +```bash +codeql pack add codeql/javascript-all +``` + +**Understand QL Basics** +QL is a declarative, logic programming language. A basic query follows this structure: +```ql +from /* ... variable declarations ... */ +where /* ... logical conditions ... */ +select /* ... expressions to output ... */ +``` + +**Write and Refine Queries** +- **Leverage Standard Libraries**: Use existing classes and predicates from the extensive CodeQL libraries to build upon. +- **Use Debugging Tools**: If a query doesn't work as expected, techniques like creating a minimal code example, using Quick Evaluation in VS Code, viewing the AST, and generating partial path graphs are invaluable for diagnosing issues. +- **Add Query Metadata**: Include a comment block with metadata like `@name`, `@description`, and `@kind` so the results are properly displayed in GitHub. + +### 🚀 Implement at Scale and Customize Further + +- **Scale Across an Organization**: For organizations needing advanced setup on many repositories, you can use a bulk configuration script to add the workflow across a group of repos efficiently. +- **Extend Analysis with Model Packs**: For code that uses custom or niche frameworks not recognized by standard queries, you can specify published **CodeQL model packs** (currently in preview for C/C++, C#, Java/Kotlin, Python, Ruby, and Rust) to extend coverage. +- **Specify a Custom Database Location**: While usually handled automatically, you can define where the CodeQL database is stored using the `db-location` parameter under the `init` action, which is useful for custom workflow steps. + +I hope this gives you a comprehensive roadmap for leveraging the advanced capabilities of CodeQL. Which aspect are you most interested in implementing first? If you're working with a specific programming language or framework, I might be able to offer more tailored guidance. +# CodeQL Advanced Implementation Guide + +## 🚀 Advanced CodeQL Configuration + +### Multi-Language Enterprise Setup + +```yaml +# .github/workflows/advanced-codeql.yml +name: "Advanced CodeQL Analysis" + +on: + push: + branches: [ main, develop, release/* ] + paths-ignore: + - '**/*.md' + - '**/*.docs' + - '**/test/**' + - '**/spec/**' + pull_request: + branches: [ main, develop ] + paths: + - 'src/**' + - 'lib/**' + - '**.java' + - '**.js' + - '**.ts' + - '**.py' + - '**.go' + schedule: + - cron: '0 2 * * 1' # Weekly on Monday at 2 AM UTC + workflow_dispatch: + inputs: + severity: + description: 'Minimum severity level' + required: true + default: 'error' + type: choice + options: + - error + - warning + - note + +env: + CODEQL_ACTION_DISABLE_SETUP: false + CODEQL_EXTRACTOR_JAVA_AGENT_DISABLE_KOTLIN: false + CODEQL_POWERFUL_ANALYSIS: true + +jobs: + analyze: + name: Advanced CodeQL Analysis + runs-on: ubuntu-latest + + strategy: + fail-fast: false + matrix: + include: + - language: java-kotlin + build-mode: autobuild + - language: javascript + build-mode: none + - language: python + build-mode: none + - language: go + build-mode: autobuild + - language: cpp + build-mode: manual + + steps: + - name: Checkout repository + uses: actions/checkout@v4 + with: + fetch-depth: 0 # Full history for better analysis + + - name: Initialize CodeQL + uses: github/codeql-action/init@v3 + with: + languages: ${{ matrix.language }} + queries: +security-and-quality,security-extended + config-file: ./.github/codeql/codeql-config.yml + tools: latest + + - name: Autobuild (if needed) + if: matrix.build-mode == 'autobuild' + uses: github/codeql-action/autobuild@v3 + + - name: Manual Build (C/C++) + if: matrix.language == 'cpp' && matrix.build-mode == 'manual' + run: | + mkdir build && cd build + cmake .. + make -j4 + + - name: Perform CodeQL Analysis + uses: github/codeql-action/analyze@v3 + with: + category: "/language:${{ matrix.language }}" + output: sarif-results + upload-database: true + skip-queries: ${{ github.event_name == 'schedule' && 'security' || '' }} + env: + CODEQL_ACTION_DISABLE_SETUP: false + + custom-queries: + name: Custom Query Analysis + runs-on: ubuntu-latest + needs: analyze + + steps: + - name: Checkout repository + uses: actions/checkout@v4 + + - name: Initialize CodeQL + uses: github/codeql-action/init@v3 + with: + languages: javascript, python + queries: ./.github/codeql/custom-queries.qls + packs: codeql/custom-queries + + - name: Run Custom Queries + uses: github/codeql-action/analyze@v3 + with: + category: "custom" + + security-dashboard: + name: Security Dashboard Update + runs-on: ubuntu-latest + needs: [analyze, custom-queries] + if: always() + + steps: + - name: Upload Results to Security Dashboard + uses: github/codeql-action/upload-sarif@v3 + with: + sarif_file: sarif-results + wait-for-processing: true + + - name: Generate Security Report + run: | + echo "## CodeQL Security Analysis Report" >> $GITHUB_STEP_SUMMARY + echo "| Language | Critical | High | Medium | Low |" >> $GITHUB_STEP_SUMMARY + echo "|----------|----------|------|--------|-----|" >> $GITHUB_STEP_SUMMARY + echo "| Complete | - | - | - | - |" >> $GITHUB_STEP_SUMMARY +``` + +## 🔧 Advanced Configuration Files + +### CodeQL Configuration +```yaml +# .github/codeql/codeql-config.yml +name: "Advanced CodeQL Configuration" + +query-filters: +- exclude: + id: java/example/too-many-loops +- include: + tags: + - security + - external/cwe/cwe-798 + - external/cwe/cwe-259 + +paths: +- src +- lib +- app + +paths-ignore: +- test +- spec +- node_modules +- vendor +- build +- dist + +languages: + java: + build-command: + - mvn compile -DskipTests + - gradle build -x test + + javascript: + build-command: npm run build + + python: + build-command: python -m py_compile **/*.py + +queries: + uses: ./.github/codeql/query-suites.yml + +packs: +- codeql/java-queries +- codeql/javascript-queries +- codeql/python-queries +- organization/custom-security-queries + +database: + location: ./.codeql/databases + cleanup: true +``` + +### Custom Query Suites +```yql +# .github/codeql/query-suites.qls +- description: "Custom Security Queries" +- queries: . + from: custom-security-queries.qls +- include: + tags: + - security + - external/cwe/cwe-89 + - external/cwe/cwe-78 + - external/cwe/cwe-79 +- exclude: + precision: very-low +``` + +## 🛠️ Custom Query Development + +### Advanced QL Query Structure +```ql +// .github/codeql/custom-queries/SQLInjection.ql +/** + * @name Potential SQL Injection + * @description Detects potential SQL injection vulnerabilities in Java code + * @kind path-problem + * @problem.severity error + * @precision high + * @id java/sql-injection + * @tags security + * external/cwe/cwe-89 + */ + +import java +import semmle.code.java.dataflow.DataFlow +import semmle.code.java.dataflow.TaintTracking +import DataFlow::PathGraph + +class SqlInjectionConfig extends TaintTracking::Configuration { + SqlInjectionConfig() { this = "SqlInjectionConfig" } + + override predicate isSource(DataFlow::Node source) { + exists(MethodAccess ma | + ma.getMethod().hasName("getParameter") and + source.asExpr() = ma + ) + } + + override predicate isSink(DataFlow::Node sink) { + exists(MethodAccess ma | + ma.getMethod().hasName("executeQuery") and + sink.asExpr() = ma.getArgument(0) + ) + } + + override predicate isSanitizer(DataFlow::Node sanitizer) { + exists(MethodAccess ma | + ma.getMethod().hasName("escapeSql") and + sanitizer.asExpr() = ma + ) + } +} + +from SqlInjectionConfig config, DataFlow::PathNode source, DataFlow::PathNode sink +where config.hasFlowPath(source, sink) +select sink.getNode(), source, sink, "Potential SQL injection vulnerability" +``` + +### Complex Data Flow Analysis +```ql +// .github/codeql/custom-queries/AdvancedTaintTracking.ql +import java +import semmle.code.java.dataflow.DataFlow +import semmle.code.java.dataflow.TaintTracking + +class UserInputSource extends DataFlow::ExprNode { + UserInputSource() { + exists(Method m | + m.hasName("getParameter") or + m.hasName("getHeader") or + m.hasName("getCookie") or + m.hasName("getAttribute") + | + this.asExpr() = m.getACall() + ) + } +} + +class DangerousSink extends DataFlow::ExprNode { + DangerousSink() { + // File operations + exists(Method m | m.hasName("write") | this.asExpr() = m.getACall()) or + // Database operations + exists(Method m | m.hasName("execute") | this.asExpr() = m.getACall()) or + // Command execution + exists(Method m | m.hasName("exec") | this.asExpr() = m.getACall()) + } +} + +class SanitizerMethod extends DataFlow::ExprNode { + SanitizerMethod() { + exists(Method m | + m.hasName("validateInput") or + m.hasName("sanitize") or + m.hasName("escapeHtml") + | + this.asExpr() = m.getACall() + ) + } +} + +from UserInputSource source, DangerousSink sink +where DataFlow::localFlow(source, sink) +select sink, "User input flows to dangerous sink without sanitization" +``` + +## 🔍 Advanced Query Patterns + +### Performance Optimization Queries +```ql +// .github/codeql/custom-queries/PerformanceIssues.ql +import java +import semmle.code.java.dataflow.DataFlow + +class ExpensiveOperationInLoop extends DataFlow::Configuration { + ExpensiveOperationInLoop() { this = "ExpensiveOperationInLoop" } + + override predicate isSource(DataFlow::Node source) { + exists(Method m | + m.hasName("compile") or + m.hasName("createConnection") or + m.hasName("parseXML") + | + source.asExpr() = m.getACall() + ) + } + + override predicate isSink(DataFlow::Node sink) { + exists(LoopStmt loop | sink.asExpr() = loop.getCondition()) + } +} + +from ExpensiveOperationInLoop config, DataFlow::PathNode source, DataFlow::PathNode sink +where config.hasFlowPath(source, sink) +select sink.getNode(), source, sink, "Expensive operation inside loop may cause performance issues" +``` + +### Security Configuration Queries +```ql +// .github/codeql/custom-queries/SecurityMisconfiguration.ql +import java + +class InsecureConfiguration extends RefType { + InsecureConfiguration() { + this.hasQualifiedName("org.springframework.security.config.annotation.web", "WebSecurityConfigurerAdapter") + } +} + +from InsecureConfiguration config, Method m +where + m.getDeclaringType() = config and + m.overridesOrInstantiates*(m.getDeclaringType().getAMethod()) and + not exists(MethodAccess ma | + ma.getMethod().hasName("csrf") and + ma.getAnArgument().toString().matches("%disable%") + ) +select m, "Security configuration might be missing CSRF protection" +``` + +## 📊 Advanced Analysis Pipeline + +### Multi-Stage Analysis Workflow +```yaml +# .github/workflows/codeql-pipeline.yml +name: "CodeQL Security Pipeline" + +on: + push: + branches: [ main ] + schedule: + - cron: '0 0 * * 0' # Weekly + +jobs: + code-scanning: + name: Code Scanning + uses: ./.github/workflows/codeql-advanced.yml + secrets: inherit + + dependency-scanning: + name: Dependency Vulnerability Scan + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: Run OWASP Dependency Check + uses: dependency-check/Dependency-Check_Action@main + with: + project: 'my-project' + path: '.' + format: 'HTML' + out: 'reports' + + container-scanning: + name: Container Image Scanning + runs-on: ubuntu-latest + steps: + - name: Run Trivy Vulnerability Scanner + uses: aquasecurity/trivy-action@master + with: + scan-type: 'fs' + scan-ref: '.' + format: 'sarif' + output: 'trivy-results.sarif' + + security-report: + name: Generate Security Report + runs-on: ubuntu-latest + needs: [code-scanning, dependency-scanning, container-scanning] + steps: + - name: Aggregate Results + run: | + python .github/scripts/aggregate-security-report.py + - name: Upload Security Report + uses: actions/upload-artifact@v4 + with: + name: security-report + path: security-report.html +``` + +## 🔒 Advanced Security Queries + +### Authentication & Authorization +```ql +// .github/codeql/custom-queries/AuthBypass.ql +import java + +class AuthenticationBypass extends Method { + AuthenticationBypass() { + this.getAnAnnotation().getType().hasQualifiedName("org.springframework.web.bind.annotation", "RequestMapping") and + not exists(MethodAccess ma | + ma.getMethod().hasName("isAuthenticated") or + ma.getMethod().hasName("hasRole") + ) + } +} + +from AuthenticationBypass method +select method, "Endpoint might be missing authentication check" +``` + +### Cryptography Issues +```ql +// .github/codeql/custom-queries/WeakCrypto.ql +import java + +class WeakCryptoAlgorithm extends MethodAccess { + WeakCryptoAlgorithm() { + this.getMethod().hasName("getInstance") and + ( + this.getArgument(0).toString().matches("%DES%") or + this.getArgument(0).toString().matches("%RC4%") or + this.getArgument(0).toString().matches("%MD5%") + ) + } +} + +from WeakCryptoAlgorithm crypto +select crypto, "Use of weak cryptographic algorithm detected" +``` + +## 📈 Performance Optimization + +### Query Performance Configuration +```yaml +# .github/codeql/performance-config.yml +optimization: + max-paths: 1000 + timeout: 300 + memory: 8192 + +analysis: + mode: deep + threads: 4 + +caching: + enabled: true + location: ./.codeql/cache + cleanup-age: 30d + +logging: + level: INFO + queries: true + performance: true +``` + +### Custom Performance Queries +```ql +// .github/codeql/custom-queries/MemoryLeaks.ql +import java + +class PotentialMemoryLeak extends Method { + PotentialMemoryLeak() { + this.getName().matches("create%") and + not exists(Method destructor | + destructor.getName().matches("destroy%") or + destructor.getName().matches("close%") and + destructor.getDeclaringType() = this.getDeclaringType() + ) + } +} + +from PotentialMemoryLeak method +select method, "Potential memory leak: creation method without corresponding cleanup method" +``` + +## 🎯 Advanced Integration + +### CI/CD Pipeline Integration +```yaml +# .github/workflows/security-gates.yml +name: "Security Quality Gates" + +on: + pull_request: + branches: [ main ] + +jobs: + security-gate: + name: Security Quality Gate + runs-on: ubuntu-latest + steps: + - name: Run CodeQL Analysis + uses: github/codeql-action/analyze@v3 + with: + output: sarif-results + + - name: Check Security Thresholds + run: | + python .github/scripts/check-security-gates.py + + - name: Fail on Critical Issues + if: steps.security-check.outputs.critical_issues > 0 + run: | + echo "❌ Critical security issues found. Blocking merge." + exit 1 +``` + +### Custom Scripts for Analysis +```python +# .github/scripts/aggregate-security-report.py +#!/usr/bin/env python3 +import json +import sys +from pathlib import Path + +def load_sarif_results(file_path): + with open(file_path, 'r') as f: + return json.load(f) + +def analyze_severity_distribution(results): + severity_count = { + 'error': 0, + 'warning': 0, + 'note': 0 + } + + for run in results.get('runs', []): + for result in run.get('results', []): + level = result.get('level', 'warning') + severity_count[level] += 1 + + return severity_count + +def generate_report(severity_distribution): + report = f""" +# Security Analysis Report +## Summary +- Critical Issues: {severity_distribution['error']} +- Warnings: {severity_distribution['warning']} +- Recommendations: {severity_distribution['note']} + +## Quality Gates +{'❌ FAIL' if severity_distribution['error'] > 0 else '✅ PASS'} - No Critical Issues +{'⚠️ WARN' if severity_distribution['warning'] > 10 else '✅ PASS'} - Warning Threshold +""" + return report + +if __name__ == "__main__": + results_dir = Path("sarif-results") + all_results = [] + + for sarif_file in results_dir.glob("*.sarif"): + all_results.extend(load_sarif_results(sarif_file)) + + severity = analyze_severity_distribution(all_results) + report = generate_report(severity) + + with open("security-report.md", "w") as f: + f.write(report) + + print("Security report generated successfully") +``` + +This advanced CodeQL implementation provides: + +## 🚀 Key Features + +1. **Multi-Language Support** - Java, JavaScript, Python, Go, C++ +2. **Custom Query Development** - Advanced security and performance queries +3. **Performance Optimization** - Caching, parallel execution, resource management +4. **Security Quality Gates** - Automated security thresholds and blocking +5. **Comprehensive Reporting** - Aggregated security dashboards +6. **Enterprise Integration** - CI/CD pipeline integration with quality gates + +## 💡 Advanced Use Cases + +- **Custom Security Rules** - Organization-specific security requirements +- **Performance Analysis** - Code performance and memory leak detection +- **Architecture Validation** - Code structure and design pattern validation +- **Compliance Checking** - Regulatory and compliance requirements +- **Dependency Analysis** - Third-party library security assessment + +This setup enables enterprise-grade code security analysis with custom rules, performance optimization, and comprehensive reporting!