diff --git a/.github/SECRET_SCANNING.md b/.github/SECRET_SCANNING.md new file mode 100644 index 0000000..162b3a9 --- /dev/null +++ b/.github/SECRET_SCANNING.md @@ -0,0 +1,262 @@ +# Secret Scanning Documentation + +## Overview + +This repository uses [Gitleaks](https://github.com/gitleaks/gitleaks) to automatically detect hardcoded secrets, credentials, and other sensitive information in the codebase. The secret scanning workflow runs on every push and pull request to prevent accidental exposure of sensitive data. + +## Detected Secret Types + +Gitleaks scans for a wide variety of secret types across all file formats, including: + +### Cloud Provider Credentials +- AWS Access Keys, Secret Keys, and Session Tokens +- Azure Storage Keys, Client Secrets, and Tenant IDs +- Google Cloud API Keys and Service Account Credentials +- DigitalOcean Personal Access Tokens + +### Version Control & CI/CD +- GitHub Personal Access Tokens, OAuth Tokens, and App Tokens +- GitLab Personal Access Tokens and Pipeline Triggers +- Bitbucket App Passwords and Access Tokens + +### Communication & Collaboration +- Slack Webhooks and API Tokens +- Discord Webhooks and Bot Tokens +- Microsoft Teams Webhooks + +### Databases & Services +- Database Connection Strings (PostgreSQL, MySQL, MongoDB) +- Redis Passwords +- JDBC Connection URLs with embedded credentials + +### API Keys & Tokens +- Generic API Keys and Secret Keys +- JWT Tokens with high entropy +- OAuth Client IDs and Secrets + +### Cryptographic Material +- RSA Private Keys +- SSH Private Keys +- PGP Private Keys +- SSL/TLS Certificates and Private Keys + +### Application-Specific +- Stripe API Keys +- SendGrid API Keys +- Twilio Auth Tokens +- Mailgun API Keys +- Passwords and secrets in configuration files (YAML, JSON, .env, etc.) + +## Trigger Conditions + +The secret scanning workflow is triggered automatically on: + +1. **Push Events**: Any push to any branch +2. **Pull Request Events**: Any pull request targeting any branch + +The workflow runs with read-only permissions (`contents: read`) following the least-privilege principle. + +## Workflow Behavior + +When secrets are detected: +- ✗ The workflow **fails** the build +- ✗ The commit or pull request is **blocked** +- ℹ️ Details about detected secrets are shown in the workflow logs + +When no secrets are detected: +- ✓ The workflow **passes** +- ✓ The commit or pull request can proceed + +## Remediation Steps + +If the workflow detects a secret, follow these steps: + +### 1. Verify the Detection + +Review the workflow logs to understand what was detected: +```bash +# In GitHub Actions, navigate to: +# Actions → Secret Scanning → [Failed Run] → Gitleaks Secret Scan +``` + +### 2. Determine if It's a Real Secret + +**If it's a real secret:** +1. **Immediately rotate the credential** on the provider's platform +2. Remove the secret from your code +3. Never commit secrets again - use environment variables or secret management tools + +**If it's a false positive:** +1. Proceed to add it to the allowlist (see below) + +### 3. Remove the Secret from Code + +For real secrets, replace them with environment variables or secret management: + +**Before:** +```python +API_KEY = "sk_live_1234567890abcdef" +``` + +**After:** +```python +import os +API_KEY = os.environ.get("API_KEY") +``` + +**In GitHub Actions:** +```yaml +env: + API_KEY: ${{ secrets.API_KEY }} +``` + +### 4. Remove from Git History + +If the secret was already committed: + +```bash +# For the most recent commit +git reset --soft HEAD~1 +# Make corrections +git add . +git commit -m "Remove exposed secret" + +# For older commits, consider using git-filter-repo or BFG Repo-Cleaner +# https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/removing-sensitive-data-from-a-repository +``` + +⚠️ **Important:** After removing a secret from git history, **you must rotate it** because it may still be accessible in previous commits. + +### 5. Managing False Positives + +If Gitleaks incorrectly flags something as a secret, add it to the allowlist in `.gitleaks.toml`: + +**By Path:** +```toml +[allowlist] +paths = [ + '''tests/fixtures/mock_credentials.py''', + '''examples/sample_config.yaml''' +] +``` + +**By Regex Pattern:** +```toml +[allowlist] +regexes = [ + '''example-key-12345''', + '''test-token-abc''' +] +``` + +**By Commit:** +```toml +[allowlist] +commits = [ + '''a1b2c3d4e5f6''' +] +``` + +**By Stopword:** +```toml +[allowlist] +stopwords = [ + '''placeholder''', + '''example''' +] +``` + +After updating `.gitleaks.toml`, commit the changes and re-run the workflow. + +## Local Testing + +Test for secrets locally before pushing to avoid CI failures: + +### Install Gitleaks + +**macOS:** +```bash +brew install gitleaks +``` + +**Linux:** +```bash +# Download latest release (replace vX.Y.Z with current version from releases page) +# Visit https://github.com/gitleaks/gitleaks/releases for the latest version +wget https://github.com/gitleaks/gitleaks/releases/download/vX.Y.Z/gitleaks_X.Y.Z_linux_x64.tar.gz +tar -xzf gitleaks_X.Y.Z_linux_x64.tar.gz +sudo mv gitleaks /usr/local/bin/ +``` + +**Windows:** +```powershell +# Using Chocolatey +choco install gitleaks + +# Or download from GitHub releases +``` + +### Run Gitleaks Locally + +**Scan uncommitted changes:** +```bash +gitleaks detect --source . --verbose +``` + +**Scan entire repository:** +```bash +gitleaks detect --source . --log-opts="--all" --verbose +``` + +**Scan with configuration file:** +```bash +gitleaks detect --source . --config .gitleaks.toml --verbose +``` + +**Scan a specific commit range:** +```bash +gitleaks detect --source . --log-opts="main..feature-branch" --verbose +``` + +### Pre-commit Hook (Optional) + +Automatically scan for secrets before each commit: + +```bash +# Create pre-commit hook +cat > .git/hooks/pre-commit << 'EOF' +#!/bin/sh +gitleaks protect --verbose --redact --staged +EOF + +# Make it executable +chmod +x .git/hooks/pre-commit +``` + +## Best Practices + +1. **Never commit secrets** - Use environment variables, secret managers (AWS Secrets Manager, Azure Key Vault, HashiCorp Vault) +2. **Use `.env` files locally** - Add `.env` to `.gitignore` (already configured in this repo) +3. **Rotate exposed secrets immediately** - Assume any committed secret is compromised +4. **Review allowlist regularly** - Ensure false positives haven't become real secrets +5. **Educate team members** - Share this documentation with all contributors +6. **Use GitHub Secrets** - For CI/CD workflows, store secrets in GitHub repository settings + +## Configuration Files + +- **`.github/workflows/secret-scanning.yml`** - GitHub Actions workflow definition +- **`.gitleaks.toml`** - Gitleaks configuration with allowlist rules + +## Resources + +- [Gitleaks Documentation](https://github.com/gitleaks/gitleaks) +- [GitHub Secret Scanning](https://docs.github.com/en/code-security/secret-scanning) +- [Removing Sensitive Data from Git](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/removing-sensitive-data-from-a-repository) +- [OWASP Secrets Management Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/Secrets_Management_Cheat_Sheet.html) + +## Support + +For issues or questions about secret scanning: +1. Check the [workflow logs](../../actions/workflows/secret-scanning.yml) for detailed error messages +2. Review this documentation for remediation steps +3. Open an issue in this repository if you need assistance diff --git a/.github/workflows/secret-scanning.yml b/.github/workflows/secret-scanning.yml new file mode 100644 index 0000000..d00f598 --- /dev/null +++ b/.github/workflows/secret-scanning.yml @@ -0,0 +1,26 @@ +name: Secret Scanning + +on: + push: + branches: ['**'] + pull_request: + branches: ['**'] + +permissions: + contents: read + +jobs: + gitleaks: + name: Gitleaks Secret Scan + runs-on: ubuntu-latest + steps: + - name: Checkout code + uses: actions/checkout@v4 + with: + fetch-depth: 0 + + - name: Run Gitleaks + uses: gitleaks/gitleaks-action@v2 + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + GITLEAKS_LICENSE: ${{ secrets.GITLEAKS_LICENSE }} diff --git a/.gitleaks.toml b/.gitleaks.toml new file mode 100644 index 0000000..adf4aa4 --- /dev/null +++ b/.gitleaks.toml @@ -0,0 +1,42 @@ +# Gitleaks Configuration File +# This file extends the default Gitleaks rules with project-specific settings + +title = "Gitleaks Configuration for HAL Project" + +# Extend default Gitleaks rules +# The default rules cover: +# - API keys (AWS, Azure, Google Cloud, etc.) +# - Authentication tokens (GitHub, GitLab, Slack, etc.) +# - Private keys (RSA, SSH, PGP) +# - Database credentials +# - Passwords and secrets in various formats + +[extend] +# Use default Gitleaks rules as base +useDefault = true + +# Allowlist section for managing false positives +# Add paths, commits, regexes, or stopwords to exclude from scanning + +[allowlist] +# Example: Exclude test files with mock credentials +# paths = [ +# '''tests/fixtures/.*''', +# '''examples/.*''' +# ] + +# Example: Exclude specific commits (e.g., initial test data setup) +# commits = [ +# '''abc123def456''' +# ] + +# Example: Allowlist specific patterns that are false positives +# regexes = [ +# '''example-key-not-real''' +# ] + +# Example: Exclude files by extension +# stopwords = [ +# '''mock''', +# '''test''' +# ]