Add documentation for .otignore feature (OT-1318)

docs/integrations/github.md

@@ -21,6 +21,8 @@ Connect GitHub to sync repository data and analyze code structure.
 - Code symbols and dependencies
 - Issues
 
+You can exclude specific files or directories from analysis using [`.otignore` files](../otignore.md).
+
 ## Permissions Required
 
 OpenTrace requests read-only access to:

docs/integrations/gitlab.md

@@ -21,6 +21,8 @@ Connect GitLab to sync repository data and analyze code structure.
 - Code symbols and dependencies
 - Issues
 
+You can exclude specific files or directories from analysis using [`.otignore` files](../otignore.md).
+
 ## Self-Hosted GitLab
 
 For self-hosted GitLab instances, you may need to configure the GitLab URL in your OpenTrace settings before connecting.

docs/otignore.md

@@ -0,0 +1,364 @@
+# Excluding Files with .otignore
+
+## Overview
+
+OpenTrace analyzes your repository to build a knowledge graph of your codebase. While this analysis is valuable, you may want to exclude certain files or directories from being analyzed. The `.otignore` file allows you to specify which files and directories OpenTrace should skip during analysis.
+
+### Why Use .otignore?
+
+Common scenarios for using `.otignore`:
+
+- **Generated code**: Protobuf files, specs, or code generated by tools
+- **Third-party dependencies**: Vendor directories, or other package directories
+- **Secrets**: API tokens, keys or similar pieces of information
+- **Large binary files**: Archives, images, or other non-code assets
+- **Performance optimization**: Reduce analysis time for large repositories
+
+### Difference to .gitignore
+
+`.otignore` serves a different purpose than `.gitignore`:
+
+- **`.gitignore`**: Controls which files Git tracks and commits
+- **`.otignore`**: Controls which *committed* files OpenTrace analyzes
+
+Since OpenTrace only has access to files that are already committed to your repository, `.otignore` helps you exclude committed files that you don't want analyzed (like vendored dependencies or generated code that you choose to commit).
+
+## Quick Start
+
+To start using `.otignore`:
+
+1. Create a file named `.otignore` (or `.opentraceignore`) in your repository root
+2. Add patterns for files or directories to exclude (uses gitignore syntax)
+3. Commit the file to your repository
+4. The next OpenTrace analysis will automatically respect your exclusions
+
+**Example `.otignore` file:**
+
+```
+# Exclude generated protobuf code
+*.pb.go
+*.pb.py
+
+# Exclude vendored dependencies
+/vendor/
+/node_modules/
+
+# Exclude build outputs
+/dist/
+/build/
+```
+
+## Configuration
+
+### Supported Filenames
+
+OpenTrace recognizes two filenames for ignore patterns:
+
+- `.otignore` (recommended)
+- `.opentraceignore` (alternative)
+
+If both files exist, patterns from both will be applied. Patterns from `.otignore` are loaded first, followed by patterns from `.opentraceignore`.
+
+### File Location
+
+The ignore file must be placed in the **repository root directory**. Ignore files in subdirectories are not currently supported.
+
+### Default Exclusions
+
+OpenTrace automatically excludes certain directories from analysis, even without an `.otignore` file. These directories are always skipped:
+
+- `.git` - Git repository metadata
+- `.venv`, `venv` - Python virtual environments
+- `node_modules` - Node.js dependencies
+- `__pycache__` - Python bytecode cache
+- `.pytest_cache` - Pytest cache
+- `.mypy_cache` - MyPy type checker cache
+- `.tox` - Tox testing environments
+- `dist` - Distribution/build output
+- `build` - Build artifacts
+- `.eggs` - Python egg artifacts
+- `vendor` - Vendored dependencies
+
+These default exclusions work alongside your `.otignore` patterns. You don't need to add these directories to your `.otignore` file - they're already excluded automatically.
+
+!!! note "Why these defaults?"
+    These directories typically contain dependencies, build artifacts, or caching data that adds noise to analysis without providing value. Excluding them by default improves performance and keeps your knowledge graph focused on your actual source code.
+
+## Pattern Syntax
+
+`.otignore` uses the same pattern syntax as `.gitignore`, powered by the `pathspec` library. This ensures familiar and predictable behavior.
+
+### Basic Patterns
+
+| Pattern | Matches |
+|---------|---------|
+| `file.txt` | Specific file named `file.txt` in any directory |
+| `*.log` | All files ending with `.log` |
+| `test` | Any file or directory named `test` |
+
+### Directory Patterns
+
+| Pattern | Matches |
+|---------|---------|
+| `logs/` | Directory named `logs` (trailing slash required) |
+| `/build/` | Directory named `build` only in repository root |
+| `**/temp/` | Directory named `temp` at any depth |
+
+### Wildcards
+
+| Wildcard | Meaning |
+|----------|---------|
+| `*` | Matches any characters except `/` |
+| `**` | Matches any characters including `/` (any depth) |
+| `?` | Matches exactly one character |
+
+**Examples:**
+
+```
+# Match all .log files
+*.log
+
+# Match all files in any __pycache__ directory
+**/__pycache__/**
+
+# Match .js files in any test directory
+**/test/*.js
+```
+
+### Negation Patterns
+
+Use `!` to re-include files that were previously excluded:
+
+```
+# Exclude all .txt files
+*.txt
+
+# But include this important one
+!important.txt
+```
+
+### Comments
+
+Lines starting with `#` are treated as comments and ignored:
+
+```
+# This is a comment explaining the next pattern
+*.tmp
+```
+
+## Examples
+
+### Example 1: Python Project
+
+```
+# Exclude Python generated files
+**/__pycache__/
+*.pyc
+*.pyo
+*.pyd
+.Python
+
+# Exclude virtual environments
+/venv/
+/.venv/
+/env/
+
+# Exclude generated protobuf code
+*_pb2.py
+*_pb2_grpc.py
+```
+
+### Example 2: JavaScript/TypeScript Project
+
+```
+# Exclude dependencies
+/node_modules/
+/bower_components/
+
+# Exclude build outputs
+/dist/
+/build/
+/.next/
+/out/
+
+# Exclude generated files
+*.generated.ts
+```
+
+### Example 3: Go Project
+
+```
+# Exclude vendored dependencies
+/vendor/
+
+# Exclude generated protobuf code
+*.pb.go
+
+# Exclude compiled binaries
+/bin/
+*.exe
+```
+
+### Example 4: Monorepo
+
+```
+# Exclude all node_modules in monorepo
+**/node_modules/
+
+# Exclude all dist directories
+**/dist/
+
+# Exclude specific generated service code
+services/api/generated/
+services/auth/generated/
+
+# But include the schema definitions
+!services/*/generated/schema.yaml
+```
+
+## Common Use Cases
+
+### Excluding Protobuf/OpenAPI Generated Code
+
+Generated API code often creates noise in analysis results:
+
+```
+# Protocol Buffers
+*.pb.go
+*.pb.py
+*_pb2.py
+*_pb2_grpc.py
+
+# OpenAPI/Swagger
+/generated/openapi/
+**/swagger-generated/
+```
+
+### Ignoring Vendored Dependencies
+
+Third-party code adds unnecessary complexity to your knowledge graph:
+
+```
+# Go vendor directory
+/vendor/
+
+# JavaScript/Node
+/node_modules/
+
+# Ruby gems
+/vendor/bundle/
+
+# Python packages
+/site-packages/
+```
+
+### Excluding Test Fixtures and Mocks
+
+Large test data files can slow down analysis:
+
+```
+# Test fixtures
+**/fixtures/
+**/testdata/
+
+# Mock data
+**/mocks/
+**/__mocks__/
+```
+
+### Performance Optimization for Large Repos
+
+For very large repositories, exclude non-essential directories:
+
+```
+# Documentation site builds
+/docs/site/
+/docusaurus/build/
+
+# IDE configurations (if committed)
+/.vscode/
+/.idea/
+
+# Large binary assets
+/assets/images/
+/public/uploads/
+```
+
+## Troubleshooting
+
+### Pattern Not Working?
+
+**Check your syntax:**
+
+- Ensure patterns follow gitignore format
+- Directory patterns need a trailing `/`
+- Use `/` as the path separator (even on Windows)
+- Test your patterns with a gitignore validator
+
+**Example issue:**
+```
+# ❌ Won't match directories
+node_modules
+
+# ✅ Correctly matches directories
+node_modules/
+```
+
+### File Still Appears in Analysis?
+
+**Check file location:**
+
+- Ensure `.otignore` is in the repository root, not a subdirectory
+- Verify the file is named exactly `.otignore` or `.opentraceignore`
+
+### Changes Not Taking Effect?
+
+**Commit your .otignore file:**
+
+- Changes only apply after the `.otignore` file is committed
+- OpenTrace reads the ignore file from the committed repository
+- Run `git add .otignore && git commit -m "Update otignore patterns"`
+
+**Wait for next analysis:**
+
+- Changes apply to new analysis runs, not retroactively
+- Trigger a new repository sync in OpenTrace to apply changes
+
+## Technical Details
+
+### Pattern Matching Implementation
+
+OpenTrace uses the `pathspec` library with gitignore-style pattern matching, ensuring behavior identical to `.gitignore` files. Patterns are applied to relative paths from the repository root.
+
+### Scope of Exclusions
+
+The `.otignore` file affects all Git-based integrations:
+
+- GitHub repository analysis
+- GitLab repository analysis
+- Any future Git-based source integrations
+
+Files excluded by `.otignore` will not appear in:
+
+- The knowledge graph
+- Code search results
+- Dependency analysis
+- Investigation context
+
+### Performance Impact
+
+Using `.otignore` to exclude large directories can significantly improve:
+
+- Analysis speed
+- Memory usage during analysis
+- Knowledge graph size and query performance
+
+For repositories with thousands of files, excluding generated code and dependencies can reduce analysis time by 50% or more.
+
+## Best Practices
+
+1. **Commit your .otignore early**: Add it when you first set up OpenTrace for your repository
+2. **Start broad, refine later**: Begin by excluding obvious directories like `node_modules/`, then add more specific patterns as needed
+3. **Document your patterns**: Use comments to explain why certain paths are excluded
+4. **Review periodically**: As your repository evolves, update `.otignore` to reflect new generated code or dependencies
+5. **Keep it simple**: Don't over-optimize - exclude only what meaningfully impacts analysis quality or performance

mkdocs.yml

@@ -78,6 +78,7 @@ nav:
     - Data Sources:
       - GitHub: integrations/github.md
       - GitLab: integrations/gitlab.md
+      - Excluding Files from Data Sources: otignore.md
       - AWS EKS (Early Access): integrations/aws-eks.md
   - What You Can Do: capabilities.md
   - Example Workflows: workflows.md