contribute back to open source project

.claude/skills/new-destination.md

@@ -0,0 +1,99 @@
+---
+description: Scaffold a new destination connector for Bizon
+---
+
+# New Destination Connector
+
+You are helping create a new Bizon destination connector. Destinations require **registration in 3 places**.
+
+## Workflow
+
+### Step 1: Gather Requirements
+
+Ask the user:
+1. **Destination name** (e.g., "postgres", "s3", "snowflake")
+2. **Connection parameters** needed (host, credentials, etc.)
+3. **Write method**: batch insert, streaming, file upload
+
+### Step 2: Read the Templates
+
+Before generating code, read these files:
+- `docs/ai-destination-guide.md` - Templates and placeholders
+- `bizon/connectors/destinations/logger/src/` - Simple reference
+- `bizon/destination/config.py` - See existing `DestinationTypes` enum
+
+### Step 3: Create Directory Structure
+
+```bash
+mkdir -p bizon/connectors/destinations/{dest_name}/src
+touch bizon/connectors/destinations/{dest_name}/src/__init__.py
+```
+
+### Step 4: Generate Files
+
+Create these files:
+
+1. **`src/config.py`**:
+   - Details config extending `AbstractDestinationDetailsConfig`
+   - Main config extending `AbstractDestinationConfig` with `Literal[DestinationTypes.X]`
+
+2. **`src/destination.py`**:
+   - Destination class extending `AbstractDestination`
+   - Implement `check_connection()`, `write_records()`, optionally `finalize()`
+
+### Step 5: Register in 3 Places (CRITICAL!)
+
+#### 5.1 Add Enum Value
+**File**: `bizon/destination/config.py`
+```python
+class DestinationTypes(str, Enum):
+    ...
+    {DEST_ENUM} = "{dest_name}"  # ADD THIS
+```
+
+#### 5.2 Update Union Type
+**File**: `bizon/common/models.py`
+- Add import at top: `from bizon.connectors.destinations.{dest_name}.src.config import {DestName}Config`
+- Add to `destination: Union[...]` field
+
+#### 5.3 Update Factory
+**File**: `bizon/destination/destination.py`
+- Add `elif config.name == DestinationTypes.{DEST_ENUM}:` branch in `get_destination()`
+
+### Step 6: Verify
+
+```bash
+# Format all modified files
+uv run ruff format bizon/connectors/destinations/{dest_name}
+uv run ruff format bizon/destination/config.py
+uv run ruff format bizon/common/models.py
+uv run ruff format bizon/destination/destination.py
+
+# Run tests
+uv run pytest tests/ -v
+```
+
+## Quick Reference
+
+| Method | Returns | Purpose |
+|--------|---------|---------|
+| `check_connection()` | `bool` | Test connectivity |
+| `write_records(df)` | `Tuple[bool, str\|None]` | Write DataFrame |
+| `finalize()` | `bool` | Optional cleanup |
+
+## DataFrame Columns
+
+The `df_destination_records` DataFrame contains:
+- `bizon_id`: str - Unique record ID
+- `bizon_extracted_at`: datetime
+- `bizon_loaded_at`: datetime
+- `source_data`: str - JSON string of record
+- `source_record_id`: str
+
+## Registration Checklist
+
+Before finishing, verify all 3 registrations:
+- [ ] `DestinationTypes` enum has new value
+- [ ] `BizonConfig.destination` Union includes new config
+- [ ] `DestinationFactory.get_destination()` has elif branch
+- [ ] All 3 use exactly the same name string

.claude/skills/new-source.md

@@ -0,0 +1,81 @@
+---
+description: Scaffold a new source connector for Bizon
+---
+
+# New Source Connector
+
+You are helping create a new Bizon source connector. Sources are auto-discovered - no registration needed!
+
+## Workflow
+
+### Step 1: Gather Requirements
+
+Ask the user:
+1. **Source name** (e.g., "stripe", "hubspot", "salesforce")
+2. **Streams to sync** (e.g., "users", "orders", "contacts")
+3. **API documentation URL** (optional but helpful)
+4. **Authentication type**: API key, OAuth, Basic, or none
+
+### Step 2: Read the Templates
+
+Before generating code, read these files for patterns:
+- `docs/ai-connector-guide.md` - Templates and decision trees
+- `docs/reference-connector.md` - Annotated production example
+- `bizon/connectors/sources/dummy/src/source.py` - Simple reference
+
+### Step 3: Create Directory Structure
+
+```bash
+mkdir -p bizon/connectors/sources/{source_name}/src
+mkdir -p bizon/connectors/sources/{source_name}/config
+touch bizon/connectors/sources/{source_name}/src/__init__.py
+```
+
+### Step 4: Generate Files
+
+Create these files:
+
+1. **`src/source.py`** - Main implementation with:
+   - Config class extending `SourceConfig`
+   - Source class extending `AbstractSource`
+   - All required methods: `streams()`, `get_config_class()`, `get_authenticator()`, `check_connection()`, `get_total_records_count()`, `get()`
+
+2. **`config/{source_name}.example.yml`** - Example YAML config
+
+### Step 5: Key Implementation Details
+
+- `streams()` returns `List[str]` of available stream names
+- `get()` returns `SourceIteration(records=[...], next_pagination={...})`
+- `next_pagination` must be `{}` (empty dict) when no more pages
+- Each record needs unique `id`: `SourceRecord(id=item["id"], data=item)`
+
+### Step 6: Verify
+
+```bash
+# Format
+uv run ruff format bizon/connectors/sources/{source_name}
+
+# Verify discovery
+uv run bizon source list
+uv run bizon stream list {source_name}
+
+# Test
+uv run bizon run bizon/connectors/sources/{source_name}/config/{source_name}.example.yml
+```
+
+## Quick Reference
+
+| Method | Returns | Purpose |
+|--------|---------|---------|
+| `streams()` | `List[str]` | Available streams |
+| `get_config_class()` | `SourceConfig` | Config class |
+| `get_authenticator()` | `AuthBase` or `None` | Auth handler |
+| `check_connection()` | `Tuple[bool, str\|None]` | Test connectivity |
+| `get_total_records_count()` | `int\|None` | Total records |
+| `get(pagination)` | `SourceIteration` | Fetch records |
+
+## Remember
+
+- Sources are **auto-discovered** - no registration needed
+- Empty `__init__.py` is required in `src/`
+- Return `{}` for `next_pagination` when done paginating

.claude/skills/run-checks.md

@@ -0,0 +1,74 @@
+---
+description: Run formatting, linting, and tests before committing
+---
+
+# Run Pre-Commit Checks
+
+Run the standard checks before committing code.
+
+## Quick Command
+
+```bash
+make format && uv run pytest tests/ -v
+```
+
+## Individual Commands
+
+### Format Code
+```bash
+uv run ruff format .
+```
+
+### Lint and Auto-Fix
+```bash
+uv run ruff check --fix .
+```
+
+### Run All Tests
+```bash
+uv run pytest tests/ -v
+```
+
+### Run Specific Tests
+```bash
+# Single file
+uv run pytest tests/path/to/test_file.py -v
+
+# Single test
+uv run pytest tests/path/to/test_file.py -k "test_name" -v
+
+# With coverage
+uv run pytest --cov=bizon tests/
+```
+
+## Makefile Shortcuts
+
+```bash
+make format    # Ruff format + lint
+make lint      # Lint only
+make test      # Run pytest
+make install   # Full dev install
+```
+
+## Common Fixes
+
+### Import Order Issues
+Ruff handles this automatically with `--fix`
+
+### Line Too Long (>120 chars)
+- Use parentheses for continuation
+- Break long strings
+- Add `# noqa: E501` if unavoidable
+
+### Type Hints
+Use `List`, `Dict`, `Optional` from `typing` for Python 3.9 compatibility
+
+## Before Committing
+
+1. Run `make format`
+2. Run `uv run pytest`
+3. Check for any remaining linting issues
+4. Commit with conventional message format:
+   - `feat(source): add Stripe connector`
+   - `fix(destination): handle timeout`
+   - `docs: update contributing guide`

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,37 @@
+---
+name: Bug Report
+about: Report a bug in Bizon
+title: '[BUG] '
+labels: bug
+---
+
+## Description
+<!-- Clear description of the bug -->
+
+## Steps to Reproduce
+1.
+2.
+3.
+
+## Expected Behavior
+<!-- What should happen -->
+
+## Actual Behavior
+<!-- What actually happens -->
+
+## Environment
+- Bizon version:
+- Python version:
+- OS:
+- Queue type (if relevant):
+- Backend type (if relevant):
+
+## Config (if relevant)
+```yaml
+# Paste your config (remove sensitive data)
+```
+
+## Error Message/Logs
+```
+# Paste relevant logs
+```

.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,21 @@
+---
+name: Feature Request
+about: Suggest a new feature
+title: '[FEATURE] '
+labels: enhancement
+---
+
+## Summary
+<!-- Brief description of the feature -->
+
+## Problem/Motivation
+<!-- What problem does this solve? -->
+
+## Proposed Solution
+<!-- How should it work? -->
+
+## Alternatives Considered
+<!-- Other approaches you've thought about -->
+
+## Additional Context
+<!-- Any other information -->

.github/ISSUE_TEMPLATE/new_connector.md

@@ -0,0 +1,25 @@
+---
+name: New Connector Request
+about: Request a new source or destination connector
+title: '[CONNECTOR] '
+labels: connector
+---
+
+## Connector Type
+- [ ] Source (extract data from)
+- [ ] Destination (load data to)
+
+## Service/API Name
+<!-- e.g., Stripe, Snowflake, etc. -->
+
+## API Documentation
+<!-- Link to API docs -->
+
+## Authentication
+<!-- How does auth work? API key, OAuth, etc. -->
+
+## Priority Streams/Tables
+<!-- What data would you sync? -->
+
+## Use Case
+<!-- Why do you need this connector? -->

.github/PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,28 @@
+## Summary
+<!-- Brief description of changes -->
+
+## Type of Change
+- [ ] Bug fix
+- [ ] New feature
+- [ ] New connector (source/destination)
+- [ ] Documentation
+- [ ] Refactoring
+- [ ] Other: ___
+
+## Changes Made
+<!-- Bullet points of what changed -->
+-
+
+## Testing
+<!-- How did you test this? -->
+- [ ] Unit tests added/updated
+- [ ] Manual testing performed
+- [ ] Existing tests pass (`uv run pytest`)
+
+## Checklist
+- [ ] Code follows project style (`make format`)
+- [ ] Self-review completed
+- [ ] Documentation updated (if needed)
+
+## Related Issues
+<!-- Link any related issues: Fixes #123 -->