Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
166 changes: 166 additions & 0 deletions .claude/commands/swp/dev/dataframe-audit.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,166 @@
---
description: Audit DataFrame usage patterns across the SolarWindPy codebase
---

## DataFrame Patterns Audit: $ARGUMENTS

### Overview

Audit SolarWindPy code for compliance with DataFrame conventions:
- MultiIndex structure (M/C/S columns)
- Memory-efficient access patterns (.xs())
- Level operation patterns

**Default Scope:** `solarwindpy/`
**Custom Scope:** Pass path as argument (e.g., `solarwindpy/core/`)

### Pattern Catalog

**1. Level Selection with .xs()**
```python
# Preferred: Returns view, memory-efficient
df.xs('p1', axis=1, level='S')
df.xs(('n', '', 'p1'), axis=1)

# Avoid: Creates copy, wastes memory
df[df.columns.get_level_values('S') == 'p1']
```

**2. Level Reordering Chain**
```python
# Required pattern after concat/manipulation
df.reorder_levels(['M', 'C', 'S'], axis=1).sort_index(axis=1)
```

**3. Level-Specific Operations**
```python
# Preferred: Broadcasts correctly across levels
df.multiply(series, axis=1, level='C')
df.pow(exp, axis=1, level='C')
df.drop(['p1'], axis=1, level='S')
```

**4. Groupby Transpose Pattern (pandas 2.0+)**
```python
# Deprecated (pandas < 2.0)
df.sum(axis=1, level='S')

# Required (pandas >= 2.0)
df.T.groupby(level='S').sum().T
```

**5. Column Duplication Prevention**
```python
# Check before concat
if new.columns.isin(existing.columns).any():
raise ValueError("Duplicate columns")

# Remove duplicates after operations
df.loc[:, ~df.columns.duplicated()]
```

**6. Empty String Conventions**
```python
# Scalars: empty component
('n', '', 'p1') # density for p1

# Magnetic field: empty species
('b', 'x', '') # Bx component

# Spacecraft: empty species
('pos', 'x', '') # position x
```

### Audit Execution

**Primary Method: ast-grep (recommended)**

ast-grep provides structural pattern matching for more accurate detection:

```bash
# Install ast-grep if not available
# macOS: brew install ast-grep
# pip: pip install ast-grep-py
# cargo: cargo install ast-grep

# Run full audit with all DataFrame rules
sg scan --config tools/dev/ast_grep/dataframe-patterns.yml solarwindpy/

# Run specific rule only
sg scan --config tools/dev/ast_grep/dataframe-patterns.yml --rule swp-df-003 solarwindpy/
```

**Fallback Method: grep (if ast-grep unavailable)**

If ast-grep is not installed, use grep for basic pattern detection:

```bash
# .xs() usage (informational)
grep -rn "\.xs(" solarwindpy/

# reorder_levels usage (check for missing sort_index)
grep -rn "reorder_levels" solarwindpy/

# Deprecated level= aggregation (pandas 2.0+)
grep -rn "axis=1, level=" solarwindpy/

# Boolean indexing anti-pattern
grep -rn "get_level_values" solarwindpy/
```

**Step 2: Check for violations**
- `swp-df-001`: Boolean indexing instead of .xs()
- `swp-df-002`: reorder_levels without sort_index
- `swp-df-003`: axis=1, level= aggregation (deprecated)
- `swp-df-004`: MultiIndex without standard names
- `swp-df-005`: Missing column duplicate checks
- `swp-df-006`: multiply without level= parameter

**Step 3: Report findings**

| File | Line | Rule ID | Issue | Severity |
|------|------|---------|-------|----------|
| ... | ... | swp-df-XXX | ... | warn/info |

### Contract Tests Reference

The following contracts validate DataFrame structure:

1. **MultiIndex names**: `columns.names == ['M', 'C', 'S']`
2. **DatetimeIndex row**: `isinstance(df.index, pd.DatetimeIndex)`
3. **xs returns view**: `not result._is_copy`
4. **No duplicate columns**: `not df.columns.duplicated().any()`
5. **Sorted after reorder**: `df.columns.is_monotonic_increasing`

### Output Format

```markdown
## DataFrame Patterns Audit Report

**Scope:** <path>
**Date:** <date>

### Summary
| Pattern | Files | Issues |
|---------|-------|--------|
| xs-usage | X | Y |
| reorder-levels | X | Y |
| groupby-transpose | X | Y |

### Issues Found

#### xs-usage (N issues)
1. **file.py:line**
- Issue: Boolean indexing instead of .xs()
- Current: `df[df.columns.get_level_values('S') == 'p1']`
- Suggested: `df.xs('p1', axis=1, level='S')`

[...]
```

---

**Reference Documentation:**
- `tmp/copilot-plan/dataframe-patterns.md` - Full specification
- `tests/test_contracts_dataframe.py` - Contract test suite
- `tools/dev/ast_grep/dataframe-patterns.yml` - ast-grep rules
126 changes: 126 additions & 0 deletions .claude/commands/swp/dev/fix-tests.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,126 @@
---
description: Diagnose and fix failing tests with guided recovery
---

## Fix Tests Workflow: $ARGUMENTS

### Phase 1: Test Execution & Analysis

Run the failing test(s):
```bash
pytest <test_path> -v --tb=short
```

Parse pytest output to extract:
- **Test name**: Function that failed
- **Status**: FAILED, ERROR, SKIPPED
- **Assertion**: What was expected vs actual
- **Traceback**: File, line number, context

### Phase 2: Failure Categorization

**Category A: Assertion Failures (Logic Errors)**
- Pattern: `AssertionError: <message>`
- Cause: Code doesn't match test specification
- Action: Review implementation against test assertion

**Category B: Physics Constraint Violations**
- Pattern: "convention violated", "conservation", "must be positive"
- Cause: Implementation breaks physics rules
- Action: Check SI units, formula correctness, edge cases
- Reference: `.claude/templates/test-patterns.py` for correct formulas

**Category C: DataFrame/Data Structure Errors**
- Pattern: `KeyError`, `IndexError`, `ValueError: incompatible shapes`
- Cause: MultiIndex structure mismatch or incorrect level access
- Action: Review MultiIndex level names (M/C/S), use `.xs()` instead of `.copy()`

**Category D: Coverage Gaps**
- Pattern: Tests pass but coverage below 95%
- Cause: Edge cases or branches not exercised
- Action: Add tests for boundary conditions, NaN handling, empty inputs

**Category E: Type/Import Errors**
- Pattern: `ImportError`, `AttributeError: has no attribute`
- Cause: Interface mismatch or incomplete implementation
- Action: Verify function exists, check import paths

**Category F: Timeout/Performance**
- Pattern: `timeout after XXs`, tests stalled
- Cause: Inefficient algorithm or infinite loop
- Action: Profile, optimize NumPy operations, add `@pytest.mark.slow`

### Phase 3: Targeted Fixes

**For Logic Errors:**
1. Extract expected vs actual values
2. Locate implementation (grep for function name)
3. Review line-by-line against test
4. Fix discrepancy

**For Physics Violations:**
1. Identify violated law (thermal speed, Alfvén, conservation)
2. Look up correct formula in:
- `.claude/docs/DEVELOPMENT.md` (physics rules)
- `.claude/templates/test-patterns.py` (reference formulas)
3. Verify SI units throughout
4. Fix formula using correct physics

**For DataFrame Errors:**
1. Check MultiIndex structure: `df.columns.names` should be `['M', 'C', 'S']`
2. Replace `.copy()` with `.xs()` for level selection
3. Use `.xs(key, level='Level')` instead of positional indexing
4. Verify level values match expected (n, v, w, b for M; x, y, z, par, per for C)

**For Coverage Gaps:**
1. Get missing line numbers from coverage report
2. Identify untested code path
3. Create test case for that path:
- `test_<function>_empty_input`
- `test_<function>_nan_handling`
- `test_<function>_boundary`

### Phase 4: Re-Test Loop

After fixes:
```bash
pytest <specific_test> -v # Verify fix
.claude/hooks/test-runner.sh --changed # Run affected tests
```

Repeat Phases 2-4 until all tests pass.

### Phase 5: Completion

**Success Criteria:**
- [ ] All target tests passing
- [ ] No regressions (previously passing tests still pass)
- [ ] Coverage maintained (≥95% for changed modules)
- [ ] Physics validation complete (if applicable)

**Output Summary:**
```
Tests Fixed: X/X now passing
Regression Check: ✅ No broken tests
Coverage: XX.X% (maintained)

Changes Made:
• <file>: <fix description>
• <file>: <fix description>

Physics Validation:
✅ Thermal speed convention
✅ Unit consistency
✅ Missing data handling
```

---

**Quick Reference - Common Fixes:**

| Error Pattern | Likely Cause | Fix |
|--------------|--------------|-----|
| `KeyError: 'p1'` | Wrong MultiIndex level | Use `.xs('p1', level='S')` |
| `ValueError: shapes` | DataFrame alignment | Check `.reorder_levels().sort_index()` |
| `AssertionError: thermal` | Wrong formula | Use `sqrt(2 * k_B * T / m)` |
| Coverage < 95% | Missing edge cases | Add NaN, empty, boundary tests |
95 changes: 95 additions & 0 deletions .claude/commands/swp/dev/implement.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
---
description: Implement a feature or fix from description through passing tests
---

## Implementation Workflow: $ARGUMENTS

### Phase 1: Analysis & Planning

Analyze the implementation request:
- **What**: Identify the specific modification needed
- **Where**: Locate target module(s) and file(s) in solarwindpy/
- **Why**: Understand purpose and validate physics alignment (if core/instabilities)

**Target Module Mapping:**
- Physics calculations → `solarwindpy/core/` or `solarwindpy/instabilities/`
- Curve fitting → `solarwindpy/fitfunctions/`
- Visualization → `solarwindpy/plotting/`
- Utilities → `solarwindpy/tools/`

Search for existing patterns and implementations:
1. Grep for similar functionality
2. Review module structure
3. Identify integration points

Create execution plan:
- Files to create/modify
- Test strategy (unit, integration, physics validation)
- Coverage targets (≥95% for core/instabilities)

### Phase 2: Implementation

Follow SolarWindPy conventions:
- **Docstrings**: NumPy style with parameters, returns, examples
- **Units**: SI internally (see physics rules below)
- **Code style**: Black (88 chars), Flake8 compliant
- **Missing data**: Use NaN (never 0 or -999)

**Physics Rules (for core/ and instabilities/):**
- Thermal speed convention: mw² = 2kT
- SI units: m/s, kg, K, Pa, T, m³
- Conservation laws: Validate mass, energy, momentum
- Alfvén speed: V_A = B/√(μ₀ρ) with proper composition

Create test file mirroring source structure:
- Source: `solarwindpy/core/ions.py` → Test: `tests/core/test_ions.py`

### Phase 3: Hook Validation Loop

After each edit, hooks automatically run:
```
PostToolUse → test-runner.sh --changed → pytest for modified files
```

Monitor test results. If tests fail:
1. Parse pytest output for failure type
2. Categorize: Logic error | Physics violation | DataFrame issue | Coverage gap
3. Fix targeted issue
4. Re-test automatically on next edit

**Recovery Guide:**
- **AssertionError**: Check implementation against test expectation
- **Physics constraint violation**: Verify SI units and formula correctness
- **ValueError/KeyError**: Check MultiIndex structure (M/C/S levels), use .xs()
- **Coverage below 95%**: Add edge case tests (empty input, NaN handling, boundaries)

### Phase 4: Completion

Success criteria:
- [ ] All tests pass
- [ ] Coverage ≥95% (core/instabilities) or ≥85% (plotting)
- [ ] Physics validation passed (if applicable)
- [ ] Conventional commit message ready

**Output Summary:**
```
Files Modified: [list]
Test Results: X/X passed
Coverage: XX.X%
Physics Validation: ✅/❌

Suggested Commit:
git add <files>
git commit -m "feat(<module>): <description>

🤖 Generated with Claude Code
Co-Authored-By: Claude <noreply@anthropic.com>"
```

---

**Execution Notes:**
- Hooks are the "Definition of Done" - no separate validation needed
- Use `test-runner.sh --physics` for core/instabilities modules
- Reference `.claude/templates/test-patterns.py` for test examples
- Check `.claude/docs/DEVELOPMENT.md` for detailed conventions
Loading
Loading