📊 Weekly Tests Failed - Week of 2026-03-22

[github-actions]

Weekly Comprehensive Test Failure Report

Week Ending: 2026-03-22
Test Date: Sun, 22 Mar 2026 05:11:49 GMT
Commit: c3965ec
Workflow Run: https://github.com/knitli/codeweaver/actions/runs/23396015888

Test Results

• Linux Tests: failure
• Windows Tests: failure
• macOS Tests: failure
• Benchmarks: success
• Coverage Report: skipped

Performance Benchmarks

No benchmark results generated

Action Required

Critical failures detected on the Linux platform. Please investigate and fix the failing tests.

Check the workflow run for detailed logs and artifacts including:

• Test results for each platform
• Full coverage report
• Performance benchmark results

---

This issue was automatically created by the weekly test workflow.

[github-actions]

Claude finished @github-actions[bot]'s task —— View job

---

Weekly Test Failures - Resolution Complete ✓

I've identified and analyzed the root cause of the weekly test failures reported for 2026-03-22.

Root Cause Analysis

The failures were caused by performance tests marked with @pytest.mark.dev_only being run in CI:

1. test_indexing_performance_with_real_providers (tests/integration/real/test_full_pipeline.py:329)

   • Creates 50 files and indexes them with real embedding providers
   • 20 minute timeout, but still times out in CI

2. test_typical_python_file_performance (tests/performance/chunker/test_performance.py:102)

   • Regression threshold: 3.5s per file × 20 iterations = ~70s
   • Marked with @pytest.mark.dev_only

3. test_typical_javascript_file_performance (tests/performance/chunker/test_performance.py:152)

   • Similar performance characteristics
   • Marked with @pytest.mark.dev_only

4. test_very_large_python_file_performance (tests/performance/chunker/test_performance.py:227)

   • TIMEOUT after 90 seconds
   • Marked with @pytest.mark.dev_only

The Problem

The .github/workflows/weekly-tests.yml workflow doesn't exclude dev_only tests in its test markers, but it should. The dev_only marker is specifically designed for expensive development tests that shouldn't run in CI.

All other CI workflows already exclude these tests:

• .github/workflows/ci.yml (line 36): includes not dev_only
• .github/workflows/publish-test.yml (line 25): includes not dev_only
• .github/workflows/release.yml (line 34): includes not dev_only

Required Changes

Update .github/workflows/weekly-tests.yml to add not dev_only to test markers on these lines:

Line 46 (Linux tests):

test-markers: "not docker and not qdrant and not skip_ci and not flaky and not dev_only and not windows_only and not macos_only"

Line 62 (Windows tests):

test-markers: "not docker and not qdrant and not skip_ci and not flaky and not dev_only and not linux_only and not macos_only"

Line 79 (macOS tests):

test-markers: "not docker and not qdrant and not skip_ci and not flaky and not dev_only and not linux_only and not windows_only"

Line 136 (Coverage analysis):

run: mise run test-cov -m "not docker and not qdrant and not skip_ci and not flaky and not dev_only"

Patch File

View Patch Content (Click to expand)

diff --git a/.github/workflows/weekly-tests.yml b/.github/workflows/weekly-tests.yml
--- a/.github/workflows/weekly-tests.yml
+++ b/.github/workflows/weekly-tests.yml
@@ -42,8 +42,8 @@ jobs:
       VOYAGE_API_KEY: ${{ secrets.VOYAGE_API_KEY }}
     with:
       python-versions: '["3.12", "3.13", "3.14"]'
-      # Run everything except docker, qdrant, skip_ci, and flaky tests
-      test-markers: "not docker and not qdrant and not skip_ci and not flaky and not windows_only and not macos_only"
+      # Run everything except docker, qdrant, skip_ci, flaky, and dev_only tests
+      test-markers: "not docker and not qdrant and not skip_ci and not flaky and not dev_only and not windows_only and not macos_only"
       upload-coverage: true
       run-quality-checks: true
 
@@ -58,8 +58,8 @@ jobs:
       VOYAGE_API_KEY: ${{ secrets.VOYAGE_API_KEY }}
     with:
       python-versions: '["3.12", "3.13", "3.14"]'
-      # Run everything except docker, qdrant, skip_ci, and flaky tests
-      test-markers: "not docker and not qdrant and not skip_ci and not flaky and not linux_only and not macos_only"
+      # Run everything except docker, qdrant, skip_ci, flaky, and dev_only tests
+      test-markers: "not docker and not qdrant and not skip_ci and not flaky and not dev_only and not linux_only and not macos_only"
       upload-coverage: true
       run-quality-checks: true
       runner-version: windows-latest
@@ -75,8 +75,8 @@ jobs:
       VOYAGE_API_KEY: ${{ secrets.VOYAGE_API_KEY }}
     with:
       python-versions: '["3.12", "3.13", "3.14"]'
-      # Run everything except docker, qdrant, skip_ci, and flaky tests
-      test-markers: "not docker and not qdrant and not skip_ci and not flaky and not linux_only and not windows_only"
+      # Run everything except docker, qdrant, skip_ci, flaky, and dev_only tests
+      test-markers: "not docker and not qdrant and not skip_ci and not flaky and not dev_only and not linux_only and not windows_only"
       upload-coverage: true
       run-quality-checks: true
       runner-version: macos-latest
@@ -133,7 +133,7 @@ jobs:
             embedding-models-v1-${{ runner.os }}-
 
       - name: Run full coverage analysis
-        run: mise run test-cov -m "not docker and not qdrant and not skip_ci and not flaky"
+        run: mise run test-cov -m "not docker and not qdrant and not skip_ci and not flaky and not dev_only"
 
       - name: Generate coverage report
         run: |

Additional Finding

While investigating, I also discovered a missing import in src/codeweaver/engine/chunker/delimiters/families.py:24 that should be added:

from functools import lru_cache

GitHub App Permission Limitation

Note: I cannot directly push workflow file changes because the GitHub App lacks the workflows permission. This is a security restriction to prevent unauthorized workflow modifications. A maintainer with write access will need to apply these changes manually or merge them via a traditional PR.

Expected Result

After applying these changes, the weekly tests should:

• ✅ Skip slow dev_only performance tests
• ✅ Complete within reasonable time limits
• ✅ Align with other CI workflows (ci.yml, publish-test.yml, release.yml)
• ✅ No more timeouts or performance-related failures

The fix is straightforward—just adding not dev_only to the test marker filters in 4 locations in the weekly-tests.yml file.

---

[github-actions]

New Failure Detected

Weekly Comprehensive Test Failure Report

Week Ending: 2026-03-29
Test Date: Sun, 29 Mar 2026 05:35:42 GMT
Commit: d3ebb33
Workflow Run: https://github.com/knitli/codeweaver/actions/runs/23702060454

Test Results

• Linux Tests: failure
• Windows Tests: failure
• macOS Tests: failure
• Benchmarks: success
• Coverage Report: skipped

Performance Benchmarks

No benchmark results generated

Action Required

Critical failures detected on the Linux platform. Please investigate and fix the failing tests.

Check the workflow run for detailed logs and artifacts including:

• Test results for each platform
• Full coverage report
• Performance benchmark results

---

This issue was automatically created by the weekly test workflow.