feat: refactor repository routes to use unified cache service and fix data accuracy issues

[jonasyr]

Closes #120

Table of Contents

• Summary
• Phase 1: Core Refactoring (Issue #120)
  • Cache Service Extension
  • Repository Routes Refactoring
  • Data Accuracy Fixes
• Phase 2: Code Deduplication & Quality Improvements
  • Helper Extraction - Cache Operations
  • Helper Extraction - Route Operations
  • Route Factory Pattern
  • Utility Extraction
• Phase 3: Testing & Documentation
  • Test Enhancements
  • Documentation Created
• Breaking Changes ⚠️
• Migration Guide for Frontend Teams
• Test Results ✅
• Documentation Files
• Lessons Learned
• Acceptance Criteria
• Related Issues
• Testing Checklist
• How to Test
• Rollback Plan

---

Summary

This PR completely refactors all repository API routes to use the unified multi-tier cache service, removes manual Redis operations, migrates from POST to GET endpoints for RESTful compliance, and resolves critical data accuracy issues. Following the initial implementation, extensive code deduplication and quality improvements were applied based on SonarQube analysis.

Total changes: 28 commits with comprehensive refactoring, deduplication, and testing enhancements.

---

📊 View Architecture Migration Diagram

Architecture Overview: Legacy → Modern

%%{init: {'theme':'base', 'themeVariables': { 'primaryColor':'#e3f2fd','primaryTextColor':'#1565c0','primaryBorderColor':'#1976d2','lineColor':'#424242','secondaryColor':'#f3e5f5','tertiaryColor':'#fff3e0'}}}%%
flowchart TB
    subgraph BEFORE["❌ Legacy Architecture"]
        direction TB
        OLD_ROUTES["POST Routes
        (request body)"]
        OLD_CACHE["Manual Redis Cache
        (inconsistent keys)"]
        OLD_GIT["Temporary Git Clones
        (shallow, incomplete)"]
        
        OLD_ROUTES -->|direct Redis operations| OLD_CACHE
        OLD_CACHE -->|depth=1000 / 346 commits| OLD_GIT
    end
    
    subgraph AFTER["✅ New Architecture"]
        direction TB
        NEW_ROUTES["GET Routes
        (query parameters)"]
        UNIFIED_CACHE["Unified Cache Service
        (normalized keys)"]
        CACHE_TIERS["Multi-Tier Cache
        Memory → Disk → Redis → Git"]
        COORDINATED_GIT["Shared Git Repositories
        (blob filter, complete)"]
        
        NEW_ROUTES -->|unified interface| UNIFIED_CACHE
        UNIFIED_CACHE -->|automatic promotion| CACHE_TIERS
        CACHE_TIERS -->|blob:none / 480 commits| COORDINATED_GIT
    end
    
    subgraph IMPROVEMENTS["🎯 Key Improvements"]
        direction LR
        FIX1["✓ Complete History
        blob filtering"]
        FIX2["✓ Consistent Caching
        normalized keys"]
        FIX3["✓ Accurate Data
        sequential execution"]
    end
    
    BEFORE -.->|refactored to| AFTER
    COORDINATED_GIT -.->|enables| FIX1
    UNIFIED_CACHE -.->|ensures| FIX2
    CACHE_TIERS -.->|provides| FIX3
    
    classDef legacyStyle fill:#ffebee,stroke:#c62828,stroke-width:2px,color:#b71c1c
    classDef modernStyle fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px,color:#1b5e20
    classDef improvementStyle fill:#fff3e0,stroke:#ef6c00,stroke-width:2px,color:#e65100
    
    class BEFORE,OLD_ROUTES,OLD_CACHE,OLD_GIT legacyStyle
    class AFTER,NEW_ROUTES,UNIFIED_CACHE,CACHE_TIERS,COORDINATED_GIT modernStyle
    class IMPROVEMENTS,FIX1,FIX2,FIX3 improvementStyle

Loading

🖼️ View Architecture Diagram (PNG)

---

Phase 1: Core Refactoring (Issue #120)

Cache Service Extension (repositoryCache.ts)

Added getCachedChurnData() - Implemented churn analysis caching using aggregated data tier with TTL of 900s (15 minutes), using withSharedRepository() for efficient Git access.

Added getCachedSummary() - Integrated repositorySummaryService with unified cache, using TTL of 7200s (2 hours) for metadata stability and preserving sparse clone optimization.

Enhanced getOrGenerateAggregatedData() - Added extensive debug logging, null/empty commit handling, type guard validation for CommitHeatmapData, and fixed filter conversion to exclude undefined properties.

Repository Routes Refactoring (repositoryRoutes.ts)

All routes migrated to unified cache with POST → GET conversion:

• ✅ GET /commits - Uses getCachedCommits() with pagination
• ✅ GET /heatmap - Uses getCachedAggregatedData() with filters
• ✅ GET /contributors - Uses getCachedContributors() with filters
• ✅ GET /churn - Uses getCachedChurnData() with filters
• ✅ GET /summary - Uses getCachedSummary()
• ✅ GET /full-data - Uses both getCachedCommits() and getCachedAggregatedData()

New Features:

• Pagination support with page and limit query parameters
• Multi-tier caching: Memory → Disk → Redis → Git
• Automatic cache promotion for hot data
• Transactional consistency with automatic rollback
• Deadlock prevention via ordered lock acquisition
• Repository coordination to prevent duplicate clones

Data Accuracy Fixes

Fix 1: Incomplete Commit History - Changed from shallow clone --depth 1000 to blob filtering strategy (--filter=blob:none), fetching complete history while saving 95-99% bandwidth (480 commits vs 346).

Fix 2: Cache Key Mismatch - Fixed filter objects with undefined properties creating inconsistent cache keys by only including defined properties.

Fix 3: Lock Contention - Changed full-data endpoint from parallel to sequential execution to prevent cache corruption.

---

Phase 2: Code Deduplication & Quality Improvements

Following SonarQube analysis identifying 3.3% code duplication, systematic refactoring was performed:

Helper Extraction - Cache Operations

recordCacheHit() / recordCacheMiss() - Centralized cache hit/miss metrics recording, eliminating ~120 lines of duplication across 8 cache locations.

handleCacheHit() - Consolidated cache hit path with logging and metrics, eliminating ~60 lines across 4 cache methods.

handleCacheMiss() - Standardized cache miss recording and logging.

handleTransactionSuccess() - Consolidated successful transaction caching operations with uniform logging and metrics.

handleTransactionError() - Standardized transaction error handling with rollback, metrics, and structured logging.

safeCacheGet() - Safe cache retrieval with standardized error handling (cacheErrorHandlers.ts).

Helper Extraction - Route Operations

setupRouteRequest() - Extracted common request initialization (logger, repoUrl, userType) used by all routes.

recordRouteSuccess() - Standardized success path with metrics, logging, and HTTP 200 responses.

recordRouteError() - Uniform error handling with metrics, logging, and error propagation.

buildCommitFilters() - Builds CommitFilterOptions from query parameters, excluding undefined properties for consistent cache keys.

buildChurnFilters() - Builds ChurnFilterOptions with proper field mapping (fromDate → since, toDate → until).

extractPaginationParams() - Consistent pagination logic with defaults (page=1, limit=100).

Route Factory Pattern

createCachedRouteHandler() - Factory function that eliminates route duplication by extracting common pattern:

1. Setup request context
2. Execute cache operation
3. Record success with metrics
4. Handle errors uniformly

buildRepoValidationChain() - Builds validation chains based on options (pagination, dates, authors, churn).

Utility Extraction

hashUrl() / hashObject() - Moved from private methods to shared utilities in hashingUtils.ts for reuse.

gitUtils.shallowClone() - Fixed to use blob filtering for complete commit history instead of depth limiting.

---

Phase 3: Testing & Documentation

Test Enhancements

Unit Tests Added:

• gitUtils.unit.test.ts - Tests for shallow clone blob filtering
• repositoryRouteFactory.unit.test.ts - Tests for route factory functions
• routeHelpers.unit.test.ts - Tests for helper functions

Enhanced repositoryRoutes.unit.test.ts - Added mock utilities and validation handlers for comprehensive coverage.

API Test Scripts:

• test-api-phase1.sh - Comprehensive API testing scenarios
• api_test_scenarios.md - Documented test scenarios for all endpoints

Documentation Created

FRONTEND_API_MIGRATION.md - Complete guide for frontend teams to migrate from POST to GET endpoints, including:

• Endpoint mapping table
• Request/response structure changes
• Code examples for each endpoint
• Filter parameter conversion

Serena MCP Memory Files:

• Architecture overview
• Codebase structure
• Coding standards
• Project context for AI assistants

---

Breaking Changes ⚠️

All repository endpoints changed from POST to GET with query parameters:

Old Endpoint	New Endpoint	Key Changes
POST /api/repositories	GET /api/repositories/commits	Query params + pagination
POST /api/repositories/heatmap	GET /api/repositories/heatmap	Query params
POST /api/repositories/contributors	GET /api/repositories/contributors	Query params
POST /api/repositories/churn	GET /api/repositories/churn	Query params
POST /api/repositories/full-data	GET /api/repositories/full-data	Query params + pagination
GET /api/repositories/summary	GET /api/repositories/summary	Updated internally

Frontend Migration Required: See FRONTEND_API_MIGRATION.md for complete migration guide.

---

📋 View Endpoint Migration Diagram (for Frontend Team)

POST → GET Migration Map

%%{init: {'theme':'base', 'themeVariables': {'fontSize':'16px'}}}%%
flowchart LR
    subgraph OLD["🔴 OLD - POST Endpoints"]
        direction TB
        P1["POST /api/repositories
        Body: { repoUrl }"]
        P2["POST /api/repositories/heatmap
        Body: { repoUrl, filterOptions }"]
        P3["POST /api/repositories/contributors
        Body: { repoUrl, filterOptions }"]
        P4["POST /api/repositories/churn
        Body: { repoUrl, filterOptions }"]
        P5["POST /api/repositories/full-data
        Body: { repoUrl, filterOptions }"]
        P6["GET /api/repositories/summary
        Query: ?repoUrl"]
    end
    
    subgraph NEW["🟢 NEW - GET Endpoints"]
        direction TB
        G1["GET /api/repositories/commits
        Query: ?repoUrl&page&limit"]
        G2["GET /api/repositories/heatmap
        Query: ?repoUrl&authors&fromDate&toDate"]
        G3["GET /api/repositories/contributors
        Query: ?repoUrl&authors&fromDate&toDate"]
        G4["GET /api/repositories/churn
        Query: ?repoUrl&fromDate&toDate&extensions"]
        G5["GET /api/repositories/full-data
        Query: ?repoUrl&page&limit&authors&dates"]
        G6["GET /api/repositories/summary
        Query: ?repoUrl"]
    end
    
    P1 -.->|"migrated + pagination"| G1
    P2 -.->|"migrated + query params"| G2
    P3 -.->|"migrated + query params"| G3
    P4 -.->|"migrated + query params"| G4
    P5 -.->|"migrated + pagination"| G5
    P6 -.->|"improved cache"| G6
    
    classDef oldStyle fill:#ffebee,stroke:#c62828,stroke-width:3px,color:#000
    classDef newStyle fill:#e8f5e9,stroke:#2e7d32,stroke-width:3px,color:#000
    
    class OLD,P1,P2,P3,P4,P5,P6 oldStyle
    class NEW,G1,G2,G3,G4,G5,G6 newStyle

Loading

🖼️ View Endpoint Migration Diagram (PNG)

---

Migration Guide for Frontend Teams

Key Changes

1. HTTP Method Change

// OLD (POST with body)
const response = await apiClient.post('/api/repositories', { repoUrl });

// NEW (GET with query params)
const params = new URLSearchParams({ repoUrl });
const response = await apiClient.get('/api/repositories/commits', { params });

2. Filter Parameters

// OLD (nested in body)
{
  filterOptions: {
    author: "john",
    fromDate: "2024-01-01"
  }
}

// NEW (flat query params)
?author=john&fromDate=2024-01-01

3. Pagination

// NEW - pagination now available
?repoUrl=...&page=1&limit=50

Important: This backend refactor is frontend-agnostic. The new GET-based API can be consumed by any frontend implementation, including the new frontend being developed in a separate branch. See FRONTEND_API_MIGRATION.md for complete migration details.

---

🔄 View Request Flow & Cache Performance Diagram

Request Flow Through Multi-Tier Cache

%%{init: {'theme':'base', 'themeVariables': { 'primaryColor':'#f5f5f5','secondaryColor':'#e3f2fd','tertiaryColor':'#fff3e0'}}}%%
flowchart LR
    subgraph CLIENT["Client"]
        REQUEST["HTTP Request"]
    end
    
    subgraph ENDPOINTS["API Endpoints"]
        direction TB
        EP1["GET /commits
        ?repoUrl&page&limit"]
        EP2["GET /heatmap
        ?repoUrl&authors&dates"]
        EP3["GET /contributors
        ?repoUrl&filters"]
        EP4["GET /churn
        ?repoUrl&filters"]
        EP5["GET /full-data
        ?repoUrl&all params"]
    end
    
    subgraph CACHE["Unified Cache Service"]
        direction TB
        L1["⚡ Memory
        ~1ms"]
        L2["💾 Disk
        ~10-50ms"]
        L3["🔴 Redis
        ~50-100ms"]
    end
    
    subgraph SOURCE["Git Repository"]
        GIT["📦 Shared Clone
        blob:none filter
        480/480 commits"]
    end
    
    REQUEST --> ENDPOINTS
    EP1 --> CACHE
    EP2 --> CACHE
    EP3 --> CACHE
    EP4 --> CACHE
    EP5 --> CACHE
    
    CACHE -->|miss| SOURCE
    L1 -.->|promotion| L2
    L2 -.->|promotion| L3
    L3 -.->|promotion| GIT
    
    GIT -.->|cache| L3
    L3 -.->|cache| L2
    L2 -.->|cache| L1
    L1 -->|hit| ENDPOINTS
    
    classDef clientStyle fill:#e8eaf6,stroke:#3f51b5,stroke-width:2px
    classDef endpointStyle fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
    classDef cacheStyle fill:#fff3e0,stroke:#f57c00,stroke-width:2px
    classDef sourceStyle fill:#e8f5e9,stroke:#388e3c,stroke-width:2px
    
    class CLIENT,REQUEST clientStyle
    class ENDPOINTS,EP1,EP2,EP3,EP4,EP5 endpointStyle
    class CACHE,L1,L2,L3 cacheStyle
    class SOURCE,GIT sourceStyle

Loading

🖼️ View Request Flow Diagram (PNG)

---

Test Results ✅

All endpoints tested with https://github.com/jonasyr/gitray.git:

Summary Endpoint

{
  "stats": {
    "totalCommits": 480,     // ✅ Correct (was null)
    "contributors": 6,       // ✅ Correct (was null)
    "status": "active"
  }
}

Heatmap Endpoint

{
  "timePeriod": "day",
  "dataPoints": 365         // ✅ Correct (was 0)
}

Contributors Endpoint

• ✅ Returns 5 contributors with accurate commit counts and line stats

Commits Endpoint

• ✅ Paginated commits return correctly

Churn Analysis Endpoint

• ✅ Returns 87 files with risk analysis

Full-Data Endpoint

• ✅ NOW FIXED: Returns proper CommitHeatmapData structure
• ✅ Sequential execution prevents lock contention

---

Documentation Files

• ✅ FRONTEND_API_MIGRATION.md - Complete frontend migration guide
• ✅ memories - Architecture, codebase structure, coding standards
• ✅ api_test_scenarios.md - Comprehensive API testing scenarios
• ✅ Route comments document cache service usage
• ✅ Validation chains documented with error messages
• ✅ Cache key patterns documented in code

---

Lessons Learned

1. Blob filtering > Depth limiting: Provides complete commit history while saving 95-99% bandwidth
2. Cache key consistency: Never include undefined properties in filter objects
3. Lock contention: Parallel cache operations with overlapping locks can corrupt data
4. Type safety: Runtime type guards detect cache corruption early
5. Sequential execution: Sometimes simplicity prevents race conditions
6. Code deduplication: Systematic refactoring improves maintainability and reduces bugs
7. Helper extraction: Common patterns should be extracted early to prevent duplication

---

Acceptance Criteria

• Remove manual redis.get/set logic from all routes
• Replace direct gitService calls with unified cache methods
• Ensure Redis remains configured as third tier
• All endpoints return accurate data (480 commits, 6 contributors)
• Add comprehensive validation for query parameters
• Document changes in route comments
• Create frontend migration guide
• Update unit/integration tests
• Fix data accuracy issues (commit count, heatmap, full-data)
• Prevent deadlocks with ordered lock acquisition
• Add defensive type checking for cache operations
• Eliminate code duplication (< 3%)
• Extract common patterns into reusable helpers
• Add comprehensive test coverage for new utilities

---

Related Issues

• Closes enhancement(scope:backend): Refactor old routes to use unified cache service (retain Redis, remove manual caching) #120 - Refactor old routes to use unified cache service

---

Testing Checklist

• All endpoints return correct data for gitray repository
• Cache hits work across all three tiers
• Validation correctly rejects invalid parameters
• Pagination works correctly for commits endpoint
• Filter options work for heatmap/contributors/churn
• Full-data endpoint returns correct structure
• No deadlocks occur under concurrent requests
• Unit tests pass for all refactored routes
• Backend runs without Redis (graceful degradation)
• Helper functions are properly tested
• Route factory pattern works correctly

---

How to Test

# Start backend
pnpm dev:backend

# Run comprehensive test suite
./scripts/test_api_complete.sh

# Test manually
curl "http://localhost:3001/api/repositories/commits?repoUrl=https://github.com/jonasyr/gitray.git&limit=5"
curl "http://localhost:3001/api/repositories/summary?repoUrl=https://github.com/jonasyr/gitray.git"
curl "http://localhost:3001/api/repositories/heatmap?repoUrl=https://github.com/jonasyr/gitray.git"

Expected results:

• Summary: 480 commits, 6 contributors
• Heatmap: 365 data points
• All endpoints return data in < 100ms on cache hits

---

Rollback Plan

If issues arise:

1. Frontend can temporarily use old endpoints if they exist
2. Backend changes are isolated to repository routes
3. No database migrations required
4. Cache can be cleared: rm -rf apps/backend/cache/*
5. Redis is optional (memory-only fallback works)

---

Reviewers: Please verify:

• Cache service integration is correct
• Validation logic is comprehensive
• Error handling is robust
• Code deduplication improves maintainability
• Helper functions are properly tested
• Tests cover edge cases
• Documentation is clear and complete

cc: @jonasyr

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[sonarqubecloud]

Quality Gate failed

Failed conditions
77.8% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube Cloud