enhancement(scope:backend): Refactor old routes to use unified cache service (retain Redis, remove manual caching)

[jonasyr]

🎯 Issue Type

• Enhancement

📋 Description

Problem/Need

Older endpoints in apps/backend/src/routes/repositoryRoutes.ts still implement their own caching logic: they manually set and get Redis keys and clone a repository via withTempRepository() on each call. This bypasses the Cache Service and HybridLRUCache described in the Caching System documentation. As a result, commit data is cloned multiple times, aggregated data is recomputed unnecessarily, and manual Redis operations ignore tier promotion and cache invalidation semantics.

Expected Behavior

• All routes should delegate caching to getCachedCommits(), getCachedAggregatedData() and getCachedContributors() from repositoryCache.ts.
• The unified cache should handle lookups across memory, disk, and Redis tiers, including automatic promotion, fallback, and invalidation.
• Redis remains part of the storage tier; the change removes only the manual cache handling from route handlers.

Current Behavior

• Endpoints like POST /api/repositories, /heatmap, /contributors, and /full-data check Redis directly, clone the repository, and recompute commits or aggregations.
• They do not leverage multi‑tier caching, resulting in duplicated Git operations and inconsistent cache invalidation.

🔄 Steps to Reproduce

1. Spin up the backend and send a POST to /api/repositories with a repo URL.
2. Immediately call POST /api/repositories/heatmap with the same URL.
3. Observe that the second call clones the repo again instead of using cached commit data (logs show duplicate gitService.getCommits() calls).
4. Compare to the new GET /api/commits and GET /api/commits/heatmap endpoints, which reuse cached data via the cache service.

🎨 Mockups/Screenshots

// Current manual caching (simplified)
const commitsKey = `commits:${repoUrl}`;
const cached = await redis.get(commitsKey);
if (!cached) {
  const commits = await withTempRepository(repoUrl, (dir) => gitService.getCommits(dir));
  await redis.set(commitsKey, JSON.stringify(commits));
}

// Proposed unified caching usage
const cacheOptions: CommitCacheOptions = { skip: 0, limit: 1000 };
const commits = await getCachedCommits(repoUrl, cacheOptions);

See Hybrid LRU Cache – Three‑Tier Storage Architecture for details on memory, disk, and Redis tiers and Cache Service for the unified API.

🧪 Acceptance Criteria

• Remove manual redis.get/set logic from older routes.
• Replace direct gitService calls with getCachedCommits, getCachedAggregatedData, and getCachedContributors.
• Ensure that Redis remains configured as the third tier in cache configuration; only manual calls are removed.
• Add or update unit/integration tests to confirm that subsequent calls reuse cached data.
• Document the change in route comments and/or docs, referencing the caching system design.

🛠 Technical Details

Affected Files/Components

• apps/backend/src/routes/repositoryRoutes.ts
• apps/backend/src/services/repositoryCache.ts (import/export usage)
• Test suites for repository routes.

Dependencies

• None, though ensure configuration (config.ts) still enables Redis if desired.

Breaking Changes

• Yes
• No

🏷 Categorization

Scope

• scope:backend

Priority

• prio:high

Effort

• effort:small
• effort:medium

📝 Additional Notes

• This refactor does not remove Redis from the caching infrastructure. It aligns the older routes with the multi‑tier cache design, as described in the Cache Invalidation and Hybrid LRU Cache pages.
• The repository’s cache configuration can still disable Redis if desired via enableRedis, but that change would be separate.

[jonasyr]

Refactoring Plan: Unified Cache Service Migration

Overview

Migrate all routes in repositoryRoutes.ts from manual Redis caching to the unified multi-tier cache service, aligning with the modern commitRoutes.ts pattern. This
includes migrating from POST to GET endpoints and extending the cache service to support all data types.

---

Phase 1: Extend Cache Service (New Functionality)

1.1 Add getCachedChurnData() to Cache Service

File: apps/backend/src/services/repositoryCache.ts

• Add churn data caching support (new Tier 3 aggregated data type)
• Signature: getCachedChurnData(repoUrl: string, filterOptions?: ChurnFilterOptions): Promise
• Follow existing pattern from getCachedAggregatedData()
• Use cache key: churn_data:${repoUrl_hash}:${filterHash}
• TTL: Use config.cacheStrategy.cacheKeys.aggregatedDataTTL (900s/15min)

1.2 Add getCachedSummary() to Cache Service

File: apps/backend/src/services/repositoryCache.ts

• Add repository summary caching support
• Signature: getCachedSummary(repoUrl: string): Promise
• Use cache key: repository_summary:${repoUrl_hash}
• TTL: Use config.cacheStrategy.cacheKeys.repositoryInfoTTL (7200s/2 hours)
• Note: Keep the sparse clone logic from repositorySummaryService but wrap it with unified cache

1.3 Update Type Exports

• Export ChurnFilterOptions type if not already exported
• Update CacheStats interface to include churn and summary cache metrics

---

Phase 2: Refactor Routes to GET + Query Params

2.1 Refactor POST / → GET /commits

File: apps/backend/src/routes/repositoryRoutes.ts

Before:
POST / with body { repoUrl }

After:
GET /commits?repoUrl=...&page=1&limit=100

Changes:

• Change from POST to GET
• Add query parameter validation (repoUrl, page, limit)
• Replace manual Redis logic with: getCachedCommits(repoUrl, { skip, limit })
• Keep existing metrics recording
• Update request priority middleware

2.2 Refactor POST /heatmap → GET /heatmap

Before:
POST /heatmap with body { repoUrl, filterOptions }

After:
GET /heatmap?repoUrl=...&author=...&authors=...&fromDate=...&toDate=...

Changes:

• Change from POST to GET
• Parse filter options from query params (author, authors, fromDate, toDate)
• Replace manual Redis logic with: getCachedAggregatedData(repoUrl, filters)
• Follow pattern from commitRoutes.ts:343-461

2.3 Refactor POST /contributors → GET /contributors

Before:
POST /contributors with body { repoUrl, filterOptions }

After:
GET /contributors?repoUrl=...&author=...&authors=...&fromDate=...&toDate=...

Changes:

• Change from POST to GET
• Parse filter options from query params
• Replace manual Redis logic with: getCachedContributors(repoUrl, filters)
• Remove withTempRepository() calls

2.4 Refactor POST /churn → GET /churn

Before:
POST /churn with body { repoUrl, filterOptions }

After:
GET /churn?repoUrl=...&minCommits=...&limit=...&fromDate=...&toDate=...

Changes:

• Change from POST to GET
• Parse churn filter options from query params
• Replace manual Redis logic with: getCachedChurnData(repoUrl, filters) (new function from Phase 1)

2.5 Refactor GET /summary

Current: Already uses GET but has manual caching via repositorySummaryService

Changes:

• Keep GET method and query params
• Replace repositorySummaryService.getRepositorySummary() with: getCachedSummary(repoUrl) (new function from Phase 1)
• Preserve: Sparse clone logic, URL parsing, validation
• Remove: Manual Redis operations from repositorySummaryService

2.6 Refactor POST /full-data → GET /full-data

Before:
POST /full-data with body { repoUrl, filterOptions }

After:
GET /full-data?repoUrl=...&author=...&authors=...&fromDate=...&toDate=...&page=1&limit=100

Changes:

• Change from POST to GET
• Call both getCachedCommits() and getCachedAggregatedData() in parallel
• Remove all manual Redis get/set logic
• Both functions will use unified cache internally

---

Phase 3: Validation & Error Handling

3.1 Add Comprehensive Validation Chains

File: apps/backend/src/routes/repositoryRoutes.ts

• Import and reuse validation chains from commitRoutes.ts:
  • repoUrlValidation() - URL format and security
  • paginationValidation() - page and limit
  • dateValidation() - fromDate and toDate
  • authorValidation() - author and authors
• Add churn-specific validation for minCommits and file limits

3.2 Update Error Handling

• Follow commitRoutes.ts pattern with handleValidationErrors middleware
• Use createRequestLogger(req) for structured logging
• Maintain existing metrics recording patterns

---

Phase 4: Update Tests

4.1 Update Unit Tests

File: apps/backend/tests/unit/routes/repositoryRoutes.unit.test.ts

Changes:

• Mock getCachedCommits, getCachedAggregatedData, getCachedContributors, getCachedChurnData, getCachedSummary instead of redis.get/set
• Update request method from POST to GET
• Update request parameters from body to query
• Add validation error test cases for query params
• Test cache hit/miss scenarios via mocked cache functions
• Follow pattern from commitRoutes.unit.test.ts

4.2 Add Cache Service Tests

File: apps/backend/tests/unit/services/repositoryCache.unit.test.ts

Add test coverage for:

• getCachedChurnData() with various filter combinations
• getCachedSummary() cache hit/miss scenarios
• Verify three-tier caching for new functions
• Test transactional rollback for new cache types

4.3 Add Integration Tests (Recommended)

New file: apps/backend/tests/integration/routes/repositoryRoutes.integration.test.ts

Test scenarios:

• Real cache service integration (not mocked)
• Verify subsequent calls use cached data
• Test cache invalidation across all route types
• Validate that Redis remains in the storage tier
• Performance comparison: manual cache vs unified cache

---

Phase 5: Remove Legacy Code

5.1 Clean Up Imports

File: apps/backend/src/routes/repositoryRoutes.ts

Remove:

• import redis from '../services/cache'
• import { withTempRepository } from '../utils/withTempRepository'
• import { body } from 'express-validator' (if no longer used)

Add:

• import { getCachedCommits, getCachedAggregatedData, getCachedContributors, getCachedChurnData, getCachedSummary, type CommitCacheOptions } from
  '../services/repositoryCache'
• import { query, validationResult, ValidationChain } from 'express-validator'
• import { createRequestLogger } from '../services/logger'

5.2 Deprecate repositorySummaryService (Optional)

File: apps/backend/src/services/repositorySummaryService.ts

• If no other consumers exist, move sparse clone logic into cache service
• Otherwise, refactor service to use getCachedSummary() internally

---

Phase 6: Documentation & Validation

6.1 Update API Documentation

File: docs/API.md

• Mark old POST endpoints as deprecated
• Document new GET endpoints with query params
• Add migration guide for API consumers
• Update example requests

6.2 Update Route Comments

File: apps/backend/src/routes/repositoryRoutes.ts

• Add JSDoc comments referencing unified cache service
• Document cache invalidation semantics
• Link to caching system documentation

6.3 Manual Validation

• Start backend: pnpm dev:backend
• Test each endpoint with sample repository URL
• Verify first call clones repository
• Verify second call uses cached data (check logs)
• Test cache invalidation endpoint
• Verify Redis contains cache entries (use redis-cli)

---

Acceptance Criteria Mapping

• ✅ Remove manual redis.get/set logic - Phase 2 (all routes)
• ✅ Replace direct gitService calls - Phase 2 (use cache service functions)
• ✅ Redis remains as third tier - No config changes, only usage pattern changes
• ✅ Add/update tests - Phase 4 (unit + integration)
• ✅ Document changes - Phase 6 (route comments + API docs)

---

Breaking Changes

⚠️ YES - API Contract Changes:

• All routes migrating from POST to GET
• Parameters moving from request body to query strings
• API consumers will need to update their clients

Migration Path:

1. Deploy new GET endpoints
2. Keep old POST endpoints temporarily with deprecation warnings
3. Monitor usage and coordinate with API consumers
4. Remove POST endpoints in future release

Alternative (if backwards compatibility required):

• Keep POST endpoints alongside new GET endpoints
• Both call the same unified cache service
• Gradually deprecate POST over time

---

Estimated Effort

• Phase 1 (Cache Service Extension): ~4-6 hours
• Phase 2 (Route Refactoring): ~6-8 hours
• Phase 3 (Validation): ~2-3 hours
• Phase 4 (Testing): ~6-8 hours
• Phase 5 (Cleanup): ~1-2 hours
• Phase 6 (Documentation): ~2-3 hours

Total: ~21-30 hours (medium effort, as categorized)

---

Dependencies & Risks

Dependencies:

• Unified cache service must remain stable during migration
• Redis configuration must be verified before deployment

Risks:

• Breaking API changes may impact frontend or external consumers
• Need coordination if other services call these endpoints
• Cache key format changes could invalidate existing cached data

Mitigation:

• Comprehensive testing at each phase
• Deploy to staging first
• Monitor cache metrics post-deployment
• Have rollback plan ready