Test: Database Recovery from Index Corruption During Build

[nickna]

Description

Test database behavior and recovery when index building is interrupted by failures like out-of-memory, disk space, or process termination.

Phase

Phase 1: Chaos Engineering and Failure Injection

Epic

Related to #202

Acceptance Criteria

• Create tests that inject failures during index building (KD-Tree, HNSW, Ball Tree)
• Verify database remains in consistent state after failure
• Test recovery mechanisms and partial index cleanup
• Ensure search operations work with fallback to linear search
• Validate database integrity after recovery

Failure Scenarios to Test

1. Out of Memory during HNSW build - Simulate memory exhaustion
2. Disk space exhaustion - Mock storage full during index save
3. Process termination - Interrupt index building mid-operation
4. Cancellation during async operations - Test proper cleanup
5. Concurrent modifications - Index building while vectors are being added

Test Structure

[Test]
[Category("Chaos")]
public async Task Database_RecoverFromIndexBuildFailure_OutOfMemory()
{
    // Arrange: Large dataset that could trigger memory issues
    var database = new VectorDatabase();
    PopulateWithLargeDataset(database, vectorCount: 50000);
    
    // Act: Inject memory pressure during index build
    using var memoryPressure = new MemoryPressureSimulator();
    var exception = await Assert.ThrowsAsync<OutOfMemoryException>(
        () => database.RebuildSearchIndexAsync(SearchAlgorithm.HNSW));
    
    // Assert: Database still functional with linear search
    Assert.That(database.Count, Is.EqualTo(50000));
    var results = database.Search(query, 5, SearchAlgorithm.Linear);
    Assert.That(results.Count, Is.EqualTo(5));
}

Mock Framework Integration

• Create IFailureInjector interface for different failure types
• Implement memory pressure simulation
• Mock file system failures
• Create cancellation token scenarios