Phase 9.1: Implement End-to-End Workflow and Policy Enforcement Tests

## 📋 Task Description

Implement comprehensive end-to-end workflow tests that validate complete process lifecycles from creation through completion or failure. Test policy enforcement scenarios including timeouts, retries, circuit breakers, and concurrency limits. Verify the entire system works correctly with real infrastructure.

## 🎯 Objectives

- Implement complete process lifecycle tests (Pending → Processing → Completed)
- Test failure and retry workflows (Pending → Processing → Retrying → Failed)
- Test timeout enforcement in process execution
- Test retry policy enforcement with backoff
- Test circuit breaker behavior under failures
- Test concurrency limit enforcement
- Test process handler execution end-to-end
- Test message broker integration (publish and consume)
- Test state persistence across workflow stages
- Test error tracking and reporting
- Verify telemetry and logging correctness
- Document workflow testing patterns

## 📦 Deliverables

### 1. Create End-to-End Workflow Tests

Create `tests/StarGate.IntegrationTests/Workflows/ProcessLifecycleTests.cs`:

```csharp
namespace StarGate.IntegrationTests.Workflows;

using FluentAssertions;
using StarGate.IntegrationTests.Builders;
using StarGate.IntegrationTests.Helpers;
using StarGate.IntegrationTests.Infrastructure;
using System.Net;
using Xunit;

/// <summary>
/// End-to-end tests for complete process lifecycles.
/// </summary>
public class ProcessLifecycleTests : IClassFixture<StarGateWebApplicationFactory>
{
    private readonly HttpClient _client;
    private readonly StarGateWebApplicationFactory _factory;

    public ProcessLifecycleTests(StarGateWebApplicationFactory factory)
    {
        _factory = factory;
        _client = factory.CreateClient();
    }

    [Fact]
    public async Task ProcessLifecycle_Should_CompleteSuccessfully_ForOrderProcess()
    {
        // Arrange - Create policy for order process type
        var policy = new
        {
            processType = "order",
            maxRetries = 3,
            timeoutSeconds = 30,
            maxConcurrentProcesses = 10,
            retentionDays = 30
        };
        await _client.PostJsonAsync("/api/policies/process-types", policy);

        // Act - Create process
        var request = new CreateProcessRequestBuilder()
            .WithProcessType("order")
            .WithMetadata("orderId", "order-12345")
            .WithMetadata("customerId", "customer-789")
            .WithMetadata("amount", "250.00")
            .Build();

        var createResponse = await _client.PostJsonAsync("/api/processes", request);
        var processId = ExtractProcessIdFromLocation(createResponse.Headers.Location!);

        // Wait for processing to complete
        await Task.Delay(TimeSpan.FromSeconds(2));

        // Assert - Verify final state
        var getResponse = await _client.GetAsync($"/api/processes/{processId}");
        var process = await getResponse.Content.ReadFromJsonAsync<ProcessResponse>();

        process.Should().NotBeNull();
        process!.Status.Should().Be("Completed");
        process.Errors.Should().BeNullOrEmpty();
    }

    [Fact]
    public async Task ProcessLifecycle_Should_HandleFailure_WithRetries()
    {
        // Arrange - Create policy with retries
        var policy = new
        {
            processType = "order",
            maxRetries = 2,
            timeoutSeconds = 10
        };
        await _client.PostJsonAsync("/api/policies/process-types", policy);

        // Act - Create process with invalid data (will fail)
        var request = new CreateProcessRequestBuilder()
            .WithProcessType("order")
            .WithMetadata("orderId", "order-invalid")
            // Missing required fields - will cause validation failure
            .Build();

        var createResponse = await _client.PostJsonAsync("/api/processes", request);
        var processId = ExtractProcessIdFromLocation(createResponse.Headers.Location!);

        // Wait for retries to complete
        await Task.Delay(TimeSpan.FromSeconds(5));

        // Assert - Verify failed state with retry attempts
        var getResponse = await _client.GetAsync($"/api/processes/{processId}");
        var process = await getResponse.Content.ReadFromJsonAsync<ProcessResponse>();

        process.Should().NotBeNull();
        process!.Status.Should().Be("Failed");
        process.RetryCount.Should().Be(2); // Policy maxRetries
        process.Errors.Should().NotBeEmpty();
        process.Errors![0].Should().Contain("Customer ID");
    }

    [Fact]
    public async Task ProcessLifecycle_Should_RespectTimeout_ForSlowOperations()
    {
        // Arrange - Create policy with short timeout
        var policy = new
        {
            processType = "slow-process",
            maxRetries = 1,
            timeoutSeconds = 2 // Very short timeout
        };
        await _client.PostJsonAsync("/api/policies/process-types", policy);

        // Act - Create process that will timeout
        var request = new CreateProcessRequestBuilder()
            .WithProcessType("slow-process")
            .Build();

        var createResponse = await _client.PostJsonAsync("/api/processes", request);
        var processId = ExtractProcessIdFromLocation(createResponse.Headers.Location!);

        // Wait for timeout and retry
        await Task.Delay(TimeSpan.FromSeconds(5));

        // Assert - Verify timeout error
        var getResponse = await _client.GetAsync($"/api/processes/{processId}");
        var process = await getResponse.Content.ReadFromJsonAsync<ProcessResponse>();

        process.Should().NotBeNull();
        process!.Status.Should().Be("Failed");
        process.Errors.Should().Contain(e => e.Contains("timeout"));
    }

    [Fact]
    public async Task ProcessLifecycle_Should_TransitionThroughStates_Correctly()
    {
        // Arrange
        var policy = new
        {
            processType = "order",
            maxRetries = 3,
            timeoutSeconds = 30
        };
        await _client.PostJsonAsync("/api/policies/process-types", policy);

        var request = new CreateProcessRequestBuilder()
            .WithProcessType("order")
            .WithMetadata("orderId", "order-state-test")
            .WithMetadata("customerId", "customer-123")
            .WithMetadata("amount", "100.00")
            .Build();

        // Act & Assert - Check state progression
        var createResponse = await _client.PostJsonAsync("/api/processes", request);
        var processId = ExtractProcessIdFromLocation(createResponse.Headers.Location!);

        // 1. Initially Pending
        var response1 = await _client.GetAsync($"/api/processes/{processId}");
        var process1 = await response1.Content.ReadFromJsonAsync<ProcessResponse>();
        process1!.Status.Should().Be("Pending");

        // Wait for processing to start
        await Task.Delay(TimeSpan.FromMilliseconds(500));

        // 2. Should be Processing
        var response2 = await _client.GetAsync($"/api/processes/{processId}");
        var process2 = await response2.Content.ReadFromJsonAsync<ProcessResponse>();
        process2!.Status.Should().BeOneOf("Processing", "Completed");

        // Wait for completion
        await Task.Delay(TimeSpan.FromSeconds(2));

        // 3. Finally Completed
        var response3 = await _client.GetAsync($"/api/processes/{processId}");
        var process3 = await response3.Content.ReadFromJsonAsync<ProcessResponse>();
        process3!.Status.Should().Be("Completed");
    }

    [Fact]
    public async Task ProcessLifecycle_Should_PersistStateAcrossRestarts()
    {
        // This test verifies state persistence in MongoDB
        // Create process, stop worker, verify state, restart worker, verify completion
        
        // Arrange
        var request = new CreateProcessRequestBuilder()
            .WithProcessType("order")
            .Build();

        // Act - Create process
        var createResponse = await _client.PostJsonAsync("/api/processes", request);
        var processId = ExtractProcessIdFromLocation(createResponse.Headers.Location!);

        // Verify process created in database
        var response1 = await _client.GetAsync($"/api/processes/{processId}");
        response1.StatusCode.Should().Be(HttpStatusCode.OK);

        // Simulate restart by creating new client
        var newClient = _factory.CreateClient();
        
        // Verify process still exists after "restart"
        var response2 = await newClient.GetAsync($"/api/processes/{processId}");
        response2.StatusCode.Should().Be(HttpStatusCode.OK);
        
        var process = await response2.Content.ReadFromJsonAsync<ProcessResponse>();
        process.Should().NotBeNull();
    }

    private static Guid ExtractProcessIdFromLocation(Uri location)
    {
        var segments = location.Segments;
        var idString = segments[^1];
        return Guid.Parse(idString);
    }
}

public class ProcessResponse
{
    public Guid ProcessId { get; set; }
    public string Status { get; set; } = string.Empty;
    public int RetryCount { get; set; }
    public List<string>? Errors { get; set; }
}
```

### 2. Create Policy Enforcement Tests

Create `tests/StarGate.IntegrationTests/Workflows/PolicyEnforcementTests.cs`:

```csharp
namespace StarGate.IntegrationTests.Workflows;

using FluentAssertions;
using StarGate.IntegrationTests.Builders;
using StarGate.IntegrationTests.Helpers;
using StarGate.IntegrationTests.Infrastructure;
using Xunit;

/// <summary>
/// Tests for policy enforcement scenarios.
/// </summary>
public class PolicyEnforcementTests : IClassFixture<StarGateWebApplicationFactory>
{
    private readonly HttpClient _client;

    public PolicyEnforcementTests(StarGateWebApplicationFactory factory)
    {
        _client = factory.CreateClient();
    }

    [Fact]
    public async Task RetryPolicy_Should_RetrySpecifiedTimes_BeforeFailing()
    {
        // Arrange - Policy with 3 retries
        var policy = new
        {
            processType = "retry-test",
            maxRetries = 3,
            timeoutSeconds = 30
        };
        await _client.PostJsonAsync("/api/policies/process-types", policy);

        // Act - Create failing process
        var request = new CreateProcessRequestBuilder()
            .WithProcessType("retry-test")
            .WithMetadata("shouldFail", "true")
            .Build();

        var createResponse = await _client.PostJsonAsync("/api/processes", request);
        var processId = ExtractProcessIdFromLocation(createResponse.Headers.Location!);

        // Wait for all retries
        await Task.Delay(TimeSpan.FromSeconds(10));

        // Assert
        var getResponse = await _client.GetAsync($"/api/processes/{processId}");
        var process = await getResponse.Content.ReadFromJsonAsync<ProcessResponse>();

        process!.RetryCount.Should().Be(3);
        process.Status.Should().Be("Failed");
    }

    [Fact]
    public async Task RetryPolicy_Should_UseExponentialBackoff()
    {
        // Arrange
        var policy = new
        {
            processType = "backoff-test",
            maxRetries = 3,
            timeoutSeconds = 30
        };
        await _client.PostJsonAsync("/api/policies/process-types", policy);

        // Act
        var request = new CreateProcessRequestBuilder()
            .WithProcessType("backoff-test")
            .WithMetadata("shouldFail", "true")
            .Build();

        var createResponse = await _client.PostJsonAsync("/api/processes", request);
        var processId = ExtractProcessIdFromLocation(createResponse.Headers.Location!);

        var startTime = DateTime.UtcNow;

        // Wait for all retries
        await Task.Delay(TimeSpan.FromSeconds(10));

        var endTime = DateTime.UtcNow;
        var totalDuration = endTime - startTime;

        // Assert - With exponential backoff (1s, 2s, 4s), minimum total time is 7s
        totalDuration.Should().BeGreaterThan(TimeSpan.FromSeconds(7));
    }

    [Fact]
    public async Task TimeoutPolicy_Should_CancelLongRunningOperations()
    {
        // Arrange - Short timeout
        var policy = new
        {
            processType = "timeout-test",
            maxRetries = 0,
            timeoutSeconds = 2
        };
        await _client.PostJsonAsync("/api/policies/process-types", policy);

        // Act - Create slow process
        var request = new CreateProcessRequestBuilder()
            .WithProcessType("timeout-test")
            .WithMetadata("delaySeconds", "10") // Longer than timeout
            .Build();

        var createResponse = await _client.PostJsonAsync("/api/processes", request);
        var processId = ExtractProcessIdFromLocation(createResponse.Headers.Location!);

        var startTime = DateTime.UtcNow;

        // Wait for timeout
        await Task.Delay(TimeSpan.FromSeconds(5));

        var endTime = DateTime.UtcNow;

        // Assert - Should timeout before 10 seconds
        var getResponse = await _client.GetAsync($"/api/processes/{processId}");
        var process = await getResponse.Content.ReadFromJsonAsync<ProcessResponse>();

        process!.Status.Should().Be("Failed");
        (endTime - startTime).Should().BeLessThan(TimeSpan.FromSeconds(5));
        process.Errors.Should().Contain(e => e.Contains("timeout"));
    }

    [Fact]
    public async Task ConcurrencyPolicy_Should_LimitSimultaneousProcesses()
    {
        // Arrange - Low concurrency limit
        var policy = new
        {
            processType = "concurrency-test",
            maxRetries = 0,
            timeoutSeconds = 30,
            maxConcurrentProcesses = 2 // Only 2 at a time
        };
        await _client.PostJsonAsync("/api/policies/process-types", policy);

        // Act - Create 5 processes simultaneously
        var tasks = Enumerable.Range(0, 5).Select(async i =>
        {
            var request = new CreateProcessRequestBuilder()
                .WithProcessType("concurrency-test")
                .WithClientProcessId($"concurrent-{i}")
                .WithMetadata("delaySeconds", "2")
                .Build();
            return await _client.PostJsonAsync("/api/processes", request);
        });

        var responses = await Task.WhenAll(tasks);

        // All should be created
        responses.Should().AllSatisfy(r => r.StatusCode.Should().Be(System.Net.HttpStatusCode.Created));

        // Wait for processing
        await Task.Delay(TimeSpan.FromSeconds(1));

        // Assert - Query processing status
        // At most 2 should be Processing simultaneously
        var processingCount = 0;
        foreach (var response in responses)
        {
            var processId = ExtractProcessIdFromLocation(response.Headers.Location!);
            var getResponse = await _client.GetAsync($"/api/processes/{processId}");
            var process = await getResponse.Content.ReadFromJsonAsync<ProcessResponse>();
            
            if (process!.Status == "Processing")
            {
                processingCount++;
            }
        }

        processingCount.Should().BeLessOrEqualTo(2);
    }

    [Fact]
    public async Task ClientOverride_Should_OverrideProcessTypePolicy()
    {
        // Arrange - Create process type policy
        var typePolicy = new
        {
            processType = "order",
            maxRetries = 2,
            timeoutSeconds = 10
        };
        await _client.PostJsonAsync("/api/policies/process-types", typePolicy);

        // Create client override with higher limits
        var clientOverride = new
        {
            clientId = "premium-client",
            processType = "order",
            maxRetries = 5,
            timeoutSeconds = 60
        };
        await _client.PostJsonAsync("/api/policies/client-overrides", clientOverride);

        // Act - Create failing process with premium client
        var request = new CreateProcessRequestBuilder()
            .WithClientId("premium-client")
            .WithProcessType("order")
            .WithMetadata("shouldFail", "true")
            .Build();

        var createResponse = await _client.PostJsonAsync("/api/processes", request);
        var processId = ExtractProcessIdFromLocation(createResponse.Headers.Location!);

        // Wait for all retries
        await Task.Delay(TimeSpan.FromSeconds(15));

        // Assert - Should use override (5 retries instead of 2)
        var getResponse = await _client.GetAsync($"/api/processes/{processId}");
        var process = await getResponse.Content.ReadFromJsonAsync<ProcessResponse>();

        process!.RetryCount.Should().Be(5); // Override value
        process.Status.Should().Be("Failed");
    }

    [Fact]
    public async Task RetentionPolicy_Should_AllowQueryingWithinRetentionPeriod()
    {
        // Arrange - Policy with 30 days retention
        var policy = new
        {
            processType = "retention-test",
            maxRetries = 0,
            timeoutSeconds = 30,
            retentionDays = 30
        };
        await _client.PostJsonAsync("/api/policies/process-types", policy);

        // Act - Create and complete process
        var request = new CreateProcessRequestBuilder()
            .WithProcessType("retention-test")
            .Build();

        var createResponse = await _client.PostJsonAsync("/api/processes", request);
        var processId = ExtractProcessIdFromLocation(createResponse.Headers.Location!);

        // Wait for completion
        await Task.Delay(TimeSpan.FromSeconds(2));

        // Assert - Process should be queryable
        var getResponse = await _client.GetAsync($"/api/processes/{processId}");
        getResponse.StatusCode.Should().Be(System.Net.HttpStatusCode.OK);

        // Note: Actual retention deletion would be tested with a background job
        // that runs periodically to clean up old processes
    }

    private static Guid ExtractProcessIdFromLocation(Uri location)
    {
        var segments = location.Segments;
        var idString = segments[^1];
        return Guid.Parse(idString);
    }
}
```

### 3. Create Message Broker Integration Tests

Create `tests/StarGate.IntegrationTests/Workflows/MessageBrokerIntegrationTests.cs`:

```csharp
namespace StarGate.IntegrationTests.Workflows;

using FluentAssertions;
using StarGate.IntegrationTests.Builders;
using StarGate.IntegrationTests.Helpers;
using StarGate.IntegrationTests.Infrastructure;
using Xunit;

/// <summary>
/// Tests for message broker integration in workflows.
/// </summary>
public class MessageBrokerIntegrationTests : IClassFixture<StarGateWebApplicationFactory>
{
    private readonly HttpClient _client;

    public MessageBrokerIntegrationTests(StarGateWebApplicationFactory factory)
    {
        _client = factory.CreateClient();
    }

    [Fact]
    public async Task ProcessCreation_Should_PublishMessageToQueue()
    {
        // Arrange
        var request = new CreateProcessRequestBuilder()
            .WithProcessType("order")
            .Build();

        // Act - Create process (publishes to queue)
        var createResponse = await _client.PostJsonAsync("/api/processes", request);
        createResponse.EnsureSuccessStatusCode();

        var processId = ExtractProcessIdFromLocation(createResponse.Headers.Location!);

        // Wait for message consumption and processing
        await Task.Delay(TimeSpan.FromSeconds(2));

        // Assert - Verify process was consumed and processed
        var getResponse = await _client.GetAsync($"/api/processes/{processId}");
        var process = await getResponse.Content.ReadFromJsonAsync<ProcessResponse>();

        process!.Status.Should().BeOneOf("Processing", "Completed");
    }

    [Fact]
    public async Task FailedProcess_Should_NackMessageForRequeue()
    {
        // Arrange - Process that will fail initially
        var request = new CreateProcessRequestBuilder()
            .WithProcessType("order")
            .WithMetadata("shouldFailOnce", "true")
            .Build();

        // Act
        var createResponse = await _client.PostJsonAsync("/api/processes", request);
        var processId = ExtractProcessIdFromLocation(createResponse.Headers.Location!);

        // Wait for initial failure and requeue
        await Task.Delay(TimeSpan.FromSeconds(3));

        // Assert - Process should be retrying
        var getResponse = await _client.GetAsync($"/api/processes/{processId}");
        var process = await getResponse.Content.ReadFromJsonAsync<ProcessResponse>();

        process!.RetryCount.Should().BeGreaterThan(0);
    }

    [Fact]
    public async Task CompletedProcess_Should_AckMessageSuccessfully()
    {
        // Arrange
        var request = new CreateProcessRequestBuilder()
            .WithProcessType("order")
            .WithMetadata("orderId", "test-ack")
            .WithMetadata("customerId", "customer-1")
            .WithMetadata("amount", "100.00")
            .Build();

        // Act
        var createResponse = await _client.PostJsonAsync("/api/processes", request);
        var processId = ExtractProcessIdFromLocation(createResponse.Headers.Location!);

        // Wait for completion
        await Task.Delay(TimeSpan.FromSeconds(2));

        // Assert - Process completed and message acked
        var getResponse = await _client.GetAsync($"/api/processes/{processId}");
        var process = await getResponse.Content.ReadFromJsonAsync<ProcessResponse>();

        process!.Status.Should().Be("Completed");
    }

    private static Guid ExtractProcessIdFromLocation(Uri location)
    {
        var segments = location.Segments;
        var idString = segments[^1];
        return Guid.Parse(idString);
    }
}
```

### 4. Create Test Documentation

Create `docs/TESTING-STRATEGY.md`:

```markdown
# Testing Strategy - StarGate

## Overview

StarGate implements a comprehensive testing strategy covering unit, integration, and end-to-end tests.

## Test Pyramid

```
      E2E Tests (10%)
       /         \
      /           \
   Integration     \
   Tests (30%)     \
    /               \
   /                 \
Unit Tests (60%)
```

## Test Types

### Unit Tests
- **Purpose:** Test individual components in isolation
- **Scope:** Single class/method
- **Speed:** Very fast (<100ms per test)
- **Coverage:** >90% for domain logic

### Integration Tests
- **Purpose:** Test component interactions
- **Scope:** Multiple components with real infrastructure
- **Speed:** Fast (<1s per test)
- **Coverage:** All API endpoints, repository operations

### End-to-End Tests
- **Purpose:** Test complete workflows
- **Scope:** Entire system from API to database
- **Speed:** Moderate (2-5s per test)
- **Coverage:** Critical business flows

## Workflow Testing Approach

### Happy Path
1. Create process via API
2. Message published to RabbitMQ
3. Worker consumes message
4. Handler executes business logic
5. State updated in MongoDB
6. Process completes successfully

### Error Path
1. Create process with invalid data
2. Handler throws exception
3. Error recorded
4. Retry attempted per policy
5. After max retries, process fails
6. Final state persisted

### Policy Enforcement
1. Timeout after specified duration
2. Retry with exponential backoff
3. Circuit breaker opens after threshold
4. Concurrency limited per policy
5. Retention period enforced

## Test Data Strategy

### Builders
Use builder pattern for test data:
```csharp
var request = new CreateProcessRequestBuilder()
    .WithClientId("test-client")
    .WithProcessType("order")
    .Build();
```

### Unique Identifiers
Use `Guid.NewGuid()` for test isolation

### Cleanup
TestContainers automatically clean up after tests

## Running Tests

```bash
# All tests
dotnet test

# Unit tests only
dotnet test --filter Category=Unit

# Integration tests only
dotnet test --filter Category=Integration

# Specific test class
dotnet test --filter FullyQualifiedName~ProcessLifecycleTests

# With coverage
dotnet test --collect:"XPlat Code Coverage"
```

## Debugging Tests

### View Logs
```bash
cat tests/**/bin/Debug/net8.0/logs/*.log
```

### Attach Debugger
1. Set breakpoint
2. Run test in debug mode
3. Containers remain running

### Inspect Containers
```bash
docker ps
docker logs <container-id>
```

## Coverage Goals

- **Overall:** >80%
- **Core Domain:** >90%
- **Infrastructure:** >85%
- **API/Server:** >75%

## Continuous Integration

Tests run automatically on:
- Every push to develop
- Every pull request
- Before merge to main

Coverage reports uploaded to Codecov.
```

## ✅ Acceptance Criteria

- [ ] Complete process lifecycle tests implemented
- [ ] Failure and retry workflow tests implemented
- [ ] Timeout enforcement tests implemented
- [ ] Retry policy with backoff tests implemented
- [ ] Circuit breaker behavior tests implemented
- [ ] Concurrency limit tests implemented
- [ ] Client policy override tests implemented
- [ ] Message broker integration tests implemented
- [ ] State persistence tests implemented
- [ ] Error tracking tests implemented
- [ ] All tests pass consistently
- [ ] Tests run in <5 minutes total
- [ ] Testing strategy documented
- [ ] Code follows CODING-CONVENTIONS.md

## 📝 Testing Instructions

```bash
# Run all workflow tests
dotnet test tests/StarGate.IntegrationTests/Workflows

# Run lifecycle tests
dotnet test --filter "FullyQualifiedName~ProcessLifecycleTests"

# Run policy enforcement tests
dotnet test --filter "FullyQualifiedName~PolicyEnforcementTests"

# Run with detailed output
dotnet test tests/StarGate.IntegrationTests/Workflows --verbosity detailed

# Watch mode for development
dotnet watch test tests/StarGate.IntegrationTests/Workflows

# Verify specific scenarios
# 1. Successful completion
dotnet test --filter "ProcessLifecycle_Should_CompleteSuccessfully"

# 2. Retry behavior
dotnet test --filter "RetryPolicy_Should_RetrySpecifiedTimes"

# 3. Timeout enforcement
dotnet test --filter "TimeoutPolicy_Should_CancelLongRunningOperations"

# 4. Concurrency limits
dotnet test --filter "ConcurrencyPolicy_Should_LimitSimultaneousProcesses"
```

## 📚 References

- [TECHNICAL-ANALYSIS.md - Phase 9.1](https://github.com/artcava/StarGate/blob/develop/docs/TECHNICAL-ANALYSIS.md)
- [Process State Machine](../docs/PROCESS-STATE-MACHINE.md)
- [Policy Configuration](../docs/POLICY-CONFIGURATION.md)
- [CODING-CONVENTIONS.md](https://github.com/artcava/StarGate/blob/main/docs/CODING-CONVENTIONS.md)

## 🏷️ Labels

`phase-9` `testing` `sprint-9.1` `e2e-tests` `workflow-tests` `policy-tests`

## ⏱️ Estimated Effort

**12-16 hours**

## 🔗 Dependencies

- Issue #110: API integration tests (infrastructure setup)
- Phase 7: ProcessWorker and handlers (must be functional)
- Phase 8: Resilience policies (for policy enforcement tests)

## 🔗 Related Issues

Part of Phase 9: Testing & Quality - Sprint 9.1: Integration Tests

## 📌 Important Notes

### Process State Transitions

```
Pending → Processing → Completed (success)
        → Retrying → Processing (retry)
        → Failed (max retries)
        → Cancelled (timeout)
```

### Timing Considerations

**Why `Task.Delay` in tests:**
- Async process execution
- Message broker latency
- Handler execution time
- State persistence delay

**Typical delays:**
- Process creation: immediate
- Worker pickup: ~500ms
- Handler execution: 1-2s
- State update: ~100ms

**Test timeouts:**
- Fast tests: 2-3s wait
- Retry tests: 5-10s wait
- Long tests: up to 15s

### Retry Backoff Calculation

**Policy: maxRetries = 3**

```
Attempt 1: Immediate
Attempt 2: +1s delay
Attempt 3: +2s delay
Attempt 4: +4s delay
Total: ~7s minimum
```

### Concurrency Testing Challenges

**Race Conditions:**
- Multiple processes starting simultaneously
- Timing-dependent behavior
- Non-deterministic test results

**Mitigation:**
- Use delays strategically
- Check ranges instead of exact values
- Retry flaky tests
- Use `Should().BeLessOrEqualTo()` instead of exact match

### Message Broker Behavior

**ACK (Acknowledge):**
- Message processed successfully
- Removed from queue
- Process completed

**NACK (Negative Acknowledge):**
- Processing failed
- Message requeued
- Retry attempted

**Reject:**
- Permanent failure
- Message moved to dead letter queue
- Process marked as Failed

### Policy Override Priority

```
1. Client Override (highest priority)
2. Process Type Policy
3. System Defaults (lowest priority)
```

**Test verification:**
- Create type policy
- Create client override
- Verify override values used

### State Persistence Testing

**Scenarios:**
1. Create process → verify in DB
2. Update state → verify persisted
3. Query process → verify from DB
4. "Restart" → verify survives

**Implementation:**
- Use same factory instance
- Create new HTTP client
- Simulates application restart
- Verifies MongoDB persistence

### Error Tracking

**Error Structure:**
```json
{
  "errors": [
    {
      "message": "Customer ID is required",
      "timestamp": "2026-02-18T14:30:00Z",
      "attemptNumber": 1
    }
  ]
}
```

**Test verification:**
- Error messages captured
- Timestamps recorded
- Attempt numbers tracked
- Accessible via API

### Performance Expectations

**Test Execution Times:**
- ProcessLifecycleTests: ~30s
- PolicyEnforcementTests: ~45s
- MessageBrokerIntegrationTests: ~15s
- **Total: ~90s**

**Optimization:**
- Parallel test execution
- Shorter test timeouts
- Faster container startup

### Flaky Test Prevention

**Common causes:**
- Insufficient wait times
- Race conditions
- Container startup delays
- Resource contention

**Solutions:**
- Generous delays (add buffer)
- Poll for state changes
- Retry flaky tests
- Use `Should().Eventually()`

### CI/CD Considerations

**GitHub Actions:**
- Containers start slower in CI
- Add extra wait time
- Increase test timeouts
- Monitor for flakiness

**Coverage Integration:**
- Upload after all tests
- Combine coverage reports
- Set threshold gates
- Block PRs below threshold

Phase 9.1: Implement End-to-End Workflow and Policy Enforcement Tests #111

Description

📋 Task Description

🎯 Objectives

📦 Deliverables

1. Create End-to-End Workflow Tests

2. Create Policy Enforcement Tests

3. Create Message Broker Integration Tests

4. Create Test Documentation

Unique Identifiers

Cleanup

Running Tests

Debugging Tests

View Logs

Attach Debugger

Inspect Containers

Coverage Goals

Continuous Integration

📚 References

🏷️ Labels

⏱️ Estimated Effort

🔗 Dependencies

🔗 Related Issues

📌 Important Notes

Process State Transitions

Timing Considerations

Retry Backoff Calculation

Concurrency Testing Challenges

Message Broker Behavior

Policy Override Priority

State Persistence Testing

Error Tracking

Performance Expectations

Flaky Test Prevention

CI/CD Considerations

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions