[CLEAN] Synthetic Benchmark PR #11008 - fix(backend): prevent duplicate graph executions across multiple executor pods

[ofir-frd]

Benchmark PR Significant-Gravitas#11008

Type: Clean (correct implementation)

Original PR Title: fix(backend): prevent duplicate graph executions across multiple executor pods
Original PR Description: ## Problem
Multiple executor pods could simultaneously execute the same graph, leading to:

• Duplicate executions and wasted resources
• Inconsistent execution states and results
• Race conditions in graph execution management
• Inefficient resource utilization in cluster environments

Solution

Implement distributed locking using ClusterLock to ensure only one executor pod can process a specific graph execution at a time.

Key Changes

Core Fix: Distributed Execution Coordination

• ClusterLock implementation: Redis-based distributed locking prevents duplicate executions
• Atomic lock acquisition: Only one executor can hold the lock for a specific graph execution
• Automatic lock expiry: Prevents deadlocks if executor pods crash or become unresponsive
• Graceful degradation: System continues operating even if Redis becomes temporarily unavailable

Technical Implementation

• Move ClusterLock to backend/executor/ alongside ExecutionManager (its primary consumer)
• Comprehensive integration tests (27 test scenarios) ensure reliability under all conditions
• Redis client compatibility for different deployment configurations
• Rate-limited lock refresh to minimize Redis load

Reliability Improvements

• Context manager support: Automatic lock cleanup prevents resource leaks
• Ownership verification: Locks can only be refreshed/released by the owner
• Concurrency testing: Thread-safe operations verified under high contention
• Error handling: Robust failure scenarios including network partitions

Test Coverage

• ✅ Concurrent executor coordination (prevents duplicate executions)
• ✅ Lock expiry and refresh mechanisms (prevents deadlocks)
• ✅ Redis connection failures (graceful degradation)
• ✅ Thread safety under high load (production scenarios)
• ✅ Long-running executions with periodic refresh

Impact

• No more duplicate executions: Eliminates wasted compute resources and inconsistent results
• Improved reliability: Robust distributed coordination across executor pods
• Better resource utilization: Only one pod processes each execution
• Scalable architecture: Supports multiple executor pods without conflicts

Validation

• All integration tests pass ✅
• Existing ExecutionManager functionality preserved ✅
• No breaking changes to APIs ✅
• Production-ready distributed locking ✅

🤖 Generated with Claude Code

Co-Authored-By: Claude noreply@anthropic.com
Original PR URL: Significant-Gravitas#11008