Phase 3: Async Pipeline for Performance

[dannycab]

Description

Implement async/concurrent processing for 10-100x performance improvement.

Current Performance

• Sequential processing: ~60-120s per page
• 50 pages = 50-100 minutes

Target Performance

• Concurrent processing: 50 pages in 5-10 minutes
• Proper retry logic and error recovery

Tasks

• Implement async HTTP client (aiohttp)
• Create job queue system (asyncio.Queue or Redis)
• Implement worker pool with ThreadPoolExecutor
• Add retry logic with exponential backoff
• Implement progress tracking
• Add rate limiting per-domain
• Create checkpoint/resume functionality
• Add --workers N flag to CLI
• Add --rate-limit flag
• Implement graceful shutdown on Ctrl+C

Success Criteria

• 10x faster for batches >20 pages
• No crashes with 100+ concurrent pages
• Proper resource limits (memory, CPU)
• Real-time progress display