Performance Optimization & Code Cleanup

[pouria-mirebrahimi]

Story MP-015: Performance Optimization & Code Cleanup

Description:
Optimize I/O, memory footprint, and stage latencies; reduce logging noise; remove debug artifacts; and ensure clean, maintainable source tree.

Tasks:

• Profile pipeline to identify hotspots (ingestion batching, NaN handling, dedup, export)
• Optimize Parquet write settings (row group size, compression) and reduce unnecessary DataFrame copies
• Tune parallelism/async where safe; ensure no race conditions in log or export writers
• Reduce log verbosity to actionable INFO; keep DEBUG only for dev toggles
• Remove dead code and debug scaffolding; enforce lint/format across repository

Acceptance Criteria:

• Measurable latency reduction on at least two critical stages
• Memory spikes mitigated under multi-symbol runs
• Logs are concise and actionable (no excessive noise)
• Lint/format checks pass; repository free of debug artifacts

Estimation: 1.5 days
Dependencies: MP-013, MP-014
Status: Planned