Perform Stage-Level Dry Runs and QA Validation

[pouria-mirebrahimi]

Story MP-012: Perform Stage-Level Dry Runs and QA Validation

Description:
Run the complete pipeline for multiple assets and verify correctness, logging, and quality assurance metrics meet defined thresholds.

Tasks:

• Run pipeline for 3–5 assets to test full integration
• Validate all stage outputs and logs for completeness
• Check QA stage produces correct metrics (≥0.95 threshold)
• Verify export files generated successfully in asset-first layout
• Compile and archive QA reports in /quality_reports/

Acceptance Criteria:

• All stages execute with no critical errors
• Logs show consistent stage completions
• Quality metrics ≥ 0.95 for all datasets
• Export files validated and readable via Parquet schema check

Estimation: 2.5 days
Dependencies: MP-009, MP-010
Status: Planned

[pouria-mirebrahimi]

Check quality of the fetched data, compare with the data at the database in the server. All timestamps, API filters, paginations, query must be evaluated and verified.

@nikbakhtenb4 @Farhad-Valipour