[BUG]: Resume preprocessing can duplicate output after crash between checkpoint saves

[AnshuPriya-1]

Bug Description

The current resume logic in extract_text_from_xml() uses last_processed_page from data/checkpoint.json to decide where to continue, and opens the output file in append mode whenever start_page > 0.

However, output is written continuously while checkpoints are only saved every 1000 pages. If the process crashes after writing additional pages but before the next checkpoint update, the output file may already contain more content than the checkpoint reflects.

On resume:

• pages up to last_processed_page are skipped
• output file is reopened in append mode
• pages written after the last checkpoint may be appended again

This can duplicate processed content in data/processed/wiki_clean.txt, which may also affect processed hashes and Merkle roots.

Steps to Reproduce

1. Create or use an XML dump with multiple <page> entries.
2. Run extract_text_from_xml() and let it process beyond the latest saved checkpoint.
3. Simulate an interruption after extra cleaned text has already been written to data/processed/wiki_clean.txt but before the next checkpoint update is saved.
4. Ensure data/checkpoint.json still contains an older last_processed_page value than the content already written to the output file.
5. Run extract_text_from_xml() again to resume processing.
6. Observe that pages written after the last checkpoint can be appended again, causing duplicated cleaned text in data/processed/wiki_clean.txt.

Logs and Screenshots

No runtime traceback is required to observe this issue.

The problem is a data-correctness bug in the resume logic:

• checkpoint progress is tracked by page number
• output progress is tracked only by the contents already appended to wiki_clean.txt
• if these get out of sync after an interruption, resume may duplicate already-written content

Environment Details

• OS: Windows 11
• Python version: 3.x
• Repository: AOSSIE-Org/OpenVerifiableLLM
• Relevant file: openverifiablellm/utils.py
• Function: extract_text_from_xml()

This issue appears to be logic-related and should be reproducible across environments.

Impact

High - Major feature is broken

Code of Conduct

• I have joined the Discord server and will post updates there
• I have searched existing issues to avoid duplicates

[AnshuPriya-1]

This appears to be related to #54, but is narrower in scope.

Issue #54 focuses on adding checkpoint/resume support generally, while this issue highlights a specific data-correctness problem in the current resume logic: if output is written beyond the latest saved checkpoint and the process crashes before the next checkpoint update, resuming in append mode may duplicate already-written content.

Linking this as a follow-up bug in the checkpoint/resume flow.

[ANUBprad]

Closes #76

Fix: Prevent duplicate output during resume in extract_text_from_xml

Problem

The resume logic relied on checkpoint.json to determine progress, but output
was written continuously. If the process crashed after writing additional
pages but before checkpoint update, the output file could contain more
pages than recorded in the checkpoint.

This caused duplicate content when resuming.

Solution

• Added logic to detect mismatch between output file and checkpoint state
• Ensured output file is truncated to match checkpoint before resuming
• Maintained consistency between logical progress and physical output

Changes

• Added helper functions for counting and truncating processed pages
• Updated extract_text_from_xml() in utils.py
• Improved checkpoint reliability

Testing

• Simulated interrupted runs (Ctrl + C)
• Verified no duplicate content after resume
• Verified correct truncation behavior when mismatch occurs
• Tested edge case where output exceeds checkpoint

Impact

Fixes a critical data consistency issue and ensures reliable resume behavior
for large dataset preprocessing