Metadata mismatch: subject/snippet from different message than body/from

[dvdsgl]

Summary

Some messages in the archive show metadata (subject, snippet) from one email while the stored raw MIME and parsed body/from are from a different email. The mismatch is visible in msgvault show-message <id> --json and in the UI: the subject line and snippet describe message A, but the body and sender are from message B.

Observed behavior

• msgvault show-message <id> --json returns:
  • subject and snippet: text that clearly belongs to a different message (e.g. an automated usage report)
  • from and body_text: correct, matching the actual message
• msgvault export-eml <id> shows the correct message: headers (From, Subject) and body all match each other and match from / body_text in the JSON.

So the bug is in the denormalized metadata in the messages table (subject, snippet). The raw MIME in message_raw and the parsed body/participants are correct.

Likely cause

From the source:

• Subject is set from the MIME parser (parsed.Subject in parseToModel).
• Snippet is set from the Gmail API response (raw.Snippet).

If the Gmail API sometimes returns a snippet (or if the parser’s subject were ever wrong) that doesn’t match the message’s actual content, the stored row would have this mixed metadata. Once written, incremental sync doesn’t re-fetch full content, so the wrong subject/snippet persist.

Environment

• msgvault v0.7.0 (commit 909fff4, 2026-02-09)
• Gmail account(s); sync via full and incremental

Suggestions

1. Repair path: A way to re-fetch or re-parse a single message (e.g. re-run MIME parse from message_raw and update messages.subject / messages.snippet) so affected rows can be fixed without a full re-sync.
2. Validation: Optionally compare stored subject/snippet to values derived from the raw MIME (or to Gmail’s envelope) and flag or auto-correct mismatches.
3. Fallback: When serving a message, if subject/snippet are inconsistent with body/from, consider deriving a preview from the parsed body instead of trusting the stored snippet.

Thanks for msgvault — the ability to verify with export-eml made it clear the raw data was correct and the issue was limited to the stored metadata.

[wesm]

Thanks, I will investigate!

[jkleske]

Additional data: systematic raw MIME duplication

I'm seeing the same class of bug but from the opposite direction: in my case, messages.subject, messages.snippet, and messages.sender_id are correct, while message_raw, message_bodies, and message_recipients contain data from a different message in the same sync batch.

Reproduction

Comparing two messages ingested in the same sync run (messages_added=2):

Field	msg 46602 (newsletter)	msg 46603 (personal reply)
messages.subject	"knee wraps vs knee sleeves" ✓	"AW: Check-in" ✓
messages.snippet	Newsletter content ✓	Personal reply content ✓
messages.sender_id → participants	mehdi@stronglifts.com ✓	marcel.nuernberger@adesso.de ✓
message_raw (SHA-256)	c813cafd8eb1b358...	c813cafd8eb1b358... identical!
message_bodies.body_text	Stronglifts content ✓	Stronglifts content ✗
message_recipients (from)	mehdi@stronglifts.com ✓	mehdi@stronglifts.com ✗

The raw MIME stored for msg 46603 is byte-identical to msg 46602. The MIME headers confirm it belongs to the newsletter (From: Mehdi, Subject: knee wraps), not to the personal reply. Meanwhile, the messages table row for 46603 has the correct metadata.

Systematic scan

I scanned all 46,623 message_raw entries by SHA-256 hash and found 41 duplicate pairs (82 affected messages). The pattern is 100% consistent:

• Every single duplicate pair was ingested in a sync run with messages_added >= 2
• No duplicates exist from sync runs that added only 1 message
• The issue started on 2026-02-16 (first affected: ID 46115) and continues in every sync run that adds 2+ messages
• The messages table metadata (subject, snippet, sender_id, sent_at) is always correct
• The message_raw, message_bodies, and message_recipients tables contain data from a sibling message in the same batch

Affected table summary

Table	Source	Correct?
messages (subject, snippet, sender_id, sent_at)	Gmail list API + DB lookup	✓ Always correct
message_raw	GetMessagesRawBatch → raw MIME	✗ Duplicated from sibling
message_bodies	Parsed from message_raw	✗ Wrong (follows raw)
message_recipients	Parsed from message_raw	✗ Wrong (follows raw)

Hypothesis

The concurrent fetch in GetMessagesRawBatch looks correct (each goroutine captures its own index). The bug is likely downstream — possibly a buffer reuse in GetMessageRaw, the base64 decode step, or in parseToModel/persistMessage where a shared slice or struct is reused across iterations of the for i, raw := range rawMessages loop in processBatch.

Query to check your own DB

-- Find duplicate raw MIMEs (requires Python for hashing, or compare sizes as approximation)
SELECT a.message_id AS id_a, b.message_id AS id_b, length(a.raw_data) AS size
FROM message_raw a
JOIN message_raw b ON a.raw_data = b.raw_data AND a.message_id < b.message_id;

Environment

• msgvault (binary, no version flag found) installed at ~/.local/bin/msgvault
• macOS (Apple Silicon), single Gmail account
• Incremental sync running every 15 minutes
• First affected message: ID 46115, archived 2026-02-16 09:11:08