Use proper last_id files when syncing chats and when restoring chats

[adept]

Currently chat sync and chat restore clobbers email last_id file, causing subsequent email operations to start from scratch.

[gaubert]

@adept looks good. oupps for the mistake. I did some quick tests but as I read the code everything seems ok. I will do more tests tomorrow and will accept the merge.

[gaubert]

@adept ok I made some tests and it is all fine. Merging now. Thanks for the fix

[absinthetized]

don't know if anyway related but if I relay on last_id for subsequent mail syncs, last id is NOT highest id just a "random" number. therefore restarting the sync leads to duplicated mails.

I've cases when something like 170 mails are duplicated at every sync (and I cron them every hour!)

let me attach a log (log format slightly adjusted to fit my needs) of a case where duplicated mails are less (just to keep logs small)

used version is v.1.9.1, accounts are corporate (google apps for business)

sync at 7am local time (+2 UTC)

[2016-09-09 05:00]:CRITICAL:credential_utils:Authentication performed with Gmail OAuth2 access token.

[2016-09-09 05:00]:CRITICAL:credential_utils:Get OAuth2 credential from [CLASSIFIED :-)].

[2016-09-09 05:00]:CRITICAL:gmv:Connect to Gmail server.

[2016-09-09 05:00]:CRITICAL:gmvault:Start synchronization (2016-09-09T07h00m48s).

[2016-09-09 05:00]:CRITICAL:gmvault:Start emails synchronization.
[2016-09-09 05:00]:CRITICAL:gmvault:Restart mode activated for emails. Need to find information in Gmail, be patient ...
[2016-09-09 05:00]:CRITICAL:gmvault:Restart from gmail id 1544706246256668156 (imap id 479).
[2016-09-09 05:00]:CRITICAL:gmvault:5 emails to be fetched.
[2016-09-09 05:00]:CRITICAL:gmvault:Process email num 0 (imap_id:480) from 2016-09.
[2016-09-09 05:00]:CRITICAL:gmvault:Process email num 1 (imap_id:481) from 2016-09.
[2016-09-09 05:00]:CRITICAL:gmvault:Process email num 2 (imap_id:490) from 2016-09.
[2016-09-09 05:00]:CRITICAL:gmvault:Process email num 3 (imap_id:491) from 2016-09.
[2016-09-09 05:00]:CRITICAL:gmvault:Process email num 4 (imap_id:479) from 2016-09.
[2016-09-09 05:00]:CRITICAL:gmvault:
Emails synchronisation operation performed in 2s.

Skip chats synchronization.
[2016-09-09 05:00]:CRITICAL:gmvault:

sync at 8am local time (same mail address, +2 UTC)

2016-09-09 06:00:CRITICAL:credential_utils:Authentication performed with Gmail OAuth2 access token.

2016-09-09 06:00:CRITICAL:credential_utils:Get OAuth2 credential from [CLASSIFIED :-)].

2016-09-09 06:00:CRITICAL:gmv:Connect to Gmail server.

2016-09-09 06:00:CRITICAL:gmvault:Start synchronization (2016-09-09T08h00m43s).

2016-09-09 06:00:CRITICAL:gmvault:Start emails synchronization.
2016-09-09 06:00:CRITICAL:gmvault:Restart mode activated for emails. Need to find information in Gmail, be patient ...
2016-09-09 06:00:CRITICAL:gmvault:Restart from gmail id 1544706246256668156 (imap id 479).
2016-09-09 06:00:CRITICAL:gmvault:5 emails to be fetched.
2016-09-09 06:00:CRITICAL:gmvault:Process email num 0 (imap_id:480) from 2016-09.
2016-09-09 06:00:CRITICAL:gmvault:Process email num 1 (imap_id:481) from 2016-09.
2016-09-09 06:00:CRITICAL:gmvault:Process email num 2 (imap_id:490) from 2016-09.
2016-09-09 06:00:CRITICAL:gmvault:Process email num 3 (imap_id:491) from 2016-09.
2016-09-09 06:00:CRITICAL:gmvault:Process email num 4 (imap_id:479) from 2016-09.
2016-09-09 06:00:CRITICAL:gmvault:
Emails synchronisation operation performed in 1s.

Skip chats synchronization.

[adept]

Both log snippets you provided list exactly same last_id. Do you have any idea why this is happening?

[absinthetized]

yes: this is the last id stored into the .info folder. gmvault reloads the file and restarts downloading from that id.

My guess is that ids are not well sorted: look at imap_id values: 479 is the last id but it is not the highest.

• vault downloads ids in this apparently random order: 480, 481, 490, 491, 479
• than stores the last id as 479 (which is right in the sense it is the last but not the highest - more recent-)
• next time it restarts and loops until a really new mail is found and in that case maybe the last id changes for the best.

In the end I mean: last id is not most recent id. I'm rather busy on an internal project, don't know if I can allocate time to sort this out. sorry. Anyway I can do some tests on master if you think this would help!

[adept]

I think that the fix is quite easy (see pull request #257).

@gaubert , I might be wrong but I think a call to save_lastid might be in order at the end of common_sync (otherwise the last id saved, I think, would be from the email that has num % 10 = 0). Can you please check if this is correct?