Successive syncs create duplicate chat objects

[netvor]

I have an issue whereby successive (full) syncs cause the same chat object to be downloaded multiple times. The compressed objects are all different, yet the plaintext contents are the same:

$ cd gmvault-db/db/chats/

$ md5sum */1437017132845624821.eml.gz
93db2cdf56f65c3393f48a7ac3822a89  subchats-2/1437017132845624821.eml.gz
7d891df7672a2346741de39773dc9810  subchats-3/1437017132845624821.eml.gz
cc2dac40e35bdfbcd06db0e6785d1f77  subchats-4/1437017132845624821.eml.gz

$ cp subchats-2/1437017132845624821.eml.gz /tmp/sc2.gz
$ cp subchats-3/1437017132845624821.eml.gz /tmp/sc3.gz
$ cp subchats-4/1437017132845624821.eml.gz /tmp/sc4.gz
$ gunzip /tmp/sc2.gz
$ gunzip /tmp/sc3.gz
$ gunzip /tmp/sc4.gz

$ md5sum /tmp/sc*
8f96d8ec223ea64c13a028cc9038a694  /tmp/sc2
8f96d8ec223ea64c13a028cc9038a694  /tmp/sc3
8f96d8ec223ea64c13a028cc9038a694  /tmp/sc4

These duplicates are not created every time. Generally when there is nothing to update (no new emails or chats) it does not happen, but when there is a new chat recorded, I usually get a duplicate. As far as I know this only affects chat objects, not mail objects.

To localize the problem better, I disabled compression and did a series of --chats-only syncs.

1. Initial sync: gmvault sync --no-compression --chats-only. 1267 chats stored in subchats-1
2. Force an update, i.e. send a chat message (I use a 3rd-party Jabber client, not the native Google app)
3. gmvault sync --no-compression --chats-only. This time 1268 chats stored in both subchats-1 and subchats-2
4. gmvault sync --no-compression --chats-only. This time no change.
5. Force an update
6. gmvault sync --no-compression --chats-only. This time 1268 chats stored in subchats-1 and subchats-2, 1269 chats stored in subchats-3
7. gmvault sync --no-compression --chats-only (so no update). This time 1268 in subchats-1 and -2, 1269 in -3 and 538 (huh?) in subchats-4

So you see the behavior is not very predictable. Another observation is that the different md5sum of the .gz duplicates is only a side-effect of gzip storing the timestamp of the .eml in the .gz file.

As to the duplicates, after accumulating these four subchats- folders, I discovered they are not always identical: if they are Content-Type: multipart/alternative, then the "boundary" string differs between duplicates. The .meta files are always identical.

I suppose my main question is: what is the logic behind creating new subchats- directories?

[Nyr]

This is happening to me too. Each time I run a full sync, gmvault downloads all chats on new folders.

[gaubert]

@netvor @Nyr Just got back from a long vacation. This seems to be a bug. I will check it asap and let you know.
Thks

[neogenix]

any news on this, I'm getting the same sort of behavior.

[adept]

I think that this should be fixed by #224

[maneatingduck]

I can confirm that duplication of chats is still occurring in 1.9.1/master on Windows (running from github source), when doing a full sync. I assume that it can't find the existing item in the subchats-x folders, and creates a new file. Emails might work fine due to then having folder names based on metadata?

I have a lot of chats, so I can test setting the limit per directory to something larger than my number of chats if that's interesting.

ls -R .\chats\ -name 1454569519569329389.meta
subchats-1\1454569519569329389.meta
subchats-6\1454569519569329389.meta

EDIT: I set the upper limit per directory to something much larger than my actual number of chats, moved all chat files to subchats-1, and deleted the remaining subchats-XX folders.

The problem seems to be with the _common_sync function only looking at the current subchat folder, not all subchat folders. It only sends the current folder to check_email_on_disk. I could probably make a fix, but a real Python dev will surely do it faster and better.

[atppp]

Yes I can confirm @maneatingduck's comment, both being still an issue with 1.9.1/master and the analysis.

[gaubert]

ok I need to have a look at that one. It is on my todo list for 1.9.2

[gaubert]

ok found the bug. Will fix it in the next coming days

[gaubert]

@atppp @maneatingduck @netvor @Nyr @neogenix @adept don't know why I missed that bug for so long 👎 Anyway I believe I fixed it and it will be 1.9.2 . Is one of you ready to test the beta version of 1.9.2 ?
Please let me know and on which platform you are and I will ship you next week the beta version.

Many thanks !!!

[atppp]

I am on Mac. I could test gmv-1.9.2 branch this weekend if that's equivalent? Thank YOU for fixing it.

[gaubert]

Yes you can do that or take the following functional gmv-1.9.2 version. There are still few things to fix but it should work. It is available from here: https://www.dropbox.com/s/dkya69kkef9xwa9/gmvault-v1.9.2-beta-macosx-intel.tar.gz?dl=0 Let me know if there are some issues.

…

On 20 January 2017 at 16:21, atppp ***@***.***> wrote: I am on Mac. I could test gmv-1.9.2 branch this weekend if that's equivalent? Thank YOU for fixing it. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#135 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAFynb1lmsGpQJF6Vi9ELmjSAF6AMpyks5rUND0gaJpZM4Atgwu> .