HTTP 404 terminates manifest update

[xKevin04]

Unfortunately, GOG continues to throw spanners in your gears.

I ran into two products that prevented me from updating my manifest via gogcli.exe manifest update --update=updates.json. The outputs stopped like this:

[manifest writer] Got all info on game with id 1174117468. 300 games left to process
[manifest writer] Got all info on game with id 1169744036. 299 games left to process
 getDownloadFileInfoWorkaroundWay(downloadPath=/downloads/spiritfarer_ost/en1patch3) -> body download handle retrieval error: did not expect status code of 404

 getDownloadFileInfoWorkaroundWay(downloadPath=/downloads/prison_architect_going_green/en1patch0) -> body download handle retrieval error: did not expect status code of 404

After this, the program terminated immediately (pretty sure it must have been via a non-zero exit code but I didn't check). It created a manifest-update-progress.json, but I couldn't find a way to make gogcli actually use that to resume the update, it restarted from scratch on every retry. I logged the output at the various available levels, but couldn't get anything more useful out of it.

As the problem was perfectly reproducible and happened on the same url every time I tried, I just removed the respective product id from my updates.json file and reran the command, rinse and repeat until I got no terminating error anymore. This might just be a temporary problem and I will retry after my downloads are done, but I wanted to report it anyway so you can decide if it's worth handling.

[xKevin04]

It seemed like the update went through fine after downloading everything (which worked perfectly, btw!). However, I realized that my update flags from GOG are too out of sync with reality by now. Neither Witcher 3 nor its Complete Edition has been updated, for example, even though I only keep the former updated via Galaxy at the moment.

So I tried to generate a fresh manifest from scratch. This terminates in exactly the same way as updating does, on the same url from Spiritfarer too. Also like updating, it creates a manifest-generation-progress.json that isn't used when I run the command again, but I have no idea why (or if this is even supposed to work on v0.21.0 already, as you committed some work on it afterwards).

Some way to automatically ignore or postpone these in a functional progress file would be great. As it stands I would have to rerun the manifest generation, appending the next terminating 404 url to --skip-url after every try, until it finally goes through completely. I have no idea how many of those I will encounter, and with thousands of games and still requiring the workaround method in a lot of cases, this can take forever.

[Magnitus-]

@xKevin04 Sorry for not noticing this issue earlier.

The tool can differentiate between files that are dead links on purpose (because GOG decided to list an entry in the files that you can't download like Republique's "Coming Soon" manual) and dead links that are accidental (ie, GOG's servers didn't load the files properly).

It ignores the former, but after getting manifests with some missing files that were temporarily unavailable on GOG's servers, I decided to make the tool stricter and not ignore the later.

Both generated manifests and manifest progress files have a "SkipUrls" entry that contains regular expressions of urls to skip. When it is an installer file that is missing, I prefer to just abort and file and issue with GOG's support, but when it is a patch file, I tend to just add it to the exceptions.

Mine looks like this right now (because of 3 bad patch files):

      "SkipUrls": [
        "^/downloads/spiritfarer_ost/en1patch3$",
        "^/downloads/heart_of_the_woods/en1patch1$",
        "^/downloads/prison_architect_cleared_for_transfer/en1patch0$"
      ],

To resume manifest generation/update (after you've added the entries to skip), you can use the manifest or update resume commands.

My next change to the tool will probably be a "--skip-missing-patches" flag to make this process automated.

[xKevin04]

Hey, no problem. Thanks for the info, I only checked the command help for manifest generate and manifest update. You're right, there are also manifest generate-resume and manifest update-resume as seperate commands. My bad for not looking more thoroughly, somehow expected them as additional arguments instead of their own subcommands. :)

I will try to generate a new manifest, resuming and adding to the SkipUrls-list along the way. If it works out as I imagine it should, then that's good enough as another workaround for GOG's problems. I'll report the broken files to GOG too, just wanted to actually have a comprehensive list of them first.

[xKevin04]

As expected it went great, I'm using my fresh manifest as I write. The total runtime is also at an acceptable level for me now, so I will go back to generating complete manifests again, instead of updating it. Hopefully they fix even more broken metadata problems until then.

Anyway, thanks again, your tool is really invaluable! You can close this issue or repurpose it for your planned --skip-missing-patches enhancement if you want. Here's the list of broken URLs I ended up with:

"SkipUrls": [
  "^/downloads/heart_of_the_woods/en1patch1$",
  "^/downloads/prison_architect_cleared_for_transfer/en1patch0$",
  "^/downloads/prison_architect_gangs/en1patch0$",
  "^/downloads/prison_architect_going_green/en1patch0$",
  "^/downloads/prison_architect_island_bound/en1patch0$",
  "^/downloads/prison_architect_psych_ward_wardens_edition/en1patch0$",
  "^/downloads/prison_architect_second_chances/en1patch0$",
  "^/downloads/spiritfarer_ost/en1patch3$"
]

There's really something going wrong with how they create these Prison Architect DLC links...

[Magnitus-]

Some games are indeed worst than others with bad installers. Clearly its an intersection between the devs and GOG's apis.

I'll keep this ticket around for the feature.