Bulk download for all members of a project results in corrupted zip archives

[CuriousF0x]

Description

We wanted to download the files of our TeQfor1-project's members via the CLI call

ohproj-download --master-token OH_TOKEN --directory INPUT_PATH

Downloads runs fine, no error messages or aborts. However, some of the members' data are "corrupted", i.e., the zip-archives are not readable and only have a file size of 50 Bytes.
A cross check with one of our project member revealed the data were uploaded correctly (checked it via the website -> My data).

@gedankenstuecke and I already had a brief mail exchange on that one. @gedankenstuecke suggested to move to single member download as a workaround. Nevertheless, for the sake of documentation and completeness I wanted to open the issue here.

[gedankenstuecke]

Thanks! I could reproduce the issue on my end as well. My own best guess for what the problem might be is that the download of the files takes so long that the pre-generated download URLs are already expired by the time that the download is tried.

@CuriousF0x if you still have one of the problematic files you might be able to see this if you try to just open it in any text-editor. If it's expired links it might just be the HTTP error message from the file hoster?

[CuriousF0x]

@gedankenstuecke I opened one of the archives with a text editor as you suggested. Indeed, there ist a little message. It reads:

<h1>You are not authorized to view this file.</h1>

Not sure if you excepted that one...

[madprime]

Yeah, that would be consistent with Bastian’s hypothesis. I think it’s almost certainly solvable, the code needs to be changed.

On Thu, Oct 14, 2021 at 6:37 AM CuriousF0x ***@***.***> wrote: @gedankenstuecke <https://github.com/gedankenstuecke> I opened one of the archives with a text editor as you suggested. Indeed, there ist a little message. It reads: <h1>You are not authorized to view this file.</h1> Not sure if you excepted that one... — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#104 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAUFR6Q6XKGZYPE6XW6DRDUG3MJ5ANCNFSM5F7HULMQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

-- http://www.madpriceball.net/ Pronoun preference: they/them

[gedankenstuecke]

Yep, that would be the expected return error one gets when the download link has expired.

Currently the command generates all download links for each and every project member at the start of the process, which means the clock runs from the very beginning the same for each member. We can check how to adapt the code that it does it member-by-member instead :-)

[gedankenstuecke]

I didn't get around to fixing the issue yet, but isolated where the problem occurs:

When provided with a master_access_token, the CLI tool runs the following:

open-humans-api/ohapi/command_line.py

Lines 122 to 123 in af31795

	if master_token:
	project = OHProject(master_access_token=master_token)

This in turn triggers the init method of the OHProject:

open-humans-api/ohapi/projects.py

Lines 17 to 24 in af31795

	class OHProject:
	"""
	Work with an Open Humans Project.
	"""
	def __init__(self, master_access_token):
	self.master_access_token = master_access_token
	self.project_data = None
	self.update_data()

And in turn the update_data call of the OHProject immediately turns the master access token in the complete list of file download-URLs here:

open-humans-api/ohapi/projects.py

Lines 49 to 51 in af31795

	url = ('https://www.openhumans.org/api/direct-sharing/project/'
	'members/?access_token={}'.format(self.master_access_token))
	results = get_all_results(url)

The resulting data is then stored in the OHProject.project_data attribute and never refreshed. As this happens as one of the first things in the download method the later links can already be expired by the time the download actually comes around for them.

Investigating a solution will maybe take a bit more than just finding the problem, as I'm not 100% sure where else the update_data function is used and whether reworking that one might have unintended consequences down the road.