Add an API call that acts as a message that a DTS transfer has finished.

[briehl]

Summary

The staging service needs an API endpoint that the DTS service can invoke when it has finished transferring files to a temporary area, so the staging service can complete the transfer by copying those files to the user's datastore. This proposes calling it complete_transfer.

Background

See also #200
The current design around how the DTS operates is that it has permission to copy files to an intermediate storage area in Globus. However, it does not have permission to write directly to users' directories, which it shouldn't. To solve this, the staging service needs to be alerted when a file copy operation is completed, so it can finish the file copy process.

Term definitions

There's a couple terms in here that'll get used, I just want to make sure they're properly defined

• DTS - Data Transfer Service - the external(ish) service that moves data around from outside KBase to inside.
• DTS user - the service account that the Data Transfer Service uses
• DTS directory - the intermediate Globus directory that the DTS drops data into

Proposal

The proposal here is a new endpoint tentatively called complete_transfer.
This should be an authenticated call, only allowed by a user with the DTS_TRANSFER auth credential [not sure if something like that exists or not yet].
This should be a POST command with the following body variables

• target_user: the user to send the files to
• subdirectory: the path to the subdirectory in the DTS staging area

The command should result in the subdirectory and all of its contents getting copied to either the user's root directory or a dts_transfers subdirectory (TBD). It should check that the files were copied correctly via checksum.

When complete, it should return:

• A 201 response if all is copied well, with body {"path": "/user/new_directory"}
• A 400 error if either variable is missing, or if the user doesn't exist.
• A 401 if the caller isn't authenticated
• A 403 if the caller doesn't have permission (either auth role or globus permissions)
• A 404 if the source directory doesn't exist
• A 500 if the copy failed for any reason. Any copied files and new directories should be removed as well (should they?).

An edge case here is what to do if the target directory already exists. It's unlikely, since DTS transfers show up in a UUID-named directory, but if so, they should just get appended with -1 and return the new directory as above.

Options to be decided

1. Either have the calling service specify the complete Globus path to the file endpoint, which might not be /data/bulk/... but some other Globus share, or specify that in a new config option. This would make the difference between a call like:

{ "target_user": "kbase_user", "subdirectory": "/data/dts/some_subdir" }
vs
{ "target_user": "kbase_user", "subdirectory": "some_subdir" }

2. Target directory. Should this be under the user's root, or a specified dts subdirectory? If in the user's root, then they'll get a UUID-named directory appearing, which is probably fine as it'll be expected, but might get lost if they have a bunch of other files present. If in a subdirectory, they'll know where to look, at least.
3. Copying vs. moving. IMO copies should be done and verified via checksum in case any file system things go wonky. Unless there's a direct globus call to do that, we'll have to make some Python shutil calls or similar. After copying is done and verified, we can either delete source files in the service call, or wait for external cleanup to happen.
4. File ownership. These should get updated as well, once copies are done. [not sure if necessary, the file upload endpoint doesn't do this]

[briehl]

Another point that comes to mind is that the DTS currently just acts on a user's behalf using a user's auth token. It doesn't have a service token, and just relies on Globus permissions to do file things.

IMO, the DTS should get its own service account with a DTS_TRANSFER auth role (or similar). Then, only authenticated calls with that role will be able to ask the service to alter the users staging area.

[briehl]

IMO for the options I put at the bottom:

1. Make the root directory configurable. This avoids cases where files can be moved from one user to another.
2. Target directory should be a subdirectory. This is just personal preference, though.
3. No preference here for what happens to the files after, but copies should be done and verified.
4. I don't feel like we'll need a file ownership change, but I don't know enough of the guts of Globus to be sure.

[briehl]

Another question: do new KBase users automatically have Globus ACLs at this point? Or will that need to get verified / throw an error if not?

[MrCreosote]

I assume that the staging service has local read to the DTS directory. If that's true, what about a system where the DTS adds a specifically named file containing the kbase user name to the root of a transfer directory when the data xfer is complete? Then the DTS just uses watchfiles (or whatever) to watch for those files and does the transfer. I assume only the DTS has access to write to that directory, so then you don't need to worry about the various permissions issues.

[MrCreosote]

I would make the staging service copy files into a subdir called data-transfer-service/ personally to give the user some idea of what's in the subdirs

[MrCreosote]

do new KBase users automatically have Globus ACLs at this point?

They certainly can't if they don't have a globus account linked to their KBase account. I'm not sure exactly what triggers creation of a globus staging service dir / acls; I assume it's lazy and doesn't happen until the staging service is invoked for the first time

[jeff-cohere]

I like Gavin's idea about using watchfiles (etc). As it happens, the DTS writes a manifest.json file to the root of a (user-specific) transfer directory already, and it would be very easy to stick the kbase username into that file. The only problem with this approach is that the manifest can be quite large, depending on the payload, so maybe it would be good to generate a smaller file just for the purpose of indicating to the staging service that everything is ready to go.

[jeff-cohere]

I also like the idea of copying via the local filesystem instead of Globus because it sidesteps all the Globus permission problems we've seen in our limited DTS explorations so far.

[briehl]

It looks like a KBase bulk share endpoint gets made by user action through the staging service. There's steps to go though to link a globus account, but I don't think it's necessary here.

[briehl]

Another +1 to using watchfiles or something like it. @jeff-cohere if it's possible to make a little empty (or nearly empty) file afterward to look for, that would make sense, too.

This then turns into putting a file watching function in the staging service and letting it handle things without a new endpoint.

My only issue with that is it becomes a little black-boxy, and if errors happen, the DTS won't know about it. Then again, if errors happen, the DTS might not be able to do much about it.

[MrCreosote]

You could add a status endpoint that an admin or user could hit with the DTS transfer ID to check the status of the transfer

[jeff-cohere]

It's really easy for me to generate and transfer a tiny file with a KBase username in it. I agree that the error reporting on the DTS side becomes a bit opaque, but as long as a user can find out what happened on the staging service side (and can retry, since all the files could be kept around in the case of failure), I don't think this is a big deal, because these transfers often take too long for feedback via an interactive request channel to be meaningful.

[jeff-cohere]

UPDATE: We're trying to use the manifest itself as the trigger. I'm adding a username field to the manifest that stores the local KBase username for the person requesting the transfer: kbase/dts#135

[briehl]

I put together a couple small scripts that watch for files and trigger an update based on their status. One uses watchfiles and the other uses watchdog. There are a couple of constraints that I'm running into, possibly due to my dev environment.

1. They both fire "file created" events quickly and consistently.
2. They're both inconsistent about "file updated" events. So if the manifest is large, or there's network lag, or whatever, there might not be another event. This is especially noticeable in watchfiles. watchdog generally passes an event when the file is done moving.
3. None of these are very clear "file is done loading" events, just a notice that the file's been modified. I'd recommend just watching the file for further change events, or size changes, until it stabilizes for a few seconds. Polling the size until it stops growing has been much more consistent than waiting on events.

Now to the obnoxious part: these tests were done and work fine on my local dev environment. When testing in a built container, no events get forwarded from the host (a mac). I set up like this:

1. Build container.
2. Run interactive with a volume mount for the made up file space.
3. Run the watcher script inside the container.

If I modify files outside the container (just touching from my host), nothing happens. If I exec in to that container and make changes that way, it works fine.

From doing a little digging, this seems to be just inherent to MacOS -> Rancher Desktop -> Docker container not passing file events through as expected. The little uncertainty I have now is will this be an issue in production, which I presume is running on a Linux host. From research, this seems like it'll be fine.

The other option here is just to have watchfiles force polling and look for updates in some interval. I'd set this up as a config so local dev can go forward.