"Blob storage" in object stores

[pfrazee]

Apps that access the object-stores need a way to store/publish non-objects such as pictures. This can be as part of the object-stores or by some other mechanism, but it needs to fit into the object-store flows.

[pfrazee]

Example of when this is needed: when you save a "contact" and need to save a profile pic.

[pfrazee]

A couple options:

1. We allow apps to freely make subfolders in the objects folders, and apply no constraints to those subfolders
2. We create extensions to json-schema which specify subfolders and the file-extensions that are allowed within them

Option 2 might be something like:

{
  // ...
  "title": "Contacts",
  "blobs": {
    "images": ["png", "jpg", "jpeg", "gif", "ico"]
  },
  "properties": {
  // ...
}

This would cause a ./images subfolder to be created, and enable writes in that folder which match the given extensions (png, jpg, etc).

[RangerMauve]

I think Option 1 would be easier for people since JSON-schema doesn't have a good way to represent files, and I'm not sure what the benefit would be of having a schema for them.

File uploads are usually open-ended so constraining them at the FS level could make life harder for a lot of use cases where you don't know the exact file extensions in advance.

[pfrazee]

The only reason I'm entertaining option 2 is to help keep apps coordinated with each other, which is a major function of the object-store. But it might not be worth trying to solve.

[RangerMauve]

I think that somehow identifying a folder for files would be cool, but I'm not sure about constraining the contents.

Also, do files really need to be inside the object store? Couldn't an application instead make use of a regular folder for raw files and link to it?

[pfrazee]

Also, do files really need to be inside the object store? Couldn't an application instead make use of a regular folder for raw files and link to it?

They could; that's why I said "This can be as part of the object-stores or by some other mechanism, but it needs to fit into the object-store flows."

[RangerMauve]

Would you mind describing an example of where you'd be using this? Say, in the context of Fritter.

You mentioned saving a picture for a contact, but I"m not sure if you mean it's your picture so you can show up as a contact, or another person's picture that you're saving the the object store. In the latter case I'm not sure why you'd do that instead of linking to their picture's dat URL.

[pfrazee]

Two usecases

1. On fritter, you're posting a fret with an image attachment. You want the app to upload the image somewhere on the public dat so that the fret can link it.
2. On fritter, you're following someone. The client wants to save their profile image so that if somebody views your followed users, that list will have profile-images. Same story: there needs to be somewhere to store the profile images. (This is the contacts example, and explains why you wouldn't just link to the original.)

One concern to consider - I we should be careful the apps can't write html/js and use that to attack the dat somehow.

[RangerMauve]

I think that in both cases, the files should be in a separate folder outside of the object store and have an object link to the URL.

I wouldn't expect /posts/ to contain somefile.png, it'd make more sense to have it in something like /media/.

You can still have the files in the public dat, you just don't add them via the object store API. And I think that would be more intuitive for developers, and would also reduce the complexity of object stores in general.

[pfrazee]

Mm, is that really the case? Putting post-attachments under /posts/ isnt that strange, and the object store API is just the filesystem API.

The questions I'm concerned about have to do with cross-app coordination, perm flows, and semantics. In the examples I gave above, we're talking about "post images" and "contact avatars," respectively. It makes a lot of sense to mediate access to those assets via the object-store APIs. That is, if you have permission to "Read and modify Public Fritter Posts" then I'd expect that to include the post images.

If we don't attach the "blob store" to the object store, then we need to come up with a separate ruleset which is equivalent or better.

[RangerMauve]

What about making the idea of "Add a file to a folder" a standalone concept?

Like, you can get permission to make posts, but then you also get permission to upload files through a different API.

I really like the idea of having different interfaces for different types of data. A file upload isn't really an object store. Not all applications with file uploads will have object stores, and not all applications with object stores will have file uploads.

[pfrazee]

Not all applications with file uploads will have object stores, and not all applications with object stores will have file uploads.

Yeah that's why the showFileDialog API exists. You could use that as a way to prompt the user to save the image file, but it's not super ideal for that flow. Example case, attaching an image to a fritter post:

• First you open an OS file picker to select the image.
• Then you use showFileDialog to save the image somewhere in the public dat.

There are two issues I can identify in that flow: first, the OS file picker can't select from dats, so we really need a file picker that unifies the OS filesystem and the dat filesystem. Second, you have to use two pickers, which just ain't right.

Saving a contact-picture is similarly janky. In that use-case, the app is copying the file from a target dat and saving it to the user's public dat, and this would happen after the user clicks "Add to contacts." It doesn't feel great to have some "save image" modal come up there.

So the trick here is thinking through these kind of flows and getting the jank out.

[RangerMauve]

First you open an OS file picker to select the image.
Then you use showFileDialog to save the image somewhere in the public dat.

I was thinking the flow would look more like

• Gain access to the dat-object-store for your fritter archive
• Gain write access to a folder in your fritter archive
• Use OS file picker
• Save to the folder without prompting the user again

[pfrazee]

• Gain write access to a folder in your fritter archive

That sounds like it could be one of resources requested in the session. We could make it scoped by origin if we wanted to be extra cautious. /media/fritter.hashbase.io.

[RangerMauve]

Yeah, exactly! Then the object store permission is a separate concern from a file uploading permission, and the two interfaces are more focused even if they're for the same archive / application.