Skip to content

TASK: Add MinIO compatibility#37

Open
etlam wants to merge 1 commit intoflownative:mainfrom
muensmedia:master
Open

TASK: Add MinIO compatibility#37
etlam wants to merge 1 commit intoflownative:mainfrom
muensmedia:master

Conversation

@etlam
Copy link
Copy Markdown

@etlam etlam commented Oct 15, 2020

This pull requests adds compatibilty for MinIO.

Problems with MinIO:

  • resources beneath resources are not supported by MinIO
  • use_path_style_endpoint=true must be used

$pathAndFilename = $object->getRelativePublicationPath() . $object->getFilename();
} elseif (isset($this->s3DefaultProfile['minio']) && $this->s3DefaultProfile['minio']) {
// If MinIO is used we have to change to url style. Creating an object beneath an object is not supported
$pathAndFilename = $object->getSha1() . '_folder/' . $object->getFilename();
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@etlam Can you point to the Minio document discussing this limititation ? I use Minio for a few project with Flow ... don't know if i'm lucky but it work with just the path style configuration.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Copy Markdown
Member

@kdambekalns kdambekalns Mar 22, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is a valid discussion, but is it actually failing for you?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, we use Sitegeist.Kaleidoscope in our project. I do not know if it would fail without Kaleidoscope.

@dfeyer
Copy link
Copy Markdown
Contributor

dfeyer commented Mar 20, 2021

@robertlemke @kdambekalns I can confirm that Minio did not have the limitation that this PR try to solve. Running a project and currently URL like http://storage-medialib.cms.ttree.localhost/public/sites/core/c31d44f75b3ba322d5219cd1e17c2ba933a6db5a/9WO0CELoWq6tSdjqCs34Y0qek7l-420x236.jpg are perfectly valid and works with latest docker image of Minio

But use_path_style_endpoint is required and the README update can be nice even if the Minio document is clear about this https://docs.min.io/docs/how-to-use-aws-sdk-for-php-with-minio-server.html

@etlam
Copy link
Copy Markdown
Author

etlam commented Mar 22, 2021

If you use for example Sitegeist.Kaleidoscope the original image gets saved as jhgsjsjhkshgezwwef and the smaller thumbnails try to get saved as jhgsjsjhkshgezwwef/<hash_of_variant>[...].
So the original image should be used as file and folder at the same time. This is not possible with MinIO because of the used underlying linux file system.

A folder and a file can not have the same name and be located in the same folder:
minio/minio#7335 (comment)
minio/minio#10160

Maybe there is a more elegant way to prevent that there is a file and a folder with the same name.

@kdambekalns
Copy link
Copy Markdown
Member

See #42

@etlam
Copy link
Copy Markdown
Author

etlam commented Jul 20, 2021

Yes, you have to use use_path_style_endpoint=true for MinIO.
But the problem with same file and folder names remains, as linked in my comment above:
minio/minio#7335 (comment)
minio/minio#10160

AWS S3 can use a file as a folder and store more files inside of it, MinIO can not do this.

We can close the pull request if using Sitegeist.Kaleidoscope with MinIO is a special case.

kdambekalns added a commit that referenced this pull request Jul 20, 2021
See #42 and #37 for the rationale.
@kdambekalns
Copy link
Copy Markdown
Member

Still not sure this is an issue with Sitegeist.Kaleidoscope, nor that it is a general issue:

  • Flow stores persistent in resources in a storage using just the hash
  • Resources and variants are published in a target using a pattern of hash/name

Since the hash is dependent on the file contents, two different files will have a different hash, and as long storage and target have separate buckets (as required for the S3 adapter), there should never be a clash…

Are you in fact using seperate buckets for storage and target, @etlam?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants