External Storage S3 - Very high Request

[mmajer]

Hi,
External Storage - S3 - has very high requests
2 Million every day!

I have an S3 bucket mounted on my Owncloud server with about 10 files in it.
I do not do file activity, they are just archived.
The S3 folder is shared on clients but nobody uses it in this test phase

Owncloud Server 10.0.10

Database: MySQL

What it should be at?

Error in configuration?
Do clients force a rescan of all the folders?

[ownclouders]

GitMate.io thinks possibly related issues are owncloud/core#13147 (External Storage S3 Very High Requests), owncloud/core#14371 (Issue adding S3 External Storage), owncloud/core#10198 (External Only Storage), owncloud/core#11156 (wrong Quota working with S3 Storage Backend (external)), and owncloud/core#12666 (S3 external storage --- Could not resolve host).

[DeepDiver1975]

The S3 folder is shared on clients but nobody uses it in this test phase

The clients are calling home for update detection. In case of s3 this results in a request to amazon to detect changes over there. This can result in quite some requests.

Still this amount is damn high.

Please reopen this issue in https://github.com/owncloud/files_external_s3/issues - THX

[patrickjahns]

@DeepDiver1975
I used github new "transfer" feature to move the case over there

[AdamLeyshon]

Has there been any implementation of the stats cache for AWS S3? I'm still seeing very high usage on S3 and enumerating the bucket for when checking for changes on the desktop client is still very slow.

version | 10.2.1.4
versionstring | 10.2.1
edition | Community
productname | ownCloud

[Linda-chan]

Looks like you should choose option «Never» for «Check for changes» in external storage mount settings. But this means you can't change anything in the bucket behind OC's back (just like local data folder).

[harveydobson]

@Linda-chan I can't see this option in the External Storage Mount settings. Is it a GUI option? Or something that has to be configured in config.php?

Any help would be much appreciated!

[Linda-chan]

@harveydobson Open settings, go to «External Storage» tab, where your connection is configured. You should see cogs icon near trash bin icon. Press it and find «Check for Changes» combo box.

[g3ntile]

Hi! I'm setting up an EC2 owncloud server, using an s3 bucket to store the files, and I stumbled upon this issue... I tried setting the "check for changes" option to "Never" for the S3, but I still get millions of requests per day, and every few minutes my desktop clients says "checking for changes in remote AmazonS3...".
I use that bucket only for owncloud, so the constant checking of every files for changes on the server side is unnecesary, it makes the s3 costs absurdly high just for sitting there.
Is there a solution for this?
Thank you!

[g3ntile]

Hi! I guess that this issue has no solution... it's a pity, because otherwise it works very well. But the insane amount of requests to the s3 I get (with "check for changes" set to "Never") makes de s3 unusable.
There must be some sort of simple solution, some way of maintaining a list of the files that changed since each client updated it's files. Something that don't involve parsing the whole external disk file by file every minute...
(But I'm not a developer, so I get that maybe It's not that easy.)
Or maybe I'm doing something wrong.
Thank you!

[AdamLeyshon]

I feel your pain, I had a $163 bill from AWS due to OwnCloud making 7.3 million API calls. I learned a very important lesson about not ignoring your CloudWatch billing alarms.