Allow link timeout to be set

[PeetMcK]

For the life of me I can't seem to work out out to set the link timeout to be greater than the 15 minute default.

I'm doing my best to wrap my head around the way this middleware and AWS4 authentication header work, but I'm just not there.

Is there a simple way to inject Amz-Expires=XXX into the link request? It seems like it's defaulting to a 900 second timeout, but I'm having issues pulling down large updates like an OS install.

I'll keep poking a stick at it, but I'm sure there have to be few folks that have hit this limit.

Thanks for the good work.

[waderobson]

Interesting! I wonder why this hasn't come up in the past and I think it may have something to do with this article I found

Finally, requests do not expire while in the middle of processing. If you send a request using either signing method that is deemed valid when it arrives but would have expired very soon thereafter, it is always allowed to run to completion -- for example, a large S3 download or an EBS snapshot creation request will not start, then fail to continue, because the expiration timer struck while the request had already started on the AWS side. If the action was authorized when requested, then it continues and succeeds as normal.
https://stackoverflow.com/questions/39551219/is-x-amz-expires-a-required-header-parameter-for-requests-to-aws#:~:text=X%2DAmz%2DExpires%20must%20not,to%20the%20AWS%20system%20clock.

This makes me wonder if the issue is not that the files are too big and therefore canceled partway but rather that there's an issue and the file needs to resume or retry outside that 15 min window.

Being able to optionally increase the time would be nice however I don't know how much time I'll be able to dedicate to this right now.

[PeetMcK]

Yeah, I think this must be the case for the URL's and I'm questioning this as the root cause of the issue as well. It wouldn't make sense to only have 15 minutes to complete the transaction that's generated by the URL.

I'm wondering if there's some sort of timeout elsewhere. Or a reconnection attempt that's happening ... i.e. curling down the the dmg ... hitting a 'hiccup' and then trying to curl the now out of date URL. I'm just trying to find some logging that supports what's going on!

That said, depending on how Munki is handling the URL's, an extended timeout may be valuable ... i.e. If munki pulls all the url's and queues them before downloading, then it would be possible to hit the 15 minute mark if you were on a slower connection and say the first 3 items downloaded take more than 15min to pull down the the next 3 items would fail, in this case, a longer timeout would be pretty important. But that said, you'd think folks would run in to this issue WAY more often. I'll be taking a closer look as I troubleshoot this.

But overall it's definitely an odd one.