How to deal with missing files upon server update

[elie222]

If you have 2 load balanced Meteor instances running for example, and you've done an update and 1 of the instances has been updated, but the other instance is still waiting to be updated, how would this work exactly using Cloudfront?

Cloudfront would try and grab the pages from the Meteor url, but may be connected to the wrong Meteor server that the user is connecting to and then the site will not be able to load. This is a real issue we've run into and was wondering if this package could help with that at all.

[kulttuuri]

I believe meteor serves the files with version hash added to the files so when you have cloudfront setup correctly it should serve old and new files. This doesn't however work if images or static assets do change.

[elie222]

If server 1 is updated but server 2 isn't. And then a user makes a request to server 1 and cloudfront gets connected to secret 2 it won't have the most up to date version yet and the user won't be able to use the site. This becomes more problematic if you're updating 20 servers and it can take 30 minutes or longer to update them all. Elie

…

On 15 Aug 2017 17:39, "Aleksi Postari" ***@***.***> wrote: I believe meteor serves the files with version hash added to the files so when you have cloudfront setup correctly it should serve old and new files. This doesn't however work if images or static assets do change. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#14 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AC8oX1RUwrTK_OHoFT8os7MSf_qACf_uks5sYa2OgaJpZM4O3nfe> .

[kulttuuri]

Are your load balanced servers running on same domain? So for ex. Users always connect to example.com and example.com has nginx or something similar that connects in the background to 1.example.com @ 2.example.com?

[elie222]

Yes that's correct.

…

On 15 Aug 2017 22:35, "Aleksi Postari" ***@***.***> wrote: Are your load balanced servers running on same domain? So for ex. Users always connect to example.com and example.com has nginx or something similar that connects in the background to 1.example.com @ 2.example.com? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#14 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AC8oXz4NHL-NYDTbBIodc4tdiHN_fc9Pks5sYfMQgaJpZM4O3nfe> .

[kulttuuri]

Could you then use same cloudfront endpoint for both servers? That way it would have both old and new content in the cache.

[kulttuuri]

BTW I hit similar problem earlier in the development on our production server. We're also running load balancers servers on similar scenario. I think it was the query parameter setting that I had to configure on Cloudfront correctly to get this working right (it was showing blank white screen for some users)

[elie222]

Galaxy does this with version numbers. But the question is how you get your load balancer to send cloudfront the new version of your app. If the load balancer comments cf to an instance that has yet to be updated you'll get the blank screens.

…

On 16 Aug 2017 07:46, "Aleksi Postari" ***@***.***> wrote: Could you then use same cloudfront endpoint for both servers? That way it would have both old and new content in the cache. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#14 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AC8oX_auCv7JNJAZO7PTP_xndCS2O9Dhks5sYnQOgaJpZM4O3nfe> .

[elie222]

@kulttuuri what query param did you set?

[elie222]

What we're seeing is that Cloudfront will cache the result and if the result isnt found it will cache that and then even once all servers have updated to the latest version, users wont be able to access the site.

[maxkferg]

@elie222. This was a problem that came up in the original development and testing of the package.
From memory, our first attempt at fixing this was to return a proper 404 for missing static files f824449.
If the CDN is configured NOT to cache 404 responses, then this problem will largely go away:

1. A new server with new content will come online
2. Meteor will request the new static content
3. If the content is fetched from an old server then it will return a 404. The page will be blank. The resource will not be cached*
4. If the content is fetched from a new server then everything will be fine and the new content will be cached.

*Step (3) may be repeated a few times until step (4) is reached

So as a first solution I suggest that you make sure 404 responses are not being cached. There is definitely an opportunity to handle this situation better - although it may be outside the scope of this package. As @elie222 and @kulttuuri mentioned, this is a problem at the load balancer level. I am not sure how to reroute load balancer traffic to old/new servers in a reliable way. Suggestions are welcome

[jlichti]

I handle this by having a staging server that I always deploy first (and then I snapshot and roll out to production). I have my CDN point to the staging server with a * behavior match on CloudFront just above the default.

If I'm not doing any production deploys for a while I just change my staging server behavior to DISABLED so the CDN will hit my load balancer.

[jfurneaux]

Did anyone come up with an answer to this that avoids having a separate environment?

[elie222]

Yeah. This is really annoying. It means you can never have servers running separate versions. It can also take a minute or so for a set of 16 servers to update for example, during this time the latest servers may not be able to load.

I think the way to properly solve this is to upload the files to an S3 bucket and have Cloudfront pull from there. As soon as a build finishes you upload the files to S3 so Cloudfront always has them. I think something like this is possible although I've never tried it and could be a little tricky to set up.

[elie222]

How we've been dealing with this over the past year is:

1. Turn off CDN setting
2. Deploy
3. Turn on CDN setting
4. MUP Reconfig

If you're not using MUP then you can't always do the simple mup reconfig to update settings.

[klokie]

hi! Did you ever get this to work without having to toggle the CDN settings upon deployment? We're using Galaxy + CloudFlare, following that procedure is not trivial.