Changes to deploy workflow

[pims]

tl;dr for @kingshyg : our deploy scripts is too heavy now, let’s update it.

It has become painfully obvious that our deploy workflow might need to get a little love.

The workflow that @jnorgan put together for suripu-admin is close to ideal.

1. Merge master into production
2. Travis sets maven version to be $TRAVIS_BUILD_NUMBER and deploys jar to our maven s3 repo.
3. [optional] maven boxfuse plugin creates a unikernel/AMI
4. sanders deploy/confirm/sunset takes care of deploying.

The one limitation of this new deploy flow is the dependency on two 3rd-party services: travis and boxfuse. To overcome this limitation, it’d be great to have a fallback plan.

The current deploy flow for suripu-app/service/workers is long, tedious and way too manual.

1. ./release.sh
2. ./config.sh X.X.X
3. Wait for travis to build the jar and push to maven repo
4. ssh into dev
5. deploy dev/staging version
6. update pkg.sh with new X.X.X version
7. run ./pkg.sh
8. cd packer-templates on local machine
9. run ./build.sh
10. wait for packer to build AMI
11. create launch config with sanders create
12. [optional] sanders canary
13. sanders deploy
14. sanders confirm
15. sanders sunset
16. [optional] do it all over again for another app.

That’s quite involved.

Given that our services are all bundled into uber-jars, we make very little use of provisioning AMIs, we hardly every install anything but ntp and open-jdk7.

If we give up a little immutability (!), we could be using user-script data from EC2 in the launch configuration to download a jar and create an upstart script. Which would replace ssh to dev to build .deb files. It also removes the need to use packer for custom immutable AMIs, they are almost all identical anyways.

We introduce a runtime dependency on S3 to download jar or .deb file built by travis, but we potentially save ~10 minutes per deploy, which is good for velocity. Fewer steps ~= fewer mistakes too.

The launch configuration could/should be created during the travis build.

⚠️ this means we might have to give the travis user higher IAM permissions (AssumeRole, etc…)
and right now I believe only I have IAM permissions to do this.

@jnorgan, do you want to give this is a shot when you’re back in office?
@jakepic1 any suggestions?

@kingshyg would that work for you?

[jakepic1]

I agree that for JVM services, we can treat the JAR as the "immutable artifact" and re-use the same AMI. This would simplify a lot of things.

There are some downsides though... if we ever need to change the configuration of that AMI, we lose that immutability. Say, if we change Java versions or something. I'm guessing that this will be a rare enough occurrence that we shouldn't try to optimize our workflow around it - we'll be extra careful (lol) when making an AMI change like that.

Overall though, I think it makes sense to optimize for the common case (deploying a new JAR) than for the weird edge cases.

[kingshyg]

reducing 16 steps to 4 works for m.

[jnorgan]

I know this is an old issue, but for the sake of future readers, here is the status:

• We eliminated logging into Dev and running pkg.sh to create the deb package by offloading this to Travis (using fpm during the Travis build only added 10-20 secs)
• Travis packages and deploys everything needed to S3 (no more separate upload of configs by the user)
• We now use a single standard AMI for all prod deploys (Java 7 & 8 installed)
• A shell script is passed to cloud-init via the userData in the AWS launch configuration. This handles the package download from S3 and installation. The service in question is started automatically after the cloud-init process finishes (as defined in the service's upstart config).

Main Points

• Eliminating Packer AMI builds & the Dev server .deb packaging saved a huge amount of time
• Travis build number is now the 'Patch' number in the application version
• It is now easier to be certain you are deploying the correct AMI. Travis build number in version and timestamp display in sanders makes confirming you are selecting the right AMI more reliable.