Support HA deployment

[emonty]

Kubernetes has support for running with multiple masters in HA mode. We should support such a deployment if the user wants.

There is a hostname naming issue that will be a potential upgrade issue - if someone has a single master deploy now their master would be named '{}-k8s-master' and if we copy the nodes pattern for masters we could expect {}-k8s-master-[0-9]. Perhaps to keep things simple we should not attempt to support shifting from single master to multi-master?

[emonty]

The openstack-infra/opendev team is using k8s-on-openstack to run a kubernetes on top of which a new version of our git farm will run. (thanks!)

We'd like to investigate running the master HA, which would obviously involve some changes here. Before we get started on that ... I thought it might be a good idea to check and make sure it would be something you're interested in accepting. If not - totally fine, I think it makes the code easy to read right now that there aren't a ton of different deployment options.

[ctrlaltdel]

I'm totally open to accept changes for implementing master HA, it wasn't just implemented before because kubedm did not support it back then.

A simple implementation which doesn't magically handle upgrade from non-HA to HA sounds reasonable.