Terraform provisioning for Kubernetes

This project includes Terraform scripts for deploying Kubernetes on top of OpenStack or libvirt. This is not using Magnum yet.

The deployment consists of:

• salt server: used to configure all the nodes
• kube-master
• kube-minions: the number of nodes can be configured

Requirements

Environment

Packages

• First and foremost, you need terraform installed.
• When using a libvirt environment, you will also need the terraform-provider-libvirt package. These two packages are available on OBS inside of the Virtualization:containers project.
• If you are using an openstack environment with cloud.suse.de, then you need to get the internal root certificates from SUSE. You can do this by installing the ca-certificates-suse package found in the ibs://SUSE:CA project.

Projects

• In order to provision the virtual machines, we use salt. In particular, we have our own repository for salt scripts needed for installing a proper Kubernetes cluster: https://github.com/kubic-project/salt. As it's described later in the Variables section, you may use the salt_dir variable to point to a local checkout of the kubic-project/salt project.

Images

One important aspect of the configuration is the image you will use for your VMs. This is specially important in some configurations and is the main source of problems, so the recommended solution is to use some of the images already provided by the Docker team.

• Since we rely on cloudinit, the image should start the cloudinit services automatically. The minimum cloud-init version supported is 0.7.7.
• When using libvirt, they should have the qemu-agent installed (otherwise they will not work in bridged mode)
• In development environments, they should be accessible with user/pass=root/vagrant

Cluster configuration

k8s-setup script

The Kubernetes infrastructure is managed with Terraform, but we use a ruby script, k8s-setup, for preprocessing the Terraform scripts, replacing variables and conditionally including some files.

This script processes all the *.tf and *.tf.erb files found in the terraforms directory (by default, $(pwd)/terraform) and generate a unique output file (by default, k8s-setup.tf). As a shortcut, it also runs terraform with the last arguments provided, so running k8-setup plan is equivalent to k8s-setup && terraform plan.

Helper script for libvirt

You should be using the k8s-setup script to manage your Kubernetes cluster. That being said, if you are going to use the libvirt driver, you might want to take a look at the k8s-libvirt script. This script assumes that you want to use libvirt, and it makes some assumptions that will allow you to deploy your cluster with a simple command.

Configuration Variables

Some aspects of the cluster can be configured by using variables. These variables can be provided to the k8s-setup script with -V variable=value arguments, or through a profile file. See the example files provided in the repository for more details.

Some important variables are:

• salt_dir

  The directory where the Salt scripts are (usually a checkout of this repo)

• ssh_key

  ssh_key is the key we will use for accessing machines (by default, the id_docker in the local ssh directory)

• cluster_prefix

  By default all the VMs provisioned by Terraform are going to be named in the same way (eg: kube-master, kube-minion1, kube-minion2,...). This makes impossible for multiple people to deploy a Kubernetes cluster on the same cloud.

  This can be solved by setting the cluster_prefix variable to something like flavio-.

• cluster_domain_name

  The cluster default domain name. It can be something like k8s.local. This domain name will be used across all the instances.

• kube_minions_size

  By default the k8s cluster has 3 k8s minions. However it's possible to change the default value by using the kube_minions_size variable.

• bridge

  Name of the bridge interface to use when creating the nodes. This is useful when the libvirt host is a remote machine different from the one running terraform.

• <component name>_memory

  The amount of memory to be assigned to the given component in MB. Possible options for components are: master, minion and salt. The default value is 512 MB. Moreover, if you want to setup the same value for all of them, you can use the memory shortcut. Note: this only works for the libvirt setup. Support for openstack is still being worked.

• docker_reg_mirror

  An (optional) Docker registry mirror (ie, myserver:5000). This can be specially helpful when you intend to download many Docker images and bandwidth is scarce.

Please take a look at the *.profile files for more variables used in our templates.

Deploying the cluster

Unfortunately there isn't yet a way to bring up the whole cluster with one single command: it's necessary to first create the infrastructure with Terraform and then to configure the machines via Salt.

Creating the infrastructure

The easiest way to configure your cluster is to use one of the included .profile configuration files and overwrite the variables you need. Then you can invoke the k8s-setup script with any of the commands accepted by Terraform.

For example:

$ ./k8s-setup -F base-openstack.profile apply

You could for example overwrite kube_minions_size by invoking it as:

$ ./k8s-setup -V kube_minions_size=6 -F base-openstack.profile apply

or with an additional configuration file:

$ echo "kube_minions_size=6" > local.profile
$ ./k8s-setup -F base-openstack.profile -F local.profile apply

If you want to review the generated k8s-setup.tf file, you can also obtain a prettified version of this file with:

$ ./k8s-setup -F base-openstack.profile fmt

and then run any terraform command with this file.

Running Salt orchestrator

Once all the virtual machines are up and running it's time to install software and configure them. We do that with the help of the Salt orchestration. Just execute:

$ ssh -i ssh/id_docker root@`terraform output ip_dashboard` \
    bash /tmp/salt/provision-dashboard.sh --finish

Then follow the instructions given by the provisioning script.

Notes:

• the certificate generated for the API server includes the list of IPs automatically detected by provisioning script. However, this is not enough in some cases when the API server will be accessed at some other IP (for example, when the server is behind a NAT or when a floating IP is assigned to it in a OpenStack cluster). In those cases, you should specify that IP in with --extra-api-ip <IP>.

Using the cluster

The Kubernetes API server can be used by configuring the kubectl with a kubeconfig file. Copy the admin.tar file from the Salt master, uncompress it and export the KUBECONFIG variable.

For example:

$ scp -i ssh/id_docker root@`terraform output ip_dashboard`:admin.tar .
$ tar xvpf admin.tar
$ KUBECONFIG=kubeconfig kubectl get nodes

License

This project is licensed under the Apache License, Version 2.0. See LICENSE for the full license text.