Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions MANIFEST.in
Original file line number Diff line number Diff line change
Expand Up @@ -4,3 +4,4 @@ include kubemarine/version
recursive-include kubemarine/patches *
recursive-include kubemarine/resources *
recursive-include kubemarine/templates *
recursive-include kubemarine/plugins/charts *
3 changes: 3 additions & 0 deletions ci/default_cluster.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,9 @@ plugins:
nginx-ingress-controller:
install: true

envoy-gateway:
install: true

kubernetes-dashboard:
install: true

Expand Down
4 changes: 4 additions & 0 deletions ci/extended_cluster.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,10 @@ services:
plugins:
kubernetes-dashboard:
install: true

envoy-gateway:
install: true

nginx-ingress-controller:
install: true
controller:
Expand Down
130 changes: 123 additions & 7 deletions documentation/Installation.md

Large diffs are not rendered by default.

37 changes: 25 additions & 12 deletions documentation/LoadBalancing.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,39 +2,44 @@ The following functions are installed in this section.

**Table of Content**

- [TLS Termination on Nginx Ingress Controller](#tls-termination-on-nginx-ingress-controller)
- [How to Install](#how-to-install)
- [TLS Termination on Envoy Gateway (or Nginx Ingress Controller)](#tls-termination-on-envoy-gateway-or-nginx-ingress-controller)
- [How to Install Nginx Ingress Controller (Not Recommended)](#how-to-install-nginx-ingress-controller-not-recommended)
- [How to Install Envoy Gateway (Recommended)](#how-to-install-envoy-gateway-recommended)
- [Using Kubemarine-provided TCP Load Balancer](#using-kubemarine-provided-tcp-load-balancer)
- [Using Custom TCP Load Balancer](#using-custom-tcp-load-balancer)
- [Advanced Load Balancing Techniques](#advanced-load-balancing-techniques)
- [Allow and Deny Lists](#allow-and-deny-lists)
- [Preserving Original HTTP Headers](#preserving-original-http-headers)
- [Preserving Original HTTP Headers on Nginx Ingress Controller](#preserving-original-http-headers-on-nginx-ingress-controller)
- [Maintenance Mode](#maintenance-mode)

## TLS Termination on Nginx Ingress Controller
## TLS Termination on Envoy Gateway (or Nginx Ingress Controller)

This is the default recommended approach to the TLS termination on kubemarine-installed environments. This approach is applicable when MTLS is not used in kubernetes and all communications between the pods are over plain HTTP.
A high-level overview of this approach is shown in the following image.

![](/documentation/images/tls-termination-nginx.png)
![](/documentation/images/tls-termination.png)

Here, the client creates a HTTPS connection to the TCP load balancer, which in turn proxies the traffic to the Nginx Ingress Controller without a TLS termination.
Nginx Ingress Controller uses a default wildcard certificate to authenticate itself to a client and to terminate the HTTPS connection.
Here, the client creates a HTTPS connection to the TCP load balancer, which in turn proxies the traffic to the Envoy Gateway (or Nginx Ingress Controller) without a TLS termination.
Envoy Gateway (or Nginx Ingress Controller) uses a default wildcard certificate to authenticate itself to a client and to terminate the HTTPS connection.
To support multiple hostnames, the certificate can use wildcard SANs.
Nginx Ingress Controller contacts applications using plain HTTP connection.
Envoy Gateway (or Nginx Ingress Controller) contacts applications using plain HTTP connection.
Thus using this approach, it is very easy to manage only one certificate in one place - TLS traffic is terminated on the Nginx Ingress Controller using the default certificate for all application ingresses.

For more information about Envoy Gateway default certificate, visit [https://gateway-api.sigs.k8s.io/guides/tls/](https://gateway-api.sigs.k8s.io/guides/tls/).

For more information about Nginx Ingress Controller default certificate, visit [https://kubernetes.github.io/ingress-nginx/user-guide/tls/#default-ssl-certificate](https://kubernetes.github.io/ingress-nginx/user-guide/tls/#default-ssl-certificate).

**Limitations**

This approach has the following limitations:

* Adding a new hostname might require to re-issue the certificate, if the new hostname does not match any previous wildcard SANs.
* Connections from the Nginx Ingress Controller to applications are through HTTP; thus, without an encryption.
* Connections from the Envoy Gateway (or Nginx Ingress Controller) to applications are through HTTP; thus, without an encryption.
* L7 load balancing options can be customized through "ingress" resources only.

### How to Install
### How to Install Nginx Ingress Controller (Not Recommended)

Nginx Ingress controller is installed by default, so you do not need to enable this plugin explicitly.

To enable TLS termination on Nginx Ingress Controller using the default certificate, it is required to customize the "nginx" plugin with a custom default certificate.
This can be done during:
Expand All @@ -44,6 +49,14 @@ This can be done during:

**Important**: The default certificate should be issued to wildcard hostnames, so that it can be used for all ingresses.

### How to Install Envoy Gateway (Recommended)

Envoy Gateway currently is not installed by default, you need to provide additional configuration in `cluster.yaml` to install it. To install Envoy Gateway with TLS termination using the default certificate, refer to [envoy-gateway plugin installation section](/documentation/Installation.md#envoy-gateway).

On an already installed Envoy Gateway, certificates could be installed or renewed using the `certs_renew` maintenance procedure. For details, refer to [certificate renew maintenance procedure](/documentation/Maintenance.md#configuring-certificate-renew-procedure-for-envoy-gateway).

**Important**: The default certificate should be issued to wildcard hostnames, so that it can be used for all ingresses.

### Using Kubemarine-provided TCP Load Balancer

Using kubemarine you can install and configure HAProxy TCP load balancers in the HA mode using VRRP.
Expand All @@ -62,7 +75,7 @@ In this case, your custom TCP load balancer should meet the following requiremen
* The load balancer should be an L4 pass-through TCP load balancer, without TLS termination.
* The load balancer should be Highly Available.
* The load balancer should have HTTPS (port 443) and Kubernetes API (port 6443) frontends.
* The HTTPS frontend should point to backend port 443 of worker nodes where Nginx Ingress Controller is installed.
* The HTTPS frontend should point to backend port 443 of worker nodes where Envoy Gateway (or Nginx Ingress Controller) is installed.
* The Kubernetes API frontend should point to backend port 6443 of all control-plane nodes.
* The load balancer backend configuration should be updated accordingly when new nodes are added or removed from a cluster.

Expand Down Expand Up @@ -122,7 +135,7 @@ spec:

In this example, the `whitelist.myservicea.foo.org` hostname is available only from IP address `172.10.0.1` and subnet `10.0.0.0/24`.

### Preserving Original HTTP Headers
### Preserving Original HTTP Headers on Nginx Ingress Controller

The TCP load balancer does not modify the HTTP response/request headers.
The Nginx Ingress Controller also does not modify **custom** HTTP headers.
Expand Down
27 changes: 23 additions & 4 deletions documentation/Maintenance.md
Original file line number Diff line number Diff line change
Expand Up @@ -204,6 +204,7 @@ It is because in this case, you take full control over the system packages and t
Patches that upgrade the OOB plugins have the following identifiers:
* `upgrade_calico` - It upgrades the Calico plugin.
* `upgrade_nginx_ingress_controller` - It upgrades the NGINX Ingress Controller plugin.
* `upgrade_envoy_gateway` - It upgrades the Envoy Gateway plugin.
* `upgrade_kubernetes_dashboard` - It upgrades the Kubernetes dashboard plugin.
* `upgrade_local_path_provisioner` - It upgrades the Local Path Provisioner plugin.

Expand Down Expand Up @@ -1292,7 +1293,7 @@ from particular namespaces will be applied.
**Warnings**:
* Be careful with the `exemptions` section it may cause cluster instability.
* Do not delete `kube-system` namespace from `exemptions` list without strong necessity.
* The PSS labels in namespaces for Kubemarine supported plugins ('nginx-ingress-controller', 'local-path-provisioner',
* The PSS labels in namespaces for Kubemarine supported plugins ('nginx-ingress-controller', 'gateway-system', 'local-path-provisioner',
'kubernetes-dashboard', and 'calico' (calico-apiserver)) are managed automatically.
They are deleted during the procedure in case of using `pod-security: disabled`, and changed accordingly in case `pss.defaults.enforce` is changed.
* Be careful with the `restart-pods: true` options it drains nodes one by one and may cause cluster instability. The best way to
Expand Down Expand Up @@ -1364,6 +1365,8 @@ link to Kubernetes docs regarding `kubelet.conf` rotation: https://kubernetes.io

For nginx-ingress-controller, the config map along with the default certificate is updated with a new certificate and key. The config map update is performed by plugin re-installation.

For Envoy Gateway, the Secret with the default certificate is updated with a new certificate and key. The Secret update is performed by Envoy Gateway chart re-installation.

For Calico, the certificate is updated for the Calico API server.

The `cert_renew` procedure also allows you to monitor Kubernetes internal certificates expiration status.
Expand All @@ -1376,6 +1379,22 @@ You can find description and examples of the accepted parameters in the next sec
The JSON schema for procedure inventory is available by [URL](../kubemarine/resources/schemas/cert_renew.json?raw=1).
For more information, see [Validation by JSON Schemas](Installation.md#inventory-validation).

#### Configuring Certificate Renew Procedure for Envoy Gateway

To update the certificate and key for `envoy-gateway`, use the following configuration:

```yaml
envoy-gateway:
cert: |
-----BEGIN CERTIFICATE-----
...(skipped)...
-----END CERTIFICATE-----
key: |
-----BEGIN RSA PRIVATE KEY-----
...(skipped)...
-----END RSA PRIVATE KEY-----
```

#### Configuring Certificate Renew Procedure for nginx-ingress-controller

To update the certificate and key for `nginx-ingress-controller`, use the following configuration:
Expand Down Expand Up @@ -1440,11 +1459,11 @@ kubernetes:
### Certificate Renew Tasks Tree

The `cert_renew` procedure executes the following sequence of tasks:

1. kubernetes
2. nginx_ingress_controller
3. calico
4. certs_overview
3. envoy_gateway
4. calico
5. certs_overview

# Procedure Execution

Expand Down
Binary file removed documentation/images/tls-termination-nginx.png
Binary file not shown.
Binary file added documentation/images/tls-termination.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
27 changes: 27 additions & 0 deletions examples/cluster.yaml/full-cluster.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -221,6 +221,7 @@ services:
net.ipv6.ip_nonlocal_bind: 1

loadbalancer:
target_backend: "envoy"
haproxy:
defaults:
timeout_connect: '10s'
Expand Down Expand Up @@ -345,6 +346,32 @@ plugins:
apiservice:
retries: 60

envoy-gateway:
install: true
installation:
priority: 2
kubectlRegistry: "ghcr.io"
externalGateway:
certificate:
cert: |
-----BEGIN CERTIFICATE-----
----------------------------------------------------------------
-----END CERTIFICATE-----
key: |
-----BEGIN RSA PRIVATE KEY-----
----------------------------------------------------------------
-----END RSA PRIVATE KEY-----
crValuesOverride:
gatewayClasses:
external:
envoyDeployment:
nodeSelector:
kubernetes.io/hostname: some-nodes
tolerations:
- key: node-role.kubernetes.io/control-plane
operator: Exists
effect: NoSchedule

nginx-ingress-controller:
install: true
installation:
Expand Down
25 changes: 24 additions & 1 deletion examples/cluster.yaml/one-cp-many-workers-cluster.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,31 @@ nodes:
roles: ["worker"]

# NodeSelector and Tolerations are required to make sure
# nginx-ingress runs on control-plane node
# envoy-gateway/nginx-ingress runs on control-plane node
plugins:
envoy-gateway:
install: true
externalGateway:
certificate:
cert: |
-----BEGIN CERTIFICATE-----
----------------------------------------------------------------
-----END CERTIFICATE-----
key: |
-----BEGIN RSA PRIVATE KEY-----
----------------------------------------------------------------
-----END RSA PRIVATE KEY-----
crValuesOverride:
gatewayClasses:
external:
envoyDeployment:
nodeSelector:
node-role.kubernetes.io/control-plane: ""
tolerations:
- key: node-role.kubernetes.io/control-plane
operator: Exists
effect: NoSchedule

nginx-ingress-controller:
controller:
nodeSelector:
Expand Down
1 change: 1 addition & 0 deletions kubemarine.spec
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ a = Analysis(['./kubemarine/__main__.py'],
'kubemarine.plugins.builtin',
'kubemarine.plugins.calico',
'kubemarine.plugins.nginx_ingress',
'kubemarine.plugins.envoy_gateway',
'kubemarine.plugins.kubernetes_dashboard',
],
pathex=[],
Expand Down
4 changes: 4 additions & 0 deletions kubemarine/core/resources.py
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@
import kubemarine.plugins.kubernetes_dashboard
import kubemarine.plugins.local_path_provisioner
import kubemarine.plugins.nginx_ingress
import kubemarine.plugins.envoy_gateway
import kubemarine.sysctl
import kubemarine.system
import kubemarine.thirdparties
Expand Down Expand Up @@ -406,6 +407,7 @@ def enrichment_functions(self) -> List[c.EnrichmentFunction]:
kubemarine.thirdparties.enrich_procedure_inventory,
kubemarine.cri.enrich_upgrade_inventory,
kubemarine.plugins.nginx_ingress.cert_renew_enrichment,
kubemarine.plugins.envoy_gateway.cert_renew_enrichment,
kubemarine.sysctl.enrich_reconfigure_inventory,
kubemarine.core.inventory.enrich_reconfigure_inventory,
# Enrichment of procedure inventory should be finished at this step.
Expand Down Expand Up @@ -438,6 +440,7 @@ def enrichment_functions(self) -> List[c.EnrichmentFunction]:
kubemarine.admission.verify_manage_enrichment,
kubemarine.k8s_certs.renew_verify,
kubemarine.plugins.nginx_ingress.verify_cert_renew,
kubemarine.plugins.envoy_gateway.verify_cert_renew,
# The functions below depend on kubemarine.kubernetes.verify_version
kubemarine.kubernetes.verify_upgrade_inventory,
# Depends on kubemarine.core.defaults.calculate_nodegroups
Expand Down Expand Up @@ -477,6 +480,7 @@ def enrichment_functions(self) -> List[c.EnrichmentFunction]:
kubemarine.plugins.kubernetes_dashboard.enrich_inventory,
kubemarine.plugins.local_path_provisioner.enrich_inventory,
kubemarine.plugins.nginx_ingress.enrich_inventory,
kubemarine.plugins.envoy_gateway.enrich_inventory,
# Depends on kubemarine.packages.enrich_inventory
kubemarine.audit.verify_inventory,
kubemarine.system.verify_inventory,
Expand Down
38 changes: 19 additions & 19 deletions kubemarine/core/utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -554,27 +554,27 @@ def minor_version_key(version: str) -> Tuple[int, int]:


def _test_version(version: str, numbers_amount: int) -> List[int]:
# catch version without "v" at the first symbol
is_rc = 0
if version.startswith('v'):
version_list: list = version[1:].split('.')
# catch version with unexpected number or parts
parts_num = len(version_list)
try:
for i, value in enumerate(version_list):
# catch release candidate version like v1.29.0-rc.1
if parts_num == 4 and i == 2 and value.endswith('-rc'):
value = value[:-3]
is_rc = 1
# whitespace required because python's int() ignores them
version_list[i] = int(value.replace(' ', '.'))
except ValueError:
pass
else:
if numbers_amount == parts_num - is_rc:
return version_list[:numbers_amount]

expected_pattern = 'v' + '.'.join('N+' for _ in range(numbers_amount))
version = version[1:]
version_list: list = version.split('.')
# catch version with unexpected number or parts
parts_num = len(version_list)
try:
for i, value in enumerate(version_list):
# catch release candidate version like v1.29.0-rc.1
if parts_num == 4 and i == 2 and value.endswith('-rc'):
value = value[:-3]
is_rc = 1
# whitespace required because python's int() ignores them
version_list[i] = int(value.replace(' ', '.'))
except ValueError:
pass
else:
if numbers_amount == parts_num - is_rc:
return version_list[:numbers_amount]

expected_pattern = '[v]' + '.'.join('N+' for _ in range(numbers_amount))
if numbers_amount == 3:
expected_pattern += '[-rc.N+]'
raise ValueError(f'Incorrect version \"{version}\" format, expected version pattern is \"{expected_pattern}\"')
Expand Down
13 changes: 13 additions & 0 deletions kubemarine/haproxy.py
Original file line number Diff line number Diff line change
Expand Up @@ -121,6 +121,19 @@ def get_config_path(group: NodeGroup) -> RunnersGroupResult:

return collector.result

def get_target_backend(inventory: dict) -> str:
"""
Returns loadbalancer target backend, with is either "nginx" or "envoy".
By default, "nginx" is returned if nginx plugin is installed. Otherwise, "envoy" is returned.
User can override this using "target_backend" field.
"""
if inventory['services']['loadbalancer']['target_backend'] != "":
return str(inventory['services']['loadbalancer']['target_backend'])

if inventory['plugins']['nginx-ingress-controller']['install']:
return "nginx"

return "envoy"

def install(group: NodeGroup) -> RunnersGroupResult:
cluster = group.cluster
Expand Down
1 change: 1 addition & 0 deletions kubemarine/patches/software_upgrade.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -30,3 +30,4 @@ plugins:
nginx-ingress-controller: []
kubernetes-dashboard: []
local-path-provisioner: []
envoy-gateway: []
2 changes: 1 addition & 1 deletion kubemarine/plugins/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -834,7 +834,7 @@ def apply_helm(cluster: KubernetesCluster, config: dict) -> None:
cluster.log.debug("Deployed release %s is not found. Installing it..." % release)
deployment_mode = "install"

command = prepare_for_helm_command + f'{deployment_mode} {release} {chart_path} --create-namespace --debug'
command = prepare_for_helm_command + f'{deployment_mode} {release} {chart_path} --create-namespace'
execute_subprocess_with_logging(cluster, command)


Expand Down
26 changes: 26 additions & 0 deletions kubemarine/plugins/charts/envoy-gateway-2.2.0/.helmignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
# Patterns to ignore when building packages.
# This supports shell glob matching, relative path matching, and
# negation (prefixed with !). Only one pattern per line.
.DS_Store
# Common VCS dirs
.git/
.gitignore
.bzr/
.bzrignore
.hg/
.hgignore
.svn/
# Common backup files
*.swp
*.bak
*.tmp
*.orig
*~
# Various IDEs
.project
.idea/
*.tmproj
.vscode/

# Template files
*.tmpl.*
Loading
Loading