Envoy Gateway support by theboringstuff · Pull Request #798 · Netcracker/KubeMarine

theboringstuff · 2026-03-19T04:08:25Z

Description

It is required to support Envoy Gateway plugin. Also, it should be possible to configure HAProxy to target Envoy instead of Ingress-NGINX. Also, dashboard Ingress should be migrated to HTTPRoute.

Solution

Manually added copies of qubership Envoy Gateway chart version 2.2.0 to KubeMarine
- There is currently no automation in scripts/thirdparties/sync.py to download Envoy Gateway charts, charts should be downloaded manually. Sync takes downloaded chart and simply updates images versions in compatibility map.
Added envoy-gateway plugin
- It is not installed by default.
- It allows to configure installation namespace, chart release name and CR chart release name
- PSS labels are configured for the namespace
- Backend and EnvoyPatchPolicy extensions are enabled by defult
- Only external gateway is installed by default, internal one is disabled.
- Certificates could be specified for External Gateway, in this case HTTPS listener is added
- External Gateway service uses ClusterIP type.
- External Gateway is daemonset with envoy-external-gateway name
- ClientTrafficPolicy allows underscore in headers and adds X-Forwarded-Host header with respective value
- Images registry could be specified as usual, but there is dedicated kubectlRegistry option for qubership kubectl image from ghcr.io
- User can overwrite chart values using valuesOverride and crValuesOverride
Added services.loadbalancer.target_backend option, which allows to select who should take HAProxy target_ports - either nginx (default) or envoy.
Added HTTPRoute for dashboard, which is created if Envoy Gateway is installed. Also Dashboard ingress now has converter ignore annotation.
All procedures are impacted by these changes, but following especially:
- install should now install envoy-gateway and dashboard HTTPRoute (if envoy is enabled)
- upgrade should support envoy-gateway upgrade, but currently there is no new envoy-gateway version to upgrade to
- cert_renew should support adding/renewing envoy-gateway certificates
- migrate_kubemarine should support envoy-gateway patches, but currently there is no patch
- check_paas checks were extended to also check envoy-gateway
- scripts/thirdparties/sync.py should support envoy-gateway in compatibility map, but chart should be downloaded manually
Build process was updated to include Envoy Gateway resources
CI process was updated to install Envoy Gateway
Existing tests were adjusted to new changes, but no new tests were added
Updated documentation
- Highlight that Envoy Gateway is recommended and Nginx is not recommended
- Highlight supported configuration options for Envoy Gateway plugins
- Highlight new option services.loadbalancer.target_backend

Test Cases

Test case 1

Install KubeMarine cluster as usual, using Ingress-NGINX, without enabling Envoy plugin.

Installation should succeed and cluster should work notmally.

Test case 2

Install KubeMarine cluster with following additional configuration (do not change anything else)

rbac:
  accounts:
  - name: superadmin
    role: cluster-admin
  admission: pss
  pss:
    pod-security: enabled
    defaults:
      enforce: restricted
      enforce-version: latest
      audit: restricted
      audit-version: latest
      warn: restricted
      warn-version: latest
    exemptions:
      usernames: []
      runtimeClasses: []
      namespaces: ["kube-system"]

plugins:
  envoy-gateway:
    install: true
    externalGateway:
      hostPorts:
        http: 80
        https: 443
      certificate:
        cert: |
          -----BEGIN CERTIFICATE-----
          ..... skipped ....
          -----END CERTIFICATE-----
        key: |-
          -----BEGIN RSA PRIVATE KEY-----
          .... skipped ....
          -----END RSA PRIVATE KEY-----

Cluster should be installed with both Ingress-NGINX and Envoy Gateway. Ingress-NGINX should continue to work as LB backend, Ingresses should work as usual. Envoy Gateway should be accessible on ports 80/443

Test case 3

Install KubeMarine cluster with following additional configuration (do not change anything else)

rbac:
  accounts:
  - name: superadmin
    role: cluster-admin
  admission: pss
  pss:
    pod-security: enabled
    defaults:
      enforce: restricted
      enforce-version: latest
      audit: restricted
      audit-version: latest
      warn: restricted
      warn-version: latest
    exemptions:
      usernames: []
      runtimeClasses: []
      namespaces: ["kube-system"]

services:
  loadbalancer:
    target_backend: "envoy"

plugins:
  envoy-gateway:
    install: true
    externalGateway:
      certificate:
        cert: |
          -----BEGIN CERTIFICATE-----
          ..... skipped ....
          -----END CERTIFICATE-----
        key: |-
          -----BEGIN RSA PRIVATE KEY-----
          .... skipped ....
          -----END RSA PRIVATE KEY-----

Cluster should be installed with both Ingress-NGINX and Envoy Gateway. Envoy Gateway will be used as LB backend, HTTPRoutes should become accessible (e.g. dashboard) instead of Ingresses. Ingresses should stop working. Ingress-NGINX pods should not use 20080/20443 hotsPorts, instead they should be used by Envoy external Gateway pods. Verify check_paas procedure succeeds.

Test case 4

Renew Envoy Gateway certificates using cert_renew procedure.

Checklist

I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
There is no breaking changes, or migration patch is provided
Integration CI passed
Unit tests. If Yes list of new/changed tests with brief description
There is no merge conflicts

alexarefev · 2026-03-27T12:16:54Z

Are we going to provide any description how to migrate from nginx-ingress-controller to envoy-gateway?

theboringstuff · 2026-03-27T14:15:40Z

Are we going to provide any description how to migrate from nginx-ingress-controller to envoy-gateway?

I think we will need to update our Gateway API Migration documentation. I forgot to do it as part of this activity, I suggest to do it in separate activity, since these changes become too big

alexarefev · 2026-03-27T12:12:07Z

kubemarine/resources/configurations/defaults.yaml

      keep_configs_updated: True
      maintenance_mode: False
      mntc_config_location: '/etc/haproxy/haproxy_mntc.cfg'
+    # By default, target backend is "nginx" if nginx plugin is enabled, otherwise it is "envoy"


There is no obligation to install envoy or nginx. They could be both disabled

alexarefev · 2026-03-29T07:40:20Z

documentation/Installation.md


+###### target_backend
+
+This section describes who should lister `target_ports` on worker nodes, either `nginx` or `envoy`. Default is `nginx` for backward compatibility, but it is recommended to use `envoy`.


Some reference to migration from nginx to envoy gateway is needed

theboringstuff added 7 commits March 19, 2026 09:07

implementation

d2729c3

fix schema and default install

28f772a

remove unused var

f224ffa

fix types

305650e

fix unit tests

0730089

fix registry

20779c7

dynamic priority

a5816ab

theboringstuff force-pushed the feature/envoy-gateway-support branch from 4042218 to a5816ab Compare March 19, 2026 09:11

theboringstuff added 6 commits March 19, 2026 18:27

update envoy 2.2.0

1b9ffbc

pss support

ecdbac9

docs/examples

f052520

remove todos

4bb36a0

update envoy 2.2.0

4de758c

do not install HTTPRoute if no envoy-gateway

69a11ee

theboringstuff assigned DmitriiRabenok Mar 27, 2026

theboringstuff requested review from alexarefev and pranavcracker March 27, 2026 08:01

theboringstuff marked this pull request as ready for review March 27, 2026 08:01

dashboard ignore annotation

4b74d8f

alexarefev reviewed Mar 29, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Envoy Gateway support#798

Envoy Gateway support#798
theboringstuff wants to merge 14 commits intomainfrom
feature/envoy-gateway-support

theboringstuff commented Mar 19, 2026 •

edited

Loading

Uh oh!

alexarefev commented Mar 27, 2026

Uh oh!

theboringstuff commented Mar 27, 2026

Uh oh!

alexarefev Mar 27, 2026

Uh oh!

alexarefev Mar 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants


		###### target_backend

		This section describes who should lister `target_ports` on worker nodes, either `nginx` or `envoy`. Default is `nginx` for backward compatibility, but it is recommended to use `envoy`.

Conversation

theboringstuff commented Mar 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Solution

Test Cases

Test case 1

Test case 2

Test case 3

Test case 4

Checklist

Uh oh!

alexarefev commented Mar 27, 2026

Uh oh!

theboringstuff commented Mar 27, 2026

Uh oh!

alexarefev Mar 27, 2026

Choose a reason for hiding this comment

Uh oh!

alexarefev Mar 29, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

theboringstuff commented Mar 19, 2026 •

edited

Loading