RFC: pod2pod network mesh implementation

[dciangot]

Summary

Definition for the definitive implementation of a network overlay at user-namespace level b/w the kubernetes pod network and the remote resources.

Background

We have tested manual workflows combining websocket tunnel technologies (wstunnel) with Wirguard VPN, to integrate ofloaded pods into the internal Kubernetes pod. This discussion is about make it accessible and maintainable.

N.B. Tunneling of a service port through wstunnel only is already supported following the suggested schema below. Therefore, the following consists in an evolution of the current scenario.

Proposed solution

┌──────────────────────────────────────────────────────┐       ┌───────────────────────────────────────────────────────────────────────┐
│                                                      │       │                                                                       │
│   K8s cluster                                        │       │    Remote site                                                        │
│                                                      │       │                                                                       │
│                                                      │       │             ┌──────────────────────┐                                  │
│           ┌────────────────────────┐                 │       │             │  linux namespace     │                                  │
│           │                        │                 │       │             │                      │                                  │
│           │                        │                 │       │             │                      │                                  │
│           │     K8s Pod            │         ┌───────┼───────┼─────────────┼wstunnel client       │                                  │
│           │                        │         │       │       │             │      ▲               │                                  │
│           │     10.222.1.12        │         │       │       │             │      │               │                                  │
│           │      ▲                 │         │       │       │             │wireguard client      │                                  │
│           │      │                 │         │       │       │             │                      │                                  │
│           │      │                 │         │       │       │             │           slirp4netns│                                  │
│           └──────┼─────────────────┘         │       │       │             └──────────────────────┘                                  │
│                  │                           │       │       │                                                                       │
│                  │                           │       │       │                                                                       │
│                  │                           │       │       │                                                                       │
│                  │                           │       │       │                                                                       │
│                  │                           │       │       │                                                                       │
│                  │                           │       │       │                                                                       │
│                  │                           │       │       │                                                                       │
│            ┌─────┼───────────────────┐       │       │       │                                                                       │
│            │     │                   │       │       │       │                                                                       │
│            │     │    k8s shadow pod ┌───────┼──────────┐    │                                                                       │
│            │     │                   │       ▼       │  │    │                                                                       │
│            │     │                   │  ws tunnel│   │  │    │                                                                       │
│            │     │                   │           ▼   │  │    │                                                                       │
│            │     │10.222.1.12        │  wireguard server│    │                                                                       │
│            │                         └──────────────────┘    │                                                                       │
│            │      K8s assigned IP    │        ▲      │       │                                                                       │
│            └─────────────────────────┴────────┘      │       │                                                                       │
│                                                      │       │                                                                       │
│                            iptables forward          │       └───────────────────────────────────────────────────────────────────────┘
│                                                      │                                                                                
│                                                      │                                                                                
│                                                      │                                                                                
│                                                      │                                                                                
│                                                      │                                                                                
│                                                      │                                                                                
└──────────────────────────────────────────────────────┘                                                                                

• Use the creation of a shadow pod (on a shadow namespace) to create a bridge already integrated into the k8s overlay network (assigned and managed by k8s natively).
• Connect the bridge through wstunnel (creating ingress endpoint through a wildcard is REQUIRED)
• Jobs will spawn a user linux namespace where connectivity is granted by slirp4netns and that joins the k8s network through a Wireguard VPN running over the wstunnel service.
• K8s pod IP will be automatically set to the Shadow pod IP, that will then forward all the request to the remote wg client endpoint.
• Remote job will route all the request to the pod subnet back thruogh the VPN as well.

Pros

• Fully automatic management of the IPs through native k8s mechanism
• Requires only ingress + wildcard DNS

Cons

• VPN reduce performances, and user ns cosumes a loopback -->The tunnel only scenario should be preferred over full mesh whenever possible

Question for discussion

This RFC is open to community discussion. Please provide feedback.

[dciangot]

TBD Using pasta instead of slirp4netns tu boost inter-job communication performance.