Kubernetes assumes that the nodes of a cluster can reach each other. That is easy inside one data center or one cloud network. It becomes a problem when the nodes are spread across different clouds and regions, or sit at the edge behind NAT and firewalls.
Our new Kubernetes guide shows how to run a single cluster across such nodes by connecting them over Netsody.
Netsody as the Only Tunnel
Every node runs the Netsody agent and joins the same network. This gives each node a stable overlay IP address and a direct encrypted path to every other node, regardless of where the node is located.
Kubernetes then binds to the Netsody interface. The guide uses k3s with the host-gw backend of flannel, which installs plain kernel routes and adds no encapsulation of its own. A packet from one pod to another keeps its destination IP address and is routed over the mesh directly. Netsody is the only tunnel involved, so pod traffic is not encapsulated a second time.
Pod Subnets as Resources
Kubernetes assigns each node its own pod subnet. To make pods on one node reachable from another, each of these subnets is published in Netsody as a resource with that node as the gateway. A policy then allows the nodes to reach these resources.
This is the same model Netsody uses for any private subnet behind a gateway. The pod network is therefore visible in the controller and subject to policy like everything else in the network.
One Thing to Know
Netsody only forwards traffic whose source is the node’s own overlay IP address, which prevents IP spoofing. A pod sends from its pod IP address, so each node masquerades pod traffic to its overlay address. The guide contains the iptables rule for this.
The consequence is that the pod source IP address is not preserved across nodes. A Kubernetes NetworkPolicy that matches cross-node traffic by source pod does not work in this setup.
Growing the Cluster
Adding a node takes three steps: add it to the Netsody network, join it to the cluster as an agent, and publish its pod subnet as a resource. For headless nodes, join keys take care of the first step without an interactive login.
The guide walks through the complete setup with a k3s server and two agents, from installation to a ping between pods on different nodes.
Try it
If you run Kubernetes across sites, or would like to, we are interested in your setup and in what is missing. Tell us on Discord or at oi.ydosten@olleh.