Networking overview
In-cluster networking
Chef 360 Platform’s embedded cluster built-in network provider is Calico. Calico is a layer 3 container networking solution that routes packets to pods.
For additional information on in-cluster networking, see Container Network Interface (CNI).
Note
Controller-worker communication
One goal of k0s is to allow for the deployment of an isolated control plane which may prevent the establishment of an IP route between controller nodes and the pod network. To enable this communication path (which is mandated by conformance tests), k0s deploys the Konnectivity service to proxy traffic from the API server (control plane) into the worker nodes. This ensures that Chef 360 Platform can always fulfill all the Kubernetes API functionalities, but still operate the control plane in total isolation from the workers.
Note
Required ports and protocols
| Protocol | Port | Service | Direction | Notes |
|---|---|---|---|---|
| TCP | 2380 | etcd peers | controller <-> controller | |
| TCP | 6443 | kube-apiserver | worker, CLI => controller | Authenticated Kube API using Kube TLS client certs, ServiceAccount tokens with RBAC |
| TCP | 179 | kube-router | worker <-> worker | BGP routing sessions between peers |
| UDP | 4789 | Calico | worker <-> worker | Calico VXLAN overlay |
| TCP | 10250 | kubelet | controller, worker => host * | Authenticated kubelet API for the controller node kube-apiserver (and heapster/metrics-server addons) using TLS client certs |
| TCP | 9443 | k0s-api | controller <-> controller | k0s controller join API, TLS with token auth |
| TCP | 8132 | konnectivity | worker <-> controller | Konnectivity is used as “reverse” tunnel between kube-apiserver and worker kubelets |
| TCP | 112 | keepalived | controller <-> controller | Only required for control plane load balancing VRRP instances for IP address 224.0.0.18. 224.0.0.18 is a multicast IP address defined in [RFC 3768]. |
You also need to enable all traffic to and from the [podCIDR and serviceCIDR] subnets on nodes with a worker role.
Note
iptables
iptables can work in two distinct modes, legacy and nftables.
k0s auto detects the mode and prefers nftables.
To check which mode k0s is configured with, run:
ls -lah /var/lib/k0s/bin/
The iptables link target reveals the mode which k0s selected.
k0s has the same logic as other Kubernetes components. Use the following commands to make sure all components use the same mode:
- kube-proxy:
nsenter -t $(pidof kube-proxy) -m iptables -V
- kube-router:
nsenter -t $(pidof kube-router) -m /sbin/iptables -V
- calico:
nsenter -t $(pidof -s calico-node) -m iptables -V
Different iptables versions have known incompatibility issues.
k0s ships a version of iptables that’s tested to interoperate with all other Kubernetes components it ships with. However, if you have other tooling (firewalls etc.) on your hosts that uses iptables and the host iptables version is different from the version k0s (and other Kubernetes components) ships with, it may cause networking issues.
Firewalld
If you are using firewalld on your hosts, ensure it’s configured to use the same FirewallBackend as Chef 360 Platform and other Kubernetes components use. Otherwise, networking will be broken in various ways.
Here’s an example configuration for a tested working networking setup:
[root@rhel-test ~]# firewall-cmd --list-all
public (active)
target: default
icmp-block-inversion: no
interfaces: eth0
sources: 10.244.0.0/16 10.96.0.0/12
services: cockpit dhcpv6-client ssh
ports: 80/tcp 6443/tcp 8132/tcp 10250/tcp 179/tcp 179/udp
protocols:
forward: no
masquerade: yes
forward-ports:
source-ports:
icmp-blocks:
rich rules: