Introduction
Inter node communication between pods on Kubernetes is enabled by networking infrastructure set up by CNI plugins. On this page is described the architecture and setup of EVIO CNI plugin enabling integration of EVIO features with Kubernetes, where inter node pod traffic is carried on encrypted p2p tincan tunnels between nodes part of the Kubernetes cluster. The plugin grafts the inter-node pod network on existing EVIO P2P overlay network connecting the nodes.
Architecture
Functionality is split in two executables.
- CNI config gen utility
- EVIO CNI plugin
Both of these executables are deployed as a daemon set on the Kubernetes cluster, where they will be run on every participating node. CNI config gen runs as an Init Container and generates a configuration file which specifies the CNI plugin to be picked up by Kubelet. EVIO CNI plugin carries out the actual task of setting up the network.
Two key aspects will of networking will have to be decided by the user
- PodCIDR - Subnet for the pod network e.g. 10.144.0.0/16
- NodeBits - Number of bits to identify the max number of allowed nodes in the cluster. e.g. 8
To illustrate, if NodeBits is 8, 8 bits will be reserved for node identification and the cluster can have 2^8 = 256 nodes. Assuming 16 bits for podCIDR this leaves us with 32 - 16 -8 = 8 bits for allocating addresses for pods on a given node. CNI config gen will use this information to generate address range for all pods instantiated on a given node, with 8 bits we can have 2^8 - 3 = 253 pod addresses on a given node, x.x.x.0 and x.x.x.255 addresses are not used, first valid address in the range is reserved for the pod bridge on the host.
CNI config gen keeps track of the address range of pods allocated to each node in a Etcd database. When a new node joins the cluster, it generates a configuration file with a non-conflicting address range for pods on this node and saves it in the default "/etc/cni/net.d" directory.
Architecture of EVIO CNI plugin is based on the Bridge CNI plugin which is extended by incorporating support for OVS bridges. Key actions taken by EVIO plugin are as follows.
- Check if EVIO bridge is up, if not set up the bridge.
- Configure the bridge with the gateway address and bring it up.
- Set up a Veth interface pair for every pod, with the host end attached to the EVIO bridge.
- Configure the pod Network Namespace with an IP address and set up routes and IP Masq, this functionality is delegated to IPAM host-local plugin.
Prerequisites
-
Instantiate a two-overlay EdgeVPN network connecting all the nodes that you intend to have in your kubernetes cluster. First overlay is used for connectivity between the host nodes and the second overlay will be used for inter node pod communication. Follow the instructions in the linked page to set up the EdgeVPN network.
Setting up two-overlay EdgeVPN network
Verify the connectivity between the nodes on the overlay before proceeding. -
Install kubeadm and associated components and container runtime before proceeding. Our setup has been tested with docker as container runtime. The versions are as shown below.
osboxes@godev:~/Downloads/EVPN/controller$ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.3", GitCommit:"2e7996e3e2712684bc73f0dec0200d64eec7fe40", GitTreeState:"clean", BuildDate:"2020-05-20T12:49:29Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
osboxes@godev:~/Downloads/EVPN/controller$ docker version
Client:
Version: 18.09.7
API version: 1.39
Go version: go1.10.8
Git commit: 2d0083d
Built: Thu Jun 27 17:56:23 2019
OS/Arch: linux/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 18.09.7
API version: 1.39 (minimum version 1.12)
Go version: go1.10.8
Git commit: 2d0083d
Built: Thu Jun 27 17:23:02 2019
OS/Arch: linux/amd64
Experimental: false
osboxes@godev:~/Downloads/EVPN/controller$
Instruction on installing kubeadm and associated tools can be found in the linked page.
Install kubeadm
Verify kubeadm has been successfully setup.
- Install etcd data-store, There might be conflicts instantiating etcd on the Kubernetes master node (reason for listening port 42000 in the following command). Make sure that the datastore is reachable by all nodes part of the Kubernetes cluster.
docker run -d -p 42000:42000 --env ALLOW_NONE_AUTHENTICATION=yes --env ETCD_ADVERTISE_CLIENT_URLS=http://0.0.0.0:42000 --env ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:42000 --name etcd bitnami/etcd
Test that the datastore is accessible on the EdgeVPN overlay.
osboxes@godev:~$ sudo apt update
osboxes@godev:~$ sudo apt install etcd-client
osboxes@godev:~$ ETCDCTL_API=3 etcdctl --endpoints http://10.10.100.1:42000 put foo bar
OK
osboxes@godev:~$
Setup Kubernetes cluster
Note: Setup assumes that Kubernetes cluster has been setup and nodes are already connected by a EVIO overlay network. These steps have been tested on an Ubuntu 18.04 LTS platform, with Kubernetes cluster setup using kubeadm v1.18.3:
- Remove existing CNI config files, otherwise kubelet will simply invoke the plugin specified by the first config file in alphabetical order.
osboxes@osboxes:~$ cd /etc/cni/net.d
osboxes@osboxes:/etc/cni/net.d$ sudo rm *.conf
osboxes@osboxes:/etc/cni/net.d$ sudo rm *.conflist
rm: cannot remove '*.conflist': No such file or directory
osboxes@osboxes:/etc/cni/net.d$
- To force kubelet to use EdgeVPN node addresses one has to manually provide the --node-ip option in the below file.
ubuntu@ip-172-31-43-219:/etc/systemd/system/kubelet.service.d$ sudo cat 10-kubeadm.conf
# Note: This dropin only works with kubeadm and kubelet v1.11+
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml --node-ip=10.10.100.2"
# This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use
# the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.
EnvironmentFile=-/etc/default/kubelet
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS
sudo systemctl daemon-reload
sudo systemctl restart kubelet
- Install Kubernetes using kubeadm, make note of podCIDR should be same as you will use for evio template, advertise address is the EdgeVPN node overlay address of the master node.
sudo swapoff -a
sudo kubeadm init --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=10.10.100.2
- If want to deploy pods on master for testing, execute the below command.
kubectl taint nodes --all node-role.kubernetes.io/master-
- You should be able to see something like below on you host.
osboxes@osboxes:~$ kubectl get pods --all-namespaces -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-system coredns-66bff467f8-db5ms 0/1 Pending 0 54s <none> <none> <none> <none>
kube-system coredns-66bff467f8-wj6mv 0/1 Pending 0 54s <none> <none> <none> <none>
kube-system etcd-osboxes 1/1 Running 0 64s 192.168.81.134 osboxes <none> <none>
kube-system kube-apiserver-osboxes 1/1 Running 0 64s 192.168.81.134 osboxes <none> <none>
kube-system kube-controller-manager-osboxes 1/1 Running 0 63s 192.168.81.134 osboxes <none> <none>
kube-system kube-proxy-9rnck 1/1 Running 0 54s 192.168.81.134 osboxes <none> <none>
kube-system kube-scheduler-osboxes 1/1 Running 0 64s 192.168.81.134 osboxes <none> <none>
osboxes@osboxes:~$
- Get evioCNI.yml and modify it as per you needs - key configuration is in the definition of configmap.
wget https://github.com/saumitraaditya/evioCNI/raw/master/evioDeployment/evioCNI.yml
net-conf.json: |
{
"cniVersion": "0.3.1",
"name": "k8s-pod-network",
"type": "evioPlugin",
"bridge": "evioB1B111B",
"bridgeType": "OVS",
"isGateway": true,
"isDefaultGateway": true,
"isIPMasq": true,
"nodeBits": "8",
"podCIDR": "10.244.0.16/16",
"dataStore": "10.10.100.1:42000",
"auth": false
}
Here bridge is the evioBridge for your respective podNetwork EdgeVPN overlay, podCIDR should match with the one in kubeadm init command and nodeBits are as described earlier. dataStore is the end point address of the etcd store on the EdgeVPN node overlay. In the example configuration access to data store is not authenticated.
If auth is set to "true", username,password authentication will be used to access the etcd store, this would entail both configuring the etcd store appropriately and configuring the username and password as base64 encoded literals in secret section of the evioCNI.yml file. etcd store can be configured using the below steps, note that the datastore prefix should be "evpn".
ETCDCTL_API=3 etcdctl --endpoints http://10.10.100.1:42000 user add root
ETCDCTL_API=3 etcdctl --endpoints http://10.10.100.1:42000 auth enable
ETCDCTL_API=3 etcdctl --endpoints http://10.10.100.1:42000 --user root:root role add evpnManager
ETCDCTL_API=3 etcdctl --endpoints http://10.10.100.1:42000 --user root:root role grant-permission evpnManager --prefix=true readwrite /evpn/
ETCDCTL_API=3 etcdctl --endpoints http://10.10.100.1:42000 --user root:root user add evpnUser
ETCDCTL_API=3 etcdctl --endpoints http://10.10.100.1:42000 --user root:root user grant-role evpnUser evpnManager
section in evioCNI.yml file to be populated is as below.
osboxes@osboxes:~/Documents/evioCNI$ echo -n 'evpnUser' | base64
ZXZwblVzZXI=
osboxes@osboxes:~/Documents/evioCNI$ echo -n 'pass' | base64
cGFzcw==
osboxes@osboxes:~/Documents/evioCNI$
apiVersion: v1
kind: Secret
metadata:
name: store-credentials
namespace: kube-system
data: # base64 encoded
username: ZXZwblVzZXI=
password: cGFzcw==
- Now deploy the plugin daemonset using the definition file.
osboxes@osboxes:~/evioCNI$ kubectl apply -f evioCNI.yml
podsecuritypolicy.policy/psp.evio.unprivileged created
serviceaccount/evio created
configmap/kube-evio-cfg created
daemonset.apps/kube-evio-ds-amd64 created
osboxes@osboxes:~/evioCNI$
- Check again if pods have connected to the network, coreDNS pods should have an IP address in the range, also there should be a populated config file as well with address range for pods on this node and a switch setup with host veth interface endpoints from pods connected.
osboxes@osboxes:~$ kubectl get pods --all-namespaces -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-system coredns-66bff467f8-pbjlm 1/1 Running 0 2m34s 10.244.0.3 osboxes <none> <none>
kube-system coredns-66bff467f8-rk2r6 1/1 Running 0 2m34s 10.244.0.2 osboxes <none> <none>
kube-system etcd-osboxes 1/1 Running 0 2m43s 192.168.81.134 osboxes <none> <none>
kube-system kube-apiserver-osboxes 1/1 Running 0 2m43s 192.168.81.134 osboxes <none> <none>
kube-system kube-controller-manager-osboxes 1/1 Running 0 2m43s 192.168.81.134 osboxes <none> <none>
kube-system kube-evio-ds-amd64-v7wmc 1/1 Running 0 71s 192.168.81.134 osboxes <none> <none>
kube-system kube-proxy-9klts 1/1 Running 0 2m34s 192.168.81.134 osboxes <none> <none>
kube-system kube-scheduler-osboxes 1/1 Running 0 2m43s 192.168.81.134 osboxes <none> <none>
osboxes@osboxes:~$ sudo ovs-vsctl show
f269da11-336f-43cc-ae9d-692fb4098c0f
Bridge "evioB1B111B"
Port "chnl-a100001"
Interface "chnl-a100001"
Port "vethcd0b0f57"
Interface "vethcd0b0f57"
Port "vethedc3a33b"
Interface "vethedc3a33b"
Port "evioB1B111B"
Interface "evioB1B111B"
type: internal
Bridge "brlA1A111A"
Port "brl-pp0"
Interface "brl-pp0"
type: patch
options: {peer="evi-pp0"}
Port "brlA1A111A"
Interface "brlA1A111A"
type: internal
Bridge "evioA1A111A"
Controller "tcp:127.0.0.1:6633"
is_connected: true
Port "tnl-a100001"
Interface "tnl-a100001"
Port "evi-pp0"
Interface "evi-pp0"
type: patch
options: {peer="brl-pp0"}
Port "evioA1A111A"
Interface "evioA1A111A"
type: internal
ovs_version: "2.9.2"
osboxes@osboxes:~$ cat /etc/cni/net.d/10-evio.conf
{
"cniVersion": "0.3.1",
"name": "k8s-pod-network",
"type": "evioPlugin",
"bridge": "evioB1B111B",
"bridgeType": "OVS",
"bridgeIPv4": "10.244.0.1/16",
"isGateway": true,
"isDefaultGateway": true,
"isIPMasq": true,
"ipam": {
"type": "host-local",
"subnet": "10.244.1.0/16",
"rangeStart": "10.244.0.2",
"rangeEnd": "10.244.0.254"
},
"nodeBits": "8",
"dataStore": "10.10.100.1:42000"
}
- Debug logs for kubelet can be observed by invoking the below command, if there is a problem with CNI it should be apparent in the logs.
sudo journalctl -u kubelet -f
- Logs for evioPlugin can be observed like below
osboxes@osboxes:/var/log$ tail -f /var/log/evioPlugin.log
- In case your pods do not have internet connectivity try the below iptables rule.
sudo iptables -A FORWARD -j ACCEPT
- If your pods cannot resolve dns names, you need to update the kube-proxy configuration.
kubectl -n kube-system edit configmap kube-proxy
In iptables section set masqueradeAll to true.
iptables:
masqueradeAll: true
masqueradeBit: null
minSyncPeriod: 0s
syncPeriod: 0s
kubectl -n kube-system edit daemonset kube-proxy
Towards the bottom of the file add "--masquerade-all"
spec:
containers:
- command:
- /usr/local/bin/kube-proxy
- --masquerade-all
- --config=/var/lib/kube-proxy/config.conf
- --hostname-override=$(NODE_NAME)
env:
- name: NODE_NAME
Make sure your kube-proxy pod has come up with no errors.
kubectl get pods -n kube-system -o wide