In this post, we will introduce MidoNet virtual networking integration with Kubernetes.
Kubernetes, as it’s stated in the official documentation, is an open-source platform for automating deployment, scaling, and operations of application containers across clusters of hosts, providing container-centric infrastructure.
Kubernetes is portable, expandable and self healing platform builds upon experience that Google has with running production workloads at scale, combined with best-of-breed ideas and practices from the community. If you are not familiar with Kubernetes basics, it may be helpful to review information about kubernetes basics and terminology.
MidoNet is an open, software-only, highly scalable and resilient, network virtualization system. MidoNet allows users to build isolated networks in software and overlays the existing network hardware infrastructure. It allows to build, run, and manage virtual networks at scale with increased control, security and flexibility. MidoNet provides fully virtualized L2 to L4 Networking,i.e. switches, routers, DHCP, NAT, load balancers and firewalls among other network services.
Kubernetes networking model is such that each pod (application ‘logical host’) has an IP in a flat shared networking namespace that has full communication with other physical computers and containers across the network. IP-per-pod creates a clean, backward-compatible model where pods can be treated much like VMs or physical hosts from the perspectives of port allocation, networking, naming, service discovery, load balancing, application configuration, and migration. Kubernetes networking model avoids using dynamic port allocation and NAT that complicates applications configuration and is not convenient for the users.
All containers within a pod can reach each other’s ports on localhost. This offers simplicity, since static ports are known. It’s quite secure, since containers inside a pod do not have unique IP addresses and hence, are reachable from the outside through the Pod only.
Every Pod gets a unique IP address in a NAT-less flat address space, therefore pods can communicate without proxies or translations. IP addresses are the same both inside and outside the pods. This makes service discovery simple. Within a pod, containers likely communicate though volumes or IPC. Inter pod communication is what this post is going to cover.
With native kubernetes networking, each node in k8s cluster has an /24 (256 IP addresses) out of the Cluster ip address space that get routed to it in addition to the node ‘main’ IP address that is usually NAT-ed for the internet access. This is the approach that many of the existing kubernetes networking implementations take, like flannel, OpenVSwitch, etc.
This approach imposes limitations on the maximal amount of Pods that can be spawned on each node.
Support Kubernetes Networking with MidoNet
Let’s check the approach we took to integrate MidoNet SDN as a kubernetes network provider through the CNI plugin. Kubernetes supports multiple virtual clusters backed by the same physical cluster. These virtual clusters called namespaces. Namespace is a way to divide cluster resources (i.e. CPU, Disk, Memory) between multiple projects, users or teams, providing a scope for resources names and resource quota. If you are familiar with OpenStack, kubernetes namespace is similar to OpenStack tenant/project notion. Although Kubernetes does not expect namespace to provide isolated networking domains while OpenStack does. In MidoNet, virtual bridges are used to isolate different tenant networks. To allow more flexible networking and as a step towards Network Policy support, kubernetes namespace is mapped to MidoNet virtual bridge with associated subnet allocated from the cluster IP addresses space.
Kubernetes native networking allocates subnet , usually /24, for each worker node.
With MidoNet integration, we allocate subnet per namespace, and there maybe Pods on different nodes which IP address is allocated from the same subnet.
In order to satisfy the requirements on the kubernetes networking model that all pods should be reachable, there is a cluster virtual route to connect all subnets. All kubernetes namespaces’ virtual bridges are automatically connected to this router.
Kubernetes Pods Virtual Topology
We use neutron API (de facto MidoNet tenant networking API) for the integration.
Implementing Kubernetes Services
The kubernetes service abstraction provides a way to group pods unders the same access policy with stable access IP address. Service is assigned virtual IP (VIP) that clients can access which is proxied to the pods in the service.
In native kubernetes networking implementation, each node runs a kube-proxy process which programs iptables rules to trap access to service IPs and redirect them to the correct backends. Kube-proxy does the job very well, but as number of services increase, its getting more and more difficult to debug and maintain as requires in deep iptables expertise.
With MidoNet, we leverage MidoNet L4 load balancer to support K8s services.
K8s service is mapped to the MidoNet Load Balancer VIP and associated pool. Every service endpoint is assigned as member in the Load Balancer pool. MidoNet Load Balancer manages the k8s services with no need to run kube-proxy on the nodes.
Kubernetes Service Virtual topology
To allow access to the cluster services from the outside world, the common practice is to use external load balancer. With MidoNet networking provider, we extend the internal service implementation by association of the floating IP (FIP) with service virtual IP (VIP).
Also for the services implementation, we use neutron API to program MidoNet virtual topology.
As you can see, with Neutron/MidoNet network provider, the k8s networking model can be fully supported.
Among the benefits of Kubernetes deployments with Neutron/MidoNet, we can list:
- Inherent multi-tenancy and micro-segmentation support
- Distributed, scalable L4 load-balancing for kubernetes services
- Flexible IPAM independent of physical location
By using MidoNet native API or via neutron API, advanced network services may be made available, such as:
- QoS (rate-limiting/noisy-neighbor protection, DSCP/service classes)
- Network function insertion
- Port mirroring
MidoNet networking will support hybrid environments allowing nested or side by side OpenStack deployments.
With MidoNet Enterprise edition (MEM) there are even more benefits availables, such as:
- L7 advanced security policies by security controllers
- Flow-level visibility of forwarding/security decisions
We will follow up with a new post on integration implementation, additional advanced features and plans to engage with OpenStack/Kuryr community.