Introduction
Have you ever wondered what’s happening under the hood of Kubernetes bootstrap process?
In this blog post I will take you on a journey through the steps of the Kubernetes cluster provisioning in general without focusing on any particular cloud provider or hardware virtualization.
It will give you an overview how the various components fit and work together.
Kubernetes components
This section assume that you are familiar with Kubernetes and know the basics.
Before we start digging into the actual bootstrap process let’s take a look at the Kubernetes components in order to understand what we’re going to provision.
As you can see we have 3 nodes — two workers and one master node.
API component is running on master node which exposes Kuberentes API. An authenticated and authorized user or service can view or modify Kubernetes objects described in manifests.
Scheduler and Controller are running on master node and are responsible for managing cluster state based on Kubernetes Objects stored in etcd (highly available key-value datastore).
Kubelet is “primary” node agent and it’s focused on running containers, it doesn’t manage containers which were not created on Kubernetes.
Kube-proxy is responsible for TCP and UDP forwarding (iptables or ipvs) for Kubernetes Services.
Pod is just a group of containers (docker, rkt) running on the same node with common life cycle.
Under the Kubernetes abstraction layer there is nothing more a operating system, container runtime, systemd services and some networking stuff.
Communication between containers is done by CNI (Container Network Interface) which manages network virtual interfaces (e.g. Flannel, Calico).
In some cases there might be virtualization environment running underneath (e.g. in the could).
Bootstrap on baremetal
In order to understand the Kubernetes bootstrap process from very beginning let’s start with bare-metal example.
It involves number of technologies, like:
- PXE enabled network boot
- DHCP
- TFTP
- OS image (Container Linux)
- OS configuration (Ignition from Container Linux)
- Docker
PXE network boot
PXE (Preboot Execution Environment) is one of the possible ways to network boot.
PXE server is nothing more that DHCP and TFTP servers which provide kernel image and root file system.
NIC (Network Interface Card) of PXE client sends DHCP request and receives network configuration like: IP, subnet, mask, DNS and gateway.
In addition it provides location of TFTP server.
Client connects to TFTP server in order to download boot image.
Client executes boot image and downloads all the files it needs (kernel image, root file system).
Optionally we can mount NFS (Network File System).
Dnsmasq
Dnsmasq is a mature project that can be used for a number of purposes:
- works as DHCP server
- also can work as DHCP in proxy mode if you already have DHCP server on your network
- TFTP server
It can be run as a container:
CoreOS and Ignition
As the operating system we’ll use CoreOS Container Linux which is:
““Minimal container operating system with basic userland utilities. Runs on nearly any platform whether physical, virtual or private/public cloud.””
It supports a container runtime (docker and rkt) out of the box.
Ignition is a provisioning utility for Container Linux based on YAML files. It executes in early boot process in initrd. It’s able to partition disk, configure network, systemd services and many more.
In order to understand when and where Ignition is executed let’s take a look at high level Linux Boot Process:
Matchbox
Matchbox came from CoreOS and actually it’s just a web and gRPC server which takes care about mapping baremetal machines based on labels like MAC address to PXE boot profiles.
Matchbox is configured by files and directory structure, by default it uses /var/lib/matchbox. It can be also configured using Terrform module.
Directory consists of:
- assets — just static files
- groups — mapping between baremetal machines and boot profiles
- profiles — kernel/initrd images
- ignition — OS configuration like: systemd services, network configuration
It can be run as a container:
For more examples please take a look at coreos/matchbox/examples.
Groups
Groups are selector for phycical machines based on label e.g. MAC address with metadata and corresponding profile.
Profiles
Profiles specify the kernel and initrd, kernel arguments, iPXE config, GRUB config or other configuration values a given machine should use.
Assets
This directory consists of static assets, like Container Linux image.
Ignition
This directory consists of ignition templates for profiles.
Kubernetes node boot process
After we covered all the building blocks needed for Kubernetes node boot process, let’s summarize it briefly:
- Power on baremetal machine.
- Receive network configuration from DHCP.
- PXE network boot.
- First OS boot with configuration via Ignition.
- Provide additional assets (TLS certificates, kubernetes manifests).
- Systemd kubelet.service starts and loads kubernetes manifests.
kubelet.service
Kubelet is configured as systemd service and runs on every node.
It loads all Kubernetes manifests from directory specified by pod-manifest-path (kube-apiserver, kube-proxy, kube-dns, etc).
On the other hand kubelet is responsible for configuring CNI network based on cni-conf-dir parameter.
Example CNI configuration (for Calico):
kube-proxy
Once kubelet.service is up and running, it applies kube-proxy manifest:
We’ve seen before that kube-proxy is responsible for TCP and UDP forwarding using iptables or ipvs rules.
That’s pretty much all what is needed to bootstrap minimal Kubernetes Cluster. We can dig deeper into kube-apiserver, kube-dns, scheduler or even more configuration files but this is not the scope of this blog post :)
Bootstrap on cloud
Bootstrap on cloud is not so different than on baremetal instead of providing physical machines we have to take care of provisioning the nodes using different tools running on higher level of abstraction.
For example in terms of AWS cloud provider we can take an advantage of CloudFormation templates in order to create and provision EC2 instances.
Note that every cloud provider use different virtualization environment which sometimes adds some level of complexity.
Summary
Obviously there are more ways how to bootstrap Kubernetes cluster like kubeadm, kubespray, tectonic-installer, etc.
I decided to write this blog post because I couldn’t find anything on the internet which covers high level overview of the bootstrap process.
