Kubernetes Cluster Bootstrap Explained

Introduction

Have you ever wondered what’s happening under the hood of Kubernetes bootstrap process?

In this blog post I will take you on a journey through the steps of the Kubernetes cluster provisioning in general without focusing on any particular cloud provider or hardware virtualization.

It will give you an overview how the various components fit and work together.

Kubernetes components

This section assume that you are familiar with Kubernetes and know the basics.

Before we start digging into the actual bootstrap process let’s take a look at the Kubernetes components in order to understand what we’re going to provision.

As you can see we have 3 nodes — two workers and one master node.

API component is running on master node which exposes Kuberentes API. An authenticated and authorized user or service can view or modify Kubernetes objects described in manifests.

Scheduler and Controller are running on master node and are responsible for managing cluster state based on Kubernetes Objects stored in etcd (highly available key-value datastore).

Kubelet is “primary” node agent and it’s focused on running containers, it doesn’t manage containers which were not created on Kubernetes.

Kube-proxy is responsible for TCP and UDP forwarding (iptables or ipvs) for Kubernetes Services.

Pod is just a group of containers (docker, rkt) running on the same node with common life cycle.

Under the Kubernetes abstraction layer there is nothing more a operating system, container runtime, systemd services and some networking stuff.

Communication between containers is done by CNI (Container Network Interface) which manages network virtual interfaces (e.g. Flannel, Calico).

In some cases there might be virtualization environment running underneath (e.g. in the could).

Bootstrap on baremetal

In order to understand the Kubernetes bootstrap process from very beginning let’s start with bare-metal example.

It involves number of technologies, like:

PXE enabled network boot
DHCP
TFTP
OS image (Container Linux)
OS configuration (Ignition from Container Linux)
Docker

PXE network boot

PXE (Preboot Execution Environment) is one of the possible ways to network boot.

PXE server is nothing more that DHCP and TFTP servers which provide kernel image and root file system.

NIC (Network Interface Card) of PXE client sends DHCP request and receives network configuration like: IP, subnet, mask, DNS and gateway.

In addition it provides location of TFTP server.

Client connects to TFTP server in order to download boot image.

Client executes boot image and downloads all the files it needs (kernel image, root file system).

Optionally we can mount NFS (Network File System).

Dnsmasq

Dnsmasq is a mature project that can be used for a number of purposes:

works as DHCP server
also can work as DHCP in proxy mode if you already have DHCP server on your network
TFTP server

It can be run as a container:

1sudo docker run --rm --cap-add=NET_ADMIN --net=host quay.io/coreos/dnsmasq \
2  -d -q \
3  --dhcp-range=192.168.1.3,192.168.1.254 \
4  --enable-tftp --tftp-root=/var/lib/tftpboot \
5  --dhcp-match=set:bios,option:client-arch,0 \
6  --dhcp-boot=tag:bios,undionly.kpxe \
7  --dhcp-match=set:efi32,option:client-arch,6 \
8  --dhcp-boot=tag:efi32,ipxe.efi \
9  --dhcp-match=set:efibc,option:client-arch,7 \
10  --dhcp-boot=tag:efibc,ipxe.efi \
11  --dhcp-match=set:efi64,option:client-arch,9 \
12  --dhcp-boot=tag:efi64,ipxe.efi \
13  --dhcp-userclass=set:ipxe,iPXE \
14  --dhcp-boot=tag:ipxe,http://matchbox.example.com:8080/boot.ipxe \
15  --address=/matchbox.example/192.168.1.2 \
16  --log-queries \
17  --log-dhcp

CoreOS and Ignition

As the operating system we’ll use CoreOS Container Linux which is

“Minimal container operating system with basic userland utilities. Runs on nearly any platform whether physical, virtual or private/public cloud.”

It supports a container runtime (docker and rkt) out of the box.

Ignition is a provisioning utility for Container Linux based on YAML files. It executes in early boot process in initrd. It’s able to partition disk, configure network, systemd services and many more.

1---
2systemd:
3  units:
4    - name: docker.service
5      enable: true
6    - name: locksmithd.service
7      mask: true
8    - name: kubelet.path
9      enable: true
10      contents: |
11        [Unit]
12        Description=Watch for kubeconfig
13        [Path]
14        PathExists=/etc/kubernetes/kubeconfig
15        [Install]
16        WantedBy=multi-user.target
17    - name: wait-for-dns.service
18      enable: true
19      contents: |
20        [Unit]
21        Description=Wait for DNS entries
22        Wants=systemd-resolved.service
23        Before=kubelet.service
24        [Service]
25        Type=oneshot
26        RemainAfterExit=true
27        ExecStart=/bin/sh -c 'while ! /usr/bin/grep '^[^#[:space:]]' /etc/resolv.conf > /dev/null; do sleep 1; done'
28        [Install]
29        RequiredBy=kubelet.service

In order to understand when and where Ignition is executed let’s take a look at high level Linux Boot Process:

BIOS: perform POST, load MBR
MBR: load GRUB Boot Loader
GRUB2: load kernel image and initramfs ← Ignition here
KERNEL: load modules and start system 1st process
SYSTEMD: read from /etc/systemd/

Matchbox

Matchbox came from CoreOS and actually it’s just a web and gRPC server which takes care about mapping baremetal machines based on labels like MAC address to PXE boot profiles.

Matchbox is configured by files and directory structure, by default it uses /var/lib/matchbox. It can be also configured using Terrform module.

Directory consists of:

assets — just static files
groups — mapping between baremetal machines and boot profiles
profiles — kernel/initrd images
ignition — OS configuration like: systemd services, network configuration

It can be run as a container:

1sudo docker run --rm quay.io/coreos/matchbox:latest \
2 -p 8080:8080 \
3 -v /var/lib/matchbox:/var/lib/matchbox:Z \
4 -address=0.0.0.0:8080 \
5 -log-level=debug

For more examples please take a look at coreos/matchbox/examples.

Groups

Groups are selector for phycical machines based on label e.g. MAC address with metadata and corresponding profile.

1{
2  "id": "node1",
3  "name": "Worker Node",
4  "profile": "worker",
5  "selector": {
6    "mac": "52:54:00:b2:2f:86"
7  },
8  "metadata": {
9    "domain_name": "node1.example.com",
10    "k8s_dns_service_ip": "10.3.0.10",
11    "pxe": "true",
12    "ssh_authorized_keys": [
13      "ssh-rsa XXXXXXXXXXXX fake-test-key-REMOVE-ME"
14    ]
15  }
16}

Profiles

Profiles specify the kernel and initrd, kernel arguments, iPXE config, GRUB config or other configuration values a given machine should use.

1{
2  "id": "worker",
3  "name": "Worker",
4  "boot": {
5    "kernel": "/assets/coreos/1465.8.0/coreos_production_pxe.vmlinuz",
6    "initrd": ["/assets/coreos/1465.8.0/coreos_production_pxe_image.cpio.gz"],
7    "args": [
8      "initrd=coreos_production_pxe_image.cpio.gz",
9      "root=/dev/sda1",
10      "coreos.config.url=http://matchbox.example.com:8080/ignition?uuid=${uuid}&mac=${mac:hexhyp}",
11      "coreos.first_boot=yes",
12      "console=tty0",
13      "console=ttyS0",
14      "coreos.autologin"
15    ]
16  },
17  "ignition_id": "worker.yaml"
18}

Assets

This directory consists of static assets, like Container Linux image.

1$ tree /var/lib/matchbox/assets
2/var/lib/matchbox/assets/
3├── coreos
4│   └── 1465.8.0
5│       ├── CoreOS_Image_Signing_Key.asc
6│       ├── coreos_production_image.bin.bz2
7│       ├── coreos_production_image.bin.bz2.sig
8│       ├── coreos_production_pxe_image.cpio.gz
9│       ├── coreos_production_pxe_image.cpio.gz.sig
10│       ├── coreos_production_pxe.vmlinuz
11│       └── coreos_production_pxe.vmlinuz.sig

Ignition

This directory consists of ignition templates for profiles.

1---
2systemd:
3  units:
4    - name: docker.service
5      enable: true
6    - name: locksmithd.service
7      mask: true
8    - name: kubelet.path
9      enable: true
10      contents: |
11        [Unit]
12        Description=Watch for kubeconfig
13        [Path]
14        PathExists=/etc/kubernetes/kubeconfig
15        [Install]
16        WantedBy=multi-user.target
17    - name: wait-for-dns.service
18      enable: true
19      contents: |
20        [Unit]
21        Description=Wait for DNS entries
22        Wants=systemd-resolved.service
23        Before=kubelet.service
24        [Service]
25        Type=oneshot
26        RemainAfterExit=true
27        ExecStart=/bin/sh -c 'while ! /usr/bin/grep '^[^#[:space:]]' /etc/resolv.conf > /dev/null; do sleep 1; done'
28        [Install]
29        RequiredBy=kubelet.service

Kubernetes node boot process

After we covered all the building blocks needed for Kubernetes node boot process, let’s summarize it briefly:

Power on baremetal machine.
Receive network configuration from DHCP.
PXE network boot.
First OS boot with configuration via Ignition.
Provide additional assets (TLS certificates, kubernetes manifests).
Systemd kubelet.service starts and loads kubernetes manifests.

kubelet.service

Kubelet is configured as systemd service and runs on every node.

1- name: kubelet.service
2  command: start
3  runtime: true
4  content: |
5    [Unit]
6    Description=Kubelet via Hyperkube ACI
7
8    [Service]
9Environment=KUBELET_IMAGE=quay.io/coreos/hyperkube:v1.9.2_coreos.0
10    ExecStart=/usr/lib/coreos/kubelet-wrapper \
11      --kubeconfig=/etc/kubernetes/kubelet-kubeconfig.yaml \
12      --require-kubeconfig \
13      --cni-conf-dir=/etc/kubernetes/cni/net.d \
14      --network-plugin=cni \
15      --lock-file=/var/run/lock/kubelet.lock \
16      --exit-on-lock-contention \
17      --pod-manifest-path=/etc/kubernetes/manifests \
18      --allow-privileged \
19      --node-labels="node-role.kubernetes.io/node",type=worker,cluster=baremetal \
20      --cni-bin-dir=/var/lib/cni/bin \
21      --minimum-container-ttl-duration=6m0s \
22      --cluster_dns=10.5.0.10 \
23      --cluster-domain=cluster.local \
24      --client-ca-file=/etc/kubernetes/ssl/ca.pem \
25      --anonymous-auth=false \
26      --register-node=true
27
28    ExecStop=-/usr/bin/rkt stop --uuid-file=/var/cache/kubelet-pod.uuid
29
30    Restart=always
31    RestartSec=10
32
33    [Install]
34    WantedBy=multi-user.target

It loads all Kubernetes manifests from directory specified by pod-manifest-path (kube-apiserver, kube-proxy, kube-dns, etc).

On the other hand kubelet is responsible for configuring CNI network based on cni-conf-dir parameter.

Example CNI configuration (for Calico):

1{
2    "name": "k8s-pod-network",
3    "type": "calico",
4    "etcd_endpoints": "__ETCD_ENDPOINTS__",
5    "etcd_key_file": "__ETCD_KEY_FILE__",
6    "etcd_cert_file": "__ETCD_CERT_FILE__",
7    "etcd_ca_cert_file": "__ETCD_CA_CERT_FILE__",
8    "log_level": "__LOG_LEVEL__",
9    "ipam": {
10        "type": "calico-ipam"
11    },
12    "policy": {
13        "type": "k8s",
14        "k8s_api_root": "https://__KUBERNETES_SERVICE_HOST__:__KUBERNETES_SERVICE_PORT__",
15        "k8s_auth_token": "__SERVICEACCOUNT_TOKEN__"
16    },
17    "kubernetes": {
18        "kubeconfig": "__KUBECONFIG_FILEPATH__"
19    }
20}

kube-proxy

Once kubelet.service is up and running, it applies kube-proxy manifest:

1- path: /etc/kubernetes/manifests/kube-proxy.yaml
2  content: |
3    apiVersion: v1
4    kind: Pod
5    metadata:
6      name: kube-proxy
7      namespace: kube-system
8      labels:
9        k8s-app: kube-proxy
10    spec:
11      containers:
12      - name: kube-proxy
13        image: quay.io/coreos/hyperkube:v1.9.2_coreos.0
14        command:
15        - ./hyperkube
16        - proxy
17        - --kubeconfig=/etc/kubernetes/kubelet-kubeconfig.yaml
18        - --proxy-mode=iptables
19        - --cluster-cidr=10.123.0.0/16
20        securityContext:
21          privileged: true
22        volumeMounts:
23        - mountPath: /etc/ssl/certs
24          name: ssl-certs-host
25          readOnly: true
26        - name: etc-kubernetes
27          mountPath: /etc/kubernetes
28          readOnly: true
29      hostNetwork: true
30      volumes:
31      - hostPath:
32          path: /usr/share/ca-certificates
33        name: ssl-certs-host
34      - name: etc-kubernetes
35        hostPath:
36          path: /etc/kubernetes

We’ve seen before that kube-proxy is responsible for TCP and UDP forwarding using iptables or ipvs rules.

That’s pretty much all what is needed to bootstrap minimal Kubernetes Cluster. We can dig deeper into kube-apiserver, kube-dns, scheduler or even more configuration files but this is not the scope of this blog post :)

Kubernetes Cluster Bootstrap

Introduction

Kubernetes components

Bootstrap on baremetal

PXE network boot

Dnsmasq

CoreOS and Ignition

Matchbox

Groups

Profiles

Assets

Ignition

Kubernetes node boot process

kubelet.service

kube-proxy

Bootstrap on cloud

Summary

Summary

Subscribe to our newsletter and never miss an article

Explore more topics

Cloud transformation with adoption of inner sourcing at scale

Internal developer portals: a springboard to platform engineering

Reference Architecture: A roadmap to efficiency and scalability