Keep your CI/CD infrastructure alive with Jenkins Operator

This short article is directly inspired by Maria Sabastian’s text “How a Jenkins Job Broke our Jenkins UI” and published recently on the Slack Engineering blog. I strongly encourage reading it for the context.

One of the key takeaways from Maria’s article is to mirror Jenkins production to test changes (i.e. Jenkins upgrades, plugin upgrades, scripts) before routing real traffic to it. Doing this can help you avoid unexpected downtime in one of the crucial services in an IT organization – CI/CD. I aim to show you how Jenkins Kubernetes Operator enables you to do that and how it opens doors for doing canary or blue/green deployment.

The Operator

For those unfamiliar with the concept of the Kubernetes operator pattern, it extends the Kubernetes API by adding Custom Resources that can be used to manage applications. To quote Kubernetes documentation:

Our team created and currently maintains an open-source Jenkins Kubernetes Operator, which recently joined the official Jenkins Organization. Among other plans, there’s an ongoing effort to release a new and improved API schema. You can find the project on GitHub: https://github.com/jenkinsci/kubernetes-operator

This project aims to provide an easy way to deploy Jenkins on the Kubernetes cluster, preconfigured, integrated into Kubernetes with recommended security hardening applied to it. In addition, the operator also enforces good practices which helps you achieve scalability and consistency of Jenkins deployment.

Besides that, we are also building commercial offerings around that. Operator Service for Jenkins helps run scalable, immutable, and secure Jenkins in your organization in a managed fashion.

Ephemeral Jenkins

The essential design choice was to assume the ephemerality of the Jenkins state. You can use this characteristic to quickly test configuration changes while being able to easily roll back to the persisted state (just kill the pod and watch it being automatically recreated). Of course, everything that is configured inside the running pod will be lost after recreation. The recommended way to persist the configuration is to use configuration files:

• Configuration as Code declarative YAML;
• job-dsl for job and pipeline definitions;
• Groovy scripts for the most advanced topics (when the above are not enough).

The last missing part would be the job history, backed up and restored when provisioning a pod.

We strongly advocate the GitOps model, and we believe that storing all of the configuration files for deployment in a repository gives a lot of benefits, e.g. the ease of collaboration, versioning, branching, and redundancy, to name a few.

Helm Chart

For brevity, we’ll focus on using Helm charts to deploy Jenkins. The official Helm chart is highly configurable. You can specify, among other things, the Jenkins container image, plugins to install, configuration as code YAML file, and backups. In case you need more control, you can use Kubernetes manifest files to define Custom Resources.

Following the guide:

1. helm repo add jenkins https://raw.githubusercontent.com/jenkinsci/kubernetes-operator/master/chart 2. kubectl create namespace jenkins-blue 3. helm install jenkins-blue jenkins/jenkins-operator -n jenkins-blue --set jenkins.namespace=jenkins-blue

The above commands will add the Helm chart repo, create a new namespace, and deploy the Jenkins instance using the default configuration. After a few seconds, you should see:

NAME: Jenkins-blue LAST DEPLOYED: Sun Jun  6 13:40:26 2021 NAMESPACE: Jenkins-blue STATUS: deployed REVISION: 1 TEST SUITE: None NOTES:

• A. Watch Jenkins instance being created:

$ kubectl --namespace jenkins-blue get pods -w

• B. Get Jenkins credentials:

$ kubectl --namespace jenkins-blue get secret jenkins-operator-credentials-jenkins -o 'jsonpath={.data.user}' | base64 -d $ kubectl --namespace jenkins-blue get secret jenkins-operator-credentials-jenkins -o 'jsonpath={.data.password}' | base64 -d

• C. Connect to Jenkins (actual Kubernetes cluster):

$ kubectl --namespace jenkins-blue port-forward jenkins-jenkins 8080:8080

And when running the first command:

kubectl --namespace jenkins-blue get pods -w 

Troubleshooting

If your Jenkins doesn’t start, check the pod logs kubectl logs -n jenkins-blue jenkins-jenkins jenkins-master, you might see there something like this:

java.io.IOException: Failed to load: Credentials Plugin (2.5) - Update required: Configuration as Code Plugin (1.46) to be updated to 1.51 or higher        at hudson.PluginWrapper.resolvePluginDependencies(PluginWrapper.java:952)        at hudson.PluginManager$2$1$1.run(PluginManager.java:549)

This means that one of your plugin’s dependencies loaded a newer version of a plugin that requires an upgrade in another plugin. To fix that, update plugin versions under basePlugins and plugins (values.yaml for reference). To avoid these kinds of issues you have to pin the exact plugin versions for all of your plugins and their dependencies. After doing this you will be able to consistently restore the Jenkins state.

We have Blue deployed, what about Green?

First, let’s pick a scenario: we would like to test a newer version of Jenkins. The easiest way to deploy that would be to run:

1. kubectl create namespace jenkins-green (if it doesn’t already exist)
2. helm install jenkins-green jenkins/jenkins-operator -n jenkins-green --set jenkins.namespace=jenkins-green,jenkins.image=jenkins/jenkins:2.289.1-lts-alpine

And after a few seconds, it should start successfully:

kubectl --namespace jenkins-green get pods -w

We were able to quickly run two separate Jenkins instances in different namespaces. At this point, you can connect to the green instance, check if everything works and decide if you want to switch it to production. After the switch, you may keep the blue deployment running for some time in case something bad happens to be able to easily switch back to it.

Real-world scenario

In a real-world scenario, I’d advise you to use a declarative configuration where possible and store it in a Git repository and then utilize the git branching model. For example, use develop for development and then merge into blue and green branches to pin the configuration revisions used for both deployments.

When using the blue/green deployment, some thought may be required to make sure the two Jenkins controller instances that may be running simultaneously are not interfering with one another.

Doing canary releases may require more work to ensure the whole team and their jobs get executed on the same instance.

Summary

Jenkins Operator makes running Jenkins on Kubernetes a breeze. There will be a new API schema released soon. It splits the current Jenkins Custom Resource into several CRs which will positively affect the future extensibility of the project.

Don’t hesitate to engage in a conversation in our community! As with everything, start small, test it out, if it works for you — great, if not, let us know what we should improve.