Keep your CI/CD infrastructure alive with Jenkins Operator

Picture of Piotr Ryba,  Lead Software Engineer

Piotr Ryba

Lead Software Engineer

7 minutes read

This short article is directly inspired by Maria Sabastian’s text “How a Jenkins Job Broke our Jenkins UI” and published recently on the Slack Engineering blog. I strongly encourage reading it for the context.

One of the key takeaways from Maria’s article is to mirror Jenkins production to test changes (i.e. Jenkins upgrades, plugin upgrades, scripts) before routing real traffic to it. Doing this can help you avoid unexpected downtime in one of the crucial services in an IT organization – CI/CD. I aim to show you how Jenkins Kubernetes Operator enables you to do that and how it opens doors for doing canary or blue/green deployment.

The Operator

For those unfamiliar with the concept of the Kubernetes operator pattern, it extends the Kubernetes API by adding Custom Resources that can be used to manage applications. To quote Kubernetes documentation:

The Operator pattern aims to capture the key aim of a human operator who is managing a service or set of services. Human operators who look after specific applications and services have deep knowledge of how the system ought to behave, how to deploy it, and how to react if there are problems.

Our team created and currently maintains an open-source Jenkins Kubernetes Operator, which recently joined the official Jenkins Organization. Among other plans, there’s an ongoing effort to release a new and improved API schema. You can find the project on GitHub:

This project aims to provide an easy way to deploy Jenkins on the Kubernetes cluster, preconfigured, integrated into Kubernetes with recommended security hardening applied to it. In addition, the operator also enforces good practices which helps you achieve scalability and consistency of Jenkins deployment.

Besides that, we are also building commercial offerings around that. Operator Service for Jenkins helps run scalable, immutable, and secure Jenkins in your organization in a managed fashion.

Ephemeral Jenkins

The essential design choice was to assume the ephemerality of the Jenkins state. You can use this characteristic to quickly test configuration changes while being able to easily roll back to the persisted state (just kill the pod and watch it being automatically recreated). Of course, everything that is configured inside the running pod will be lost after recreation. The recommended way to persist the configuration is to use configuration files:

  • Configuration as Code declarative YAML;
  • job-dsl for job and pipeline definitions;
  • Groovy scripts for the most advanced topics (when the above are not enough).

The last missing part would be the job history, backed up and restored when provisioning a pod.

We strongly advocate the GitOps model, and we believe that storing all of the configuration files for deployment in a repository gives a lot of benefits, e.g. the ease of collaboration, versioning, branching, and redundancy, to name a few.

Helm Chart

For brevity, we’ll focus on using Helm charts to deploy Jenkins. The official Helm chart is highly configurable. You can specify, among other things, the Jenkins container image, plugins to install, configuration as code YAML file, and backups. In case you need more control, you can use Kubernetes manifest files to define Custom Resources.

Following the guide:

1. helm repo add jenkins 2. kubectl create namespace jenkins-blue 3. helm install jenkins-blue jenkins/jenkins-operator -n jenkins-blue --set jenkins.namespace=jenkins-blue

The above commands will add the Helm chart repo, create a new namespace, and deploy the Jenkins instance using the default configuration. After a few seconds, you should see:

NAME: Jenkins-blue LAST DEPLOYED: Sun Jun  6 13:40:26 2021 NAMESPACE: Jenkins-blue STATUS: deployed REVISION: 1 TEST SUITE: None NOTES:

  • A. Watch Jenkins instance being created:

$ kubectl --namespace jenkins-blue get pods -w

  • B. Get Jenkins credentials:

$ kubectl --namespace jenkins-blue get secret jenkins-operator-credentials-jenkins -o 'jsonpath={.data.user}' | base64 -d $ kubectl --namespace jenkins-blue get secret jenkins-operator-credentials-jenkins -o 'jsonpath={.data.password}' | base64 -d

  • C. Connect to Jenkins (actual Kubernetes cluster):

$ kubectl --namespace jenkins-blue port-forward jenkins-jenkins 8080:8080

And when running the first command:

kubectl --namespace jenkins-blue get pods -w 

jenkins-blue-jenkins-operator-f9787f6c9-62twj     0/1  ContainerCreating 05s
jenkins-blue-jenkins-operator-f9787f6c9-62twj      1/1  Running 06s
jenkins-jenkins         0/2Pending 00s
jenkins-jenkins         0/2Pending 00s
jenkins-jenkins         0/2ContainerCreating 00s
jenkins-jenkins         1/2 Running 019s
jenkins-jenkins         2/2 Running 055s


If your Jenkins doesn’t start, check the pod logs kubectl logs -n jenkins-blue jenkins-jenkins jenkins-master, you might see there something like this: Failed to load: Credentials Plugin (2.5) - Update required: Configuration as Code Plugin (1.46) to be updated to 1.51 or higher        at hudson.PluginWrapper.resolvePluginDependencies(        at hudson.PluginManager$2$1$

This means that one of your plugin’s dependencies loaded a newer version of a plugin that requires an upgrade in another plugin. To fix that, update plugin versions under basePlugins and plugins (values.yaml for reference). To avoid these kinds of issues you have to pin the exact plugin versions for all of your plugins and their dependencies. After doing this you will be able to consistently restore the Jenkins state.


We have Blue deployed, what about Green?

First, let’s pick a scenario: we would like to test a newer version of Jenkins. The easiest way to deploy that would be to run:

  1. kubectl create namespace jenkins-green (if it doesn’t already exist)
  2. helm install jenkins-green jenkins/jenkins-operator -n jenkins-green --set jenkins.namespace=jenkins-green,jenkins.image=jenkins/jenkins:2.289.1-lts-alpine

And after a few seconds, it should start successfully:

kubectl --namespace jenkins-green get pods -w


We were able to quickly run two separate Jenkins instances in different namespaces. At this point, you can connect to the green instance, check if everything works and decide if you want to switch it to production. After the switch, you may keep the blue deployment running for some time in case something bad happens to be able to easily switch back to it.

Real-world scenario

In a real-world scenario, I’d advise you to use a declarative configuration where possible and store it in a Git repository and then utilize the git branching model. For example, use develop for development and then merge into blue and green branches to pin the configuration revisions used for both deployments.

When using the blue/green deployment, some thought may be required to make sure the two Jenkins controller instances that may be running simultaneously are not interfering with one another.

Doing canary releases may require more work to ensure the whole team and their jobs get executed on the same instance.



Jenkins Operator makes running Jenkins on Kubernetes a breeze. There will be a new API schema released soon. It splits the current Jenkins Custom Resource into several CRs which will positively affect the future extensibility of the project.

Don’t hesitate to engage in a conversation in our community! As with everything, start small, test it out, if it works for you — great, if not, let us know what we should improve.

Liked the article?

Share it with others!

explore more on

Take the first step to a sustained competitive edge for your business

Let's connect

VirtusLab's work has met the mark several times over, and their latest project is no exception. The team is efficient, hard-working, and trustworthy. Customers can expect a proactive team that drives results.

Stephen Rooke
Stephen RookeDirector of Software Development @ Extreme Reach

VirtusLab's engineers are truly Strapi extensions experts. Their knowledge and expertise in the area of Strapi plugins gave us the opportunity to lift our multi-brand CMS implementation to a different level.

facile logo
Leonardo PoddaEngineering Manager @

VirtusLab has been an incredible partner since the early development of Scala 3, essential to a mature and stable Scala 3 ecosystem.

Martin OderskyHead of Programming Research Group @ EPFL

VirtusLab's strength is its knowledge of the latest trends and technologies for creating UIs and its ability to design complex applications. The VirtusLab team's in-depth knowledge, understanding, and experience of MIS systems have been invaluable to us in developing our product. The team is professional and delivers on time – we greatly appreciated this efficiency when working with them.

Michael GrantDirector of Development @ Cyber Sec Company