Manage Jenkins agents with Kubernetes native PodTemplate

Published on 2022-08-05 by 赵晓杰(Rick)

Jenkins has great integration with Kubernetes. No matter the controller or agents, they all work well in Kubernetes Pods. Configuration as Code is an awesome idea. People don’t need to open the UI page of Jenkins again and again. We can put all the system configuration into a git repository. You might already feel tired to edit a super huge Kubernetes YAML file if there are many many Jenkins agents that need to be maintained. It’s pretty likely to make a mistake in a big YAML file.

I’m going to provide a solution to make our life easier maintaining Jenkins . You only need to update a ConfigMap if you want to make any changes to the Jenkins system configuration. And you only need to create a Kubernetes native PodTemplate resource if you want to add a new Jenkins agent. To achieve this goal, you only need to install an extra Kubernetes deployment.

Please allow me to show you how to do it. Before going forward, let’s assume you already have a Kubernetes cluster, and you have enough permission to access the relevant resources.

Firstly, put all the Jenkins system config into a ConfigMap, such as:

apiVersion: v1
data:
  jenkins_user.yaml: |
    jenkins:
      mode: EXCLUSIVE
      numExecutors: 0
      scmCheckoutRetryCount: 2
      disableRememberMe: true
      clouds:
        - kubernetes:
            name: "kubernetes"
            serverUrl: "https://kubernetes.default"
            skipTlsVerify: true
kind: ConfigMap
metadata:
  name: jenkins-casc-config
  namespace: kubesphere-devops-system

Secondly, let your Jenkins load its configuration from a ConfigMap. Please see also the following important parts:

spec:
  template:
    spec:
      containers:
      - image: ghcr.io/linuxsuren/jenkins:lts
        env:
        - name: CASC_JENKINS_CONFIG
          value: "/var/jenkins_home/casc_configs/"          # loading config file from a directory that was mount from a ConfigMap
        volumeMounts:
        - mountPath: /var/jenkins_home/casc_configs
          name: casc-config                                 # mount from a volume
      volumes:
      - configMap:
          defaultMode: 420
          name: jenkins-casc-config                         # clamin a ConfigMap volume, all the CasC YAML content will be here
        name: casc-config

then, the core part is the Kubernetes controller. Please see the following code snippet:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: jenkins-agent
  namespace: kubesphere-devops-system
spec:
  template:
    spec:
      containers:
      - image: kubespheredev/devops-controller:dev-v3.2.1-rc.3-6726130
        name: controller
        args:
        - --enabled-controllers
        - all=false,jenkinsagent=true,jenkinsconfig=true        # only enable the necessary features of this controller

This controller will watch all the PodTemplates (containing the label jenkins.agent.pod), then convert it to Jenkins-style PodTemplate, merge it into the CasC YAML file finally.

When everything is setup, you can try to add a Kubernetes native PodTemplate. A few seconds later, you can use it in your Pipeline. Considering there are many potential technical details, I already put all the necessary files to this git repository. Please feel free to mainain this project if it is helpful in your work.

About the Author
赵晓杰(Rick)

Rick is a big fan of Jenkins, also as a contributor leading the Jenkins China community.