云原生之旅 – 11)基于 Kubernetes 动态伸缩 Jenkins Build Agents

前言

上一篇文章云原生之旅 – 10)手把手教你安装 Jenkins on Kubernetes我们介绍了在 Kubernetes 上安装 Jenkins,本文介绍下如何设置k8s pod作为Jenkins 构建job的 agent。

Jenkins master 和 agent 均以 pod 的形式运行在Kubernetes 节点上。Master 运行在其中一个节点上,其配置数据 Jenkins home 使用存储卷挂载,master pod重启不会导致数据丢失。agent 运行在各个节点上,根据需求动态创建并自动释放。这样做的好处很多,比如高可用,高伸缩性,资源利用率高。

关键词:Jenkins on Kubernetes 实践,Jenkins 和 Kubernetes,在Kubernetes上安装Jenkins,Jenkins 高可用安装,Jenkins 动态伸缩构建, Kubernetes Pod as Jenkins build agent

准备

  1. 已搭建 Jenkins master on kubernetes云原生之旅 – 10)手把手教你安装 Jenkins on Kubernetes
  2. 准备一个 Service Account,对目标 cluster 具有k8s admin权限,以便部署。
  3. 防火墙已开通 Jenkins 出站到Docker hub,方便 push/pull image
  4. 防火墙已开通 Jenkins 到 目标 cluster,以便部署。

插件安装

  • Kubernetes Plugin
  • Google Kubernetes Engine Plugin (我的例子是部署到 GKE cluster)

Jenkins 配置

Manage Nodes and Clouds

1. Go to `Manage Jenkins` –> `Manage Nodes and Clouds`
2.Click `Configure Clouds`
3.Add a new Cloud select `Kubernetes`
4. Click `Kubernetes Cloud Detail
5. Enter `jenkins` namespace in `Kubernetes Namespace` field
6. Click `Test Connection` –> result show `Connected to Kubernetes v1.22.12-gke.2300`
7. Click `Save`
8. Enter `http://jenkins-service.jenkins.svc.cluster.local:8080` in `Jenkins URL` field
9. Enter `jenkins-agent:50000` in `Jenkins tunnel` field

云原生之旅 - 11)基于 Kubernetes 动态伸缩 Jenkins Build Agents

10. Click `Add Pod Template` then `Pod Template Details`

11. Input `Name`=`jenkins-agent`, `Namespace`=`jenkins`, `Labels`=`kubeagent`

云原生之旅 - 11)基于 Kubernetes 动态伸缩 Jenkins Build Agents

12. (Optional) 如果不添加 container template, the Jenkins Kubernetes plugin will use the default JNLP image from the Docker hub to spin up the agents.
如果你要覆盖默认的jnlp image 可以 Click `Add Container` to add Container Template,
输入 `Name`=`jnlp`, `Docker Image`=`your_registry/jenkins/inbound-agent:4.11-1-jdk11`

云原生之旅 - 11)基于 Kubernetes 动态伸缩 Jenkins Build Agents

Ensure that you remove the sleep and 9999999 default argument from the container template.

Manage Credentials

  • Add `Usernames with password` for docker hub account/pwd,比如 wade_test_dockerhub
  • Add `Google Service Account from private key` 比如gcp_sa_json_key

Credentials 会在Jenkinsfile里面用到。

### 本文首发于博客园https://www.cnblogs.com/wade-xu/p/16863955.html

Test a freestyle project

Go to Jenkins home –> New Item and create a freestyle project,命名为 quick-test
在 job description 部分, add the label `kubeagent` for `Restrict where this project can be run`.

云原生之旅 - 11)基于 Kubernetes 动态伸缩 Jenkins Build Agents

这个label 和我们上面创建 pod template时用的label一致. 这样的话 Jenkins就知道用哪个pod template 作为agent container.

随便添加一个shell 作为build steps

云原生之旅 - 11)基于 Kubernetes 动态伸缩 Jenkins Build Agents

点Build Now

查看Console Output
Agent jenkins-agent-l7hw9 is provisioned from template jenkins-agent
......
Building remotely on jenkins-agent-l7hw9 (kubeagent) in workspace /home/jenkins/agent/workspace/quick-test
[quick-test] $ /bin/sh -xe /tmp/jenkins17573873264046707236.sh
+ echo test pipeline
test pipeline
Finished: SUCCESS

### 本文首发于博客园https://www.cnblogs.com/wade-xu/p/16863955.html

Jenkinsfile

CI

接着我们用Jenkinsfile 写一个Declarative pipeline -build/push docker image 到docker hub
首先需要定义一个 pod.yaml 作为启动 agent 的container
云原生之旅 - 11)基于 Kubernetes 动态伸缩 Jenkins Build Agents云原生之旅 - 11)基于 Kubernetes 动态伸缩 Jenkins Build Agents

kind: Pod
spec:
  containers:  # list of containers that you want present for your build, you can define a default container in the Jenkinsfile
    - name: maven
      image: maven:3.5.4-jdk-8-slim
      command: ["tail", "-f", "/dev/null"]  # this or any command that is bascially a noop is required, this is so that you don't overwrite the entrypoint of the base container
      imagePullPolicy: Always # use cache or pull image for agent
      resources:  # request and limit the resources your build contaienr
        requests:
          memory: 4Gi
          cpu: 2
        limits:
          memory: 4Gi
          cpu: 2
      volumeMounts:
        - mountPath: /root/.m2 # maven .m2 cache directory
          name: maven-home
    - name: git
      image: bitnami/git:2.38.1
      imagePullPolicy: IfNotPresent
      command: ["tail", "-f", "/dev/null"]
      resources: # limit the resources your build contaienr
        limits:
          cpu: 100m
          memory: 256Mi
    - name: kubectl-kustomize
      image: line/kubectl-kustomize:1.25.3-4.5.7
      imagePullPolicy: IfNotPresent
      command: ["tail", "-f", "/dev/null"]
      resources: # limit the resources your build contaienr
        limits:
          cpu: 100m
          memory: 256Mi
    - name: docker
      image: docker:18.06.1
      command: ["tail", "-f", "/dev/null"]
      imagePullPolicy: Always
      volumeMounts:
        - name: docker
          mountPath: /var/run/docker.sock # We use the k8s host docker engine
  volumes:
    - name: docker
      hostPath:
        path: /var/run/docker.sock
    - name: maven-home
      persistentVolumeClaim:
        claimName: maven-repo-storage

build-pod.yaml

在Jenkinsfile里面定义agent 使用这个yaml file

  agent {
    kubernetes {
      idleMinutes 3  // how long the pod will live after no jobs have run on it
      yamlFile './build-pod.yaml'  // path to the pod definition relative to the root of our project 
      defaultContainer 'docker'  // define a default container if more than a few stages use it, otherwise default to jnlp container
    }

下面步骤是 docker login/build/tag/push

environment {
DOCKER_HUB_REGISTRY=’https://index.docker.io/v1/’
DOCKER_HUB_CREDS = credentials(‘wade_test_dockerhub’)
}
stage('Build and Push Docker Image') {
      steps {
        script {
          dir(dir_path) {
            container('docker') {
                // docker login, Using single-quotes instead of double-quotes when referencing these sensitive environment variables prevents this type of leaking.
                sh 'echo $DOCKER_HUB_CREDS_PSW | docker login -u $DOCKER_HUB_CREDS_USR --password-stdin $DOCKER_HUB_REGISTRY'
                // build image with git tag
                sh """
                docker build -t $PROJECT_IMAGE_WITH_TAG .
                docker tag $PROJECT_IMAGE_WITH_TAG $DOCKER_HUB_CREDS_USR/$PROJECT_IMAGE_WITH_TAG
                """
                // push image_tag to docker hub
                sh """
                docker push $DOCKER_HUB_CREDS_USR/$PROJECT_IMAGE_WITH_TAG
                """
            }
          }
        }
      }
    }

我这里没有选择用docker.withRegistry

docker.withRegistry("$DOCKER_HUB_REGISTRY", "$DOCKER_HUB_CREDENTIAL") {}

因为会有不安全的log提示

WARNING! Using --password via the CLI is insecure. Use --password-stdin.

CI + Kustomize + CD

这个例子是上面的 CI 之后 加上 – 利用 Kustomize build K8S resource manifests 然后 CD 到一个 Cluster

Kustomize 可以参考云原生之旅 – 6)不能错过的一款 Kubernetes 应用编排管理神器 Kustomize

云原生之旅 - 11)基于 Kubernetes 动态伸缩 Jenkins Build Agents云原生之旅 - 11)基于 Kubernetes 动态伸缩 Jenkins Build Agents

    // assume your k8s manifests in another repo, mine is same repo, just in order to show git clone step
    stage('Checkout K8S manifests') {
      steps {
        script {
          dir(dir_path) {
            container('git') {
              if (! fileExists('learning_by_doing/README.md')) {
                sh """
                git clone https://github.com/wadexu007/learning_by_doing.git
                ls -lhrt
                """
              } else {
                  sh 'echo manifes repo already exist.'
              }
            }
          }
        }
      }
    }
    stage('Build manifests with Kustomize') {
      steps {
        script {
          dir(dir_path) {
            container('kubectl-kustomize') {
                sh """
                cd learning_by_doing/Kustomize/demo-manifests/services/demo-app/dev/
                kustomize edit set image $DOCKER_HUB_CREDS_USR/$PROJECT_IMAGE_WITH_TAG
                kustomize build > $WORKSPACE/$dir_path/deployment.yaml
                """
            }
          }
        }
      }
    }
    stage('Deploy to GKE test cluster') {
			environment{
				PROJECT_ID = 'xperiences-eng-cn-dev'
        CLUSTER_NAME = 'xpe-spark-test-gke'
        REGION = 'asia-east2'
        CREDENTIALS_ID = 'gcp_sa_json_key'
      }
      steps {
        script {
          dir(dir_path) {
            container('kubectl-kustomize') {
                sh """
                chown 1000:1000 deployment.yaml
                echo start to deploy to cluster $CLUSTER_NAME
                """
                step([
                  $class: 'KubernetesEngineBuilder',
                  projectId: env.PROJECT_ID,
                  clusterName: env.CLUSTER_NAME,
                  location: env.REGION,
                  manifestPattern: 'deployment.yaml',
                  credentialsId: env.CREDENTIALS_ID,
                  verifyDeployments: false])
                  // verifyDeployments does not work for non-default namespace
            }
          }
        }
      }
    }

View Code

Pipeline: Input Step

这个例子是利用 Jenkins pipeline的 Input step 来做一个人工介入Approve的步骤。 然后再来一个多cluster 部署,选不同region 部署到不同的cluster的示例。
    stage('Wait for SRE Approval') {
      steps {
        timeout(time:72, unit:'HOURS') {
          input message: "Approved Prod deployment?", submitter: 'sre-team'
        }
      }
    }
    // deployment to multipe k8s clusters
    stage('Deploy to GKE Prod cluster') {
			environment{
				PROJECT_ID = 'sre-cn-dev'
        CREDENTIALS_ID = 'gcp_sa_json_key'
        CLUSTER_COMMON_NAME = 'demo-gke-prod'
      }
      steps {
        script {
          env.REGION = input message: 'Choose which region you want to deploy?',
                             parameters: [choice(name: 'Region',
                                                description: 'Select Region to Deloy',
                                                choices: ['europe-west1', 'us-central1'])
                                          ]
          dir(dir_path) {
            if ( env.REGION == "europe-west1" ) {
              def eu_cluster_name = env.CLUSTER_COMMON_NAME + "-eu"
              container('kubectl-kustomize') {
                  sh "echo deploy to cluster $eu_cluster_name in region: $REGION"
              }
            }
            if ( env.REGION == "us-central1" ) {
              def us_cluster_name = env.CLUSTER_COMMON_NAME + "-us"
              container('kubectl-kustomize') {
                  sh "echo deploy to cluster $us_cluster_name in region: $REGION"
              }
            }
          }
        }
      }
    }

所有例子均在我的 github repo。

### 本文首发于博客园https://www.cnblogs.com/wade-xu/p/16863955.html

测试

现在你可以创建一个 Pipeline 或者 Multibranch Pipeline job 来测试。
Repository URL = `https://github.com/wadexu007/learning_by_doing`
Script Path, e.g. `Jenkins/k8s_pod_as_build_agent/demo-app-java/Jenkinsfile`
你会看到每启动一个job 都会相应的产生一个pod 来作为Jenkins agent运行,结束后根据idleMinutes自动释放。

云原生之旅 - 11)基于 Kubernetes 动态伸缩 Jenkins Build Agents

总结

如果你已经成功创建并测试 CI/CD pipeline,可以继续加强,比如加上 Post notifications

最佳实践

  • 设置resource requests and limits on each container in your Pod
  • 如果使用maven 构建 java项目,.m2 cache目录需要 mount 出来,这样加快后面的maven build速度。
  • 使用Jenkins Shared Libraries抽取Pipeline的共用代码
  • 在容器里构建容器化应用(Run docker in docker) 我的例子是通过 mountdocker.sock 利用k8s 主机 docker engine来实现的,这种方式需要privileges mode 不安全,推荐使用Kaniko,下一篇文章会介绍。
感谢阅读,如果您觉得本文的内容对您的学习有所帮助,您可以打赏和推荐,您的鼓励是我创作的动力。
© 版权声明
好牛新坐标
版权声明:
1、IT大王遵守相关法律法规,由于本站资源全部来源于网络程序/投稿,故资源量太大无法一一准确核实资源侵权的真实性;
2、出于传递信息之目的,故IT大王可能会误刊发损害或影响您的合法权益,请您积极与我们联系处理(所有内容不代表本站观点与立场);
3、因时间、精力有限,我们无法一一核实每一条消息的真实性,但我们会在发布之前尽最大努力来核实这些信息;
4、无论出于何种目的要求本站删除内容,您均需要提供根据国家版权局发布的示范格式
《要求删除或断开链接侵权网络内容的通知》:https://itdw.cn/ziliao/sfgs.pdf,
国家知识产权局《要求删除或断开链接侵权网络内容的通知》填写说明: http://www.ncac.gov.cn/chinacopyright/contents/12227/342400.shtml
未按照国家知识产权局格式通知一律不予处理;请按照此通知格式填写发至本站的邮箱 wl6@163.com

相关文章