在ACS集群中使用GitLab Runner构建CI/CD环境

背景信息

GitLab Runner是一个开源的、用于执行GitLab CI/CD流水线作业的项目，作为一个可外置的任务执行框架，它提供了多样的执行器如Shell、Kubernetes等，让您可以在私有环境或云上执行具体的CI作业，并将结果反馈回GitLab中。

Gitlab Runner的核心配置分为两部分，一部分是Manager Pod的配置和初始化设置，另一部分是Kubernetes Executor的配置。更多信息，请参见配置极狐 GitLab Runner。

实践路径

本文的全部流程如下：

前提条件

已使用kubectl连接Kubernetes集群。具体操作，请参见获取集群KubeConfig并通过kubectl工具连接集群。

安装步骤

本文将基于17.3.1版本的gitlab-runner进行，对应Chart版本为0.68.1。关于常规gitlab-runner安装步骤，请参见极狐GitLab Runner Helm Chart。

1. 获取GitLab Chart包。

   1. 添加GitLab Helm仓库。

      helm repo add gitlab https://charts.gitlab.io

   2. 更新Chart。

      helm repo update gitlab

   3. 获取GitLab。

      helm pull gitlab/gitlab-runner --version 0.68.1 && tar zvxf gitlab-runner-0.68.1.tgz

2. 注册Runner。

   您需要在GitLab控制台对项目或群组生成Runner Token，并记录Token信息。

3. 初始化设置values.yaml。

   展开查看YAML内容

   ## GitLab Runner Image
   ##
   ## By default it's using registry.gitlab.com/gitlab-org/gitlab-runner:alpine-v{VERSION}
   ## where {VERSION} is taken from Chart.yaml from appVersion field
   ##
   ## ref: https://gitlab.com/gitlab-org/gitlab-runner/container_registry/29383?orderBy=NAME&sort=asc&search[]=alpine-v&search[]=
   ##
   ## Note: If you change the image to the ubuntu release
   ##       don't forget to change the securityContext;
   ##       these images run on different user IDs.
   ##
   ...
   
   ## The GitLab Server URL (with protocol) that want to register the runner against
   ## ref: https://docs.gitlab.com/runner/commands/index.html#gitlab-runner-register
   ##
   gitlabUrl: https://jihulab.com/
   
   ## DEPRECATED: The Registration Token for adding new Runners to the GitLab Server.
   ##
   ## ref: https://docs.gitlab.com/ee/ci/runners/new_creation_workflow.html
   ##
   # runnerRegistrationToken: ""
   
   ## The Runner Token for adding new Runners to the GitLab Server. This must
   ## be retrieved from your GitLab instance. It is the token of an already registered runner.
   ## ref: (we don't have docs for that yet, but we want to use an existing token)
   ##
   runnerToken: "glrt-t3_sz6xxxxxxxxxDsWF77"
   #
   
   ## Unregister all runners before termination
   ##
   ## Updating the runner's chart version or configuration will cause the runner container
   ## to be terminated and created again. This may cause your Gitlab instance to reference
   ## non-existant runners. Un-registering the runner before termination mitigates this issue.
   ## ref: https://docs.gitlab.com/runner/commands/index.html#gitlab-runner-unregister
   ##
   unregisterRunners: true
   
   ## When stopping the runner, give it time to wait for its jobs to terminate.
   ##
   ## Updating the runner's chart version or configuration will cause the runner container
   ## to be terminated with a graceful stop request. terminationGracePeriodSeconds
   ## instructs Kubernetes to wait long enough for the runner pod to terminate gracefully.
   ## ref: https://docs.gitlab.com/runner/commands/#signals
   terminationGracePeriodSeconds: 3600
   
   ## Set the certsSecretName in order to pass custom certficates for GitLab Runner to use.
   ## Provide resource name for a Kubernetes Secret Object in the same namespace,
   ## this is used to populate the /home/gitlab-runner/.gitlab-runner/certs/ directory
   ## ref: https://docs.gitlab.com/runner/configuration/tls-self-signed.html#supported-options-for-self-signed-certificates-targeting-the-gitlab-server
   ##
   # certsSecretName:
   
   ## Configure the maximum number of concurrent jobs
   ## ref: https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-global-section
   ##
   concurrent: 10
   
   ...
   
   ## For RBAC support:
   rbac:
     ## Specifies whether a Role and RoleBinding should be created
     ## If this value is set to `true`, `serviceAccount.create` should also be set to either `true` or `false`
     ##
     create: true
     ## Define the generated serviceAccountName when create is set to true
     ## It defaults to "gitlab-runner.fullname" if not provided
     ## DEPRECATED: Please use `serviceAccount.name` instead
     generatedServiceAccountName: ""
   ...
   
   ## Configuration for the Pods that the runner launches for each new job
   ##
   runners:
     # runner configuration, where the multi line string is evaluated as a
     # template so you can specify helm values inside of it.
     #
     # tpl: https://helm.sh/docs/howto/charts_tips_and_tricks/#using-the-tpl-function
     # runner configuration: https://docs.gitlab.com/runner/configuration/advanced-configuration.html
     config: |
       [[runners]]
         [runners.kubernetes]
           namespace = "{{.Release.Namespace}}"
           image = "alpine"
   
     ## Absolute path for an existing runner configuration file
     ## Can be used alongside "volumes" and "volumeMounts" to use an external config file
     ## Active if runners.config is empty or null
     configPath: ""
   ...

   配置项	配置描述
   gitlabUrl	注册Runner的极狐GitLab服务器的完整URL。例如，https://gitlab.example.com。
   runnerToken	上一步中注册Runner时产生的Token。此Token是每一个Runner的身份标识，Manager通过此Token信息关联由其创建的Pod、Secret等信息。
   rbac	是否启用RBAC支持。启用后会自动创建相关ServiceAccount。 rbac: create: true
   concurrent	配置作业的并发度，默认值是10。Manager Pod本身在管理作业时会有一定的内存开销，如果您的并发度较高，需要适当调大Manager Pod的资源规格。
   unregisterRunners	在Manager Pod退出时执行unregister，此配置一般用于使用runnerRegistrationToken进行注册的场景，避免每次重新生成token导致的作业失联问题。详细信息，请参见FAQ。
   runners.config	Runner运行配置，以多行字符串形式作为runner运行模板，您可以按需调整执行器的相关配置。

4. 安装GitLab Runner到ACS集群。

   helm install --namespace default gitlab-runner -f values.yaml --version 0.68.1 gitlab/gitlab-runner

5. 执行以下命令，确认GitLab Runner的运行状态。

   kubectl get pod | grep gitlab

   预期输出：

   gitlab-runner-7c5b4xxxxx-xxxxx   1/1     Running     0          5m17s

   出现此输出表明安装已经完成。

镜像构建

本节我们将结合ACS的部分特性，分别以Docker-in-Docker和使用Kaniko的方式进行镜像构建。使用到的示例工程，请参见Java demo。

结合ACS的弹性策略降低CI成本

您可以通过以下几种方式使用ACS提供的best-effort实例，进一步降低CI/CD流程中的作业成本。

1. 配置全局或工程维度使用best-effort实例。

   建议您全局默认采用best-effort类型，部分特殊场景按需覆盖配置使用其他实例类型。

   1. 通过runners配置进行全局配置。

      ...
      runners:
        config: |
          [[runners]]
            [runners.kubernetes]
              ...
              pod_labels_overwrite_allowed = ".*" #允许工程内通过variable覆盖Labels
            [[runners.kubernetes.pod_labels]]
            "app" = "acs-gitlab-runner"
            "alibabacloud.com/compute-class" = "general-purpose"
            "alibabacloud.com/compute-qos" = "best-effort" #声明使用best-effort型
      ...      

   2. 您也可以在每个仓库下通过gitlab-ci.yml进行按需配置，例如使用Kaniko以非特权模式进行镜像构建中的variables配置。

2. 配置ResourcePolicy，提高弹性确定性。

   ResourcePolicy可以提供自定义的调度策略，您可以使用其提供的成本优先策略，对CI作业优先使用best-effort实例。当地域内此实例类型售罄时，可以自动降级至default型实例，确保CI作业的连续可用。

   apiVersion: scheduling.alibabacloud.com/v1alpha1
   kind: ResourcePolicy
   metadata:
     name: rp-demo
     namespace: default
   spec:
     selector: # 在selector中标记Pod，表示带有app=stress标签的Pod将遵循此调度策略
       app: acs-gitlab-runner
     units: # 在units中定义调度顺序
     - resource: acs # 优先申请best-effort类型的资源
       podLabels:
         alibabacloud.com/compute-class: general-purpose
         alibabacloud.com/compute-qos: best-effort
     - resource: acs # 前者库存不足时，申请default类型的资源
       podLabels:
         alibabacloud.com/compute-class: general-purpose
         alibabacloud.com/compute-qos: default

FAQ

当Manager Pod重建或重启时，如何处理集群内出现Pod残留？

问题原因

一种可能的情况是，重启或重建Manager Pod后使用了和之前不同的一个RunnerToken进行启动，RunnerToken作为唯一的身份标识信息，当产生变化后，无法继续接管原有的作业Pod。

解决办法

• 您可以在GitLab控制台重新创建Runner并将RunnerToken持久化到安装的配置文件中。

• 如果是通过registrationToken在启动期间进行的Runner注册，建议挂载一个Secret，并将首次注册产生的Token信息持久化到Secret中，确保每次重启时优先进行读取操作。

• 建议开启featureGate FF_USE_POD_ACTIVE_DEADLINE_SECONDS，为每个worker增加TTL，作为备用的资源回收策略。