Use GitLab Runner to build a CI/CD environment in an ACS cluster

Step 1: Get the GitLab Runner chart

helm repo add gitlab https://charts.gitlab.io
helm repo update gitlab
helm pull gitlab/gitlab-runner --version 0.68.1 && tar zvxf gitlab-runner-0.68.1.tgz

Step 2: Create values.yaml

Three parameters are required to get GitLab Runner running:

The following values.yaml shows a working configuration with recommended settings:

## GitLab Runner Image
##
## By default it's using registry.gitlab.com/gitlab-org/gitlab-runner:alpine-v{VERSION}
## where {VERSION} is taken from Chart.yaml from appVersion field
##
## ref: https://gitlab.com/gitlab-org/gitlab-runner/container_registry/29383?orderBy=NAME&sort=asc&search[]=alpine-v&search[]=
##
## Note: If you change the image to the ubuntu release
##       don't forget to change the securityContext;
##       these images run on different user IDs.
##
...

## The GitLab Server URL (with protocol) that want to register the runner against
## ref: https://docs.gitlab.com/runner/commands/index.html#gitlab-runner-register
##
gitlabUrl: https://jihulab.com/

## DEPRECATED: The Registration Token for adding new Runners to the GitLab Server.
##
## ref: https://docs.gitlab.com/ee/ci/runners/new_creation_workflow.html
##
# runnerRegistrationToken: ""

## The Runner Token for adding new Runners to the GitLab Server. This must
## be retrieved from your GitLab instance. It is the token of an already registered runner.
## ref: (we don't have docs for that yet, but we want to use an existing token)
##
runnerToken: "glrt-t3_sz6xxxxxxxxxDsWF77"
#

## Unregister all runners before termination
##
## Updating the runner's chart version or configuration will cause the runner container
## to be terminated and created again. This may cause your Gitlab instance to reference
## non-existant runners. Un-registering the runner before termination mitigates this issue.
## ref: https://docs.gitlab.com/runner/commands/index.html#gitlab-runner-unregister
##
unregisterRunners: true

## When stopping the runner, give it time to wait for its jobs to terminate.
##
## Updating the runner's chart version or configuration will cause the runner container
## to be terminated with a graceful stop request. terminationGracePeriodSeconds
## instructs Kubernetes to wait long enough for the runner pod to terminate gracefully.
## ref: https://docs.gitlab.com/runner/commands/#signals
terminationGracePeriodSeconds: 3600

## Set the certsSecretName in order to pass custom certficates for GitLab Runner to use.
## Provide resource name for a Kubernetes Secret Object in the same namespace,
## this is used to populate the /home/gitlab-runner/.gitlab-runner/certs/ directory
## ref: https://docs.gitlab.com/runner/configuration/tls-self-signed.html#supported-options-for-self-signed-certificates-targeting-the-gitlab-server
##
# certsSecretName:

## Configure the maximum number of concurrent jobs
## ref: https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-global-section
##
concurrent: 10

...

## For RBAC support:
rbac:
  ## Specifies whether a Role and RoleBinding should be created
  ## If this value is set to `true`, `serviceAccount.create` should also be set to either `true` or `false`
  ##
  create: true
  ## Define the generated serviceAccountName when create is set to true
  ## It defaults to "gitlab-runner.fullname" if not provided
  ## DEPRECATED: Please use `serviceAccount.name` instead
  generatedServiceAccountName: ""
...

## Configuration for the Pods that the runner launches for each new job
##
runners:
  # runner configuration, where the multi line string is evaluated as a
  # template so you can specify helm values inside of it.
  #
  # tpl: https://helm.sh/docs/howto/charts_tips_and_tricks/#using-the-tpl-function
  # runner configuration: https://docs.gitlab.com/runner/configuration/advanced-configuration.html
  config: |
    [[runners]]
      [runners.kubernetes]
        namespace = "{{.Release.Namespace}}"
        image = "alpine"

  ## Absolute path for an existing runner configuration file
  ## Can be used alongside "volumes" and "volumeMounts" to use an external config file
  ## Active if runners.config is empty or null
  configPath: ""
...

The following table describes the key parameters:

Step 3: Install GitLab Runner

helm install --namespace default gitlab-runner -f values.yaml --version 0.68.1 gitlab/gitlab-runner

Verify the installation

kubectl get pod | grep gitlab

Expected output:

gitlab-runner-7c5b4xxxxx-xxxxx   1/1     Running     0          5m17s

Method 1: Configure BestEffort QoS for all pods in the cluster

We recommend that you configure the BestEffort QoS class for all pods in the cluster. You can also select other QoS classes based on your business requirements.

Add pod labels in values.yaml to apply BestEffort QoS to every job pod the runner creates:

...
runners:
  config: |
    [[runners]]
      [runners.kubernetes]
        ...
        pod_labels_overwrite_allowed = ".*" # Allow the system to overwrite the labels of project pods based on variables.
      [[runners.kubernetes.pod_labels]]
      "app" = "acs-gitlab-runner"
      "alibabacloud.com/compute-class" = "general-purpose"
      "alibabacloud.com/compute-qos" = "best-effort" # Configure the BestEffort QoS class for pods.
...

To apply different QoS classes per project, add the corresponding KUBERNETES_POD_LABELS_* variables in the project's .gitlab-ci.yml instead. See the kaniko example above.

Method 2: Configure a ResourcePolicy for automatic failover

A ResourcePolicy lets you define a custom scheduling priority. Job pods are scheduled to BestEffort capacity first. When BestEffort pods are exhausted in a region, the system automatically creates Default pods to ensure the continuity and availability of your CI jobs.

apiVersion: scheduling.alibabacloud.com/v1alpha1
kind: ResourcePolicy
metadata:
  name: rp-demo
  namespace: default
spec:
  selector: # Specify the label selector used to select pods. In this example, the ResourcePolicy is applied to pods that have the app=stress label.
    app: acs-gitlab-runner
  units: # Specify the priorities of different types of nodes for pod scheduling.
  - resource: acs # Prioritize BestEffort resources.
    podLabels:
      alibabacloud.com/compute-class: general-purpose
      alibabacloud.com/compute-qos: best-effort
  - resource: acs # Apply for Default resources when the preceding resources are out of stock.
    podLabels:
      alibabacloud.com/compute-class: general-purpose
      alibabacloud.com/compute-qos: default

Why are there residual pods after the runner manager pod restarts?

When the manager pod is recreated with a different runner token, it loses the ability to manage the job pods created under the previous token. Those pods remain but are no longer tracked.

To avoid this:

• Recreate the runner in GitLab and update the runner token in the configuration file.

• If you use registration tokens, mount a Secret to the manager pod containing the token from the initial registration. This ensures the same token is reused across restarts.

• Enable FF_USE_POD_ACTIVE_DEADLINE_SECONDS. This sets a TTL on each job pod equal to the job timeout, so orphaned pods are cleaned up automatically.