Use Terraform to manage add-ons

Add-on types

ACK add-ons are either system add-ons or optional add-ons. See Add-ons.

System add-ons

Optional add-ons

Optional add-ons are not required but can be installed to extend cluster capabilities. They are grouped into categories including:

• Application management

• Logging and monitoring

• Storage

• Networking

• Security

Generate Terraform configuration parameters from the console

The ACK console can automatically generate the Terraform configuration for managing cluster add-ons.

1. Log on to the ACK console. In the left navigation pane, click Clusters.

2. On the Clusters page, click the name of your cluster. In the left navigation pane, click Components and Add-ons.

3. On the Add-ons page, find the target add-on and click Install or Configuration.

4. In the dialog box, click Console-to-Code. Click the Terraform tab and copy the displayed configuration.

Best practices for add-on management

Specify add-ons at cluster creation and manage their lifecycle afterward. The following sections cover lifecycle management and best practices for different scenarios.

Specify add-ons at cluster creation

The following Terraform resources correspond to different cluster types:

• ACK managed cluster: alicloud_cs_managed_kubernetes

• ACK dedicated cluster: alicloud_cs_kubernetes

• ACK Edge cluster: alicloud_cs_edge_kubernetes

• ACK Serverless cluster: alicloud_cs_serverless_kubernetes

Set the addons object in the Resource to specify add-ons. The following code block shows the addons object attributes:

# This example uses an ACK managed cluster. 
resource "alicloud_cs_managed_kubernetes" "default" {
  # Other parameters. 
  # ...

  # The addons object is a list. You can set the addons object in a Resource to specify the add-ons to be installed when the system creates the cluster. 
  addons {
    # The name of the add-on. You can query the name of an add-on by using alicloud_cs_kubernetes_addons of Data Source. 
    # The add-ons that are already installed and can be installed in the cluster and the versions of the add-ons. 
    name = "XXX"

    # Custom add-on parameters. You can set this attribute for cluster add-ons that support custom parameters. For more information, see the Modify the custom parameters of a cluster add-on section. 
    config = jsonencode(
      {
        ....
      }
    )

    # The value is of Boolean type and the default is false. By default, ACK automatically installs specific add-ons for you to manage the cluster. If you do not want ACK to install add-ons when ACK creates the cluster, set disabled=true. 
    disabled = XXX
  }
}

The following table lists add-ons configurable with Terraform.

Manage add-ons after cluster creation

Ensure you have an ACK cluster before managing add-on lifecycle.

Use the alicloud_cs_kubernetes_addon resource to install, update, uninstall, and configure add-ons. The following code block shows the alicloud_cs_kubernetes_addon attributes:

resource "alicloud_cs_kubernetes_addon" "addon-example" {
  # The ID of the cluster. 
  cluster_id = "XXXX"

  # The name of the add-on. You can query the add-ons that are already installed and can be installed and their versions by using alicloud_cs_kubernetes_addons of Data Source. 
  name = "XXXX"

  # The version of the add-on. 
  version = "XXXX"

  # Custom add-on parameters in a JSON string. You can use the jsonencode method of Terraform to specify the parameters or directly specify the parameters in a JSON string. Pay attention to character escaping if you directly specify the parameters in a JSON string. You can set this attribute for cluster add-ons that support custom parameters. For more information, see the Modify the custom parameters of a cluster add-on section. 
  config = jsonencode(
    {
      ....
    }
  )
}

You can also specify custom parameters directly in a JSON string. Pay attention to character escaping. For example, configure nginx-ingress-controller using either method:

• Use jsonencode to configure custom parameters:

  config = jsonencode(   
    {       
      IngressSlbNetworkType="internet"       
      IngressSlbSpec="slb.s2.small"     
    }  
  )

• Directly specify custom parameters in a JSON string:

  config = "{\"IngressSlbNetworkType\":\"internet\",\"IngressSlbSpec\":\"slb.s2.small\"}"

Import installed add-ons into Terraform

Use terraform import to import cluster add-ons into Terraform for management. The following example uses nginx-ingress-controller.

1. Create a .tf file and define a Resource. If you already have a .tf file, define the Resource there.

   The alicloud_cs_kubernetes_addon resource manages cluster add-ons. Leave it empty initially.

   resource "alicloud_cs_kubernetes_addon" "nginx-ingress-controller" {
   }

2. Import nginx-ingress-controller:

   Terraform pulls the nginx-ingress-controller configuration from the cluster into the .state.

   terraform import alicloud_cs_kubernetes_addon.nginx-ingress-controller <cluster_id>:nginx-ingress-controller

3. Run terraform plan. The output shows the configuration difference between nginx-ingress-controller and the Resource.

   Update the Resource from Step 1 based on the diff and the .state. When terraform plan shows no difference, the import is complete.

   resource "alicloud_cs_kubernetes_addon" "nginx-ingress-controller" {
     cluster_id = "XXXXX"
     name = "nginx-ingress-controller"
     version = "v1.2.1-aliyun.1"
     config = jsonencode(
       {
         IngressSlbNetworkType = "internet"
         IngressSlbSpec        = "slb.s2.small"
       }
     )
   }

Install cluster add-ons

Use the alicloud_cs_kubernetes_addon resource to install add-ons. The following example uses Gatekeeper.

1. Specify the add-on in your .tf:

   • The cluster ID.

   • The name and version of the add-on:

     Query installable add-ons and versions with alicloud_cs_kubernetes_addons Data Source. Only the latest version is shown; for earlier versions, check the add-on release notes.

   • (Optional) Custom add-on configuration:

     Customize the config field with jsonencode. Query supported parameters with alicloud_cs_kubernetes_addon_metadata Data Source. See Customize add-on parameters.

   Click to view details

   resource "alicloud_cs_kubernetes_addon" "gatekeeper" {
     cluster_id = "ce36b7c61e126430b8b245730ca6d****"
     name = "gatekeeper"
     version = "v3.8.1.113-geb7947ef-aliyun"
     config = jsonencode(
       {
         AdmissionPodCpuLimit      = "1000m"
         AdmissionPodCpuRequest    = "100m"
         AdmissionPodMemoryLimit   = "512Mi"
         AdmissionPodMemoryRequest = "256Mi"
         AdmissionPodNumber        = 3
         AuditInterval             = 1800
         AuditPodCpuLimit          = "1000m"
         AuditPodCpuRequest        = "100m"
         AuditPodMemoryLimit       = "512Mi"
         AuditPodMemoryRequest     = "256Mi"
         EnableAuditPod            = false
         EnableMutatingWebhook     = false
       }
     )
   }

2. Install the add-on:

   terraform apply

   Expected output:

   Plan: 1 to add, 0 to change, 0 to destroy.
   
   Do you want to perform these actions?
     Terraform will perform the actions described above.
     Only 'yes' will be accepted to approve.
   
     Enter a value: yes
   
   alicloud_cs_kubernetes_addon.gatekeeper: Creating...
   alicloud_cs_kubernetes_addon.gatekeeper: Still creating... [10s elapsed]
   alicloud_cs_kubernetes_addon.gatekeeper: Creation complete after 16s [id=XXXXX:gatekeeper]
   
   Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

   When Apply complete! appears, the add-on is installed.

Update cluster add-ons

Query available versions with alicloud_cs_kubernetes_addons Data Source, then change the version number to update. The following example uses Gatekeeper.

resource "alicloud_cs_kubernetes_addon" "gatekeeper" {
  cluster_id = "ce36b7c61e126430b8b245730ca6d****"
  name = "gatekeeper"

  # Change version to the version number that you want to use. 
  version = "XXXXXXXXX"
  config = jsonencode(
    {
      AdmissionPodCpuLimit      = "1000m"
      AdmissionPodCpuRequest    = "100m"
      AdmissionPodMemoryLimit   = "512Mi"
      AdmissionPodMemoryRequest = "256Mi"
      AdmissionPodNumber        = 3
      AuditInterval             = 1800
      AuditPodCpuLimit          = "1000m"
      AuditPodCpuRequest        = "100m"
      AuditPodMemoryLimit       = "512Mi"
      AuditPodMemoryRequest     = "256Mi"
      EnableAuditPod            = false
      EnableMutatingWebhook     = false
    }
  )
}

Run terraform apply to update the add-on. Apply complete! indicates the update succeeded.

Customize add-on parameters

Modify the config field in alicloud_cs_kubernetes_addons to customize add-on configuration. The following example uses Gatekeeper.

resource "alicloud_cs_kubernetes_addon" "gatekeeper" {
  cluster_id = "ce36b7c61e126430b8b245730ca6d****"
  name = "gatekeeper"
  version = "v3.8.1.113-geb7947ef-aliyun"

  # You can modify and apply the attributes in Config to customize the configuration of a cluster add-on. 

}

Query supported custom parameters with alicloud_cs_kubernetes_addon_metadata Data Source, which returns a JSON schema. For example, to query Gatekeeper parameters, add the following to your .tf file:

# Define Data Source to obtain the schema that includes the custom parameters supported by the Gatekeeper add-on. 
data "alicloud_cs_kubernetes_addon_metadata" "default" {
  cluster_id = "ce36b7c61e126430b8b245730ca6d****"
  name       = "gatekeeper"
  version    = "v3.8.1.113-geb7947ef-aliyun"
}

# Output the schema. 
output "addon_config_schema" {
  value = data.alicloud_cs_kubernetes_addon_metadata.default.config_schema
}

addon_config_schema = <<EOT
{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "properties": {
    "AdmissionPodCpuLimit": {
      "default": "1000m",
      "description": "cpu limit for gatekeeper",
      "pattern": "^(|[1-9][0-9]*(m|\\.\\d+)?)$",
      "type": "string"
    },
    "AdmissionPodCpuRequest": {
      "default": "100m",
      "description": "cpu request for gatekeeper",
      "pattern": "^[1-9][0-9]*(m|\\.\\d+)?$",
      "type": "string"
    },
    "AdmissionPodMemoryLimit": {
      "default": "512Mi",
      "description": "memory limit for gatekeeper",
      "pattern": "^(|[1-9][0-9]*(\\.\\d+)?(K|Ki|M|Mi|G|Gi|T|Ti)?)$",
      "type": "string"
    },

    ......
  },
  "title": "Config",
  "type": "object"
}
EOT

Configure network add-ons

ACK clusters use Terway for container networking. See Overview.

# Configure the Terway network add-on and enable the Assign One ENI to Each Pod mode. This is the default mode. 
# In this mode, the number of pods on a node is limited by the elastic network interface (ENI) quota of Elastic Compute Service (ECS) instances. 

resource "alicloud_cs_managed_kubernetes" "default" {
  # Other parameters. 
  # ...

  addons {
    name = "terway-eniip"
  }
}

# Configure the Terway network add-on and enable the IPVLAN mode. 
# You can use only the Alibaba Cloud Linux 2 operating system because the One ENI for Multi-Pod mode uses the IPVLAN + eBPF virtualization technology. 
resource "alicloud_cs_managed_kubernetes" "default" {
  # Other parameters. 
  # ...

  addons {
    name   =  "terway-eniip"
    config = "{\"IPVlan\":\"true\",\"NetworkPolicy\":\"false\"}"
  }
}

# Configure the Terway network add-on and enable Kubernetes network policies in IPVLAN mode. 
# You can use only the Alibaba Cloud Linux 2 operating system because the One ENI for Multi-Pod mode uses the IPVLAN + eBPF virtualization technology. 
# The IPVLAN mode provides network access control based on Kubernetes network policies. 
resource "alicloud_cs_managed_kubernetes" "default" {
  # Other parameters. 
  # ...

  addons {
    name   =  "terway-eniip"
    config = "{\"IPVlan\":\"true\",\"NetworkPolicy\":\"true\"}"
  }
}

Configure volume add-ons

# The CSI volume add-on consists of csi-plugin and csi-provisioner. If you use the CSI add-on and want ACK to create a default NAS file system and CNFS file system for dynamically provisioned volumes, you must also install the storage-operator add-on. 
resource "alicloud_cs_managed_kubernetes" "default" {
  # Other parameters. 
  # ...

  addons {
    name = "csi-plugin"
  }
  addons {
    name = "csi-provisioner"
  }
  addons {
    name   = "storage-operator"
    config = "{\"CnfsOssEnable\":\"false\",\"CnfsNasEnable\":\"true\"}"
  }
}

Configure logging add-ons

# Use the automatically created SLS project. 
resource "alicloud_cs_managed_kubernetes" "default" {
  # Other parameters. 
  # ...

  addons {
    name = "loongcollector"
  }
}

# Use the automatically created SLS project. Enable the automatically created Ingress dashboard. 
resource "alicloud_cs_managed_kubernetes" "default" {
  # Other parameters. 
  # ...

  addons {
    name = "loongcollector"
    config = "{\"IngressDashboardEnabled\":\"true\"}"
  }
}


# Use an existing SLS project. Enable the automatically created Ingress dashboard. 
resource "alicloud_cs_managed_kubernetes" "default" {
  # Other parameters. 
  # ...

  addons {
    name = "loongcollector"
    config = "{\"IngressDashboardEnabled\":\"true\",\"sls_project_name\":\"k8s-log-c55c35ff493df47b88783bea48827****\"}"
  }
}

# Install and configure node-problem-detector. 
# Configure node-problem-detector to use the automatically created SLS project to store events. 
resource "alicloud_cs_managed_kubernetes" "default" {
  # Other parameters. 
  # ...

  addons {
    name   = "ack-node-problem-detector"
    config = "{\"sls_project_name\":\"\"}"
  }
}

# Install and configure node-problem-detector. 
# Configure node-problem-detector to use an existing SLS project to store events. node-problem-detector can share the Logstore used by loongcollector. 
resource "alicloud_cs_managed_kubernetes" "default" {
  # Other parameters. 
  # ...

  addons {
    name   = "ack-node-problem-detector"
    config = "{\"sls_project_name\":\"k8s-log-c55c35ff493df47b88783bea48827****\"}"
  }
}

Configure monitoring add-ons

# Install the Managed Service for Prometheus. 
resource "alicloud_cs_managed_kubernetes" "default" {
  # Other parameters.
  # ...
  addons {
    name = "arms-prometheus"
  }
}
# Install the CloudMonitor agent on ECS nodes. 
resource "alicloud_cs_kubernetes_node_pool" "default" {
  # Other parameters.
  # ...
  install_cloud_monitor = true
}

Configure Ingress add-ons

# Configure nginx-ingress-controller. 
# If you use an Internet-facing Server Load Balancer (SLB) instance, you must set the IngressSlbNetworkType parameter to internet in the Config. 
# If you use an internal-facing SLB instance, you must set the IngressSlbNetworkType parameter to intranet in the Config. 
resource "alicloud_cs_managed_kubernetes" "default" {
  # Other parameters. 
  # ...

  addons {
    name   =  "nginx-ingress-controller"
    config = "{\"IngressSlbNetworkType\":\"internet\",\"IngressSlbSpec\":\"slb.s2.small\"}"
  }
}

# Configure alb-ingress-controller. 
resource "alicloud_cs_managed_kubernetes" "default" {
  # Other parameters. 
  # ...

  addons {
    name   = "alb-ingress-controller"
    config = "{\"albIngress\":{\"CreateDefaultALBConfig\":false}}"  # Not created right now.
#   config = "{\"albIngress\":{\"LoadBalancerId\":\"alb-vl8uiXXXXXxdr\",\"CreateDefaultALBConfig\":true}}" # Specify an existing ALB instance.
#   config = "{\"albIngress\":{\"AddressType\":\"Internet\",\"ZoneMappings\":{\"cn-hangzhou-l\":[\"vsw-uf6XXXXXoyb4qe\"],\"cn-hangzhou-m\":[\"vsw-uf6XXXX0rlkiq\"]},\"CreateDefaultALBConfig\":true}}" # Create a new ALB instance. You must specify at least two available zones.
 
  }
}

Disable default add-on installation

# Disable the system to install the nginx-ingress-controller add-on. 

resource "alicloud_cs_managed_kubernetes" "default" {
  # Other parameters. 
  # ...

  addons {
    name     =  "nginx-ingress-controller"
    disabled = true
  }
}

Add-ons installed by default

Without explicit addons configuration, the system installs the following add-ons:

Common configuration examples

# Do not install any add-ons when you create the cluster. In this scenario, only the default add-ons are installed. 
# A simple template. Replace the variables with the desired values. 
resource "alicloud_cs_managed_kubernetes" "default" {
  name                         = var.name
  cluster_spec                 = "ack.pro.small"
  is_enterprise_security_group = true
  pod_cidr                     = "172.20.0.0/16"
  service_cidr                 = "172.21.0.0/20"
  worker_vswitch_ids           = [var.vswitch_id]
}

# Create a cluster that has Terway installed. 
# Enable the Assign One ENI to Each Pod mode. 

resource "alicloud_cs_managed_kubernetes" "default" {
  name                         = var.name
  cluster_spec                 = "ack.pro.small"
  is_enterprise_security_group = true
  pod_vswitch_ids              = [var.vswitch_id]
  service_cidr                 = "172.21.0.0/20"
  worker_vswitch_ids           = [var.vswitch_id]

  addons {
    name = "terway-eni"
  }
}

# Create a cluster that has Terway installed. 
# Enable the IPVLAN mode and enable Kubernetes network policies. 
resource "alicloud_cs_managed_kubernetes" "default" {
  name                         = var.name
  cluster_spec                 = "ack.pro.small"
  is_enterprise_security_group = true
  service_cidr                 = "172.21.0.0/20"
  pod_vswitch_ids              = [var.vswitch_id]
  worker_vswitch_ids           = [var.vswitch_id]

  addons {
    name   =  "terway-eniip"
    config = "{\"IPVlan\":\"true\",\"NetworkPolicy\":\"true\"}"
  }
}

# A template for installing the Terway, CSI, and nginx-ingress-controller add-ons. 

resource "alicloud_cs_managed_kubernetes" "default" {
  name                         = var.name
  cluster_spec                 = "ack.pro.small"
  is_enterprise_security_group = true
  service_cidr                 = "172.21.0.0/20"
  pod_vswitch_ids              = [var.vswitch_id]
  worker_vswitch_ids           = [var.vswitch_id]

  addons {
    name = "terway-eniip"
    config = "{\"IPVlan\":\"true\",\"NetworkPolicy\":\"false\"}"
  }
  addons {
    name = "csi-plugin"
  }
  addons {
    name = "csi-provisioner"
  }
  addons {
    name = "storage-operator"
    config = "{\"CnfsOssEnable\":\"false\",\"CnfsNasEnable\":\"true\"}"
  }
  addons {
    name = "loongcollector"
    config = "{\"IngressDashboardEnabled\":\"true\"}"
  }
  addons {
    name = "ack-node-problem-detector"
    config = "{\"sls_project_name\":\"\"}"
  }
  addons {
    name = "nginx-ingress-controller"
    config = "{\"IngressSlbNetworkType\":\"internet\",\"IngressSlbSpec\":\"slb.s2.small\"}"
  }
  addons {
    name = "ack-node-local-dns"
  }
  addons {
    name = "arms-prometheus"
  }
  addons {
    name = "alicloud-monitor-controller"
    config = "{\"group_contact_ids\":\"[10619]\"}"
  }
}