Use Terraform to implement cloud automation

The following is the video transcript:

Hello, welcome back to Auto Talk, the Alibaba Cloud Open Platform Automation Series. In this episode, we explore how to efficiently achieve cloud automation by using Terraform. I am Xiahuai from the Alibaba Cloud Open Platform team.

First, let's look at the concept of IaC. "I" stands for Infrastructure, which refers to cloud resources, and "C" refers to code. Infrastructure as Code (IaC) means defining and managing infrastructure through readable configuration files or code. This includes resource definitions. Resources are not limited to Infrastructure-as-a-Service (IaaS) resources, such as cloud instances or OSS buckets, but also include Platform-as-a-Service (PaaS) resources, such as Kubernetes namespaces, and Software-as-a-Service (SaaS) resources, such as DNS entries.

In IaC, we generally classify methods into two categories: imperative and declarative. Imperative methods focus on the specific process of creating infrastructure by describing the exact commands to be executed in code. On the other hand, declarative methods focus on the final state or goal of infrastructure creation, describing the desired state through code.

Many tools are available for IaC. Any framework or tool that uses programming methods to make changes or configure infrastructure in a declarative or imperative manner can be considered an IaC tool. Alibaba Cloud has its own tools, as well as third-party tools. For example, Alibaba Cloud provides Resource Orchestration Service (ROS). Among third-party tools, we support widely used development tools like Terraform. Terraform is an open-source orchestration tool introduced by HashiCorp, and Alibaba Cloud is one of its four official major providers.

Terraform has several key features. First, in terms of orchestration, Terraform provides a unified orchestration language called HCL, and it also supports JSON. Resources are defined in code, and the underlying CRUD operations for resources are implemented by the cloud provider. Second, in terms of management, Terraform handles the writing of code templates. Defining resources and managing templates equates to managing resources. Terraform parses and executes code and manages the state information of resources. Third, in terms of standards, Terraform uses a resource model to manage resources, security audits, and other capabilities, allowing them to align with this model through a codified approach. Finally, in terms of processes, Terraform can integrate with VCS, ITSM, and CI/CD pipelines, enabling self-service management of cloud resources. Additionally, these processes can help make the entire system and business more automated and standardized.

Terraform has several core components. First, providers. Alibaba Cloud is one of the four major providers of HashiCorp. A provider represents a cloud service provider or a Terraform provider that implements CRUD interfaces and orchestrates resource-related APIs. Users need to specify the provider in their templates. Currently, Terraform supports over 3,000 providers.

Another important concept in Terraform is the resource. Any specific resource or service can be defined as a resource. It can be an IaaS-level resource, such as an ECS instance or a VPC, or a SaaS-level resource, such as a DNS route. Each resource has a complete CRUD lifecycle. In Terraform, the relationship between one resource and another is often also defined as a resource, such as attaching a disk to an ECS instance.

Another important concept is the data source. Data sources provide dynamic querying capabilities for resources. For example, you can obtain existing ECS instances, available zones, instance specifications, or instance images by using data sources. The fourth concept is the state. After operations are performed on resources defined in a template, their attributes and management information are saved to a state file. Two types of states exist: local and remote. Remote states are typically stored in OSS or AWS S3.

The last commonly used concept is the module. You can encapsulate multiple resources into a ready-to-use package. Similar to encapsulating SDKs or classes in Java, modules improve reusability, shareability, and general applicability.

Terraform has several main operations. The init operation initializes Terraform, loading the dependencies required by a template, including providers, modules, and remote states. It is a prerequisite for all other commands. The plan command previews the resources defined in the template. When you run the command, Terraform shows you exactly which resources will be created or modified. The plan command is for previewing, and the apply command actually executes the creation and modification of resources.

The destroy command destroys all resources defined in the template. The import command brings existing resources that were not previously managed by Terraform into the state file. You can then supplement your Terraform templates so that subsequent resource management, especially continuous management, can be done through Terraform. These are the steps to achieve cloud automation.

First, we prepare the environment. For example, we need to download and install providers and modules, configure cloud vendor accounts and environments, and download some IDE plugins to help us write Terraform code more conveniently. During this process, we can refer to documentation or examples, and even try out some operations in playground environments. For instance, Alibaba Cloud has Terraform integrated into its Cloud Shell, where you can experiment with it.

Once you finish debugging, you can move on to real business integration. For example, you can import existing resources into the state file while supplementing the template information. During this process, you may need to consider how to organize your code, and as mentioned earlier, you can use modules to reuse code. Additionally, during the phase of improving the business, you can use testing methods to ensure that the template execution results meet your expectations. You can also integrate open-source security and compliance components to perform static code checks.

Next, configure the production environment, including managing state files. As mentioned earlier, you need to manage local and remote state files. For remote state files, activate OSS and configure the state files. You can also integrate Terraform command execution into CI/CD pipelines. Additionally, when creating resources on Alibaba Cloud, some permission-related operations may be required. These can be encapsulated into templates and executed beforehand.

Finally, during the continuous operation and monitoring phase, focus on a few key points. First, errors may occur, and you need to use the error diagnosis tools provided by the cloud vendor to troubleshoot issues. Second, set up monitoring and alerting mechanisms specifically for Terraform.

Another common issue is configuration drift. For example, if resources managed by Terraform are modified through the console by other teams, or new resources are created outside Terraform, those resources fall outside its control scope. This is called configuration drift. To address this, align the Terraform template with the actual state of the cloud resources. Otherwise, running Terraform commands again may cause discrepancies, and states modified through the console may be overwritten.

Lastly, let's talk about best practices for using Terraform. There are two key points here. First, we recommend that you use Terraform together with a code repository management system, such as GitLab or GitHub, and host your Terraform code in these tools. Managing infrastructure changes essentially means managing code. At the same time, we can encapsulate the entire pipeline into a service and integrate it into a self-service portal. This way, the capabilities provided by the infrastructure team can allow developers to apply for and manage resources through self-service. The platform team only needs to review code or approve requests to achieve resource control and production goals for the development team.

This approach has several characteristics. First, it provides self-service capabilities, allowing development teams to apply for and manage resources independently, improving resource management efficiency. Even without a self-service portal, development teams can submit pull requests directly in the code repository. The code is then reviewed, merged, and Terraform commands are executed through a pipeline.

Second, the entire process — from submitting Terraform code, triggering the plan command, to executing the apply command — is orchestrated through a pipeline. This standardizes the workflow and platformizes Terraform capabilities. Third, through this IaC management architecture, integration with cloud platforms and cloud resources becomes much simpler. You no longer need to orchestrate a large number of APIs, effectively avoiding bottlenecks in resource provisioning efficiency.

That concludes the content of this episode. If you have any questions or ideas about automation, feel free to scan the QR code below the screen to join our DingTalk group and communicate with us. Looking forward to seeing you in the next episode!