LingoChamp's automated practices with Terraform

About LingoChamp

LingoChamp® (NYSE: LAIX) is a leading technology-driven education company co-founded in September 2012 by Dr. Yi Wang, Zheren Hu, and Dr. Hui Lin. As a pioneer in AI-powered education, the company has an exceptional artificial intelligence team. Its proprietary AI English teacher uses deep learning technology to provide personalized and adaptive learning courses for every user. In the nine years since its founding, LingoChamp® has launched a variety of products that cover different interests and user segments. These include English education apps, such as "LingoChamp® English" and "LingoChamp® Reading", and English learning products, such as "LingoChamp® Dongni English® A+" and "LingoChamp® Pronunciation". As of June 2021, LingoChamp's® products had accumulated over 200 million registered users across 175 countries and 384 cities in China. The company has also built a massive "Chinese Speakers' English Speech Database" that has accumulated over 3.9 billion minutes of dialogue and 53.7 billion recorded sentences.

The driving force for automation

In a rapidly changing market, business teams want to respond quickly to market demands and create more innovative products. This requires the Cloud Infrastructure team to provide a more agile infrastructure and improve cloud management in the following ways:

• Accelerate resource provisioning: Enable business teams to quickly obtain cloud resources in a self-service manner.

• Reduce total costs: Use the elastic capabilities of the cloud to improve resource utilization and lower overall costs.

• Improve operations and maintenance (O&M) and management efficiency: Increase O&M efficiency and reduce repetitive manual operations.

LingoChamp's automation practices

Our automation was not built overnight. It was a gradual process. We followed these principles:

1) Automate routine, repetitive tasks. This helps everyone experience the value of automation and builds a culture around it.

2) Prioritize automation tasks based on value. This unlocks the business value of automation.

We explored automation from the following three dimensions. As our automation capabilities matured, so did our management practices:

• Deployment automation

• Management automation

• Administration automation

Deployment automation: Automated and process-driven resource provisioning

Provisioning cloud resources is the first step for all projects and is one of our daily tasks. Before automation, all operations were performed in the console. This process seems simple, but it caused many problems in practice:

• Resources could not be managed centrally. Without a unified repository to record information such as resource ownership and specification changes, it was easy for chaos to ensue when changes occurred.

• Manual change errors affected the normal operation of online services and were difficult to roll back. Excessive human interaction turned convenience into a source of errors. The worst-case scenario was making a mistake and forgetting the original configuration.

• Creating duplicate resources required repetitive manual operations on the web page. This process was time-consuming and could not be standardized.

Therefore, we used Terraform, Luban (our self-developed management platform), and GitLab to fully automate resource provisioning. This reduced provisioning time from hours to minutes and increased O&M support efficiency by 100%.

The overall architecture is as follows:

• Automate the process with Luban and use Chatbox to improve efficiency and user experience.

• Use GitLab to centrally manage the Infrastructure as Code (IaC) resource configuration repository.

• Use Terraform to create and change resources on Alibaba Cloud.

The detailed architecture and implementation are described below.

1) Request resources on the Luban platform

We put all the links for the research and development (R&D) team on a platform called Luban. The requester simply selects the necessary parameters on the frontend and submits the request. After submission, they just need to wait for the approval result. For example, to request an Alibaba Cloud ECS instance, you would have to write a Terraform file like the following:

resource "alicloud_instance" "instance" {
  # cn-beijing
  availability_zone = "cn-beijing-b"
  security_groups   = alicloud_security_group.group.*.id

  # Series III
  instance_type              = "ecs.n4.large"
  system_disk_category       = "cloud_efficiency"
  system_disk_name           = "test_foo_system_disk_name"
  system_disk_description    = "test_foo_system_disk_description"
  image_id                   = "ubuntu_18_04_64_20G_alibase_20190624.vhd"
  instance_name              = "test_foo"
  vswitch_id                 = alicloud_vswitch.vswitch.id
  internet_max_bandwidth_out = 10
  data_disks {
    name        = "disk2"
    size        = 20
    category    = "cloud_efficiency"
    description = "disk2"
    encrypted   = true
    kms_key_id  = alicloud_kms_key.key.id
  }
}

Although this looks easy to understand, some parts can be confusing. You would have to check the Alibaba Cloud provider documentation to understand each parameter and make changes. This process is difficult for someone who has not written Terraform before. To simplify this, we created a frontend request page where you only need to select the required parameters. The Luban backend then generates the code in the corresponding directory according to predefined rules and triggers the GitOps flow. The requester only needs to wait for the approval result.

2) Run a Terraform Plan check

This step runs automatically in the background and triggers the mobius webhook to perform a terraform plan. mobius is a critical engine in the Luban backend. It integrates with GitLab webhooks to handle merge request events, such as create, update, merge, and cancel. It can also handle the init, plan, and apply steps of the Terraform flow and output logs.

After a git commit, mobius automatically runs the terraform plan pipeline.

The Tech Leader can view the detailed progress of the resource request in the request history on the frontend. They check if the plan result meets expectations. If it does, they click Approve to proceed to the next step, where an infrastructure team member conducts a final review.

3) Notify the on-duty staff to approve the resource change

After the plan runs successfully, the process automatically moves to the next stage. The Luban Bot module notifies the on-duty staff of the Infrastructure team to approve the resource change.

4) Automatically change online resources

After the on-duty staff from the Infrastructure team approves the request in GitLab, the mobius webhook is triggered to run terraform apply. This applies the changes online. After the apply succeeds, mobius automatically merges the code. This completes the request flow. The requester receives a notification from the Luban Bot in the internal chat that the resource request was successful.

These four steps show that resource provisioning is not only fully automated but also rigorous. Staff no longer need to learn how to write Terraform. The focus is on reviewing the terraform plan output to see if the changes are as expected. Triggered notifications from the ChatBot also improve efficiency.

Management automation: Automated elastic scaling

Adding cloud resources during business peaks and releasing them during troughs helps us reduce cloud costs.

Manually adjusting resources based on business activity plans or passively configuring them when the load is high is not optimal for the user experience. This approach also does not dynamically respond in real time to reduce costs. Therefore, we began to explore how to automate elastic scaling.

After a period of exploration, we were able to automatically adjust the number of resources based on rule configurations or business metric monitoring. In practice, this has resulted in cost savings of over 20% for the same business payload.

Technically, we have implemented automated elastic scaling at the container, application server, and database layers:

• Container layer: Use Horizontal Pod Autoscaler (HPA) to scale pods. Scaling is triggered by monitoring metrics, business metrics, and scheduled tasks.

• Server layer: Use Auto Scaling groups to scale ECS instances based on server metrics.

• Database layer: Use cloud-native databases to achieve elasticity. For example, EMR can scale out ECS instances on a schedule or based on CPU/memory metrics.

Administration automation: Automated cost management

For a company like LingoChamp that was built on the cloud, cost management is a key task. Our primary challenges are to effectively manage cloud spending and use technology to eliminate resource waste. Automating cost allocation to various business teams is the foundation of our cost management strategy.

1) Cost allocation based on tags

Cloud cost allocation is not as complicated as it might seem. Cloud providers have tag systems for their cloud resources, and Kubernetes (K8s) resources have a label system. These two tagging systems can be easily associated. Alibaba Cloud also provides bill-related APIs that can be called.

We developed a Catalog system to bind resources to applications (apps). This system also clarifies the owner and team for each app. It serves as the single source of truth for resource ownership and usage statistics within the company. With this in place, we were able to perform cost allocation based on the data managed in the Catalog source.

When resource ownership changes, we only need to modify the Catalog system, which is extremely lightweight and efficient. Some resources are shared by multiple teams due to long-term accumulation. We first allocate costs for most resources with clear ownership. Then, we use the resulting proportions to allocate the costs of unallocatable resources. This approach has been agreed upon with all lines-of-business.

2) Automated cost analysis reports

We allocate the costs of public support teams, such as big data, infrastructure, and Business Mid-end, to each line-of-business based on their established cost proportions. We also calculate the R&D cost proportion for each business and prepare year-over-year and month-over-month comparisons for all cost data.

In summary, we clarify resource relationships using Catalog and tags/labels. We use mostly accurate data for allocation, which solves the difficulty of allocating public resources. Finally, a detailed, automated cost analysis report is provided to each line-of-business monthly, and a simple real-time dashboard is also available.

In addition, we use historical monitoring data from Prometheus for various resources to compile statistics on CPU and memory utilization for resources, and CPU, memory, and IOPS utilization for storage resources. These reports are sent automatically on a weekly basis. The core report template is as follows:

We use this data to downgrade underutilized resources. To date, we have not experienced any performance regressions or production incidents.

Summary

By applying automation, our Cloud Infrastructure team has improved work efficiency and management capabilities. This has also brought significant benefits to the business, including faster resource delivery and more transparent cost spending.

About the author

The Cloud Infrastructure team from the Technology Department at LingoChamp