Automate host operating system patching using OOS-Well-Architected Framework(WAF)-阿里云帮助中心

Overview

Enterprises often have compliance requirements for their IT assets. They must promptly fix system vulnerabilities in Elastic Compute Service (ECS) instances to prevent security attacks and keep software packages updated to the latest versions.

This document describes a solution that helps enterprises automate host operating system patching in the cloud. This solution meets the operations and maintenance (O&M) requirements for centralized patch scanning, monitoring, and installation in a multi-account environment.

Advantages

Centralized O&M for operating system patches

You can define patch baselines in a shared service account or an operations account and then distribute the baselines to ensure that all accounts use the same one. You can use a CloudOps Orchestration Service (OOS) automation flow that is defined in the shared service account or operations account to automatically install or upgrade operating system patches on specified ECS instances across all accounts. This provides automated and centralized patch O&M.

Improved business security and compliance

You can use Cloud Config and tag policies to continuously audit and ensure patch and tag compliance. You can promptly identify and handle ECS instances that do not comply with patch baselines. This ensures that ECS instances for all services meet the enterprise's overall patch baseline requirements and prevents security and compliance risks from delayed patch installation or updates.

Customer scenarios

Unified internal O&M

Scenario description

An enterprise needs to uniformly manage operating system patches for ECS instances. This ensures that all instances have a consistent patch baseline and allows for unified installation and upgrades.

Applicable customers

Enterprises that use a resource directory to manage multiple Alibaba Cloud accounts.
Enterprises that use multiple Alibaba Cloud accounts.
Enterprises with centralized O&M and security teams.
Enterprises with multiple ECS instances in their Alibaba Cloud accounts.

Solution architecture

This solution automates host operating system patching in a multi-account environment. Patch O&M is performed from a shared service account or an operations account. First, you can define a unified patch baseline. After you define the baseline, you can distribute it to the required business accounts (Step 1 in the figure). You can use Cloud Config to continuously audit the patch compliance of business accounts (Step 2). You can also send audit results as notifications through EventBridge or integrate them into a monitoring system using Simple Log Service (SLS). To install patches, you can trigger an automation task in OOS (Step 5). During the process, you can include manual approval (Step 3) and notification (Step 4) flows to further ensure the security and compliance of the O&M process. Finally, you can distribute tag policies (Step 6) to ensure that ECS instances in business accounts have the required tags for patch O&M.

Product costs and terms

Product costs

Product Name	Product Description	Product Costs
Resource Directory (RD)	Resource Directory (RD) is a service that allows enterprises to manage the relationships between multiple accounts and resources.	Free. For more information, see Pricing.
Cloud Config	Cloud Config is a resource auditing service. It helps you achieve continuous compliance monitoring for your infrastructure when you have many resources.	Free. For more information, see Pricing.
Resource Access Management (RAM)	Resource Access Management (RAM) is an Alibaba Cloud service that manages user identities and resource access permissions.	Free.
CloudOps Orchestration Service (OOS)	CloudOps Orchestration Service (OOS) is an automated O&M service from Alibaba Cloud that can automatically manage and execute tasks.	Free. For more information, see Pricing.
Alibaba Cloud Resource Orchestration Service (ROS)	Resource Orchestration Service (ROS) helps you simplify the management of cloud computing resources. By following the template specifications that are defined by ROS, you can define the collection of required cloud computing resources and their dependencies. ROS can automatically create and configure all resources to achieve automated deployment and O&M.	Free. For more information, see Pricing.

Glossary

Name	Description
Management account	When an enterprise has multiple Alibaba Cloud accounts, this refers to the administrator account that has permission to manage resources in other accounts. It is used to manage multiple accounts, uniformly configure identity and permissions for multiple accounts, view bills for all cloud accounts, and uniformly configure and distribute audit rules to member accounts.
Shared service account	Enterprise shared services, such as network deployments, are deployed in this account. We recommend that the costs for this account be borne by a unified team, such as the infrastructure team.
ROS template	A template is a text file in JSON, YAML, or Terraform format, encoded in UTF-8. You can save these files with specified extensions, such as .json, .yaml, .template, .txt, or .tf. Templates are used to create stacks and serve as blueprints for your infrastructure and architecture. Template editors define Alibaba Cloud resources, configuration details, and resource dependencies in the template.
Stack	A stack is a management unit for ROS resources. You can create, update, and delete a set of resources by creating, updating, and deleting stacks.
Stack group	A stack group is a management unit for ROS stacks. Using a stack group, you can use one ROS template to create stacks across multiple Alibaba Cloud accounts and regions. All resources in each stack are defined by the ROS template of the stack group.
OOS template	A file in YAML or JSON format that includes one or more tasks.

Security

Resource orchestration security

ROS is integrated with ActionTrail. You can query management events that are generated by user operations on ROS in ActionTrail. ActionTrail supports the delivery of management events to a Logstore in SLS or a bucket in OSS. This meets the requirements for real-time auditing and issue analysis.

ROS also supports compliance checks for ROS templates before resource creation. For a list of Alibaba Cloud services that support compliance dry runs, see List of managed rules.

Notes

Patch baseline limits

The OOS patch management service provides a predefined default patch baseline for each operating system. You can set a custom patch baseline as the default for an operating system to apply custom patch scanning and installation rules. Note the following limits when you use patch baselines:

Each operating system can have only one default patch baseline.
Patch scanning and installation are performed strictly according to the rules defined in the default patch baseline. You cannot specify other non-default patch baselines.

Automation templates

Template introduction

This solution provides automation templates based on Terraform and ROS. The code structure of the templates is as follows.

├── step1-preparation						       // Creates the required roles and permissions in the shared service account and delegates the ROS administrator to the shared service account.
└── step2-create-cross-account-role    // Uses an ROS template to create a stack group for ROS in the shared service account. This creates the required roles for all business accounts that need unified O&M across accounts.
└── step3-distribute-patch-baseline	   // Uses an ROS template to create a stack for ROS in the shared service account to uniformly distribute patch baselines across accounts.
└── step4-automation-deployment	       // Creates a custom OOS task template in the shared service account.

Template address

For the template location, see the code repository.

Procedure

Preparations

Ensure that you have enabled a resource directory. For more information, see Enable a resource directory.
Ensure that you have enabled the control policy feature in the management account. For more information, see the Resource Directory topic Enable the control policy feature.
Ensure that you have created the patch baselines to be distributed to target accounts in the shared service account. For more information, see Patch baselines.

Implementation duration

After the preparations are complete, the estimated implementation time for this solution is 60 minutes.

Procedure

Prerequisites

Create a delegated administrator account

Using a delegated administrator account lets you separate organization management tasks from business management tasks. The management account performs organization management tasks for the resource directory, while the delegated administrator account performs business management tasks for ROS. This aligns with security best practices. You can set the shared service account or operations account of the resource directory as the delegated administrator account. This account acts as the ROS administrator to deploy stacks for member accounts.

Log on to the Resource Management console using the management account.
In the navigation pane on the left, choose Resource Directory > Trusted Services.
On the Trusted Services page, click Manage in the Actions column for the Resource Orchestration Service (ROS) service.
In the Delegated Administrator Account section, click Add.
In the Add Delegated Administrator Account panel, select the member account (shared service account).
Click OK. After the account is added, you can use this delegated administrator account to access the multi-account management module of the corresponding trusted service to perform management operations within the scope of the resource directory.

Enable trusted access

After you enable a resource directory, you must enable trusted access to grant service-managed permissions when you use the management account or a delegated administrator account of the resource directory to create a stack group in service-managed permission mode. Trusted access must be enabled for both the management account and the delegated administrator account.

Log on to the ROS console separately using the management account and the shared service account (delegated administrator).
In the navigation pane on the left, click Stack Groups.
On the Stack Groups page, click Enable Trusted Access in the upper-right corner.

Resource and O&M planning (Optional)

The entire patch O&M process requires participation from both business and O&M teams. Each business application is usually deployed in multiple environments, such as development, staging (for pre-release acceptance), and production. The business and O&M teams need to plan O&M windows for patches in each environment. This ensures that when a patch is applied to the production environment, the patch has already been tested in the development and staging environments and is confirmed to not adversely affect the production application. Therefore, proper resource planning and strict division of application environments are fundamental to the entire patch O&M process. This solution uses tags to divide resources into different environments. The following flow provides an example of how to plan O&M windows and configure tags for a business application deployed in multiple environments.

The O&M and business teams must jointly plan O&M time windows for resources, such as ECS instances, in different environments and configure tags for resource grouping on these ECS instances. Ensure that resources within the same tag group have the same O&M window. If you have ECS instances that use different operating systems, we recommend that you configure corresponding tags based on the operating system. This helps ensure that the correct patch baseline is deployed to the correct ECS instances.
Using the patch management service of OOS, you can scan or install patches on ECS instances based on default or custom patch baselines. The patch management service provides a predefined default patch baseline for each operating system. A patch baseline defines the rules for patches that you install on an instance and can specify the following:
1. The product type of the operating system
2. The classification of the patch
3. The severity level of the patch
4. Conditions for automatic approval
You can customize a patch baseline and set it as the default for an operating system to apply custom patch scanning and installation rules.
When you need to apply a new patch or upgrade an existing patch, you must first apply it to the development environment. After patching is complete, the business team must test the application and verify that it works as expected. If the application encounters any issues during the patching process, the business team must request the O&M team to pause the current patching work.
After the patch is successfully applied in the development environment, you can deploy it to other non-production environments. Similar to the development environment, the applications in these environments must be tested and validated.
After the patch is successfully applied in all non-production environments, you can apply it to the production environment.

For more information about resource planning, see Manage cloud resources in a single account using resource groups and tags.

Continuous compliance audit (Optional)

Resource planning audit

In cloud IT management, IT resources on the cloud must have one or more specific tags. These tags are used to manage resources in scenarios such as permission isolation, bill splitting, and automated O&M. You can use the Cloud Config service to continuously audit resource compliance:

Log on to the Cloud Config console using the management account.
In the navigation pane on the left, click Compliance Audit > Rules > New Rule.
Select Create from Template, search for "tag", and create rules such as "has-at-least-one-tag", "has-all-tags", or "matches-tags" to audit the tagging status of resources for compliance.

Patch baseline inspection

If you want to set patch baselines for ECS instances across multiple accounts and continuously scan them to view instances with missing patches and their reports, see Use Cloud Config and Function Compute to implement risk inspection for cloud hosts.

Configure tag policies (Optional)

A tag policy is a type of policy that helps enterprises implement tag standardization. With tag policies, enterprises can enforce compliant tags that must be attached to resources. Compliant tags improve management efficiency in scenarios such as tag-based billing, tag-based authorization, and automated O&M. Tag policies support tag policies for the current account and for the resource directory, which meets enterprise needs for tag standardization and control at different stages.

Problems such as forgetting to attach tags after you create a resource, attaching only some of the required tags, or misspelling tags can occur. These issues can cause resources to be excluded from automated O&M processes. Tag policies provide solutions for the following scenarios:

Automatic tag detection: After a resource is created, you can use tag policies to monitor resource changes and promptly identify resources that have non-compliant tags.
Automatic tag remediation: Based on the results of automatic tag detection, if your rule meets the conditions for automatic remediation and you have enabled this feature, the tag policy automatically fixes the tags of non-compliant resources without manual intervention.
Pre-emptive blocking of non-compliant tags: You can use tag policies to enforce the attachment of compliant tags when resources are created and to block non-compliant tags.
Automatic inheritance of resource group tags to resources within the group: After you attach tags to a resource group, when you create a resource in or add a resource to the group, the resource automatically inherits the tags of the resource group.

To learn how to enable and create tag policies, see Quick start for tag policies.

Create required roles

Create an automation O&M role

Create a RAM role that OOS automation tasks use to run automated O&M tasks.

Log on to the RAM console using the shared service account.
In the navigation pane on the left, choose Identities > Roles.
On the Roles page, click Create Role.
On the Create Role page, set Trusted Entity Type to Alibaba Cloud Service, and then click Next.
Set Role Type to Service Role. Enter a role name, such as EcsPatchingAutomationTriggerRole. Set Trusted Service to CloudOps Orchestration Service. Click Complete.

Next, you must grant permissions to this role. OOS assumes this role to trigger the execution of the automation O&M template.

Log on to the RAM console using the shared service account.
In the navigation pane, choose Policies and click Create Policy.
On the Script tab, replace the existing script in the configuration box with the following content. Note: Replace ${oos_cross_account_assume_role} with your actual value, such as EcsPatchingAutomationRole. This policy allows the automation O&M role (EcsPatchingAutomationTriggerRole) that you created in the previous step to assume the patch baseline execution role (created in the Create patch baseline execution roles in batches section) across accounts to execute the patch baseline.
```
  {
    "Version": "1",
    "Statement": [
      {
        "Action": [
          "sts:AssumeRole"
        ],
        "Resource": "acs:ram:*:*:role/${oos_cross_account_assume_role}",
        "Effect": "Allow"
      }
    ]
  }
```
Click Continue to edit basic information and enter a name for the custom policy, such as EcsPatchingAutomationTriggerRolePolicy. Click OK to complete the creation.
After the policy is created, go to the Roles page, find the role that you created in the previous step, and click Grant Permission.
Set Authorization Scope to Alibaba Cloud Account. The permission takes effect within the current Alibaba Cloud account. Set Principal to EcsPatchingAutomationTriggerRole, which is the role that you created in the previous step. For Custom Policy, select EcsPatchingAutomationTriggerRolePolicy, which is the policy that you created in the previous step.
Click OK to complete the creation.

Create patch baseline execution roles in batches

Use an ROS template to create a stack group. This stack group creates RAM roles in batches for business accounts within your resource directory. During patch deployment, an OOS automation task assumes this role through chained role assumption to apply patches to the corresponding ECS instances in the business accounts across accounts.

Log on to the ROS console using the shared service account.
In the navigation pane on the left, click Stack Groups.
From the region drop-down list in the top menu bar, select the region where you want to create the stack group.
On the Stack Groups page, click Create Stack Group.
In the Select Template section, select Select an Existing Template.
Set Template Source to Enter Template and Template Content to ROS > YAML.
Copy the content of this ROS template into the text box, and then click Next.
Enter a Stack Group Name and Stack Group Description. On the Configure Parameters page, you can keep the default parameter values or modify them as needed, and then click Next.
On the Configure Stack Group page, select Service-managed permissions, and then click Next.
On the Set Deployment Options page, set Deployment Targets to the resource folders of the target accounts in the resource directory, and enable Automatic deployment.
Configure parameters such as Regions, Resource Group, Tags, and Deployment Options as needed. For more information, see Parameter description. Then, click Next.
On the Review page, review the stack group information. If the information is correct, click Create Stack Group.
After the stack group is created, you can view the status of the instances on the Instances tab of the stack group details page. A status of Latest indicates that the roles are created in the business accounts.

Distribute patch baselines

Log on to the ROS console using the shared service account.
In the navigation pane on the left, click Stacks.
From the region drop-down list in the top menu bar, select the region where you want to create the stack.
On the Stacks page, click Create Stack and select Use ROS.
In the Select Template section, select Select an Existing Template.
Set Template Source to Enter Template and Template Content to ROS > YAML.
Copy the content of this ROS template into the text box, and then click Next.

Enter a Stack Name. In the Configure Parameters section, configure the required parameters as needed, and then click Next.

Parameter Name	Example Value	Description
Stack Group Name	distribute-patch-baseline	The name of the stack group to be created for distributing patch baselines.
Patch Baseline to Distribute	ACS-CentOS-DefaultPatchBaseline	The patch baseline to distribute. Enter the name of the patch baseline in OOS of the shared service account. You can view it in Patch Management.
Destination RD Folders	["fd-1xxx", "fd-2xxx"]	Select the destination RD folders. The patch baseline will be distributed to all accounts in these RD folders.
Destination Regions	["cn-hangzhou", "cn-shanghai"]	The regions where the patch baseline is distributed.
Automatic Deployment	true	Automatically creates or deletes patch baselines when members are added to or removed from the RD folders.

On the Configure Stack page, configure the parameters as needed. For more information, see Parameter description.
Click Create. After the stack is created successfully, the Status column shows Create Succeeded.

Unified patch O&M

Deploy the patch automation O&M template

This step uses Terraform code for automated deployment. Go to the source code repository and obtain the source code from the `step4-automation-deployment` directory. Create a `tfvars.json` file, configure the parameters, and then run the terraform apply -var-file=tfvars.json command to deploy the template.

Ensure that your runtime environment can run Terraform.
Run the command using the identity of the shared service account.

The parameter settings are as follows:

Parameter Name	Example Value	Description
region	cn-shanghai	The region where the automation O&M template is deployed.
oss_assume_role	EcsPatchingAutomationTriggerRole	The role created in the shared service account in Step 1. OOS assumes this role to trigger the automation O&M template.
oos_cross_account_assume_role	EcsPatchingAutomationRole	The role created in other business accounts that require unified O&M in Step 2. OOS assumes this role to perform cross-account patch O&M.
approverRamUserName	approver	The RAM user who is allowed to approve the O&M flow. You can also grant this user the AliyunOOSFullAccess permission. This RAM user can approve or deny the O&M flow. You can modify this parameter when you trigger the O&M flow.
approverWebHookUrl	A valid webhook URL	When manual approval is required, a notification message is sent through this webhook. You can modify this parameter when you trigger the O&M flow.
patchingWebHookUrl	A valid webhook URL	When the O&M flow is running, a notification message is sent through this webhook. You can use this webhook to send notifications to the business team.

Alternatively, you can use ROS for deployment:

Log on to the ROS console using the shared service account.
In the navigation pane on the left, click Stacks.
From the region drop-down list in the top menu bar, select the region where the stack is located.
On the Stacks page, click Create Stack and select Use Terraform.
In the Select Template section, select Select an Existing Template.
Set Template Source to Enter Template and Template Content to Terraform.
Copy the content of the Terraform file into the text box.
In the directory bar, click +, click Create File, enter the file name variable.tf, copy the content of the Terraform file into the text box, and then click Next.
Enter a Stack Name. On the Configure Parameters page, configure the required parameters as needed.
Click Create. After the stack is created successfully, the Status column shows Create Succeeded.

Execute the patch O&M flow

Log on to the OOS console using the shared service account.
In the navigation pane on the left, click Custom Task Templates.
From the region drop-down list in the top menu bar, select the region where the template is located. You can see the task template that is created in the Deploy the patch automation O&M template step. Click Create Execution.
On the Basic Information page, set Execution Mode to Pause on Failure, and then click Next.
Follow the prompts to configure the required parameters, and then click Next.
Review the information. If the information is correct, click Create to create the automated O&M task for operating system patches. The process then proceeds with manual approval, sends patch O&M notifications, and executes patch O&M operations.

The following figure shows the architecture of the entire automation flow.

You can trigger the installation flow in several ways: manually, on a schedule, or using an API in your CI/CD pipeline. The automation flow works as follows.

When a patch installation task is triggered, an approval notification is sent using a webhook (Step 1 in the figure). The approver receives the notification and can approve or deny the patch installation flow.
After approval, a patch installation notification is sent to the corresponding business team using a webhook (Step 2). The business team must promptly test the application and verify that it works as expected. If the application encounters any issues during the patching process, the business team must request the O&M team to pause the current patching work.
By assuming the RAM role in the business account, a Cloud Assistant command is executed across accounts. Cloud Assistant installs or updates patches for the specified ECS instances in batches based on the patch baseline definition (Step 3). When you execute the command in batches, we recommend that the O&M team choose a concurrency control mode that pauses between batches. This allows the business team to test and validate the application in a timely manner.

Update patch baselines (Optional)

After a patch baseline is distributed, you must create a new baseline and distribute it again to overwrite the old one in each business account if you want to modify its configuration. After the new baseline is distributed, you can delete the old one.

For example, a new patch baseline named `centos-default-update` is created in the shared service account and needs to replace the previously distributed patch baseline.

Follow the steps in Distribute patch baselines. Note that you must set a unique Stack Group Name in the Configure Parameters section. After the stack group is created, the new patch baseline is distributed and set as the default. Then, on the Stacks page, you can delete the stack that was created to distribute the previous patch baseline.

Log on to the OOS console using a target account, select Patch Management, and then choose the region. You can see that the patch baseline is updated.

Troubleshooting

Why can't I enable a resource directory?

There are two possible reasons:

The current account has not completed enterprise identity verification. For more information about enterprise identity verification, see Enterprise identity verification.
The current account is already in a resource directory and cannot be enabled again.

Why can't I use the service-managed permission model in ROS?

If you use a member account of a resource directory to perform business management tasks for ROS, you must set the member account as a delegated administrator account. This account acts as the ROS administrator to deploy stacks for other member accounts. For more information about ROS delegated administrators, see Set a delegated administrator account.
After you enable a resource directory, you must enable trusted access to grant service-managed permissions when you use the management account or a delegated administrator account of the resource directory to create a stack group in service-managed permission mode. For more information about trusted access, see Enable trusted access.

Why does the OOS O&M task fail?

For information about common issues with OOS, see OOS Service Support.