This document describes how to use CloudOps Orchestration Service (OOS) to automate the O&M of host system and software configurations in a multi-account environment.-Well-Architected Framework(WAF)-阿里云帮助中心

Overview

Enterprises often have compliance requirements for IT assets. These requirements state that Elastic Compute Service (ECS) instances must maintain a consistent system and software configuration baseline. This baseline includes settings such as host kernel parameters, host time zones, and software versions.

This document describes a solution for enterprise customers to automate the operations and maintenance (O&M) of host system and software configurations on the cloud. This solution enables centralized scanning, monitoring, and remediation of host configurations in a multi-account environment.

Benefits

Centralized O&M for system and software configurations

You can define CloudOps Orchestration Service (OOS) automation flows in a shared service account or an operations account. These flows automatically run custom commands in batches on specified ECS instances across all your accounts to modify system and software configurations. This achieves automated and centralized host configuration O&M. The process also includes manual approval and notification steps to ensure the O&M process is secure and compliant.

Improved business security and compliance

You can use CloudConfig to continuously audit and ensure the compliance of host system and software configurations. This lets you promptly identify and manage ECS instances that do not meet your enterprise configuration baseline. This process ensures that all your business ECS instances comply with the enterprise baseline and prevents security risks that arise from outdated configurations.

Customer scenarios

Unified internal O&M for enterprises

Scenario description

Enterprises need to maintain a unified configuration baseline for host systems and software. This practice ensures that all host instances have a consistent configuration baseline and allows for unified monitoring and remediation of host configurations.

Applicable customers

Enterprise customers who use Resource Directory to manage multiple cloud accounts.
Enterprise customers who use multiple Alibaba Cloud accounts.
Enterprise customers that have a central O&M team or security team.
You have multiple ECS instances in your Alibaba Cloud account.

Solution architecture

This solution uses a multi-account structure to automate host system and software configuration O&M. O&M tasks are run in a shared service account or an operations account. You can use CloudConfig to continuously audit the configuration compliance of all host instances in your business accounts (1 in the figure). You can also send audit results as notifications through EventBridge or integrate them into a monitoring system using Simple Log Service (SLS). If a host configuration needs to be fixed, you can trigger an automation task in OOS (4 in the figure). During execution, you can add manual approval (2 in the figure) and notification (3 in the figure) steps to make the O&M process more secure and compliant. Finally, you can apply a tag policy (5 in the figure) to ensure that ECS host instances in business accounts have the required tags for configuration O&M.

Products and pricing

Pricing

Product Name	Description	Pricing
Resource Directory (RD)	Resource Directory (RD) is a service that provides multi-level account and resource relationship management for enterprise customers.	Free. For more information, see Pricing.
CloudConfig	CloudConfig is a resource-oriented audit service. If you have many resources, CloudConfig helps you achieve continuous compliance monitoring of your infrastructure.	Free. For more information, see Pricing.
Resource Access Management (RAM)	Resource Access Management (RAM) is an Alibaba Cloud service for managing user identities and resource access permissions.	Free.
CloudOps Orchestration Service (OOS)	CloudOps Orchestration Service (OOS) is an automated O&M service provided by Alibaba Cloud that can automatically manage and execute tasks.	Free. For more information, see Pricing.
Alibaba Cloud Resource Orchestration Service (ROS)	Resource Orchestration Service (ROS) helps you simplify the management of cloud computing resources. By following the template specifications that are defined by ROS, you can define the collection of required cloud computing resources and the dependencies between them. ROS can automatically create and configure all resources to achieve automated deployment and O&M.	Free. For more information, see Pricing.

Glossary

Name	Description
Management account	When an enterprise has multiple Alibaba Cloud accounts, this refers to the administrator account that has permission to manage resources in other accounts. It is used to manage multiple accounts, centrally configure identity and permissions for multiple accounts, view bills for all cloud accounts, and centrally configure audit rules to apply to member accounts.
Shared service account	Enterprise shared services, such as network deployments, are deployed in this account. We recommend that the costs for this account be borne by a single team, such as the infrastructure team.
ROS template	A template is a text file in JSON, YAML, or Terraform format that uses UTF-8 encoding. You can save these files with a specified extension, such as .json, .yaml, .template, .txt, or .tf. A template is a blueprint for describing your infrastructure and architecture. You use it to create a stack. In the template, you define the Alibaba Cloud resources, configuration details, and dependencies between resources.
Stack	A stack is a management unit for ROS resources. You can create, update, and delete a group of resources by creating, updating, and deleting a stack.
Stack group	A stack group is a management unit for ROS stacks. Using a stack group, you can use one ROS template to create stacks across multiple Alibaba Cloud accounts and regions. All resources in each stack are defined by the ROS template of the stack group.
OOS template	A file in YAML or JSON format that includes one or more tasks.

Security

Resource orchestration security

Resource Orchestration Service (ROS) is integrated with ActionTrail. You can query management events that are generated by user operations on ROS in ActionTrail. ActionTrail can deliver management events to a Logstore in Simple Log Service (SLS) or a bucket in Object Storage Service (OSS). This delivery meets the requirements for real-time auditing and issue analysis.

ROS also supports compliance checks before you create resources. For a list of Alibaba Cloud services that support compliance dry runs, see List of managed rules.

Notes

Differences between Cloud Assistant and local command execution

Running commands with Cloud Assistant differs from running them locally on an ECS instance. The differences are related to supported features, environment variables, and command results. These differences and their solutions vary by operating system. Before you start, you must read and understand the details for:

Automation templates

Template introduction

This solution provides automation templates based on Terraform and Resource Orchestration Service (ROS). The code structure is as follows.

├── step1-preparation						       // Create required roles and permissions in the shared service account and delegate the ROS administrator to the shared service account.
└── step2-create-cross-account-role    // Use an ROS template to create a Resource Orchestration Service (ROS) stack group in the shared service account. This creates the required roles for all business accounts that need unified O&M.
└── step3-automation-deployment	       // Create a custom OOS task template in the shared service account.

Template location

For the template location, see the code repository.

Implementation steps

Preparations

Ensure that you have enabled Resource Directory. For more information, see Enable a resource directory.
Ensure that you have enabled the control policy feature in the management account. For more information, see Enable the control policy feature in the Resource Directory documentation.

Implementation duration

After you complete the preparations, this solution takes about 60 minutes to implement.

Procedure

Prerequisites

Create a delegated administrator account

A delegated administrator account lets you separate organization management tasks from business management tasks. The management account performs organization management tasks for Resource Directory, and the delegated administrator account performs business management tasks for ROS. This approach follows security best practices. You can set the shared service account or operations account in Resource Directory as the delegated administrator account. This account then acts as the ROS administrator account to deploy stacks for member accounts.

Log on to the Resource Management console using your management account.
In the navigation pane on the left, choose Resource Directory > Trusted Services.
On the Trusted Services page, click Manage in the Actions column for the Resource Orchestration Service (ROS) service.
In the Delegated Administrator Account section, click Add.
In the Add Delegated Administrator Account panel, select the member (shared service account).
Click OK. After the account is added, you can use this delegated administrator account to access the multi-account management module of the corresponding trusted service. You can then perform management operations within the scope of the resource directory.

Enable trusted access

If you have enabled Resource Directory, you must enable trusted access when you create a stack group in service-managed permissions mode. This requirement applies whether you use the management account or a delegated administrator account of Resource Directory. Enabling trusted access grants service-managed permissions. Both the management account and the delegated administrator account must enable trusted access.

Log on to the ROS console separately using the management account and the shared service account (delegated administrator).
In the navigation pane on the left, click Stack Groups.
On the Stack Groups page, click Enable Trusted Access in the upper-right corner.

Resource and O&M planning (Optional)

Business and O&M teams must work together on the host configuration O&M process. Each business application is usually deployed in multiple environments, such as development, staging, and production. The teams need to plan an O&M window for host configuration in each environment. This planning ensures that when a command is run in the production environment, it has already been tested in the development and staging environments and is confirmed to not adversely affect the production application. Therefore, proper resource planning and strict separation of application environments are fundamental to the entire host configuration O&M process. This solution uses tags to separate resources by environment. The following flow provides an example of how to plan O&M windows and configure tags for a business application that is deployed in multiple environments.

The O&M team and the business team must work together to plan O&M time windows for resources (ECS instances) in different environments. They also need to configure tags for these ECS instances for resource grouping. Ensure that resources with the same tag group are allowed the same O&M time window. If you have ECS instances with different operating systems, we recommend that you configure corresponding tags based on the operating system. This helps ensure that the correct script commands are deployed to the correct ECS instances.
After you identify non-compliant host configurations through a host configuration inspection (see the Continuous compliance audit section), you can customize remediation script commands based on the operating system of the ECS host. This script command is run on the ECS host instance through Cloud Assistant.
When you need to run a custom script command, you must first run it in the development environment. After execution, the business team needs to test the application and verify that it works correctly. If the application encounters any problems during command execution, the business team must ask the O&M team to pause the current configuration remediation work.
After the command is successfully applied in the development environment, you can run the script command in other non-production environments. Similar to the development environment, the applications in these environments need to be tested and verified.
After the command is successfully executed in all non-production environments, you can apply the script command to the production environment.

For more information about resource planning, see Manage cloud resources in a single account using resource groups and tags.

Continuous compliance audit (Optional)

Resource planning audit

Cloud IT management requires that cloud resources have specific tags. These tags are used for permission isolation, bill splitting, and automated O&M. You can use the CloudConfig service to continuously audit resource compliance.

Log on to the CloudConfig console using your management account.
In the navigation pane on the left, choose Compliance Audit > Rules > New Rule.
Select Create from Template and search for "tag". You can create rules such as "at-least-one-tag-is-attached", "all-specified-tags-are-attached", and "tags-are-matched" to audit the tagging compliance of resources.

Host software configuration inspection

Enterprises must maintain a unified security and compliance baseline for host resources, operating systems, and software configurations. To ensure that all host instances have a consistent configuration baseline, you need to continuously inspect host resources for risks and monitor them from a central dashboard. For more information, see Inspect cloud hosts for risks using CloudConfig and Function Compute.

Configure a tag policy (Optional)

A tag policy helps you standardize tags. You can use tag policies to define the compliant tags that must be attached to resources. Compliant tags improve management efficiency for bill splitting, authorization, and automated O&M. Tag policies can be applied to the current account or across a Resource Directory. This feature meets enterprise needs for tag control at different stages.

Forgetting to attach tags after you create a resource, attaching only some tags, or misspelling tags can cause problems. These issues can prevent resources from being automatically managed during automated O&M. Tag policies provide solutions for the following scenarios:

Automatic tag detection: After a resource is created, you can use a tag policy to monitor resource changes and promptly identify resources with non-compliant tags.
Automatic tag remediation: Based on the results of automatic tag detection, if your rule meets the conditions for automatic remediation and you have enabled this feature, the tag policy automatically fixes the tags of non-compliant resources without manual intervention.
Proactive blocking of non-compliant tags: You can use a tag policy to enforce the attachment of compliant tags when resources are created and block non-compliant tags.
Automatic inheritance of resource group tags to resources within the group: After you attach tags to a resource group, when you create a resource in or add a resource to the resource group, the resource automatically inherits the tags of the resource group.

For more information about how to enable and create a tag policy, see Quick Start for Tag Policy.

Create required roles

Create an automation O&M role

You need to create a RAM role. The OOS automation task uses this role to run automated O&M tasks.

Log on to the RAM console using the shared service account.
In the navigation pane on the left, choose Identity > Roles.
On the Roles page, click Create Role.
On the Create Role page, select Alibaba Cloud Service as the trusted entity type, and then click Next.
Set Role Type to Service Role, enter a role name, such as EcsCommandRunningAutomationTriggerRole, and select CloudOps Orchestration Service as the trusted service. Click OK.

Next, you need to grant permissions to this role. OOS will assume this role to trigger the execution of the automated O&M template.

Log on to the RAM console using the shared service account.
Choose the Policies menu and click Create Policy.
In the Script editor, replace the existing script in the configuration box with the following content. Ensure that you replace ${oos_cross_account_assume_role} with your actual value, such as EcsCommandRunningAutomationRole. This policy allows the automation O&M role (EcsCommandRunningAutomationTriggerRole) that is created in the previous step to assume the script command execution role (created in the Create script command execution roles in batches section) across accounts to execute O&M script commands.
```
  {
    "Version": "1",
    "Statement": [
      {
        "Action": [
          "sts:AssumeRole"
        ],
        "Resource": "acs:ram:*:*:role/${oos_cross_account_assume_role}",
        "Effect": "Allow"
      }
    ]
  }
```
Click Continue to edit basic information and enter a name for the custom permission policy. For example, EcsCommandRunningAutomationTriggerRolePolicy. Click OK to create the policy.
On the Roles page, find the role that you created in the previous step and click the New Authorization button.
Set Grant Scope to Alibaba Cloud Account. The permission takes effect within the current Alibaba Cloud account. Set Principal to EcsCommandRunningAutomationTriggerRole, which is the role that you created in the previous step. For Custom Policy, select EcsCommandRunningAutomationTriggerRolePolicy, which is the policy that you created in the previous step.
Click OK to complete the creation.

Create script command execution roles in batches

You can use an ROS template to create a stack group. This stack group creates the required RAM roles in batches for all business accounts in your resource directory that need unified O&M. OOS assumes this role to run script commands across accounts and complete the configuration O&M.

Log on to the ROS console using the shared service account.
In the navigation pane on the left, click Stack Groups.
From the region drop-down list in the top menu bar, select the region where the stack group is located.
On the Stack Groups page, click Create Stack Group.
In the Select Template section, select Select an Existing Template.
Set Template Source to Enter Template and set Template Content to ROS > YAML.
Copy the text content of this ROS template into the text box, and then click Next.
Enter a Stack Group Name and Stack Group Description. On the Configure Template Parameters page, you can keep the default parameter values or modify them as needed, and then click Next.
On the Configure Stack Group page, select Service-managed permissions, and then click Next.
On the Set Deployment Options page, set Deployment targets to the resource folder of the target account in Resource Directory, and enable Automatic deployment.
Fill in information such as Regions, Resource Group, Tags, and Deployment options as needed. For more information, see Parameter description. Then click Next.
On the Check and Confirm page, check that the stack group information is correct, and then click Create Stack Group.
After the stack group is created, you can view the status of the instances on the Instances tab of the stack group's details page. Latest indicates that the role has been created in the business account.

Unified automated O&M

Deploy the automation O&M template

This step uses Terraform code for automated deployment. Go to the source code repository and obtain the source code from the `step3-automation-deployment` directory. You can create a `tfvars.json` file to configure parameters and then deploy the template by running the terraform apply -var-file=tfvars.json command.

Ensure that your runtime environment can run Terraform.
Run the command using the identity of the shared service account.

The parameter configurations are described as follows:

Parameter	Example Value	Description
region	cn-shanghai	The region where the automation O&M template is deployed.
oss_assume_role	EcsCommandRunningAutomationTriggerRole	The role created in the shared service account in the "Create an automation O&M role" step. OOS assumes this role to trigger the automation O&M template.
oos_cross_account_assume_role	EcsCommandRunningAutomationRole	The role created in other business accounts that require unified O&M in the "Create script command execution roles in batches" step. OOS assumes this role to run script commands across accounts and complete configuration O&M.
approverRamUserName	approver	The RAM user who is allowed to approve the O&M flow. You can also grant the AliyunOOSFullAccess permission to this user. This RAM user can approve or reject the O&M flow. You can change this parameter when you trigger the O&M flow.
approverWebHookUrl	A valid webhook URL	When manual approval is required, a notification is sent through this webhook. You can change this parameter when you trigger the O&M flow.
commandRunningWebHookUrl	A valid webhook URL	When the O&M flow is running, a notification is sent through this webhook. You can use this webhook to notify the business team.

Alternatively, you can deploy using Resource Orchestration Service (ROS):

Log on to the ROS console using the shared service account.
In the navigation pane on the left, click Stacks.
From the region drop-down list in the top menu bar, select the region where the stack is located.
On the Stacks page, click Create Stack and select Use Terraform.
In the Select Template section, select Select an Existing Template.
Set Template Source to Enter Template and set Template Content to Terraform.
Copy the text content of the Terraform file into the text box.
In the directory bar, click +, click Create File, enter the file name variable.tf, copy the text content of the Terraform file into the text box, and then click Next.
Enter a Stack Name. In the Configure Template Parameters section, fill in the required parameters as needed.
Click Create. After the stack is created, the Status column shows Create Succeeded.

Execute the automated O&M flow

Log on to the OOS console using the shared service account.
In the navigation pane on the left, click Custom Task Templates.
From the region drop-down list in the top menu bar, select the region where the template is located. You can see the task template that was created in the Deploy the automation O&M template step. Click Create Execution.
On the Basic Information page, set the execution mode to Suspend on Failure and click Next.
Set the required parameters as prompted, and click Next.
Check that the information is correct and click Create to create the automated O&M task for host system and software configuration. The following steps are performed in sequence: manual approval, sending automated O&M notifications, and executing automated O&M script commands.

The following figure shows the architecture of the entire automation flow.

You can trigger the script command execution flow in several ways: manually, on a schedule, or through an API in your CI/CD pipeline. The automation flow works as follows.

When the script command execution task is triggered, an approval notification is sent via webhook (1 in the figure). The approver receives the notification and can approve or reject the command execution flow.
After approval, a notification about the command execution is sent to the corresponding business team via webhook (2 in the figure). The business team needs to promptly test the application and verify that it is working correctly. If the application encounters any problems during command execution, the business team must ask the O&M team to pause the current execution.
By assuming the RAM role in the business account, Cloud Assistant commands are executed across accounts. Cloud Assistant runs the custom command on the specified ECS instances in batches (3 in the figure). For batch execution, we recommend that the O&M team choose a concurrency control mode that pauses between batches. This approach allows the business team to test and verify the application in a timely manner.

Troubleshooting

Why can't I enable Resource Directory?

There are two possible reasons:

The current account has not completed enterprise identity verification. For more information about enterprise identity verification, see Enterprise real-name registration.
The current account is already in a resource directory and cannot be enabled again.

Why can't I use the service-managed permissions mode of ROS?

If you use a member account of Resource Directory to perform business management tasks for ROS, you need to set the member account as a delegated administrator account. This account then acts as the ROS administrator account to deploy stacks for other member accounts. For more information about ROS delegated administrators, see Create a delegated administrator account.
If you have enabled Resource Directory, you must enable trusted access when you create a stack group in service-managed permissions mode. This requirement applies whether you use the management account or a delegated administrator account of Resource Directory. For more information about trusted access, see Enable trusted access.

Why did my OOS O&M task fail?

For common issues with OOS, see OOS Service Support.