Resource profiling for container configuration recommendations

Step 1: Enable resource profiling

1. Create a file named recommendation-profile.yaml with the following YAML content to enable resource profiling for a workload.

   A RecommendationProfile CRD enables resource profiling for a workload and provides resource specification data for its containers. You can control the profiling scope by specifying namespaces and workload types. The final scope is the intersection of the two.

   apiVersion: autoscaling.alibabacloud.com/v1alpha1
   kind: RecommendationProfile
   metadata:
     # The object name. A namespace is not required for this cluster-scoped object.
     name: profile-demo
   spec:
     # The workload types for which to enable resource profiling.
     controllerKind:
     - Deployment
     # The namespaces for which to enable resource profiling.
     enabledNamespaces:
     - default

   The following table describes the configuration fields.

   Parameter	Type	Description
   metadata.name	String	The name of the object. A namespace is not required because RecommendationProfile is a cluster-scoped (non-namespaced) object.
   spec.controllerKind	String	The workload types for which resource profiling is enabled. Supported workload types include Deployment, StatefulSet, and DaemonSet.
   spec.enabledNamespaces	String	The namespaces for which resource profiling is enabled.

2. Apply the profile configuration.

   kubectl apply -f recommendation-profile.yaml

3. Create a file named cpu-load-gen.yaml with the following content.

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: cpu-load-gen
     labels:
       app: cpu-load-gen
   spec:
     replicas: 2
     selector:
       matchLabels:
         app: cpu-load-gen-selector
     template:
       metadata:
         labels:
           app: cpu-load-gen-selector
       spec:
         containers:
         - name: cpu-load-gen
           image: registry.cn-zhangjiakou.aliyuncs.com/acs/slo-test-cpu-load-gen:v0.1
           command: ["cpu_load_gen.sh"]
           imagePullPolicy: Always
           resources:
             requests:
               cpu: 8 # The CPU request for this application is 8 cores.
               memory: "1Gi"
             limits:
               cpu: 12
               memory: "2Gi"

4. Deploy the cpu-load-gen application.

   kubectl apply -f cpu-load-gen.yaml

5. Get the resource profiling results.

   kubectl get recommendations -l \
     "alpha.alibabacloud.com/recommendation-workload-apiVersion=apps-v1, \
     alpha.alibabacloud.com/recommendation-workload-kind=Deployment, \
     alpha.alibabacloud.com/recommendation-workload-name=cpu-load-gen" -o yaml

   ack-koordinator generates a resource profile for each profiled workload and stores the results in a Recommendation CRD. The following is a sample resource profile for the cpu-load-gen workload.

   apiVersion: autoscaling.alibabacloud.com/v1alpha1
   kind: Recommendation
   metadata:
     labels:
       alpha.alibabacloud.com/recommendation-workload-apiVersion: apps-v1
       alpha.alibabacloud.com/recommendation-workload-kind: Deployment
       alpha.alibabacloud.com/recommendation-workload-name: cpu-load-gen
     name: f20ac0b3-dc7f-4f47-b3d9-bd91f906****
     namespace: recommender-demo
   spec:
     workloadRef:
       apiVersion: apps/v1
       kind: Deployment
       name: cpu-load-gen
   status:
     recommendResources:
       containerRecommendations:
       - containerName: cpu-load-gen
         target:
           cpu: 4742m
           memory: 262144k
         originalTarget: # Intermediate result of the resource profiling algorithm. Do not use directly.
          # ...

   To simplify retrieval, the Recommendation object is created in the same namespace as the workload. It also includes labels that specify the API version, kind, and name of the workload, as described in the following table.

   Label key	Description	Example
   alpha.alibabacloud.com/recommendation-workload-apiVersion	The API version of the workload. The forward slash (/) is replaced with a hyphen (-) to comply with Kubernetes label syntax.	apps-v1 (from apps/v1)
   alpha.alibabacloud.com/recommendation-workload-kind	The type of the workload, such as Deployment or StatefulSet.	Deployment
   alpha.alibabacloud.com/recommendation-workload-name	The name of the workload. It must be no more than 63 characters long to comply with Kubernetes label syntax.	cpu-load-gen

   The resource profiling results for each container are stored in status.recommendResources.containerRecommendations. The following table describes the fields.

   Field	Description	Format	Example
   containerName	The name of the container.	string	cpu-load-gen
   target	The profiled resource specifications, including CPU and memory.	map[ResourceName]resource.Quantity	cpu: 4742m memory: 262144k
   originalTarget	An intermediate result from the profiling algorithm. Do not use this field directly.	-	-

   Note

   The minimum profiled CPU value per pod is 0.025 cores, and the minimum memory value is 250 MB.

   By comparing the declared resource specifications in the cpu-load-gen application with the profiling results, you can see that the CPU request is over-provisioned. You can reduce the request to save cluster resources.

   Category	Original specification	Profiled specification
   CPU	8 cores	4.742 cores

Step 2: (Optional) View the results in Prometheus

Resource profiling algorithm

Suitable application types

Resource profiling is best suited for online service applications.

Currently, the profiling results prioritize ensuring a container has sufficient resources to cover the vast majority of its usage samples. However, this approach can be conservative for certain application types. For offline applications, such as batch processing tasks that prioritize overall throughput and can tolerate some resource contention to improve cluster utilization, the profiling results may appear too conservative. In addition, for critical system components deployed in an active-passive configuration, the passive replicas are idle for long periods, and their low resource usage can interfere with the profiling algorithm. For these scenarios, review and adjust the profiling results as needed before you apply them. We recommend staying up-to-date with product updates for resource profiling.

Use profiled values for requests and limits

This depends on your specific workload. The profiled values provide a summary of your application's current resource demand. You should use them as a baseline and adjust them based on your application's characteristics and business requirements.

For example, for applications that need to handle traffic spikes or require seamless failover in an active-active architecture, you must add a resource buffer. For resource-sensitive applications that do not perform well on hosts with high load, you should also increase the resource allocation beyond the profiled value.

View metrics in self-managed Prometheus

The ack-koord-manager module of the ack-koordinator component exposes resource profiling metrics as a Prometheus-formatted HTTP endpoint. You can get the pod IP address and access the metrics data.

1. Get the pod IP address.

   kubectl get pod -A -o wide | grep koord-manager

   Expected output:

   kube-system    ack-koord-manager-b86bd47d9-92f6m                                 1/1     Running     0               16h     10.10.0.xxx   cn-hangzhou.10.10.0.xxx   <none>           <none>
   kube-system    ack-koord-manager-b86bd47d9-vg5z7                                 1/1     Running     0               16h     10.10.0.xxx   cn-hangzhou.10.10.0.xxx   <none>           <none>

2. Run the following command to view the metrics data (note that ack-koord-manager runs in a dual-replica active-passive mode and data is available only on the primary replica Pod). For the port port (default: 9326), refer to the ack-koord-manager Deployment configuration.

   Ensure that the server on which you run the command can communicate with the cluster's container network.

   curl -s http://10.10.0.xxx:9326/all-metrics | grep slo_manager_recommender_recommendation_workload_target
   # If you use an ack-koordinator version earlier than v1.5.0-ack1.12, run the following command to view the metrics data.
   curl -s http://10.10.0.xxx:9326/metrics | grep slo_manager_recommender_recommendation_workload_target

   Expected output:

   # HELP slo_manager_recommender_recommendation_workload_target Recommendation of workload resource request.
   # TYPE slo_manager_recommender_recommendation_workload_target gauge
   slo_manager_recommender_recommendation_workload_target{container_name="xxx",namespace="xxx",recommendation_name="d2169dbf-fb36-4bf4-99d1-673577fb85c1",resource="cpu",workload_api_version="apps/v1",workload_kind="Deployment",workload_name="xxx"} 0.025
   slo_manager_recommender_recommendation_workload_target{container_name="xxx",namespace="xxx",recommendation_name="d2169dbf-fb36-4bf4-99d1-673577fb85c1",resource="memory",workload_api_version="apps/v1",workload_kind="Deployment",workload_name="xxx"} 2.62144e+08

After the ack-koordinator component is installed, it automatically creates Service and ServiceMonitor objects that are associated with the corresponding pods. If you use Managed Service for Prometheus, the service automatically collects and displays these metrics on the corresponding Grafana dashboard.

Prometheus supports multiple collection methods. If you use a self-managed Prometheus instance, refer to the official Prometheus documentation for configuration and use the process described above for debugging. After debugging, you can refer to Step 2: (Optional) View the results in Prometheus to configure a Grafana dashboard in your environment.

Delete profiling results and rules

Recommendation CRDs store profiling results, and RecommendationProfile CRDs store profiling rules. Run the following commands to delete all results and rules.

# Delete all profiling results.
kubectl delete recommendation -A --all
# Delete all profiling rules.
kubectl delete recommendationprofile -A --all

Grant permissions to RAM users

ACK authorization has two layers: RAM authorization for basic resource access and RBAC (Role-Based Access Control) for permissions within the cluster. For an overview, see Authorization best practices. To grant a RAM user permissions to use resource profiling, you must configure permissions at both levels:

1. RAM authorization

   Log on to the RAM console with your Alibaba Cloud account and grant the AliyunCSFullAccess built-in system policy to the RAM user. For detailed instructions, see Grant permissions.

2. RBAC authorization

   After completing RAM authorization, grant the RAM user the developer role or higher in the target cluster. For instructions, see Use RBAC to authorize operations on cluster resources.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: recommendation-clusterrole
rules:
- apiGroups:
  - "autoscaling.alibabacloud.com"
  resources:
  - "*"
  verbs:
  - "*"