Use the cluster API server auditing feature

Audit center overview

The audit center overview report displays a general summary of events in an ACK cluster and provides details about important events, such as RAM user operations, public network access, command executions, resource deletions, Secret access, and Kubernetes CVE security risks.

Resource operation overview

The resource operation overview report shows statistics for operations such as create, update, delete, and access on common ACK cluster compute, network, and storage resources, including:

• Compute resources: Deployments, StatefulSets, Jobs, CronJobs, Pods, and DaemonSets.

• Network resources: Services and Ingresses.

• Storage resources: ConfigMaps, Secrets, and PersistentVolumeClaims.

• Access control resources: Roles, ClusterRoles, RoleBindings, and ClusterRoleBindings.

Resource operation details

This report displays a detailed list of operations for a specific resource type in an ACK cluster. Select or enter a resource type to run a real-time query. The report shows the total number of events for each operation type, the distribution across namespaces, the success rate, time-series trends, and a detailed list of operations.

Kubernetes CVE security risks

This report displays potential Kubernetes CVE security risks in the current cluster. Select or enter a RAM user ID to perform a real-time query. The report shows the Kubernetes CVE security risks associated with the specified account. For more information about CVE details and solutions, see [CVE Security] Vulnerability Fix Announcement.

Example 1: Alert on command execution in a container

A company has strict Kubernetes cluster usage policies and does not allow users to log on to containers or execute commands in them. If a user executes a command, an alert is sent immediately. The alert notification must include information such as the specific container accessed, the command executed, the operator, the event ID, the timestamp, and the source IP address.

• Query statement:

  verb : create and objectRef.subresource:exec and stage:  ResponseStarted | SELECT auditID as "Event ID", date_format(from_unixtime(__time__), '%Y-%m-%d %T' ) as "Operation Time",  regexp_extract("requestURI", '([^\?]*)/exec\?.*', 1)as "Resource",  regexp_extract("requestURI", '\?(.*)', 1)as "Command" ,"responseStatus.code" as "Status Code",
   CASE 
   WHEN "user.username" != 'kubernetes-admin' then "user.username"
   WHEN "user.username" = 'kubernetes-admin' and regexp_like("annotations.authorization.k8s.io/reason", 'RoleBinding') then regexp_extract("annotations.authorization.k8s.io/reason", ' to User "(\w+)"', 1)
   ELSE 'kubernetes-admin' END  
   as "Operator Account", 
  CASE WHEN json_array_length(sourceIPs) = 1 then json_format(json_array_get(sourceIPs, 0)) ELSE  sourceIPs END
  as "Source Address" order by "Operation Time" desc  limit 10000

• Trigger condition: "Event ID" =~ ".*".

Example 2: Alert on failed public API server access

A cluster has public access enabled. To prevent malicious attacks, you need to monitor the number of public access attempts and the failure rate. When the number of access attempts reaches a specified threshold (for example, 10) and the failure rate exceeds a specified threshold (for example, 50%), an alert is sent immediately. The alert notification must include information such as the region of the user's IP address, the source IP address, and whether it is a high-risk IP address.

• Query statement:

  * | select ip as "Source Address", total as "Access Count", round(rate * 100, 2) as "Failure Rate %", failCount as "Illegal Access Count", CASE when security_check_ip(ip) = 1 then 'yes' else 'no' end  as "Is High-Risk IP",  ip_to_country(ip) as "Country", ip_to_province(ip) as "Province", ip_to_city(ip) as "City", ip_to_provider(ip) as "Carrier" from (select CASE WHEN json_array_length(sourceIPs) = 1 then json_format(json_array_get(sourceIPs, 0)) ELSE  sourceIPs END
  as ip, count(1) as total,
  sum(CASE WHEN "responseStatus.code" < 400 then 0 
  ELSE 1 END) * 1.0 / count(1) as rate,
  count_if("responseStatus.code" = 403) as failCount
  from log  group by ip limit 10000) where ip_to_domain(ip) != 'intranet' and ip not LIKE '%,%' and not try(is_subnet_of('<your_subnet_ip_address>')) ORDER by "Access Count" desc limit 10000

• Trigger condition: "Source Address" =~ ".*".

Change SLS project

To migrate API server audit logs to another SLS project, use the Change Log Service Project feature in cluster auditing.

1. Log on to the ACS console. In the left navigation pane, click Clusters.

2. On the Clusters page, click the name of the target cluster. In the left navigation pane, choose Security > Audit.

3. In the upper-right corner of the cluster auditing page, click Change Log Service Project to migrate the cluster audit logs to another SLS project.

Disable API server auditing

To disable the API server auditing feature, use the following method.

1. Log on to the ACS console. In the left navigation pane, click Clusters.

2. On the Clusters page, click the name of the target cluster. In the left navigation pane, choose Security > Audit.

3. In the upper-right corner of the Cluster Auditing page, click Disable Cluster Auditing to disable the auditing feature for the current cluster.

Use a third-party logging solution in an ACK cluster

We recommend that you use Log Service (SLS) to record ACK cluster audit logs. However, if you want to use a third-party logging service, you can choose not to use SLS when you create the cluster and then integrate another logging solution to collect and retrieve audit logs.

Audit policy

The event log collection rules vary by audit level.

Save the following audit policy as a YAML file and apply it to the API server by using the --audit-policy-file startup flag.

apiVersion: audit.k8s.io/v1 # Required. Use audit.k8s.io/v1 for clusters that run Kubernetes v1.24 or later, and audit.k8s.io/v1beta1 for clusters that run earlier versions.
kind: Policy
# Do not generate audit events for requests at the RequestReceived stage.
omitStages:
  - "RequestReceived"
rules:
  # The following types of requests are frequent and have low potential risks. We recommend that you set the level to None to skip auditing.
  - level: None
    users: ["system:kube-proxy"]
    verbs: ["watch"]
    resources:
      - group: "" # core
        resources: ["endpoints", "services"]
  - level: None
    users: ["system:unsecured"]
    namespaces: ["kube-system"]
    verbs: ["get"]
    resources:
      - group: "" # core
        resources: ["configmaps"]
  - level: None
    users: ["kubelet"] # legacy kubelet identity
    verbs: ["get"]
    resources:
      - group: "" # core
        resources: ["nodes"]
  - level: None
    userGroups: ["system:nodes"]
    verbs: ["get"]
    resources:
      - group: "" # core
        resources: ["nodes"]
  - level: None
    users:
      - system:kube-controller-manager
      - system:kube-scheduler
      - system:serviceaccount:kube-system:endpoint-controller
    verbs: ["get", "update"]
    namespaces: ["kube-system"]
    resources:
      - group: "" # core
        resources: ["endpoints"]
  - level: None
    users: ["system:apiserver"]
    verbs: ["get"]
    resources:
      - group: "" # core
        resources: ["namespaces"]
  # For read-only URLs, such as /healthz*, /version*, and /swagger*, set the level to None to skip auditing.
  - level: None
    nonResourceURLs:
      - /healthz*
      - /version
      - /swagger*
  # Set the level to None for events to skip auditing.
  - level: None
    resources: 
      - group: "" # core
        resources: ["events"]
  # For interfaces such as Secrets, ConfigMaps, and TokenReviews that may contain sensitive information or binary files, set the level to Metadata.
  - level: Metadata
    resources:
      - group: "" # core
        resources: ["secrets", "configmaps"]
      - group: authentication.k8s.io
        resources: ["tokenreviews"]
  # Requests may return large amounts of data. Set the level to Request to not collect the response body.
  - level: Request
    verbs: ["get", "list", "watch"]
    resources:
      - group: "" # core
      - group: "admissionregistration.k8s.io"
      - group: "apps"
      - group: "authentication.k8s.io"
      - group: "authorization.k8s.io"
      - group: "autoscaling"
      - group: "batch"
      - group: "certificates.k8s.io"
      - group: "extensions"
      - group: "networking.k8s.io"
      - group: "policy"
      - group: "rbac.authorization.k8s.io"
      - group: "settings.k8s.io"
      - group: "storage.k8s.io"
  # For known Kubernetes APIs, the level is set to RequestResponse by default to return the request and response bodies.
  - level: RequestResponse
    resources:
      - group: "" # core
      - group: "admissionregistration.k8s.io"
      - group: "apps"
      - group: "authentication.k8s.io"
      - group: "authorization.k8s.io"
      - group: "autoscaling"
      - group: "batch"
      - group: "certificates.k8s.io"
      - group: "extensions"
      - group: "networking.k8s.io"
      - group: "policy"
      - group: "rbac.authorization.k8s.io"
      - group: "settings.k8s.io"
      - group: "storage.k8s.io"
  # For all other requests, the level is set to Metadata by default.
  - level: Metadata

Note

After a request is received, logging does not start immediately. It starts after the response header is sent.

The system does not audit a large number of redundant requests, such as kube-proxy watch requests, GET requests for nodes from kubelet and system:nodes, operations on endpoints by kube-system components, and GET requests for namespaces from the API server.

For sensitive interfaces such as authentication, rbac, certificates, autoscaling, and storage, the system records the corresponding request and response bodies for read and write operations.

Audit backend

The audit backend stores collected events as log files in standard JSON format.