Audit operations inside containers

Updated at:

Audit commands that organization members or applications run inside containers to detect threats before they escalate.kubectl exec or shell access to detect threats before they escalate.

Enable the feature, view audit reports and logs, configure alerts, and disable the feature when no longer needed.

This is a whitelist feature. Submit a ticket to enable access before proceeding.

Billing

The feature itself is free. After enablement, Simple Log Service (SLS) usage fees apply. For pricing details, see Billing overview.

Limitations

Cluster types: ACK managed Pro cluster, ACK managed Basic cluster, and ACK dedicated cluster.

Operating system and kernel: Alibaba Cloud Linux, Ubuntu, or ContainerOS with a kernel version later than 4.19.

OS

Minimum cluster version

Additional requirements

Alibaba Cloud Linux

1.18

—

ContainerOS

1.24

—

Ubuntu

1.30 (see Manually upgrade a cluster)

Automatic OS upgrades must be disabled during node initialization. The /etc/resolv.conf symlink must point to /run/systemd/resolve/stub-resolv.conf, and DNS must be configured by DHCP. Cloud Parallel File System (CPFS) persistent volumes (PVs), image acceleration plugins, and security hardening are not supported.

Step 1: Enable the feature

Enabling the feature installs two add-ons:

  • Log collection add-on — collects audit logs, sends them to SLS, and creates a default audit report. See Log collection component.

  • ack-advanced-audit add-on — audits operations within containers.

A Logstore named advaudit-${cluster_id} is created in the log collection project. Audit logs are retained for 180 days by default. To adjust the retention period, see Manage a Logstore.

To enable the feature:

  1. Log in to the ACK console and click Clusters in the left navigation pane.

  2. Click the name of your cluster. In the left navigation pane, choose Security > Audit.

  3. On the Audit page, click the Container Audit tab and then click Install.

Step 2: View audit reports

On the Audit page, click the Container Audit tab and then click the Container Audit Overview tab.

The overview report shows:

  • Pod access frequency and pod details.Report 1

  • Kubernetes accounts, commands run after access, and common threats.Report 2Threat listReport

Step 3: View detailed log records

Two methods are available depending on your goal:

Method

Best for

Audit report page

Quickly reviewing a recent individual event

Logstore page with search statements

Historical queries and complex filters

View logs from the audit report page

On the Container Audit Overview tab, scroll to the Suspicious Process Event List section. Click a link in the traceId or eventId column:

  • traceId — shows all commands run after a container was accessed.

  • eventId — shows the details of one specific command.

View logs from the Logstore page

  1. Log in to the ACK console and click Clusters in the left navigation pane.

  2. Click your cluster name. In the left navigation pane, choose Security > Audit.

  3. On the Audit page, click the Container Audit tab and then click the Query Container Audit Logs tab.

  4. Enter a search statement in the search box. Query by pod — replace <namespace> and <pod_name> with actual values:

    Category

    Field

    Description

    Pod

    k8s.pod.namespace

    Namespace of the pod

    Pod

    k8s.pod.name

    Name of the pod

    Process

    process.name

    Name of the process that was executed

    * and k8s.pod.namespace: <namespace> and k8s.pod.name: <pod_name>

    Query by process name — replace <name> with the program name:

    * and process.name: <name>

    The table above lists key query fields. For all supported methods, see Log search and analysis.

  5. Set a time range and click Search & Analyze.

(Optional) Step 4: Configure alerts

Use the SLS alerting feature to monitor container operations in real time. Supported alert channels include SMS messages, DingTalk chatbot, emails, custom webhooks, and the notification center.

This topic uses DingTalk alerts as an example to describe how to configure container internal operation audit alerts. For other alert configuration methods, see Alerting.

  1. Log on to the Simple Log Service console.

  2. Go to the Webhook Integration page.

    1. In the Projects section, click the project you want.

    2. In the left-side navigation pane, click Alerts.

    3. On the Alert Center page, click the Notification Objects tab and then click Webhook Integration.

  3. Create a webhook.

    1. On the Webhook Integration page, click Create.

    2. In the Create Webhook dialog box, configure the following parameters and click OK.

      Parameter

      Description

      ID

      The webhook ID. Must be unique. This example uses ack-container-operation-audit-alert.

      Name

      The name of the webhook. This example uses Kubernetes intra-container operation audit alert.

      Type

      The webhook type. This example selects DingTalk.

      Request URL

      The webhook URL.

      Create a custom chatbot in DingTalk and obtain its webhook URL. See Custom robot access.

  4. Create an alert template.

    1. On the Alert Center page, click the Notification Management tab and then click Alert Template.

    2. On the Alert Template page, click Create.

    3. Configure the ID and name for the alert template. Then, configure the title and content for the DingTalk alert notification.

      See Create an alert template. Configure two templates as follows.

      ID

      Name

      Title

      Content

      ack-container-operation-audit-alert-enter

      Kubernetes container access alert

      Kubernetes intra-container operation audit alert

      - Alert Rule: ${alert_name}
      - Trigger Time: ${alert_time}
      - Severity: ${severity}
      - Cluster ID: {{ alert.results[0].fire_result['clusterid'] }}
      
      An account accessed a container by using kubectl exec or the terminal feature. Check for any abnormal activities.
      Number of operations: {{ alert.fire_results_count }}. The following provides details about one of the operations:
      
      - Operation Time: {{ alert.results[0].fire_result['time'] }}
      {% if alert.results[0].fire_result['k8s.user.aliuid'] -%}
      - Account: {{ alert.results[0].fire_result['k8s.user.aliuid'] }}
      {% else -%}
      - Account: {{ alert.results[0].fire_result['k8s.user.username'] }}
      {% endif -%}
      - Namespace: {{ alert.results[0].fire_result['k8s.pod.namespace'] }}
      - Pod: {{ alert.results[0].fire_result['k8s.pod.name'] }}
      - Container: {{ alert.results[0].fire_result['kubeobject.operation.podexecoptions.container'] }}
      - Command: {{ alert.results[0].fire_result['kubeobject.operation.podexecoptions.commandstr'] }}
      
      [[Details](${query_url})]
      [[Settings](${alert_url})]

      ack-container-operation-audit-alert-run-danger-cmd

      Kubernetes alert for running risky programs after container access

      Kubernetes intra-container operation audit alert

      - Alert Rule: ${alert_name}
      - Trigger Time: ${alert_time}
      - Severity: ${severity}
      - Cluster ID: {{ alert.results[0].fire_result['clusterid'] }}
      
      An account ran a risky program after accessing a container by using kubectl exec or the terminal feature. Check for any abnormal activities.
      Number of operations: {{ alert.fire_results_count }}. The following provides details about one of the operations:
      
      - Operation Time: {{ alert.results[0].fire_result['time'] }}
      {% if alert.results[0].fire_result['k8s.user.aliuid'] -%}
      - Account: {{ alert.results[0].fire_result['k8s.user.aliuid'] }}
      {% else -%}
      - Account: {{ alert.results[0].fire_result['k8s.user.username'] }}
      {% endif -%}
      - Namespace: {{ alert.results[0].fire_result['k8s.pod.namespace'] }}
      - Pod: {{ alert.results[0].fire_result['k8s.pod.name'] }}
      - Container: {{ alert.results[0].fire_result['k8s.container.name'] }}
      - Working Directory: {{ alert.results[0].fire_result['process.cwd'] }}
      - Command: {{ alert.results[0].fire_result['process.cmdline'] }}
      
      [[Details](${query_url})]
      [[Settings](${alert_url})]
  5. Configure alerts.

    1. Log on to the Simple Log Service console.

    2. In the project list, find the project that the cluster uses (k8s-log-${cluster_id}) and click the project name.

    3. In the left-side navigation pane, click the Dashboard icon, select Dashboards, and then click Kubernetes Container Internal Operation Audit.

    4. On the Kubernetes Container Internal Operation Audit report page, choose Alert > Create Alert Monitoring Rule.

      Alert

    5. On the Alert Monitoring Rule page, configure the alert rules for Kubernetes container access alert and Kubernetes alert for running risky programs after container access, and then click OK.

      The following values are for reference. Adjust based on your requirements.

      • Kubernetes container access alert

        Parameter

        Description

        Rule Name

        Kubernetes container access alert

        Check Frequency

        Fixed Interval: 1 minute.

        Query Statistics

        Click Add, open the Advanced Settings tab, configure the parameters, and click OK.

        • Logstore: Search for and select advaudit-${cluster_id}, for example, advaudit-c76da730c08ca45adb90fad86fb******.

        • Query Time Range: Select 1 Minute (Relative).

        • Search: Displayed after you specify the Logstore. Enter the following query:

          kind: Kubernetes and kubeObject.operation.kind: PodExec | select "clusterid", "time", "traceId", "eventId", "k8s.user.aliuid", "k8s.user.username", json_extract(k8s, '$.user.groups') as "k8s.user.groups", "host.hostip", "host.nodename", "k8s.pod.namespace", "k8s.pod.name", json_extract(k8s, '$.pod.labels') as "k8s.pod.labels", "kubeobject.operation.podexecoptions.container", json_extract(kubeobject, '$.operation.podExecOptions.command') as "kubeobject.operation.podexecoptions.command", "kubeobject.operation.podexecoptions.commandstr" from log

        Trigger Condition

        Select Data is returned. Then, set Severity to Report.

        Destination

        Select Simple Log Service Notification.

        Enable

        Turn on the switch.

        Alert Policy

        Select Simple Mode and configure the following parameters.

        • Notification Method: Select DingTalk.

        • Select Webhook: Select Kubernetes intra-container operation audit alert.

        • Notified Contacts: Select No Reminder.

        • Alert Template: Search for and select Kubernetes container access alert.

        • Period: Select Any Time.

      • Kubernetes alert for running risky programs after container access

        Parameter

        Description

        Rule Name

        Kubernetes alert for running risky programs after container access

        Check Frequency

        Fixed Interval: 1 minute.

        Query Statistics

        Click Add, open the Advanced Settings tab, configure the parameters, and click OK.

        • Logstore: Search for and select advaudit-${cluster_id}, for example, advaudit-c76da730c08ca45adb90fad86fb74****.

        • Query Time Range: Select 1 Minute (Relative).

        • Search: Displayed after you specify the Logstore. Enter the following query:

          kind: Command | select "clusterid", "time", "traceId", "eventId", "k8s.user.aliuid", "k8s.user.username", json_extract(k8s, '$.user.groups') as "k8s.user.groups", "host.hostip", "host.nodename", "k8s.pod.namespace", "k8s.pod.name", json_extract(k8s, '$.pod.labels') as "k8s.pod.labels", "k8s.container.name", "k8s.container.image", "process.cwd", "process.name", "process.cmdline", json_extract(process, '$.pid') as "process.pid", "process.user.uid", json_extract(process, '$.parentPid') as "process.parentpid", "process.parentname" from log where "process.name" in ('rm', 'sudo', 'su', 'nsenter', 'curl', 'wget', 'yum', 'apt-get', 'apt', 'apk', 'dpkg', 'nc', 'ncat', 'ssh', 'scp', 'nmap', 'docker', 'crictl', 'nerdctl', 'podman', 'kubectl', 'helm', 'mysql', 'redis', 'psql', 'redis-cli', 'pip', 'npm', 'gem')

        Trigger Condition

        When Data is returned, set Severity to medium.

        Destination

        Select Simple Log Service Notification.

        Enable

        Turn on the switch.

        Alert Policy

        Select Simple Mode and configure the following parameters.

        • Notification Method: Select DingTalk.

        • Select Webhook: Select Kubernetes intra-container operation audit alert.

        • Notified Contacts: Select No Reminder.

        • Alert Template: Select Kubernetes alert for running risky programs after container access.

        • Period: Select Any Time.

  6. Test the alert rules.

    1. Run kubectl exec -it <pod_name> -- bash to enter a container. This triggers the Kubernetes container access alert rule.

      Container access alert

    2. In the container, run touch a.txt && rm a.txt. This triggers the Kubernetes alert for running risky programs after container access rule.

      Command execution alert

Disable the feature

Uninstalling the ack-advanced-audit add-on disables the feature.

Important

Disabling the feature does not delete the advaudit-${cluster_id} Logstore. If you leave it in place, SLS storage charges continue to accrue. To stop billing, log in to the SLS console and manually delete the Logstore. See Stop billing for or delete a Logstore.

To uninstall the add-on:

  1. Log in to the ACK console and click Clusters in the left navigation pane.

  2. Click your cluster name. In the left navigation pane, click Add-ons.

  3. On the Add-ons page, find ack-advanced-audit and click Uninstall.

Next steps