This topic uses Project k8s-log-<id> as an example to walk through the complete Generate SLS SOP (Standard Operating Procedure) workflow, from generating a structured knowledge base to applying it for log analysis.
Background
Generate SLS SOP automatically extracts, selects, and structures log-based knowledge from a Simple Log Service (SLS) Project into a structured knowledge base that the SOP Agent can use directly.
Generate SLS SOP produces SKILL documents, and the SOP Agent loads and executes the query templates from these documents for troubleshooting and auditing.
The Project k8s-log-<id> is automatically created after you enable container observability for an Alibaba Cloud Container Service for Kubernetes (ACK) cluster. It includes built-in resources covering core scenarios such as Kubernetes audits, events, and container logs.
SKILL system structure:
SKILL entry point document (
<root>/<project>/SKILL.md): The top-level document the SOP Agent loads for a given Project.Reference documents (
references/<logstore>.md): Per-Logstore files containing structured query templates the SOP Agent executes.Dependent SKILLs: Troubleshooting SKILLs that reference generated SKILLs to inherit query capabilities without duplicating log structure knowledge.
The complete data-to-knowledge pipeline has two stages:
Knowledge generation: Generate SLS SOP automatically extracts, selects, and structures log-based knowledge from an SLS Project into a knowledge base the SOP Agent can use directly.
Knowledge application: The generated SKILLs support two modes — template matching for standardized scenarios such as routine monitoring and auditing, and knowledge reference for complex tasks that require multi-turn reasoning, such as troubleshooting.
In both modes, you do not need to understand the underlying log structure. The necessary query capabilities are already structured and embedded within the SKILLs, allowing the SOP Agent to automatically orchestrate queries across different troubleshooting scenarios.
Prerequisites
Generate SLS SOP is distributed as a SKILL — a reusable instruction module you can install on an AI Agent. Install it on supported Agent platforms such as Codex, Claude Code, or Qoder using the npx skills add command:
# Project-level installation (recommended, run in the root directory of the target workspace)
npx skills add https://github.com/aliyun/sop-chat --skill generate-sls-skill -a codex claude-code qoder -y
# Global installation
npx skills add https://github.com/aliyun/sop-chat --skill generate-sls-skill -a codex claude-code qoder -g -yPrerequisites include Python 3 and the Alibaba Cloud CLI (version 3.0.308 or later, required for SLS mode). For detailed usage instructions, see the README.
Part 1: Knowledge generation
Send an instruction to the Agent such as "Generate a SKILL document for Project k8s-log-c8f3002d205724aa0aa692ab153ac30b8 and validate the query." The generate-sls-skill SKILL loads and runs three phases: global preparation (Phase A), parallel processing (Phase B), and global aggregation (Phase C).
Phase A: Global preparation
Step 0: Recovery check
On the first run, the system detects no previous checkpoints and automatically starts from Step 1.
Step 1: Resource filtering
The system filters the Logstores in the Project, skipping those with no associated resources (such as dashboards, alerts, or scheduled_sql jobs) and internal Logstores (such as those with the internal- prefix). Only Logstores valuable for generating the knowledge base are retained.
Step 3: Configuration confirmation
This is an interactive checkpoint where you confirm the simplified name and output path for each Logstore. The system then automatically locates the root directory, skills, which contains existing SKILL documents.
Phase B: Parallel processing
The system starts parallel Task Agents to process the Logstores.
Fault tolerance: If a Task Agent fails — for example, due to model rate limiting — you do not need to restart the entire process. The step-level checkpoint mechanism lets you rerun the SKILL, and the system automatically resumes from the point of failure.
Phase C: Global aggregation
After reference documents for all Logstores are generated, the system creates or updates the Project's SKILL entry point document (<root>/<project>/SKILL.md). Query templates for each Logstore are saved as a reference document at references/<logstore>.md.
The generated reference documents contain structured query templates you can use directly for log analysis.
Part 2: Knowledge application
The generated SKILLs are not tied to a specific cluster — they provide general-purpose log query capabilities applicable to any cluster of the same type. This section demonstrates how to use these SKILLs with another Kubernetes cluster, ce76790c8d8dc4eed99b2326112f9a29a.
Choosing a mode:
Use template matching when a single query returns the answer — routine monitoring, auditing, or status checks.
-
Use knowledge reference when the investigation requires multiple queries or branches based on intermediate results — complex troubleshooting where the next step depends on what you find.
Template matching
The SOP Agent reads and executes the generated SKILL documents directly. A user issues a command, the SOP Agent finds and runs a matching query template, and returns the results.
Scenario 1: Audit Secret access records
User instruction: View the access records for Secrets in the Kubernetes cluster
ce76790c8d8dc4eed99b2326112f9a29aover the last 24 hours.The SOP Agent automatically matches the relevant query template in the audit reference document, generates and runs an audit log query, and returns access records for all Secrets. The records include the operating account, the accessed Secret, the source address, and the access frequency.
Scenario 2: Monitor command execution
User instruction: List the commands executed in the Kubernetes cluster
ce76790c8d8dc4eed99b2326112f9a29aover the last 24 hours.The SOP Agent locates the
exec-related query template in thek8s-auditreference and returns allkubectl execoperation records within the cluster. The records include the executor, target pod, executed command, and timestamp — useful for security reviews and compliance audits.Scenario 3: Detect pod OOM events
User instruction: Show the pod OOM (Out of Memory) events in the Kubernetes cluster
ce76790c8d8dc4eed99b2326112f9a29aover the last 24 hours.The SOP Agent matches the OOM event query template in the
k8s-eventreference and returns a list of pods terminated due to insufficient memory. The list includes the pod name, namespace, node, and the time of the OOM event, helping you quickly identify and address memory-related issues.Knowledge reference
Template matching works when a single query returns the answer. Real-world troubleshooting often requires multiple log queries along different investigation paths depending on intermediate results. For these scenarios, author a dedicated troubleshooting SKILL that references the SKILLs generated by Generate SLS SOP. Declare the dependency by name to inherit its query capabilities — no knowledge of the underlying log structure required.
Scenario 1: Troubleshoot pod eviction issues
Authoring a troubleshooting SKILL: A troubleshooting SKILL gains query capabilities by referencing other SKILLs created by Generate SLS SOP, such as the
k8s-logSKILL.--- name: pod-eviction-troubleshooting description: Troubleshoots Kubernetes pod eviction issues and identifies the root cause based on the eviction type. --- ## Dependent SKILLs This troubleshooting process references the k8s-log SKILL to inherit query capabilities. ## Troubleshooting flow ### Step 1: Query pod eviction events **Data source**: Kubernetes events **Information to extract**: - Eviction type (for example, Evicted, TaintManagerEviction) - Eviction reason details - Node where the pod is located **Criteria**: - If the eviction type is `Evicted` → Go to Path A - If the eviction type is `TaintManagerEviction` → Go to Path B --- ### Path A: Evicted (kubelet resource pressure eviction) **Scenario**: The node has insufficient resources, and the kubelet actively evicts the pod. **Step A1**: Get the reason from the eviction event - Example reason: `The node was low on resource: memory` - Extract the node information where the pod was located at the time of eviction. **Step A2**: Query node events to confirm the root cause - **Data source**: Kubernetes events - **Query target**: Find resource pressure events related to the node. --- ### Path B: TaintManagerEviction (taint-based eviction) **Scenario**: A node is marked with a NoExecute taint (such as NotReady), and the control plane evicts the pod. **Step B1**: Get the pod deletion record from the audit log - **Data source**: Kubernetes audit - **Information to extract**: Find the deletion record and the node for the corresponding pod. **Step B2**: Query node events to confirm the root cause - **Data source**: Kubernetes events - **Query target**: Find taint or status change events for the node.Troubleshooting in practice:
User instruction: The pod chaos-daemon-dp7kv in the Kubernetes cluster
ce76790c8d8dc4eed99b2326112f9a29awas evicted. Investigate the cause.The SOP Agent loads the
pod-eviction-troubleshootingSKILL. By referencing thek8s-logSKILL, it gains the ability to query pod eviction events. The agent discovers that the pod was scheduled on a specific node (Node A), then queries for resource pressure events on Node A and examines the pod's entire lifecycle. Based on this analysis, the agent provides a final conclusion.