Using the SysAK toolkit

Environment requirements

• Operating system: Alibaba Cloud Linux 2/3, Anolis OS 8.4 ANCK, CentOS 7, or other Linux systems with a kernel version of 3.10 or later.

  Note

  Run the uname -a command to check the kernel version of your instance.

• Hardware: x86_64.

Installation and deployment

Usage

SysAK modes

SysAK supports two modes: monitoring and diagnosis. In monitoring mode, SysAK runs in the background and provides various system metrics for O&M engineers. In diagnosis mode, which you can enable as needed, SysAK analyzes the root causes of system issues. For more information, see SysAK system monitoring and SysAK diagnosis features.

Start monitoring

Monitoring metrics

• System resources

  Metric category	Category description		Specific metrics	Enhanced feature
  Compute resource	CPU	user	User-mode CPU usage	None
  		sys	Kernel-mode CPU usage
  		hirq	Hardware interrupt CPU usage
  		sirq	Software interrupt CPU usage
  	LOAD	load*	Load average over 1, 5, and 15 seconds
  Memory resource	Memory	free	Free memory
  		used	Used memory
  		buffer	Memory used by buffers
  		cache	Memory used by cache
  		total	Total memory
  		mem.util	Memory usage
  	swap	swpin	Number of pages swapped in
  		swapout	Number of pages swapped out
  		total	Total number of swap pages
  		swap.util	Swap usage
  I/O resource	I/O access	rrqms	Number of merged read requests sent to the device per second
  		wrqms	Number of merged write requests sent to the device per second
  		rs	Number of read requests sent to the device per second
  		ws	Number of write requests sent to the device per second
  		rsecs	Number of sectors read from the device per second
  		wsecs	Number of sectors written to the device per second
  		rqsize	Average size of requests sent to the device
  		qusize	Average queue length of requests sent to the device
  		svctm	Average service time for I/O requests sent to the device
  		io.util	Percentage of CPU time spent on I/O requests
  	Disk space	bfree	Remaining blocks
  		bused	Number of used data blocks
  		btotl	Total number of data blocks
  		patition.util	Database Utilization
  		ifree	Number of free inodes
  		itotl	Total number of inodes
  		iutil	Inode usage
  Network resource	Network traffic	bytin	Bytes of inbound traffic
  		bytout	Bytes of outbound traffic
  		pktin	Total number of received packets
  		pktout	Total number of sent packets
  	TCP	active	Number of active TCP connection initiations
  		pasive	Number of passive TCP connection acceptances
  		iseg	Number of received TCP segments
  		outseg	Number of sent TCP segments
  	UDP	idgm	Number of received UDP datagrams
  		odgm	Number of sent UDP datagrams

• System bottlenecks

  Metric category	Category description		Specific metrics	Enhanced feature
  I/O bottleneck	Read/write latency	await	Average I/O wait time	None
  		rawait	Average I/O read wait time
  		wawait	Average I/O write wait time
  Memory bottleneck	Cache reclaim and defragmentation	kswapd	Number of kswapd reclaim operations
  		pg_kr	Number of asynchronously reclaimed pages
  		pg_dr	Number of directly reclaimed pages
  		kcompd	Number of kcompactd compaction operations
  		dc_all	Number of direct memory compactions
  		dc_fin	Number of completed direct memory compactions
  		oom	Number of out-of-memory (OOM) events
  Network bottleneck	Network transmission	pkterr	Number of system error packets
  		pktdrp	Number of system dropped packets
  		EstReset	Number of resets in the ESTABLISHED state
  		AtmpFail	Number of failed TCP connection attempts
  		retran	TCP retransmission rate
  		noport	Number of times a UDP destination port/address was not found
  		idmerr	Number of invalid UDP packets
  CPU bottleneck	Multitasking concurrency	cswch	Number of system context switches
  		proc	Number of system forks
  	Long waits in the ready queue	rqslow.dltnum	Number of times the ready queue wait time exceeded the threshold	SysAK
  		rqslow.dlttm	Total delay when the ready queue wait time exceeded the threshold	SysAK
  System software bottleneck	Access to critical kernel resources	noschd.dltnum	Number of times the CPU residency in kernel mode exceeded the threshold	SysAK
  		noschd.dlttm	Total delay when the CPU residency in kernel mode exceeded the threshold	SysAK

• System interference

  Metric category	Category description		Specific Metrics	Enhanced feature
  System interference	Long interrupt-off latency	irqoff.dltnum	Number of times the system interrupt-off duration exceeded the threshold	SysAK
  		irqoff.dlttm	Total delay when the system interrupt-off duration exceeded the threshold	SysAK

• Container scenarios (metrics for each container)

  Metric category	Category description		Specific metrics	Enhanced feature
  Compute resource	CPU resource	usr/sys/hriq/sirq	Container CPU usage: user mode, kernel mode, hardware interrupt, and software interrupt	None
  	Load information	nrun	Number of tasks in the ready state in the container	Alibaba Cloud Linux, Anolis OS
  		nunint	Number of tasks in the D (uninterruptible sleep) state in the container
  		load*	Container load average over 1, 5, and 10 seconds
  Memory resource	Memory resource	total/free/used/cache/buffer	Container memory: total, free, used, cache, and buffer	None
  	Memory bottleneck	pgfault	Number of page faults in the container
  		pgmajfault	Number of major page faults (from disk swap or file mapping) in the container
  		mfailcnt	Number of failed memory allocation requests in the container
  		drgl*	Latency distribution of global memory reclaim	Alibaba Cloud Linux, Anolis OS
  		drml*	Latency distribution of memory reclaim in the container
  		dcl*	Latency distribution of in-memory compaction in the container
  I/O resource	I/O metrics	riops	Number of read I/O operations in the container	None
  		wiops	Number of read and write I/O operations in the container
  		rbps	Number of bytes read in the container
  		wbps	Number of bytes written in the container
  		rwait	Read wait time in the container	Alibaba Cloud Linux, Anolis OS
  		wwait	Write wait time in the container
  		rsrv	Read service time in the container
  		wsrv	Write service time in the container
  		rioq	Number of queued read I/O operations in the container
  		wioq	Number of queued write I/O operations in the container
  		rioqsz	Size in bytes of queued read I/O operations in the container
  		wioqsz	Size in bytes of queued write I/O operations in the container
  		rarqsz	Average size in bytes of read I/O operations in the container
  		warqsz	Average size in bytes of write I/O operations in the container
  Hardware resource	Resource bottleneck	llcref	Number of LLC accesses in the container	None
  		llcmis	Number of LLC misses in the container
  		CPI	Container CPI (cycles per instruction)

Service configuration

You can configure the metrics collected by the SysAK monitoring service in its configuration file. The file is located at /usr/local/sysak/sysakmon.conf. After you modify the configuration, run the systemctl restart sysak command to restart the service.

View data

User scenario diagnosis

Specialized low-level system diagnosis

• Scheduling

  Command category	Feature	Command and description
  nosched	Detects when the kernel holds a CPU for a long time, which prevents tasks from being scheduled to run.	sysak nosched [--help] [-t THRESH(ms)] [-f LOGFILE] [-s duration(s)] -t THRESH: The threshold for the non-scheduling period in the kernel. If the threshold is exceeded, a record is made. Unit: ms. Optional. The default is 10 ms. -f LOGFILE: Record the log to a specified file. Optional. The default is /var/log/sysak/nosched/nosched.log. -s durations: The runtime of the program. Unit: s. Optional. By default, the program runs indefinitely. For more information, see nosched feature description.
  irqoff	Detects when interrupts are disabled for a long time in the system.	sysak irqoff [--help] [-t THRESH(ms)] [-f LOGFILE] [duration(s)] -t THRESH: The threshold for the interrupt-off period. If the threshold is exceeded, a record is made. Unit: ms. Optional. The default is 10 ms. -f LOGFILE: Record the log to a specified file. Optional. The default is /var/log/sysak/irqoff/irqoff.log. durations: The runtime of the program. Unit: s. Optional. By default, the program runs indefinitely. For more information, see irqoff feature description.
  runqslower	Detects high task scheduling latency in the system.	sysak runqslower [-s SPAN] [-t TID] [-f LOGFILE] [-P] [THRESH] -s SPAN: The runtime of the program. Unit: s. Optional. By default, the program runs indefinitely. THRESH: The threshold for task preemption time. If the threshold is exceeded, a record is made. Unit: ms. Optional. The default is 20 ms. -f LOGFILE: Record the log to a specified file. Optional. The default is /var/log/sysak/runqslow/runqslow.log. -t TID: A filter option that specifies the ID of the thread to monitor. Optional. By default, all threads are monitored. -P: Record the name and TID of the previous task. Optional. By default, this information is not recorded. For more information, see runqslower feature description.

• Memory

  Command category	Feature	Command and description
  memleak	Diagnose kernel memory leaks, including slab, vmalloc, and buddy system memory leaks, and find the source of the leak.	sysak memleak [-t type] [-i internal] [-c] -t: The type of memory leak. slab: Detect slab allocation memory leaks. vmalloc: Detect vmalloc allocation memory leaks. page: Detect buddy system memory allocation leaks. -i: The detection duration. The default is 300 s. -c: If you specify this parameter, the tool only performs a quick check to determine if a memory leak exists, without diagnosing the source of the leak.
  mmaptrace	Capture user-mode memory leak points and provide the user-mode memory allocation call stack.	sysak mmaptrace [ option ] [ args ] -p <pid>: Monitor the memory allocation of a specified process ID (PID). -l: Set the size of malloc/mmap allocations to monitor. -s: Display the user-mode call stack for memory allocation.
  memgraph	Memory graph tool.	sysak memgraph [ option ] -g: Display the overall memory usage graph. -f: Display detailed page cache information. -a: Display detailed anonymous memory information. -k: Detect memory leaks. -l: Display the memory usage of system threads. -c: Display the memory usage of system cgroups.

• I/O

  Command category	Feature	Command and description
  iofsstat	Collect I/O statistics at the process and file levels.	sysak iofsstat [-h] [-T TIMEOUT] [-t TOP] [-u UTIL_THRESH] [-b BW_THRESH] [-i IOPS_THRESH] [-c CYCLE] [-d DEVICE] [-p PID] [-j] [-f] -T TIMEOUT: The runtime of the command (secs). -t TOP: The top N reports of maximum I/O resource usage. -u UTIL_THRESH: A filter condition that filters out information where utilization is below this threshold. -b BW_THRESH: A filter condition that filters out information where bandwidth is below this threshold. -i IOPS_THRESH: A filter condition that filters out information where IOPS is below this threshold. -c CYCLE: The refresh rate (secs). -d DEVICE: The name of the disk to monitor. -p PID: The PID of the task to follow. -j,--json: Output in JSON format. -f,--fs: Monitor and report information for a specified partition.

• Network

  Command category	Feature	Command and description
  pingtrace	A tool to probe and pinpoint network latency.	sysak pingtrace [ options ] -v,--version: Display the version number. -h,--help: Display help information. -s,--server: Run in server mode. -c,--client ip: Run in client mode. -C,--count UINT: The number of probe packets. Unlimited by default. -i <interval_us>: The interval for sending packets, in microseconds (us). -t < UINT >: The runtime of the program, in seconds (s). -m,--maxdelay us: The threshold for ping latency. Only data from packets that exceed this value is recorded. The default is 0. -b <INT=556 >: The size of the probe packet to send. The minimum size is 144 bytes. --log TEXT=./pingtrace.log: The name of the log file. --logsize INT: The maximum disk space for the log file. --logbackup INT=3: The maximum number of log file backups. --mode auto/pingpong/compact: The running mode of PingTrace. -o,--output image/json/log/imagelog: The output format for PingTrace data. -n,--namespace: Probe information related to the net namespace. --nslocal: When probing net namespace information, this informs the PingTrace client and server that they are running on the same host to avoid redundant data. --userid UINT: When probing net namespace information, specify different user IDs for different hosts. This helps PingTrace identify and correct time synchronization issues between hosts. --debug: Print relevant debug information, mainly libbpf information.
  skcheck	TCP and socket leak detection tool.	sysak skcheck [ options ] [ cmdargs ] -s: Start leak detection. -i: Set the threshold for open sockets. The default is 2000. -l: Set the threshold for closed sockets. The default is 500.

• Performance

  Command category	Feature	Command and description
  numa_access	Display NUMA information for a specified PID and CPU.	sysak numa_access [ options ] [ cmdargs ] -p <pid>: Specify a PID. -c <cpu>: Specify a CPU. -i <time>: Set the display interval.
  hw_event	Hardware event information for Docker.	sysak hw_event [ options ] [ cmdargs ] -c <name>: Specify the name of a Docker container. By default, all Docker containers are included. -s <time>: Set the runtime. The default is 5 s.

• Virtualization

  Command category	Feature	Command and description
  kvmexittime	Trace and diagnose virtual machine VM-exit events.	sysak kvmexittime [--help] [-p PID] [-t TID] [interval] -p <PID>: Specify a PID. -t <TID>: Specify a thread ID (TID). interval: The duration for tracing and diagnosis. --help: Output help information for the current command.

• Generic

  Command category	Feature	Command and description
  syscall_slow	Analyzes application thread lock contention issues that cause slow system calls.	sysak syscall_slow [-t THRESH(ms)] [-n sys_NR] <[-c COMM] [-p tid]> [-f LOGFILE][duration(s)] -t: Records system calls that take longer than this threshold. Unit: ms. Optional. The default value is 10 ms. -n: Detects a specified system call. Optional. By default, all system calls are detected. -c/-p: Detects a specific task by its name or thread ID. You must specify either -c or -p. Required. -f: Records the log to a specified file. Optional. The default path is /var/log/sysak/syscall_slow/syscall_slow.log. durations: The runtime of the program in seconds. This parameter is optional. By default, the program runs indefinitely. For more information, see syscall_slow feature description.
  ulockcheck	Analyzes application thread lock contention issues.	sysak ulockcheck -p <pid> | -s <thread pid>| -a | -t <0|1> | -d -p: Starts monitoring thread lock contention for a specified process. -a: Displays the current lock owner and the top five lock requesters. -s: Displays the lock contention status of the monitored thread. -t: Enables the print switch. If a lock wait exceeds 100 ms, the user-mode call stack of the thread is printed. -d: Stops monitoring.
  cpuirq	Outputs the interrupt binding and runtime status of a CPU.	sysak cpuirq [-c cpu -b ] [ -t [ -i interval ] ] -c: Specifies the CPU to view. -b: Outputs the interrupt binding information for the specified CPU. -t: Views the most frequent interrupts within a specified period. -i: Specifies the time interval for softirq.
  softirq	Collects statistics on system software interrupts, such as count and rate.	sysak softirq [options] [args] -s: Specifies the initial data source file. -r: Outputs to a specified file.