Use tcpdump to capture packets and troubleshoot network issues

Background information

tcpdump is a widely used packet capture tool that captures packets transmitted through a network interface and analyzes them in real time. O&M engineers can use tcpdump to monitor and record network traffic and obtain detailed packet information such as source address, destination address, and transport protocol. This information provides critical clues for identifying network issues, diagnosing latency, packet loss, or abnormal connections, and helps quickly locate and resolve network failures.

Procedure

The following example deploys a sample application and manually injects a temporary container with necessary tools such as tcpdump to demonstrate how to capture network packets for the sample application from within the temporary container.

1. Create acs-tcpdump.yaml with the following content.

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: acs-tcpdump
     namespace: default
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: acs-tcpdump
     strategy:
       rollingUpdate:
         maxSurge: 25%
         maxUnavailable: 25%
       type: RollingUpdate
     template:
       metadata:
         labels:
           alibabacloud.com/compute-class: general-purpose
           alibabacloud.com/compute-qos: default
           app: acs-tcpdump
       spec:
         containers:
         - image: registry.openanolis.cn/openanolis/nginx:1.14.1-8.6
           imagePullPolicy: IfNotPresent
           name: tcpdump
           resources:
             limits:
               cpu: 500m
               memory: 1Gi
             requests:
               cpu: 500m
               memory: 1Gi
         dnsPolicy: ClusterFirst
         restartPolicy: Always
         schedulerName: default-scheduler
         securityContext: {}
         terminationGracePeriodSeconds: 30

2. Run the following command to deploy the application.

   kubectl apply -f acs-tcpdump.yaml

3. Inject a temporary container into the sample application.

   kubectl debug -n default acs-tcpdump-7d959xxxxx-xxxxx --image=registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/acs-tcpdump-demo:alpha.v1 -c debug-container --profile=netadmin

   Important

   If the permissions provided by the netadmin profile do not meet your needs during validation, change the profile value to sysadmin. Using the sysadmin profile requires enabling privileged mode for the pod. Exercise caution in production environments and ensure thorough auditing and monitoring.

   ACS clusters disable privileged mode by default. To enable privileged mode, submit a ticket.

   The image used by the temporary container includes the following tools:

   Tool name	Purpose	Is Privilege-Dependent
   Trace-CMD	A command-line tool for tracing kernel functions. For more information, see trace-cmd.	Required
   perf	perf is a system performance optimization tool included with the Linux kernel. For more information, see perf.	Required
   SysAK	SysAK (System Analysis Kit) is a set of O&M tools provided by the OpenAnolis community. It includes tools for issue diagnosis, event monitoring/tracing, and system and service operations.	Partially required
   Strace	Strace is a Linux user space tracer for diagnostics and debugging. Use it to monitor interactions between user space processes and the kernel, such as system calls, signal delivery, and process status changes. For more information, see strace.	Required
   tcpdump	tcpdump is a powerful packet capture tool for capturing and analyzing network traffic. It runs from the command line and provides detailed network packet information to help diagnose network issues, monitor traffic, and perform security analytics.	Required
   IProute2	IProute2 is a collection of command-line tools for network configuration and management, designed to replace legacy net-tools utilities such as ifconfig, route, and netstat.	Required
   Nmap-ncat	Nmap-ncat is a tool from the Nmap project, commonly known as ncat. Ncat is a powerful networking utility for reading from and writing to network connections. It is similar to netcat (nc) but offers additional features and improvements.	Required

4. Confirm the injection status of the temporary container.

   kubectl describe pod acs-tcpdump-7d959xxxxx-xxxxx | grep Containers: -A 10

   Expected output:

   Containers:
     tcpdump:
       Container ID:   containerd://71da93ea68ec02a2bcd468b7835ecd12d8180d4a05ac80d77ff4dcab61aea701
       Image:          registry.openanolis.cn/openanolis/nginx:1.14.1-8.6
       Image ID:       registry.openanolis.cn/openanolis/nginx@sha256:bb8fe1e8d4353f29768340bdcbe2bfbc105081d550d259fc11d07d2abe53b5d1
       Port:           <none>
       Host Port:      <none>
       State:          Running
         Started:      Thu, 13 Feb 2025 10:23:33 +0800
       Ready:          True
       Restart Count:  0
   --
   Ephemeral Containers:
     debug-container:
       Container ID:   containerd://5a26877432ab93a6a056b48d34e6c888a4f98c0662627728cf8d8ee4486f3342
       Image:          registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/acs-tcpdump-demo:alpha.v1
       Image ID:       registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/acs-tcpdump-demo@sha256:64ee96ff1b0e000dc5ca0711685e09fe3a08e62a9c9bb1f354682b82f35bddee
       Port:           <none>
       Host Port:      <none>
       State:          Running
         Started:      Thu, 13 Feb 2025 10:24:38 +0800
       Ready:          False
       Restart Count:  0

   You can see that the debug-container has been successfully injected.

5. Verify packet capture.

   1. Enter the temporary container.

      kubectl exec -it acs-tcpdump-7d959xxxxx-xxxxx -c debug-container -- bash

   2. Use tcpdump to capture all TCP packets to port 80 on the local host.

      tcpdump -i eth0 'tcp port 80'

      Expected output:

      dropped privs to tcpdump
      tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
      listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
      03:09:10.752193 IP acs-tcpdump-7d959bxxxxx-xxxxx.35656 > 100.xxx.xxx.209.http: Flags [S], seq 202950596, win 64240, options [mss 1460,sackOK,TS val 3648381756 ecr 0,nop,wscale 7], length 0
      03:09:10.753944 IP 100.xxx.xxx.209.http > acs-tcpdump-7d959bxxxxx-xxxxx.35656: Flags [S.], seq 270454912, ack 202950597, win 29200, options [mss 1440,nop,nop,sackOK,nop,wscale 9], length 0
      03:09:10.753966 IP acs-tcpdump-7d959bxxxxx-xxxxx.35656 > 100.xxx.xxx.209.http: Flags [.], ack 1, win 502, length 0
      03:09:10.754020 IP acs-tcpdump-7d959bxxxxx-xxxxx.35656 > 100.xxx.xxx.209.http: Flags [P.], seq 1:614, ack 1, win 502, length 613: HTTP: POST /logstores/kernel-logs/shards/lb HTTP/1.1
      03:09:10.755776 IP 100.xxx.xxx.209.http > acs-tcpdump-7d959bxxxxx-xxxxx.35656: Flags [.], ack 614, win 60, length 0
      03:09:10.755825 IP 100.xxx.xxx.209.http > acs-tcpdump-7d959bxxxxx-xxxxx.35656: Flags [P.], seq 1:26, ack 614, win 60, length 25: HTTP: HTTP/1.1 100 Continue
      03:09:10.755829 IP acs-tcpdump-7d959bxxxxx-xxxxx.35656 > 100.xxx.xxx.209.http: Flags [.], ack 26, win 502, length 0
      03:09:10.755867 IP acs-tcpdump-7d959bxxxxx-xxxxx.35656 > 100.xxx.xxx.209.http: Flags [P.], seq 614:1654, ack 26, win 502, length 1040: HTTP
      03:09:10.758773 IP 100.xxx.xxx.209.http > acs-tcpdump-7d959bxxxxx-xxxxx.35656: Flags [P.], seq 26:243, ack 1654, win 66, length 217: HTTP: HTTP/1.1 200 OK
      03:09:10.758786 IP acs-tcpdump-7d959bxxxxx-xxxxx.35656 > 100.xxx.xxx.209.http: Flags [.], ack 243, win 501, length 0
      03:09:10.758854 IP acs-tcpdump-7d959bxxxxx-xxxxx.50624 > 100.xxx.xxx.112.http: Flags [F.], seq 318721902, ack 4047870907, win 501, length 0
      03:09:10.763400 IP 100.xxx.xxx.112.http > acs-tcpdump-7d959bxxxxx-xxxxx.50624: Flags [F.], seq 1, ack 1, win 83, length 0
      03:09:10.763413 IP acs-tcpdump-7d959bxxxxx-xxxxx.50624 > 100.xxx.xxx.112.http: Flags [.], ack 2, win 501, length 0
      03:09:11.219397 IP acs-tcpdump-7d959bxxxxx-xxxxx.50638 > 100.xxx.xxx.112.http: Flags [P.], seq 2989834529:2989835691, ack 1755317949, win 501, length 1162: HTTP: POST /logstores/memory-metrics/shards/lb HTTP/1.1
      03:09:11.219451 IP acs-tcpdump-7d959bxxxxx-xxxxx.50644 > 100.xxx.xxx.112.http: Flags [P.], seq 1733121609:1733122681, ack 40604405, win 501, length 1072: HTTP: POST /logstores/disk-metrics/shards/lb HTTP/1.1

      The temporary container includes OSSUtil. Use the tcpdump parameter -w packets.pcap to generate a packet file, then use OSSUtil to upload the captured packets to OSS for local analysis.

      > kubectl exec -it acs-tcpdump-xxx -c debug-container -- bash
      [root@acs-tcpdump-xxx app]# ll
      total 4
      drwxr-xr-x 2 root root 4096 Feb 14 08:03 tools
      [root@acs-tcpdump-xxx app]# cd tools/
      [root@acs-tcpdump-xxx tools]# ll
      total 11040
      -rwxr-xr-x 1 root root 11304833 May 17  2024 ossutil
      [root@acs-tcpdump-xxx tools]#

      Note

      The temporary container in this example includes a built-in countdown of 3600 seconds. After completing packet capture and debugging, wait for the temporary container to exit automatically or manually delete the pod to clean up the debug container. Perform these operations during off-peak hours to avoid impacting production workloads.

Related content

The following Dockerfile was used to build the temporary container in this topic. You can use it as a reference and customize it based on your needs.

FROM alibaba-cloud-linux-3-registry.cn-hangzhou.cr.aliyuncs.com/alinux3/alinux3:latest
# Update system software
RUN dnf update --security -y && dnf upgrade --security -y && dnf clean all && rm -rf /var/cache/dnf/ && rm -rf /core.*
# Install essential network and other tool packages
RUN yum install net-tools iputils tcpdump wget iproute bind-utils nmap-ncat procps-ng -y && yum update -y
# Install kernel analysis and tracing tools
RUN yum install perf sysak strace trace-cmd -y && yum clean all && rm -rf /var/cache/yum/
# Add tools for easier Kubernetes debugging
# osscmd
COPY tools/ossutil64 /home/app/tools/ossutil
# Set working directory
WORKDIR /home/app
ENTRYPOINT ["/bin/bash","-c","sleep 3600"]