NAS performance troubleshooting tool-File Storage NAS(NAS)-阿里云帮助中心

How it works

On a Linux client, the NFS client module in the kernel records metrics, such as operation latency, in the local /proc/self/mountstats file. The nasiostat tool works by reading and parsing this file to present the metrics in an easy-to-understand format. This makes the tool lightweight and ensures it does not consume excessive CPU or memory resources during runtime.

Performance metrics

Parameter	Description
ops/s	The number of I/O or metadata operations completed per second.
kB/s	The data throughput per second.
kB/op	The average block size of an I/O or metadata operation.
avg RTT (ms)	The average round-trip time for an I/O or metadata request. This includes the network transfer time from the NFS client to the NAS server (T2), the server processing time (T3), and the network transfer time for the response to return to the NFS client (T4).
avg exe (ms)	The average execution time for an entire I/O or metadata operation. This is the sum of the queuing time in the NFS client kernel (T1) and the average RTT.
avg queue (ms)	The average time an I/O or metadata request waits in the NFS client kernel queue before it is sent (T1).

Timing diagram

Download the nasiostat tool

Download the nasiostat tool script.
- If your ECS instance has public network access, run the following command to download the tool:
```
wget https://raw.githubusercontent.com/aliyun/nas-client-tools/master/linux_client/nasiostat -O /tmp/nasiostat
```
- If your ECS instance does not have public network access, download the tool to your computer and then upload it to the instance.
  1. Download nasiostat to your computer.
  2. Upload the nasiostat script to your ECS instance. For instructions, see Upload or download files on a Linux instance.
Make the tool script executable.
```
sudo chmod +x /tmp/nasiostat
```

View performance metrics

Run the following command to view NFS performance metrics.

/tmp/nasiostat 1 10 -A write,read,getattr,setattr,access,rename,lookup,create,readdir,readdirplus,remove <NAS mount point>

The following list describes the key parameters. Replace placeholders with your actual values.

1: The sampling interval in seconds.
10: The total number of samples to collect.
-A: The NFS operations to sample, such as write, read, and getattr. For a list of common operations, see Common NFS commands.
<NAS mount point>: The local mount point for the NAS file system. You can run the mount -l | grep nas command to find it.

For example, in 1ca404****.cn-hangzhou.nas.aliyuncs.com:/ /mnt:
- 1ca404****.cn-hangzhou.nas.aliyuncs.com: The address of the mount target. You can find this address in the File Storage NAS console. Go to the File System List page, click Manage next to the target file system, and then go to the Mount Targets page. For more information, see Manage mount targets.
- /: The directory in the NAS file system.
- /mnt: The local mount point on your instance.

Sample output:

           ops/s     concurrency      bklogqueue       sendqueue       pendqueue
         777.085               2               0               1               1
op                 ops/s            kB/s           kB/op         retrans    avg RTT (ms)    avg exe (ms)  avg queue (ms)
write              0.722         738.623        1023.263        0 (0.0%)         155.534         823.704         667.887
read             776.335        3296.389           4.246        0 (0.0%)           7.971           8.085           0.091
getattr            0.005           0.001           0.218       1 (14.3%)           2.857           2.857           0.000
setattr            0.001           0.000           0.262        0 (0.0%)           1.000           1.000           0.000
access             0.004           0.001           0.230        0 (0.0%)           0.200           0.200           0.000
rename             0.000           0.000           0.000        0 (0.0%)           0.000           0.000           0.000
lookup             0.001           0.000           0.234        0 (0.0%)           0.000           0.000           0.000
create             0.001           0.000           0.379        0 (0.0%)           1.000           1.000           0.000
readdir            0.000           0.000           0.000        0 (0.0%)           0.000           0.000           0.000
readdirplus        0.000           0.000           0.000        0 (0.0%)           0.000           0.000           0.000
remove             0.000           0.000           0.000        0 (0.0%)           0.000           0.000           0.000

Common NFS commands

Command	Description
getattr	Gets file attributes. This procedure retrieves the attributes of a specified file system object. The object is identified by a file handle, which the server returns in the response to a LOOKUP, CREATE, MKDIR, SYMLINK, MKNOD, or READDIRPLUS procedure (or from the MOUNT service described elsewhere).
setattr	Sets file attributes. This procedure changes one or more attributes of a file system object on the server. SETATTR is not guaranteed to be an atomic operation. A failed SETATTR might partially change the file's attributes. Changing the file size by using SETATTR also changes the mtime.
access	Checks access permissions. This procedure determines the access permissions that a user (identified by credentials in the request) has for a file system object. The client encodes the permissions to check as a bitmask. The server checks the permissions encoded in the bitmask. In normal cases, the server returns the NFS3_OK status and a bitmask that contains the permissions granted to the client.
lookup	Looks up a file name. This procedure searches for a specific name in a directory and returns the file handle of the corresponding file system object.
read	Reads data from a file. This procedure reads data from a file based on its file handle.
write	Writes data to a file. This procedure writes data to a file based on its file handle.
create	Creates a file. This procedure creates a regular file. The NFS version 3 protocol introduced the MKNOD procedure to create device files and FIFO files.
mkdir	Creates a directory. This procedure creates a subdirectory. Many server implementations do not allow using the file names `.` or `..` as the target of an MKDIR operation. In such cases, the server should return `NFS3ERR_EXIST`.
remove	Deletes a file. This procedure removes an entry from a directory. If the entry is the last reference to the corresponding file system object, the object might be destroyed. The REMOVE operation typically deletes non-directory file objects, while RMDIR deletes directories.
rename	Renames a file or directory. The RENAME operation moves and renames a file or directory. To ensure consistency and integrity, the source and destination directories must be on the same file system and server.
readdir	Reads a directory. This procedure retrieves a variable number of entries from a directory in sequence. It returns the name and file identifier of each entry. It also provides information that allows the client to retrieve more directory entries in subsequent READDIR requests. For example, the `ls -l` command calls this operation to traverse a directory.
readdirplus	Reads an extended directory. This procedure retrieves a variable number of entries from a directory and returns complete information for each entry. READDIRPLUS differs from READDIR only in the amount of information returned. READDIR returns only the file name and file ID. READDIRPLUS returns the name, file ID, attributes (including the file ID), and file handle for each entry. The `ls -l` command, for example, calls this operation to traverse a directory.

Troubleshooting scenarios

Scenario 1: High avg exe, high avg RTT, and low avg queue

This pattern indicates that the bottleneck is not on the NFS client. The issue is likely a slow response from the NAS server or high network latency. To resolve this, submit a ticket.

Scenario 2: High avg exe, low avg RTT, and high avg queue

This pattern indicates that the bottleneck is on the client. Requests are queuing on the NFS client. Improve the throughput from your ECS instance to NAS. For instructions, see How to improve NAS performance on Linux.

Use nethogs to troubleshoot abnormal NAS access speed

The nethogs tool monitors network traffic by process. Use nethogs to identify the process that is the source of abnormal NAS access speed.

Installation and usage

Install nethogs based on the operating system of your ECS instance:

CentOS or RHEL: yum install -y nethogs
Ubuntu or Debian: apt-get install -y nethogs

Run the nethogs eth0 command to monitor network traffic on a specific network interface. Replace eth0 with the name of your network interface.

Criteria for identifying issues

The throughput of a NAS file system is capped by the internal bandwidth specification of the ECS instance to which it is mounted. For example, the internal bandwidth of an ecs.t6-c1m1.large instance is about 80 Mbit/s (10 MB/s). In contrast, the throughput of a 4 TB Standard NAS file system can reach more than 600 MB/s. In this case, the internal bandwidth of the ECS instance is the bottleneck. You can call the DescribeInstanceTypes operation to query the internal bandwidth specification of an ECS instance.
If nethogs shows that the network traffic of a process consistently stays close to the internal bandwidth limit of the ECS instance, that process may be the source of the NAS access bottleneck. When the read/write throughput exceeds the specification threshold of the NAS file system, NAS automatically throttles the traffic. You can view the IOPS and throughput of a file system on the performance monitoring page of the NAS console.

Complementary troubleshooting with nasiostat

Use nethogs to identify the process with high network traffic.
View the IOPS and throughput of the file system on the performance monitoring page of the NAS console.
Use nasiostat to collect the latency of NFS operations.