Test ESSD IOPS performance-Elastic Compute Service(ECS)-阿里云帮助中心

This topic explains how to configure test conditions and use FIO to benchmark the IOPS of an ESSD as a raw disk.

Prerequisites

Test tool: FIO.

Note
fio is an open source I/O benchmarking tool that can test block storage performance metrics such as random and sequential read/write operations.
Instance type: An ecs.g7se.32xlarge ECS instance is recommended. For more information, see General-purpose instance families (g-series).
Image: A recent version of a Linux public image. This topic uses Alibaba Cloud Linux 3.

Note
Tests show that some Linux distributions may not deliver the expected performance. We recommend using the official Alibaba Cloud Linux 3 image from Alibaba Cloud for best results.
ESSD:
- Testing a raw disk provides more accurate performance metrics for a cloud disk. We recommend running FIO directly on a raw disk to test its performance.
- An ESSD PL3 is recommended. For more information about ESSDs, see ESSDs.
Important
- If a block storage device contains partitions, file systems, or data, fio may corrupt the file systems and cause data loss. Create snapshots before testing. See Create snapshot manually.
- Do not test with the system disk or any data disk that contains data. Use a newly created, uninitialized, empty data disk instead.
- Test results are obtained in a test environment and are for reference only. In production, cloud disk performance may vary due to factors such as network conditions and concurrent access.
- After testing the new disk:
  - This item is reserved and cannot be used directly. See Re-initialize a data disk.
  - If you do not need the disk, detach and then release it.

Procedure

Connect to an ECS instance.

For more information, see Log on to a Linux instance using Workbench.

Run the following command to list the available block storage devices.

sudo fdisk -lu

The following output is an example where /dev/vda is a system disk, and /dev/vdb and /dev/vdc are data disks.

[ecs-a                          ]$ sudo fdisk -lu
Disk /dev/vda: 40 GiB, 42949672960 bytes, 83886080 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: F51132A7-67B1-4650-806D-FD0DE6E1210C
Device      Start      End  Sectors  Size Type
/dev/vda1    2048     6143     4096    2M BIOS boot
/dev/vda2    6144   415743   409600  200M EFI System
/dev/vda3  415744 83886046 83470303 39.8G Linux filesystem
Disk /dev/vdb: 30 GiB, 32212254720 bytes, 62914560 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: C36DF120-8650-4188-8043-AEF9C85F31EF
Device     Start      End  Sectors Size Type
/dev/vdb1   2048 62912511 62910464  30G Linux filesystem
Disk /dev/vdc: 40 GiB, 42949672960 bytes, 83886080 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

The preceding output shows that the instance has one system disk /dev/vda and two data disks /dev/vdb and /dev/vdc.

Run the following command to check whether the block storage device has partitions or a file system.

sudo blkid

[ecs-a]$ sudo blkid
/dev/vdb1: UUID="9c32c24f-d2b8-4aa8-8xxx" BLOCK_SIZE="4096" TYPE="ext4" PARTLABEL="primary" PARTUUID="4bd66635-f5f4-4dc0-9bdd-664fd5b8d2fb"
/dev/vda2: SEC_TYPE="msdos" UUID="7E" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="82a50cd6-9899-41eb-91fe-7027bf257086"
/dev/vda3: LABEL="root" UUID="beef9d8d-ba84-46d9-8xxx" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="e0d4fa20-912d-4e86-943e-6b0866xxx"
/dev/vda1: PARTUUID="d083a7cd-a7ea-4898-89d5-8e1510bed584"

As shown above, the block storage devices /dev/vda and /dev/vdb have partitions and file systems. No information is returned for /dev/vdc, which indicates that it has no partitions or file systems.

Before you test the block storage performance, make sure that you have backed up the data of the test object to prevent data loss. For more information, see Create a manual snapshot.

Note
You are charged for using snapshots. For more information, see Snapshot pricing.
Run the following command to install the libaio library and the FIO tool. Select a command based on the operating system.
Alibaba Cloud Linux 2/3 and CentOS 6 or later

Note
CentOS 6 reached end of life (EOL). In accordance with Linux community rules, all content was removed from the following CentOS 6 repository address: http://mirror.centos.org/centos-6/. If you continue to use the default CentOS 6 repository on Alibaba Cloud, an error is reported. To use specific installation packages of CentOS 6, change the CentOS 6 repository address. For more information, see How do I change CentOS 6 repository addresses?
```
sudo yum install libaio libaio-devel fio -y
```
Debian 9 or later and Ubuntu 14 or later

Important
Because Debian 9 and 10 have reached their end of life (EOL), you must first change the repository address on any ECS instance running these versions. For more information, see Change the repository addresses for CentOS/Debian EOL.
```
sudo apt-get update
sudo apt-get install libaio* fio -y
```
Go to the /tmp directory.
```
cd /tmp
```
Create the test100w.sh script.
```
sudo vim test100w.sh
```

Paste the following content into the test100w.sh file. For more information, see Test100w.sh script.

#!/bin/bash
DEV_NODE=your_device
DEV_NAME=/dev/$DEV_NODE
function CheckHasFS
{
    local device=$1  # The path of the device.
    # Check whether the device exists.
    if [ ! -b "$device" ]; then
        echo "Error: Device $device does not exist"
        exit 1
    fi
    # Use the `blkid` command to check the partition table and file system type.
    local pt_type=$(sudo blkid -o value -s PTTYPE "$device")
    local fs_type=$(sudo blkid -o value -s TYPE "$device")
    if [ -n "$pt_type" ] || [ -n "$fs_type" ]; then
        return 1
    else
        return 0
    fi
}
CheckHasFS "$DEV_NAME"
if [ $? -eq 1 ]; then
    echo "$DEV_NAME contains a partition table or a file system. Stop the fio script!"
    exit 1
fi
function RunFio
{
 numjobs=$1   # The number of test threads in the instance. Example: 10.
 iodepth=$2   # The maximum number of I/O requests that can be submitted at the same time. Example: 64.
 bs=$3        # The size of a single I/O block. Example: 4k.
 rw=$4        # The read/write policy for the test. Example: randwrite.
 size=$5
 filename=$6  # The name of the test file. Example: /dev/your_device.
 nr_cpus=`cat /proc/cpuinfo |grep "processor" |wc -l`
 if [ $nr_cpus -lt $numjobs ];then
     echo "Numjobs is more than cpu cores, exit!"
     exit -1
 fi
 let nu=$numjobs+1
 cpulist=""
 for ((i=1;i<10;i++))
 do
     list=`cat /sys/block/$DEV_NODE/mq/*/cpu_list | awk '{if(i<=NF) print $i;}' i="$i" | tr -d ',' | tr '\n' ','`
     if [ -z $list ];then
         break
     fi
     cpulist=${cpulist}${list}
 done
 spincpu=`echo $cpulist | cut -d ',' -f 2-${nu}`
 echo $spincpu
 fio --ioengine=libaio --runtime=30s --numjobs=${numjobs} --iodepth=${iodepth} --bs=${bs} --size=${size} --rw=${rw} --filename=${filename} --time_based=1 --direct=1 --name=test --group_reporting --cpus_allowed=$spincpu --cpus_allowed_policy=split
}
echo 2 > /sys/block/$DEV_NODE/queue/rq_affinity
sleep 5
RunFio 10 128 4k randwrite 1024g $DEV_NAME

Modify the test100w.sh script as needed.
- Replace all your_device parameters with the actual device name, such as nvme1n1.
- Set the values for 10 (numjobs), 64 (iodepth), 4k (bs), randwrite (rw), and /dev/your_device in the RunFio 10 64 4k randwrite /dev/your_device command based on your actual situation.
- The numjobs value must not exceed the number of CPU cores. To find the number of CPU cores, run the following command:
```
cat /proc/cpuinfo |grep "processor" |wc -l
```

Run the following command to test the performance of the ESSD.

sudo sh test100w.sh

In the command output, the line containing IOPS=*** shows the IOPS of the ESSD.

job1: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128
...
  fio-3.7
  Starting 6 processes
  Jobs: 6 (f=6): [w(6)][100.0%][r=0KiB/s,w=3932MiB/s][r=0,w=1007k IOPS][eta 00m:00s]
job1: (groupid=0, jobs=6): err= 0: pid=42693: Thu Jun 23 15:32:28 2022
  write: IOPS=1005k, BW=3925MiB/s (4116MB/s)(38.4GiB/10015msec)
    slat (nsec): min=728, max=9562.2k, avg=5157.38, stdev=47881.11
    clat (usec): min=82, max=25001, avg=758.61, stdev=843.67
     lat (usec): min=86, max=25042, avg=763.86, stdev=844.85
    clat percentiles (usec):
     |  1.00th=[  190],  5.00th=[  347], 10.00th=[  445], 20.00th=[  537],
     | 30.00th=[  586], 40.00th=[  635], 50.00th=[  652], 60.00th=[  676],
     | 70.00th=[  693], 80.00th=[  717], 90.00th=[  824], 95.00th=[ 1254],
     | 99.00th=[ 4424], 99.50th=[ 6849], 99.90th=[10814], 99.95th=[15008],
     | 99.99th=[19530]
   bw (  KiB/s): min=573448, max=732136, per=16.69%, avg=670799.74, stdev=26159.23, samples=120
   iops        : min=143362, max=183034, avg=167699.93, stdev=6539.81, samples=120
  lat (usec)   : 100=0.01%, 250=2.26%, 500=11.66%, 750=71.40%, 1000=8.11%
  lat (msec)   : 2=3.38%, 4=2.02%, 10=1.04%, 20=0.11%, 50=0.01%
  cpu          : usr=14.45%, sys=24.23%, ctx=401609, majf=0, minf=236
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=110.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=0,10062326,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=128
Run status group 0 (all jobs):
  WRITE: bw=3925MiB/s (4116MB/s), 3925MiB/s-3925MiB/s (4116MB/s-4116MB/s), io=38.4GiB (41.2GB), run=10015-10015msec
Disk stats (read/write):
  nvme1n1: ios=4Z/11063917, merge=0/0, ticks=5/8026436, in_queue=9632482, util=100.00%

If you see the following output, the test disk has a partition or file system. The FIO script stops to ensure data security. Use a new, empty data disk for the test.
```
[[ecs-user@ecs tmp]$ sudo sh test100w.sh
/dev/vdb contains a partition table or a file system. Stop the fio script!
```
Warning
If the test object has a partition, a file system, or other data, running FIO directly on the test object causes file system exceptions and data loss. If your data disk has a partition and a file system, we recommend that you create a new, empty data disk for testing:
- You can create a pay-as-you-go cloud disk that has the same configurations and attach the disk to the instance for testing. For more information, see Create a data disk.
- After the test is complete, you can detach and release the disk as needed. For more information, see Detach a data disk and Release a cloud disk.

Test100w.sh script

The following command sets the block device's rq_affinity parameter to 2.

echo 2 > /sys/block/your_device/queue/rq_affinity

Values of rq_affinity	Value description
1	When a block device completes an I/O operation, the completion event is sent to the same vCPU group that submitted the request. In a multi-threaded and concurrent scenario, this creates a bottleneck and limits performance.
2	When a block device completes an I/O operation, the completion event is processed on the vCPU that submitted the request. In a multi-threaded and concurrent scenario, the performance of each vCPU can be fully utilized.

The following command binds different jobs to different CPU cores.

fio -ioengine=libaio -runtime=30s -numjobs=${numjobs} -iodepth=${iodepth} -bs=${bs} -rw=${rw} -filename=${filename} -time_based=1 -direct=1 -name=test -group_reporting -cpus_allowed=$spincpu -cpus_allowed_policy=split

Note

In normal mode, a device has only one request queue. In a multi-threaded and concurrent I/O scenario, the only request queue can become a performance bottleneck. Multi-queue mode allows a device to use multiple request queues to process I/O, fully using the back-end storage performance. To use this feature effectively, you must bind I/O threads to different CPU cores that correspond to different request queues.

Parameters	Note	Example
`numjobs`	The number of I/O threads.	10
`/dev/your_device`	The device name of the ESSD.	/dev/nvme1n1
`cpus_allowed_policy`	FIO provides the `cpus_allowed_policy` and `cpus_allowed` parameters to bind vCPUs.	split

The preceding command runs multiple jobs and binds them to different CPU cores that correspond to different Queue_Id values. To query the cpu_core_id value to which a Queue_Id value is bound, run the following commands:

Run the ls /sys/block/your_device/mq/ command. In the command, replace your_device with the actual device name, such as nvme1n1. This command queries the Queue_Id values for the specified device.
Run the cat /sys/block/your_device/mq/*/cpu_list command. In the command, replace your_device with the actual device name, such as nvme1n1. This command shows which CPU core is bound to each request queue for the specified device.

Prerequisites

Procedure

Alibaba Cloud Linux 2/3 and CentOS 6 or later

Debian 9 or later and Ubuntu 14 or later

Test100w.sh script