Testing ESSDPL-X cloud disk performance-Elastic Compute Service(ECS)-阿里云帮助中心

Alibaba Cloud ESSD PL-X cloud disks deliver high performance across multiple dimensions, including ultra-high IOPS, ultra-high throughput, and ultra-low latency. This topic demonstrates how to benchmark the latency and IOPS of a cloud disk used as a bare disk.

Benchmarking conditions

Test tool: FIO.

Note
fio is an open source I/O benchmarking tool that can test block storage performance metrics such as random and sequential read/write operations.
Instance type: The ecs.g8ise.12xlarge instance type is recommended. For more information, see Storage-enhanced general-purpose instance family g8ise.
Image: A recent Alibaba Cloud Linux public image. Alibaba Cloud Linux 3 is used in the following examples.

Note
ESSDs may not achieve expected IOPS on certain Linux distributions. Attach ESSDs to ECS instances that use Alibaba Cloud Linux 3 images.
ESSD PL-X cloud disk:
- Benchmarking a bare disk provides a more accurate measure of cloud disk performance. Use FIO to directly benchmark the bare disk.
- To ensure accurate performance test results, provision sufficient IOPS. For information about the specifications of ESSD PL-X cloud disks, see ESSD PL-X cloud disks (invitational preview).
  - When you benchmark latency, provision at least 40,000 IOPS.
  - When you benchmark IOPS, we recommend using a cloud disk of at least 3,000 GiB with 3,000,000 IOPS.
Important
- If a block storage device contains partitions, file systems, or data, fio may corrupt the file systems and cause data loss. Create snapshots before testing. See Create snapshot manually.
- Do not test with the system disk or any data disk that contains data. Use a newly created, uninitialized, empty data disk instead.
- Test results are obtained in a test environment and are for reference only. In production, cloud disk performance may vary due to factors such as network conditions and concurrent access.
- After testing the new disk:
  - This item is reserved and cannot be used directly. See Re-initialize a data disk.
  - If you do not need the disk, detach and then release it.

Benchmark ESSD PL-X cloud disk latency

Connect to an ECS instance.

For more information, see Log on to a Linux instance using Workbench.

Run the following command to query the device names of your block storage devices.

sudo fdisk -lu

In the command output, /dev/vda is the system disk, and /dev/vdb and /dev/vdc are data disks.

[ecs-a                          ]$ sudo fdisk -lu
Disk /dev/vda: 40 GiB, 42949672960 bytes, 83886080 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: F51132A7-67B1-4650-806D-FD0DE6E1210C
Device      Start      End  Sectors  Size Type
/dev/vda1    2048     6143     4096    2M BIOS boot
/dev/vda2    6144   415743   409600  200M EFI System
/dev/vda3  415744 83886046 83470303 39.8G Linux filesystem
Disk /dev/vdb: 30 GiB, 32212254720 bytes, 62914560 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: C36DF120-8650-4188-8043-AEF9C85F31EF
Device     Start      End  Sectors Size Type
/dev/vdb1   2048 62912511 62910464  30G Linux filesystem
Disk /dev/vdc: 40 GiB, 42949672960 bytes, 83886080 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

This instance has three block storage devices: the system disk /dev/vda and the data disks /dev/vdb and /dev/vdc.

Run the following command to check whether the block storage devices have partitions and a file system.

sudo blkid

[ecs-a                          ]$ sudo blkid
/dev/vdb1: UUID="9c32c24f-d2b8-4aa8-8xxx" BLOCK_SIZE="4096" TYPE="ext4" PARTLABEL="primary" PARTUUID="4bd66635-f5f4-4dc0-9bdd-664fd5b8d2fb"
/dev/vda2: SEC_TYPE="msdos" UUID="7E" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="82a50cd6-9899-41eb-91fe-7027bf257086"
/dev/vda3: LABEL="root" UUID="beef9d8d-ba84-46d9-8xxx" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="e0d4fa20-912d-4e86-943e-6b06dxxxx"
/dev/vda1: PARTUUID="d083a7cd-a7ea-4898-89d5-8e1510bed584"

As shown in the output, the block storage devices /dev/vda and /dev/vdb have partitions and a file system. No information about /dev/vdc is returned, which indicates that /dev/vdc has no partition or file system.

Before you benchmark the ESSD PL-X cloud disk, back up data on the test target to prevent data loss. ESSD PL-X cloud disks do not support snapshots. You can back up data in the same way you back up local disks, for example, by using periodic backups with Cloud Backup, or by backing up data to OSS, a cloud disk, or NAS. For more information, see Back up files on a local disk.
Run the following command to install the libaio library and the FIO tool.
```
sudo yum install libaio libaio-devel fio -y
```
Adjust the kernel boot parameters.

Important
This process is risky and misconfiguration can make the operating system unusable. Before you proceed, back up your data. Note that ESSD PL-X cloud disks do not support snapshots. For information on creating snapshots for other disk types, you can create a manual snapshot.
1. Run the following command to open /etc/default/grub.
```
sudo vim /etc/default/grub
```
2. Find the line that contains GRUB_CMDLINE_LINUX="XXX" and append idle=poll to the end of the line.
  If another idle=xxx parameter already exists in the kernel configuration, delete it.
```
GRUB_TIMEOUT=1
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_TERMINAL_OUTPUT="console"
GRUB_DISABLE_RECOVERY="true"
GRUB_ENABLE_BLSCFG=true
GRUB_CMDLINE_LINUX=" rhgb quiet $(cat /usr/share/alinux-base-setup/cmdline | xargs) biosdevname=0 net.ifnames=0 console=tty0 timeout=300 vring_force_dma_api idle=poll"
```
3. Run the following command to update the grub configuration.
```
sudo grub2-mkconfig -o /boot/grub2/grub.cfg
```
4. Run the following command to restart the instance for the changes to take effect.
```
sudo reboot
```
Run the following command to switch the directory.
```
cd /tmp
```
Run the following command to create a script named test_plx_lat.sh.
```
sudo vim test_plx_lat.sh
```

Add the following content to the test_plx_lat.sh script.

#!/bin/bash
DEV_NODE=$1
DEV_NAME=/dev/$DEV_NODE
function CheckHasFS
{
    local device=$1  # Device path
    # Check if the device exists.
    if [ ! -b "$device" ]; then
        echo "Error: Device $device does not exist"
        exit 1
    fi
    # Use the `blkid` command to check the partition table and file system type.
    local pt_type=$(sudo blkid -o value -s PTTYPE "$device")
    local fs_type=$(sudo blkid -o value -s TYPE "$device")
    if [ -n "$pt_type" ] || [ -n "$fs_type" ]; then
        return 1
    else
        return 0
    fi
}
CheckHasFS "$DEV_NAME"
if [ $? -eq 1 ]; then
    echo "$DEV_NAME contains a partition table or file system. Stopping the FIO script."
    exit 1
fi
echo 2 > /sys/block/$DEV_NODE/queue/rq_affinity
sleep 5
DEVICE=`ls -al /sys/block/$DEV_NODE/device | awk '{print $11}' | sed 's%../%%g'`
INTERRUPT=`cat /proc/interrupts | grep $DEVICE | shuf -n 1 | awk '{print $1}' | sed 's/://g'`
EFFECTIVE_SMP=`cat /proc/irq/$INTERRUPT/effective_affinity_list`
fio --ioengine=libaio --runtime=30s --numjobs=1 --iodepth=1 --bs=4k --rw=randwrite --filename=$DEV_NAME --time_based=1 --direct=1 --name=test --group_reporting=1 --cpus_allowed_policy=split --cpus_allowed=$EFFECTIVE_SMP

For a detailed description of the script, see the test_plx_lat.sh script description below.

test_plx_lat.sh script

This section describes the test_plx_lat.sh script, used for testing the latency of an ESSD PL-X cloud disk. Configure the parameters as needed.

The following command changes the value of the block device system parameter rq_affinity to 2.

echo 2 > /sys/block/$DEV_NODE/queue/rq_affinity

The following table describes the values of the rq_affinity parameter.

Value	Description
0	The block device routes I/O completion events to any vCPU. When testing latency, this can cause significant kernel context switching and scheduling overhead, affecting single-channel latency.
1	The block device routes I/O completion events to the vCPU group that processed the I/O submission. When testing latency, this may cause some kernel context switching and scheduling overhead, affecting single-channel latency.
2	The block device processes the I/O completion on the same vCPU that submitted it. When testing latency, this minimizes kernel context switching and scheduling overhead to maximize single-channel latency performance.

The following command obtains the device name of the block device.

DEVICE=`ls -al /sys/block/$DEV_NODE/device | awk '{print $11}' | sed 's%../%%g'`

The following command obtains an interrupt number for the I/O queue of the block device.

INTERRUPT=`cat /proc/interrupts | grep $DEVICE | shuf -n 1 | awk '{print $1}' | sed 's/://g'`

The following command obtains the thread ID of the vCPU that handles the interrupt.
```
EFFECTIVE_SMP=`cat /proc/irq/$INTERRUPT/effective_affinity_list`
```

The following command starts an FIO test for single-channel random write latency and restricts I/O requests to be submitted on the obtained vCPU thread.

fio --ioengine=libaio --runtime=30s --numjobs=1 --iodepth=1 --bs=4k --rw=randwrite --filename=/dev/$DEV_NODE --time_based=1 --direct=1 --name=test --group_reporting=1 --cpus_allowed_policy=split --cpus_allowed=$EFFECTIVE_SMP

The following table describes the parameters in the command.

Parameter	Description	Example
`numjobs`	The number of I/O threads. To measure single-channel latency, set this to 1.	1
`iodepth`	The I/O queue depth. To measure single-channel latency, set this to 1.	1
`filename`	The file to be benchmarked. When benchmarking a bare ESSD PL-X cloud disk, set this parameter to the device name of the cloud disk.	/dev/vdc
`cpus_allowed`	Specifies the thread ID of the vCPU to use for the benchmark. This parameter is used with `cpus_allowed_policy=split`. When testing single-channel latency, set this to the thread ID of the vCPU that handles interrupts.	4

Benchmark the latency of the ESSD PL-X cloud disk.

sudo sh test_plx_lat.sh your_device

Replace your_device with the actual device name of the ESSD PL-X cloud disk, such as vdc.

In the output, find the lat (usec): min=xx, max=xxxx, avg=xx.xx, stdev=x.xx line. This shows the latency of the ESSD PL-X cloud disk.

root@xxx:/tmp# sudo sh test_plx_lat.sh vdc
test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][w=144MiB/s][w=36.9k IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=36816: Tue Mar  8 16:20:45 2022
  write: IOPS=36.9k, BW=144MiB/s (151MB/s)(4325MiB/30001msec); 0 zone resets
    slat (nsec): min=1445, max=39912, avg=2015.59, stdev=392.88
    clat (usec): min=14, max=4937, avg=24.60, stdev= 8.77
     lat (usec): min=22, max=4939, avg=26.69, stdev= 8.77
    clat percentiles (usec):
     |  1.00th=[    23],  5.00th=[    23], 10.00th=[    24], 20.00th=[    24],
     | 30.00th=[    24], 40.00th=[    25], 50.00th=[    25], 60.00th=[    25],
     | 70.00th=[    25], 80.00th=[    26], 90.00th=[    26], 95.00th=[    27],
     | 99.00th=[    30], 99.50th=[    32], 99.90th=[    92], 99.95th=[   115],
     | 99.99th=[   128]
   bw (  KiB/s): min=141048, max=148936, per=100.00%, avg=147646.34, stdev=1235.91, samples=59
   iops        : min=35262, max=37234, avg=36911.56, stdev=308.97, samples=59
  lat (usec)   : 20=0.02%, 50=99.86%, 100=0.01%, 250=0.10%, 500=0.01%
  lat (usec)   : 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%
  cpu          : usr=5.66%, sys=10.51%, ctx=1107304, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,1107300,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
  WRITE: bw=144MiB/s (151MB/s), 144MiB/s-144MiB/s (151MB/s-151MB/s), io=4325MiB (4536MB), run=30001-30001msec
Disk stats (read/write):
  vdc: ios=51/1102570, merge=0/0, ticks=3/26754, in_queue=25568, util=85.26%

If output similar to the following example is returned, it indicates that the test target has a partition or file system. To ensure data security, the FIO script stops. We recommend that you use a newly created, empty data disk for the test.
```
[[ecs-user@ecs tmp]$ sudo sh test_plx_lat.sh vdc
/dev/vdc contains a partition table or file system. Stopping the FIO script.
```
Warning
If the test target contains a partition, a file system, or other data, running an FIO benchmark directly can corrupt the file system and cause data loss. If your data disk has a partition and a file system, create a new, empty data disk for the test:
- You can create a pay-as-you-go cloud disk with the same configurations and attach it to the instance for testing. For more information, see Create an empty data disk.
- After the test is complete, you can detach and release the disk as needed. For more information, see Detach a data disk and Release a cloud disk.

Benchmark ESSD PL-X cloud disk IOPS

Connect to an ECS instance.

For more information, see Log on to a Linux instance using Workbench.
Run the following command to query the device names of your block storage devices.
```
sudo fdisk -lu
```
This instance has three block storage devices: the system disk /dev/vda and the data disks /dev/vdb and /dev/vdc.
Run the following command to check whether the block storage devices have a partition and a file system.
```
sudo blkid
```
The block storage devices /dev/vda and /dev/vdb have partitions and a file system. No information about /dev/vdc is returned, which indicates that /dev/vdc has no partition or file system.
Before you benchmark the ESSD PL-X cloud disk, back up data on the test target to prevent data loss. ESSD PL-X cloud disks do not support snapshots. You can back up data in the same way you back up local disks, for example, by using periodic backups with Cloud Backup, or by backing up data to OSS, a cloud disk, or NAS. For more information, see Back up files on a local disk.
Run the following command to install the libaio library and the FIO tool.
```
sudo yum install libaio libaio-devel fio -y
```
Run the following command to switch the directory.
```
cd /tmp
```
Run the following command to create a script named test_plx_iops.sh.
```
sudo vim test_plx_iops.sh
```

Add the following content to the test_plx_iops.sh script.

#!/bin/bash
DEV_NODE=$1
DEV_NAME=/dev/$DEV_NODE
function CheckHasFS
{
    local device=$1  # Device path
    # Check if the device exists.
    if [ ! -b "$device" ]; then
        echo "Error: Device $device does not exist"
        exit 1
    fi
    # Use the `blkid` command to check the partition table and file system type.
    local pt_type=$(sudo blkid -o value -s PTTYPE "$device")
    local fs_type=$(sudo blkid -o value -s TYPE "$device")
    if [ -n "$pt_type" ] || [ -n "$fs_type" ]; then
        return 1
    else
        return 0
    fi
}
CheckHasFS "$DEV_NAME"
if [ $? -eq 1 ]; then
    echo "$DEV_NAME contains a partition table or file system. Stopping the FIO script."
    exit 1
fi
echo 0 > /sys/block/$DEV_NODE/queue/rq_affinity
sleep 5
fio --ioengine=libaio --runtime=30s --numjobs=64 --iodepth=64 --bs=4k --rw=randwrite --filename=/dev/$DEV_NODE --time_based=1 --direct=1 --name=test --group_reporting=1

For a detailed description of the script, see the test_plx_iops.sh script description below.

test_plx_iops.sh script

This section describes the test_plx_iops.sh script, used for testing the IOPS of an ESSD PL-X cloud disk. Configure the parameters as needed.

Optional: The following command changes the value of the block device system parameter rq_affinity to 0.

echo 0 > /sys/block/$DEV_NODE/queue/rq_affinity

The following table describes the rq_affinity parameter.

Value	Description
0	The block device routes I/O completion events to any vCPU. This behavior maximizes guest CPU parallelism for sending and receiving I/O. As cloud disk controller capabilities improve, guest parallelism must also increase to avoid performance bottlenecks.
1	The block device routes I/O completion events to the vCPU group that processed the I/O submission. This behavior can limit the parallelism of I/O processing within the guest.
2	The block device processes the I/O completion on the same vCPU that submitted it. This behavior significantly limits the parallelism of I/O processing within the guest.

The following command starts an FIO test for random write IOPS.

fio --ioengine=libaio --runtime=30s --numjobs=64 --iodepth=64 --bs=4k --rw=randwrite --filename=/dev/$DEV_NODE --time_based=1 --direct=1 --name=test --group_reporting=1

The following table describes the parameters in the command.

Parameter	Description	Example
`numjobs`	The number of I/O threads. Because the purpose of this test is to measure IOPS, you need to maximize the concurrency within the guest. Therefore, a value of 64 is used.	64
`iodepth`	The I/O queue depth. Because the purpose of this test is to measure IOPS, you need to increase the pressure on a single queue. Therefore, an I/O queue depth of 64 is used.	64
`filename`	The file to be benchmarked. When you benchmark a bare ESSD PL-X cloud disk, set this parameter to the actual device name of the cloud disk.	/dev/vdc

Run the following command to test the IOPS of the ESSD PL-X cloud disk.

sudo sh test_plx_iops.sh your_device

Replace your_device with the actual device name of the ESSD PL-X cloud disk, such as vdc.

In the output, find the IOPS=*** line. This shows the IOPS of the ESSD PL-X cloud disk.

root@xxx:olvbjfw7boZ:/tmp# sudo sh test_plx_iops.sh vdc
test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.12
Starting 64 processes
Jobs: 64 (f=64): [w(64)][100.0%][w=11.5GiB/s][w=3002k IOPS][eta 00m:00s]
test: (groupid=0, jobs=64): err= 0: pid=37033: Tue Mar  8 16:23:16 2022
  write: IOPS=2943k, BW=11.2GiB/s (12.1GB/s)(337GiB/30005msec); 0 zone resets
    slat (nsec): min=1155, max=111174k, avg=20741.77, stdev=148830.33
    clat (usec): min=56, max=72869, avg=1367.51, stdev=1385.04
     lat (usec): min=62, max=72871, avg=1388.33, stdev=1399.51
    clat percentiles (usec):
     |  1.00th=[  379],  5.00th=[  437], 10.00th=[  482], 20.00th=[  545],
     | 30.00th=[  594], 40.00th=[  676], 50.00th=[  807], 60.00th=[ 1057],
     | 70.00th=[ 1467], 80.00th=[ 1958], 90.00th=[ 2704], 95.00th=[ 4047],
     | 99.00th=[ 7046], 99.50th=[ 7701], 99.90th=[ 8979], 99.95th=[ 9765],
     | 99.99th=[23462]
   bw (  KiB/s): min=33824, max=700984, per=1.56%, avg=183556.61, stdev=93310.00, samples=3793
   iops        : min= 8456, max=175246, avg=45889.13, stdev=23327.49, samples=3793
   lat (usec)   : 100=0.01%, 250=0.09%, 500=12.26%, 750=33.70%, 1000=12.05%
   lat (msec)   : 2=22.60%, 4=14.23%, 10=5.03%, 20=0.03%, 50=0.01%
   lat (msec)   : 100=0.01%
  cpu          : usr=6.36%, sys=19.32%, ctx=7351060, majf=0, minf=653
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,88318243,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
  WRITE: bw=11.2GiB/s (12.1GB/s), 11.2GiB/s-11.2GiB/s (12.1GB/s-12.1GB/s), io=337GiB (362GB), run=30005-30005msec
Disk stats (read/write):
  vdc: ios=157/88157518, merge=0/0, ticks=28/56378887, in queue=74292632, util=100.00%

Note

You can calculate the cloud disk throughput from the benchmarked IOPS. For the calculation formula, see ESSD PL-X cloud disks (invitational preview).

If output similar to the following example is returned, it indicates that the test target has a partition or file system. To ensure data security, the FIO script stops. We recommend that you use a newly created, empty data disk for the test.
```
[[ecs-user@ecs tmp]$ sudo sh test_plx_iops.sh vdc
/dev/vdc contains a partition table or file system. Stopping the FIO script.
```
Warning
If the test target contains a partition, a file system, or other data, running an FIO benchmark directly can corrupt the file system and cause data loss. If your data disk has a partition and a file system, create a new, empty data disk for the test:
- You can create a pay-as-you-go cloud disk with the same configurations and attach it to the instance for testing. For more information, see Create an empty data disk.
- After the test is complete, you can detach and release the disk as needed. For more information, see Detach a data disk and Release a cloud disk.