Use the VFIO driver instead of the UIO driver-Elastic Compute Service(ECS)-阿里云帮助中心

Replace the UIO driver with the VFIO driver on ECS and ECS Bare Metal Instance (EBM) to fix DPDK runtime errors.

Prerequisites

Huge pages are configured on the instance.
DPDK is installed on the instance.

Background

DPDK applications on sixth-generation instance types (such as g6, c6, and r6) and later may encounter runtime errors. For example, pktgen-dpdk may fail to detect the NIC bound to the igb_uio driver. If you see the following error, replace the UIO driver with the VFIO driver.

EAL: eal_parse_sysfs_value(): cannot open sysfs value /sys/bus/pci/devices/0000:00:06.0/uio/uio0/portio/port0/start

Procedure

Connect to your ECS or EBM instance. See Log on to a Linux instance using Workbench.
Check the GRUB configuration.
```
cat /proc/cmdline
```
- If the output includes intel_iommu=on but not iommu=pt, skip to Step 4.
- Otherwise, go to Step 3 to add the parameter.
  
  In the following example, the output includes neither intel_iommu=on nor iommu=pt, so intel_iommu=on must be added.
```
[test@iZ2ze9h6rn63          ~]$ cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-4.19.91-27.4.al7.x86_64 root=UUID=90179e0e-4e13-4c59-a6cc-0ce5cxxx ro crashkernel=0M-2G:0M
```

Add intel_iommu=on to /etc/default/grub.

Open the file.
```
sudo vim /etc/default/grub
```

Press i to enter insert mode. Add intel_iommu=on to the GRUB_CMDLINE_LINUX parameter. Save and close the file.

Example of the modified configuration:

GRUB_TIMEOUT=1
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="crashkernel=auto rhgb quiet idle=halt biosdevname=0 net.ifnames=0 console=tty0 console=ttyS0,115200n8 noibrs intel_iommu=on"
GRUB_DISABLE_RECOVERY="true"

Apply the configuration.

sudo grub2-mkconfig -o /boot/grub2/grub.cfg

[test@iZ2ze9h6rn          ~]$ sudo grub2-mkconfig -o /boot/grub2/grub.cfg
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-4.19.91-27.4.al7.x86_64
Found initrd image: /boot/initramfs-4.19.91-27.4.al7.x86_64.img
Found linux image: /boot/vmlinuz-0-rescue-2cea6e3dd20d4251bf6880159559f2c9
Found initrd image: /boot/initramfs-0-rescue-2cea6e3dd20d4251bf6880159559f2c9.img
done

Restart the instance and reconnect after it starts.
```
reboot
```
Warning
The restart operation stops the instance for a short period of time and may interrupt services that are running on the instance. We recommend that you restart the instance during off-peak hours.

Load the VFIO and VFIO-PCI drivers.

sudo modprobe vfio && \
sudo modprobe vfio-pci

Enable noiommu_mode.

sudo bash -c 'echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode'

Get the bus-info value of the NIC.

ethtool -i ethX

Replace ethX with the target NIC ID. Example for eth1:

The bus-info field in the output is the bus address of the network interface (in this example, 0000:00:06.0).

[root@izxxx ~]# ethtool -i eth1
driver: virtio_net
version: 1.0.0
firmware-version:
expansion-rom-version:
bus-info: 0000:00:06.0
supports-statistics: no
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

Bind the NIC to the vfio-pci driver. Navigate to the usertools directory in your DPDK installation directory and run:

sudo ./dpdk-devbind.py -b vfio-pci 0000:00:06.0

Note

Replace 0000:00:06.0 with the actual bus-info value of your NIC.

Run ./dpdk-devbind.py --status to verify. The following output shows 0000:00:06.0 bound to vfio-pci:

[root@izxxxxxxxxxxxxx usertools]# ./dpdk-devbind.py --status

Network devices using DPDK-compatible driver
=============================================
0000:00:06.0 'Virtio network device 1000' drv=vfio-pci unused=virtio pci

Network devices using kernel driver
====================================
0000:00:05.0 'Virtio network device 1000' if=eth0 drv=virtio-pci unused=virtio_pci,vfio-pci *Active*

Start the DPDK application. Navigate to the build/app directory in your DPDK installation directory and run the command for your version.

For DPDK 18.02 and later:

sudo ./testpmd -w 0000:00:06.0 -c 0x3 -- --total-num-mbufs=2048 -ai

For versions earlier than DPDK 18.02:

sudo ./testpmd -w 0000:00:06.0 -c 0x3 -- --total-num-mbufs=2048 --disable-hw-vlan -ai

Note

-w 0000:00:06.0 specifies the NIC. total-num-mbufs=2048 must match your huge page size. Replace both with your actual values.

Example of a running DPDK application:

[    @izxxxxxxxxxxxZ app]# ./testpmd -w 0000:00:06.0 -c 0x3 -- --total-num-mbufs=2048 -ai
EAL: Detected 2 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:00:06.0 on NUMA socket -1
EAL:   Invalid NUMA socket, default to 0
EAL:   probe driver: 1af4:1000 net_virtio
EAL:   using IOMMU type 8 (No-IOMMU)
Auto-start selected
Interactive-mode selected
testpmd: create a new mbuf pool <mbuf_pool_socket_0>: n=2048, size=2176, socket=0
testpmd: preferred mempool ops selected: ring_mp_mc

Warning! port-topology=paired and odd forward ports number, the last port will pair with itself.

Configuring Port 0 (socket 0)
EAL: Error disabling MSI-X interrupts for fd 18
Port 0: 00:16:3E:35:CC:C4
Checking link statuses...
Done
Start automatic packet forwarding
io packet forwarding - ports=1 - cores=1 - streams=1 - NUMA support enabled, MP allocation mode: native
Logical Core 1 (socket 0) forwards packets on 1 streams:
  RX P=0/Q=0 (socket 0) -> TX P=0/Q=0 (socket 0) peer=02:00:00:00:00:00

  io packet forwarding packets/burst=32
  nb forwarding cores=1 - nb forwarding ports=1
  port 0: RX queue number: 1 Tx queue number: 1
    Rx offloads=0x0 Tx offloads=0x0
    RX queue: 0
      RX desc=0 - RX free threshold=0
      RX threshold registers: pthresh=0 hthresh=0  wthresh=0
      RX Offloads=0x0
    TX queue: 0
      TX desc=0 - TX free threshold=0
      TX threshold registers: pthresh=0 hthresh=0  wthresh=0
      TX offloads=0x0 - TX RS bit threshold=0
testpmd>