Use the VFIO driver instead of the UIO driver

Procedure

1. Connect to your ECS or EBM instance. See Log on to a Linux instance using Workbench.

2. Check the GRUB configuration.

   cat /proc/cmdline

   • If the output includes intel_iommu=on but not iommu=pt, skip to Step 4.

   • Otherwise, go to Step 3 to add the parameter.

     In the following example, the output includes neither intel_iommu=on nor iommu=pt, so intel_iommu=on must be added.

     [test@iZ2ze9h6rn63          ~]$ cat /proc/cmdline
     BOOT_IMAGE=/boot/vmlinuz-4.19.91-27.4.al7.x86_64 root=UUID=90179e0e-4e13-4c59-a6cc-0ce5cxxx ro crashkernel=0M-2G:0M

3. Add intel_iommu=on to /etc/default/grub.

   1. Open the file.

      sudo vim /etc/default/grub

   2. Press i to enter insert mode. Add intel_iommu=on to the GRUB_CMDLINE_LINUX parameter. Save and close the file.

      Example of the modified configuration:

      GRUB_TIMEOUT=1
      GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
      GRUB_DEFAULT=saved
      GRUB_DISABLE_SUBMENU=true
      GRUB_TERMINAL_OUTPUT="console"
      GRUB_CMDLINE_LINUX="crashkernel=auto rhgb quiet idle=halt biosdevname=0 net.ifnames=0 console=tty0 console=ttyS0,115200n8 noibrs intel_iommu=on"
      GRUB_DISABLE_RECOVERY="true"

   3. Apply the configuration.

      sudo grub2-mkconfig -o /boot/grub2/grub.cfg

      [test@iZ2ze9h6rn          ~]$ sudo grub2-mkconfig -o /boot/grub2/grub.cfg
      Generating grub configuration file ...
      Found linux image: /boot/vmlinuz-4.19.91-27.4.al7.x86_64
      Found initrd image: /boot/initramfs-4.19.91-27.4.al7.x86_64.img
      Found linux image: /boot/vmlinuz-0-rescue-2cea6e3dd20d4251bf6880159559f2c9
      Found initrd image: /boot/initramfs-0-rescue-2cea6e3dd20d4251bf6880159559f2c9.img
      done

   4. Restart the instance and reconnect after it starts.

      reboot

      Warning

      The restart operation stops the instance for a short period of time and may interrupt services that are running on the instance. We recommend that you restart the instance during off-peak hours.

4. Load the VFIO and VFIO-PCI drivers.

   sudo modprobe vfio && \
   sudo modprobe vfio-pci

5. Enable noiommu_mode.

   sudo bash -c 'echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode'

6. Get the bus-info value of the NIC.

   ethtool -i ethX

   Replace ethX with the target NIC ID. Example for eth1:

   The bus-info field in the output is the bus address of the network interface (in this example, 0000:00:06.0).

   [root@izxxx ~]# ethtool -i eth1
   driver: virtio_net
   version: 1.0.0
   firmware-version:
   expansion-rom-version:
   bus-info: 0000:00:06.0
   supports-statistics: no
   supports-test: no
   supports-eeprom-access: no
   supports-register-dump: no
   supports-priv-flags: no

7. Bind the NIC to the vfio-pci driver. Navigate to the usertools directory in your DPDK installation directory and run:

   sudo ./dpdk-devbind.py -b vfio-pci 0000:00:06.0

   Note

   Replace 0000:00:06.0 with the actual bus-info value of your NIC.

   Run ./dpdk-devbind.py --status to verify. The following output shows 0000:00:06.0 bound to vfio-pci:

   [root@izxxxxxxxxxxxxx usertools]# ./dpdk-devbind.py --status
   
   Network devices using DPDK-compatible driver
   =============================================
   0000:00:06.0 'Virtio network device 1000' drv=vfio-pci unused=virtio pci
   
   Network devices using kernel driver
   ====================================
   0000:00:05.0 'Virtio network device 1000' if=eth0 drv=virtio-pci unused=virtio_pci,vfio-pci *Active*

8. Start the DPDK application. Navigate to the build/app directory in your DPDK installation directory and run the command for your version.

   • For DPDK 18.02 and later:

     sudo ./testpmd -w 0000:00:06.0 -c 0x3 -- --total-num-mbufs=2048 -ai

   • For versions earlier than DPDK 18.02:

     sudo ./testpmd -w 0000:00:06.0 -c 0x3 -- --total-num-mbufs=2048 --disable-hw-vlan -ai

   Note

   -w 0000:00:06.0 specifies the NIC. total-num-mbufs=2048 must match your huge page size. Replace both with your actual values.

   Example of a running DPDK application:

   [    @izxxxxxxxxxxxZ app]# ./testpmd -w 0000:00:06.0 -c 0x3 -- --total-num-mbufs=2048 -ai
   EAL: Detected 2 lcore(s)
   EAL: Detected 1 NUMA nodes
   EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
   EAL: No free hugepages reported in hugepages-1048576kB
   EAL: Probing VFIO support...
   EAL: VFIO support initialized
   EAL: PCI device 0000:00:06.0 on NUMA socket -1
   EAL:   Invalid NUMA socket, default to 0
   EAL:   probe driver: 1af4:1000 net_virtio
   EAL:   using IOMMU type 8 (No-IOMMU)
   Auto-start selected
   Interactive-mode selected
   testpmd: create a new mbuf pool <mbuf_pool_socket_0>: n=2048, size=2176, socket=0
   testpmd: preferred mempool ops selected: ring_mp_mc
   
   Warning! port-topology=paired and odd forward ports number, the last port will pair with itself.
   
   Configuring Port 0 (socket 0)
   EAL: Error disabling MSI-X interrupts for fd 18
   Port 0: 00:16:3E:35:CC:C4
   Checking link statuses...
   Done
   Start automatic packet forwarding
   io packet forwarding - ports=1 - cores=1 - streams=1 - NUMA support enabled, MP allocation mode: native
   Logical Core 1 (socket 0) forwards packets on 1 streams:
     RX P=0/Q=0 (socket 0) -> TX P=0/Q=0 (socket 0) peer=02:00:00:00:00:00
   
     io packet forwarding packets/burst=32
     nb forwarding cores=1 - nb forwarding ports=1
     port 0: RX queue number: 1 Tx queue number: 1
       Rx offloads=0x0 Tx offloads=0x0
       RX queue: 0
         RX desc=0 - RX free threshold=0
         RX threshold registers: pthresh=0 hthresh=0  wthresh=0
         RX Offloads=0x0
       TX queue: 0
         TX desc=0 - TX free threshold=0
         TX threshold registers: pthresh=0 hthresh=0  wthresh=0
         TX offloads=0x0 - TX RS bit threshold=0
   testpmd>