整卡直通虚拟化ICN隔离指南

1. 背景

宿主机上的PPU卡之间通过ICN link连接来完成多张PPU卡之间的高速互联功能，在宿主机上的PPU驱动安装过程中会初始化ICN连接并建立ICN拓扑表，接下来用户就可以在宿主机上正常使用ICN互联功能。但是在rund多租场景下用户可能会选择宿主机上的一部分PPU卡直通到虚拟机内使用，这种用法下需要注意务必要把不属于当前虚拟机内PPU卡之间的ICN link断开，这么做主要有2点原因：

1. 避免不同VM内PPU内核驱动安装过程中ICN初始化过程的相互影响和冲突，最坏情况下可能会导致VM内PPU内核驱动安装失败。

2. 从VM安全性的角度来说如果两个VM之间存在ICN物理链路，那么就存在理论上某一个VM内的恶意程序可以查看或攻击另外一个VM内的用户。

所以用户在宿主机直通PPU整卡之前，一定要做好ICN link的隔离。

2. 操作命令

考虑到宿主机上不一定安装了PPU驱动，所以我们提供了一种在宿主机上通过PPU固件进行隔离的方法并提供了相应的python脚本来方便用户使用，完整的python脚本请参考本文最后的附录部分。

$ python3 icn_link_configure.py -h
usage: icn_link_configure.py [-h] [-g] [-v] [--vm_devices VM_DEVICES] [-r]

VM topo link configure script.
 When passthrough PPUs into VM, set icn link configuration by topo.

optional arguments:
  -h, --help            show this help message and exit
  -g, --gen_topo        generate a topo config file, should reset link configuration in advance.
  -v, --verbose         debug mode, enable more logging
  --vm_devices VM_DEVICES
                        define how devices are assigned to different VMs,
                        it will change configuration in all device you set.
                        can use bdf or ppu id.
                        e.g. --vm_devices "0,1|2,3|4,5,7|6"
                        e.g. --vm_devices "0,1,2"
                        e.g. --vm_devices "0000:08:00.0,0000:7e:00.0,0000:a2:00.0,0000:c6:00.0|0001:09:00.0,0001:7f:00.0|0001:aa:00.0,0001:ce:00.0"
                        e.g. --vm_devices "0000:08:00.0,0000:7e:00.0"
  --check_devices CHECK_DEVICES
                        check link configure, the parameters are the same as 'vm_devices'
                        it will check configuration in all device you set.
  -r, --reset           reset link configure, use default link mask.

==========device0==========
ppuid: 0
link[0]: nbrPpuid:6
link[3]: nbrPpuid:2
link[4]: nbrPpuid:3
link[5]: nbrPpuid:1
==========device1==========
ppuid: 1
link[0]: nbrPpuid:7
link[3]: nbrPpuid:0
link[4]: nbrPpuid:2
link[5]: nbrPpuid:3
==========device2==========
ppuid: 2
link[0]: nbrPpuid:4
link[3]: nbrPpuid:0
link[4]: nbrPpuid:1
link[5]: nbrPpuid:3
==========device3==========
ppuid: 3
link[0]: nbrPpuid:5
link[3]: nbrPpuid:2
link[4]: nbrPpuid:0
link[5]: nbrPpuid:1
==========device4==========
ppuid: 4
link[0]: nbrPpuid:2
link[3]: nbrPpuid:6
link[4]: nbrPpuid:7
link[5]: nbrPpuid:5
==========device5==========
ppuid: 5
link[0]: nbrPpuid:3
link[3]: nbrPpuid:4
link[4]: nbrPpuid:6
link[5]: nbrPpuid:7
==========device6==========
ppuid: 6
link[0]: nbrPpuid:0
link[3]: nbrPpuid:4
link[4]: nbrPpuid:5
link[5]: nbrPpuid:7
==========device7==========
ppuid: 7
link[0]: nbrPpuid:1
link[3]: nbrPpuid:6
link[4]: nbrPpuid:4
link[5]: nbrPpuid:5

附录：python脚本

import sys
import os
import subprocess
import argparse
import logging
from argparse import RawTextHelpFormatter

PPUDBG_OUTPUT_CODEC = "UTF-8"
MCU_LINK_CONFIG_REG = 0xbe382
MCU_LINK_STATUS_REG = 0xbe383
# need specify the ppudbg and icn_topo.cfg path
PPUDBG = "~/ppudbg"
ICN_TOPO_FILE1 = 'icn_topo.cfg'
ICN_TOPO_FILE2 = '/root/icn_topo.cfg'

def get_icn_topo_file():
    icn_topo_file = ICN_TOPO_FILE1
    if not os.path.exists(icn_topo_file):
        icn_topo_file = ICN_TOPO_FILE2
    if not os.path.exists(icn_topo_file):
        logging.error("Can not find icn_topo.cfg")
        sys.exit(-1)
    return icn_topo_file

def ppudbg_icn_iso(ppu_args):
    cmd = "sudo {} --direct --micnop iso {} {}".format(PPUDBG, get_icn_topo_file(), ppu_args)
    logging.debug(cmd)
    proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True)
    ret = proc.wait()
    if ret != 0:
        logging.error("configure failed!")
        print(proc.stdout.read().decode(PPUDBG_OUTPUT_CODEC, errors="ignored"))
        sys.exit(-1)
    else:
        logging.info("configure success!")

def ppudbg_icn_link_status(ppuids):
    cmd = ("sudo {} --direct --micnop topo {} " + "{}," * len(ppuids)).format(PPUDBG, get_icn_topo_file(), *ppuids).strip(',')
    logging.debug(cmd)
    proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True)
    ret = proc.wait()
    if ret != 0:
        logging.error("ppudbg cmd: \"{}\" failed!".format(cmd))
        print(proc.stdout.read().decode(PPUDBG_OUTPUT_CODEC, errors="ignored"))
        sys.exit(-1)
    else:
        logging.info("link status:")
        print(proc.stdout.read().decode(PPUDBG_OUTPUT_CODEC, errors="ignored"))

def topo_cfg():
    cmd = "{} --topo_read".format(PPUDBG)
    logging.debug(cmd)
    proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True)
    ret = proc.wait()
    if ret == 0:
        ppudbg_topo = proc.stdout.read().decode(PPUDBG_OUTPUT_CODEC, errors="ignored")
        with open("icn_topo.cfg", "w") as f:
            f.write(ppudbg_topo)
    else:
        logging.error("read topo failed")
        sys.exit(-1)

def vm_configure(vm_devices):
    ppudbg_icn_iso(vm_devices)

def reset_icn_configuration():
    cmd = ("sudo {} --direct --micnop iso reset").format(PPUDBG)
    logging.debug(cmd)
    proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True)
    ret = proc.wait()
    if ret != 0:
        logging.error("reset failed! Error log:")
        print(proc.stdout.read().decode(PPUDBG_OUTPUT_CODEC, errors="ignored"))
    else:
        logging.info("reset success!")

def main():
    parser = argparse.ArgumentParser(formatter_class=RawTextHelpFormatter, description="VM topo link configure script.\n \
When passthrough PPUs into VM, set icn link configuration by topo.")
    parser.add_argument("-g", "--gen_topo", action='store_true',
                        help="generate a topo config file, should reset link configuration in advance.")
    parser.add_argument("-v", "--verbose", action="store_true",
                        help='debug mode, enable more logging')
    parser.add_argument('--vm_devices', type=str,
                        help='''define how devices are assigned to different VMs,
it will change configuration in all device you set.
can use bdf or ppu id.
e.g. --vm_devices \"0,1|2,3|4,5,7|6\"
e.g. --vm_devices \"0,1,2\"
e.g. --vm_devices \"0000:08:00.0,0000:7e:00.0,0000:a2:00.0,0000:c6:00.0|0001:09:00.0,0001:7f:00.0|0001:aa:00.0,0001:ce:00.0\"
e.g. --vm_devices \"0000:08:00.0,0000:7e:00.0\"
''')
    parser.add_argument('-r', '--reset', action='store_true',
                        help="reset link configure, use default link mask.")

    args = parser.parse_args()

    if args.verbose:
        level = logging.DEBUG
    else:
        level = logging.INFO
    logging.getLogger("").setLevel(level)
    log_hdl = logging.StreamHandler()
    log_hdl.setFormatter(logging.Formatter("%(levelname)-8s %(message)s"))
    logging.getLogger("").addHandler(log_hdl)

    logging.debug("Param vm_devices: %s", args.vm_devices)
    logging.debug("Param verbose: %s", args.verbose)

    if args.gen_topo:
        topo_cfg()
        logging.info("Generate topo file \"icn_topo.cfg\" done")
        sys.exit(0)

    if args.vm_devices != None:
        vm_configure(args.vm_devices)
        sys.exit(0)

    if args.reset:
        reset_icn_configuration()

if __name__ == "__main__":
    try:
        main()
    except Exception as err:
        logging.error("Unknown exception: %s", err)
        sys.exit(-1)