PPU-SMI Device Management Tool (v1.5)

1. Overview

PPU-SMI (PPU System Management Interface) is a command-line tool based on HGML (HanGuang Management Library) that lets you manage and monitor PPU devices. With PPU-SMI, you can:

• Modify device configurations and feature switches.

• Query runtime parameters and feature enablement status for a specific PPU device.

• Collect runtime data and specific events, and export them to tables.

• Analyze device resource utilization by individual applications.

• Query topology information across multiple PPU devices.

2. Installation and usage

The PPU_SDK software package includes PPU-SMI. After installing the PPU_SDK, change to its directory and run the following script to set the required environment variables:

source envsetup.sh

After setting the environment variables, run ppu-smi -v to check the version information:

root@122d8d7a7e37:~# ppu-smi -v
T-Head ppu-smi version v1.0
Build date: Sep  7 2022, 10:43:22
Code node: 4a1989d

Run ppu-smi -h to view the usage information:

root@122d8d7a7e37:~# ppu-smi -h
PPU System Management Interface -- v1.0

PPU-SMI provides monitoring information for T-head PPU devices.
The data is presented in either a plain text or an XML format, via stdout or a file.
PPU-SMI also provides several management operations for changing the device state.
Note that the functionality of PPU-SMI is exposed through the HGML C-based library.

ppu-smi [OPTION1 [ARG1]] [OPTION2 [ARG2]] ...
    -h,   --help
        Print usage information and exit.
    -v,   --version
        Print version information and exit.

  LIST OPTIONS:

    -L,   --list-ppus
        Display a list of PPUs connected to the system.
...

3. Device summary

3.1 Device list

Run ppu-smi -L to list the PPU devices in the system.

root@0549cf16bb85:~# ppu-smi -L
PPU 0: PPU (UUID: GPU-019ea108-c110-0828-0000-0000c07e1a46)
PPU 1: PPU (UUID: GPU-019ea108-c120-040c-0000-0000c0267f1e)

The list includes the following information:

• device index: A zero-based index assigned to each device at power-on.

• Device name

• device UUID

3.1.1 MIG device information

When MIG mode is enabled, ppu-smi -L also displays information about associated MIG devices.

root@0549cf16bb85:~# ppu-smi -L
PPU 0: PPU (UUID: GPU-019ea108-c110-0828-0000-0000c07e1a46)
PPU 1: PPU (UUID: GPU-019ea108-c120-040c-0000-0000c0267f1e)
    MIG g1.c1.i1, Device 0 (UUID: MIG-79c62632-04cc-574b-af7b-cb2e307121c85)
    MIG g1.c0.i0, Device 1 (UUID: MIG-79c62632-04cc-574b-af7b-cb2e307121c84)

3.2 Basic information summary

Run ppu-smi with no arguments to display a summary of PPU devices in the system.

root@122d8d7a7e37:~# ppu-smi
Wed Sep  7 17:04:22 2022
+-------------------------------------------------------------------------------+
| PPU-SMI 1.2           Driver Version: 543.21.01     HGGC Version: 11.5        |
+---------------------------------+----------------------+----------------------+
| PPU  Name        Persistence M. | Bus-Id               | Volatile Uncorr. ECC |
| Fan  Temp  Perf   Pwr:Usage/Cap | Memory-Usage         | PPU-Util  Compute M. |
|                                 |                      |               MIG M. |
+=================================+======================+======================+
| 0  alixpu                N/A    | 00000000:5E:00.0     |                    0 |
| N/A  35C   P0       100W / 200W | 114MiB / 134473MiB   | 10%          Default |
|                                 |                      |             Disabled |
+---------------------------------+----------------------+----------------------+
| 1  alixpu                N/A    | 00000000:5E:00.1     |                    0 |
| N/A  34C   P0       100W / 200W | 4MiB / 134473MiB     | 10%          Default |
|                                 |                      |             Disabled |
+---------------------------------+----------------------+----------------------+

+-------------------------------------------------------------------------------+
| Processes:                                                                    |
| PPU    GI   CI   PID   Type   Process name                         PPU Memory |
|        ID   ID                                                     Usage      |
+===============================================================================+
| 0      N/A  N/A  25759 C      cuda_test1                               151MiB |
| 0      N/A  N/A  25677 C      cuda_test2                               151MiB |
+-------------------------------------------------------------------------------+

The query result is divided into two tables: a PPU device list and a process list. Information for unsupported features appears as N/A.

The following list describes the fields in the PPU device list table:

• PPU: The zero-based device index, assigned at power-on.

• Name: The device name.

• Persistence M.: Indicates whether persistence mode is enabled.

• Fan: The target fan speed as a percentage of its maximum speed.

• Temp: The current temperature of the device.

• Perf: The performance state of the device. P0 indicates the highest performance level, while P1, P2, and so on indicate progressively lower performance levels. If not supported, this field displays N/A.

• Pwr:Usage/Cap: The current power usage (W) and the power limit (W) of the device.

• Bus-Id: The PCI bus ID of the device.

• Memory-Usage: Used and total device memory (MiB).

• Volatile Uncorr. ECC: The total number of uncorrectable ECC errors on the device since the driver was loaded.

• PPU-Util: The current PPU stream processor utilization.

• Compute M.: The current compute mode, such as Default, Prohibited, or Exclusive Process (displayed as E. Process).

• MIG M.: The current MIG mode.

The following list describes the fields in the process list table:

• PPU: The zero-based device index, assigned at power-on.

• CI ID: The ID of the compute instance used by the process.

• PID: The process ID.

• Type: The type of the process. C indicates a compute process.

• Process name: The name of the process.

• PPU Memory Usage: The amount of PPU memory (MiB) used by this process.

3.2.1 MIG device information

If MIG mode is enabled on any PPU device, running ppu-smi displays a table of existing MIG devices.

+-----------------------------------------------------------------------------+
| MIG devices:                                                                |
+------------------+---------------------+----------+-------------------------+
| PPU  GI  CI  MIG |        Memory-Usage |      Vol |         Shared          |
|      ID  ID  DEV |                     |  CU  Unc | CpyEng ENC DEC OFA JPEG |
|                  |                     |      ECC |                         |
+==================+=====================+==========+=========================+
|  0   0   0    5  |   106MiB / 33618MiB |  64    0 |   4     2   2   0   0   |
+------------------+---------------------+----------+-------------------------+
|  1   1   2    3  |     6MiB / 16809MiB |  32    0 |   2     0   0   0   0   |
+------------------+---------------------+----------+-------------------------+
|  1   1   3    2  |     6MiB / 16809MiB |  32    0 |   2     0   0   0   0   |
+------------------+---------------------+----------+-------------------------+

The following list describes the fields in the MIG devices table:

• PPU: The zero-based device index, assigned at power-on.

• GI ID: The ID of the GPU instance associated with the MIG device.

• CI ID: The ID of the compute instance associated with the MIG device.

• MIG DEV: The enumeration number of the MIG device.

• Memory-Usage: Used and total device memory (MiB) for this MIG device.

• CU: The number of exclusive CU resources for this MIG device.

• Vol Unc ECC: The total number of uncorrectable ECC errors on the device since the driver was loaded.

• Shared: The resources listed below are shared by this MIG device and other MIG devices on the same GPU instance.

  • CpyEng: The number of shared copy engines.

  • ENC: The number of shared encoders.

  • DEC: The number of shared decoders.

  • OFA: The number of shared OFA processing units.

  • JPEG: The number of shared JPEG processing units.

3.2.2 Other options

4. Query device information

root@122d8d7a7e37:~# ppu-smi -q

==============PPUSMI LOG==============

Timestamp                                   : Wed Sep  7 19:13:36 2022
Driver Version                              : 510.39.01
HGGC Version                                : 11.6
SDK Version                                 : 1.4.45-5188b0

Attached PPUs                               : 1
PPU 00000000:01:00.0
    Product Name                            : alixpu
    Persistence Mode                        : Disabled
    MIG Mode
        Current                             : N/A
        Pending                             : N/A
    Serial Number                           : N/A
    PPU UUID                                : PPU-7f53d39f-ce6e-dc78-c3d4-4c18653c19c0
    PCI
        Bus                                 : 0x01
        Device                              : 0x00
        Domain                              : 0x0000
        Device Id                           : 0x1E0410DE
...

root@122d8d7a7e37:~# ppu-smi -q -d MEMORY
...
PPU 00000000:01:00.0
    HBM Vendor                              : Samsung
    Memory Usage
        Total                               : 11264 MiB
        Used                                : 248 MiB
        Free                                : 11016 MiB

root@2b92dd1ad851:~# ppu-smi -q -d UTILIZATION
...
PPU 00000000:3E:00.0
    Utilization
        Ppu                                 : 38 %
        Core                                : 30 %
        Memory                              : 0 %
        Encoder                             : 0 %
        Decoder                             : 0 %
    PPU Utilization Samples
        Duration                            : 19.89 sec
        Number of Samples                   : 99
        Max                                 : 56 %
        Min                                 : 1 %
        Avg                                 : 31 %
    Memory Utilization Samples
        Duration                            : 19.89 sec
        Number of Samples                   : 99
        Max                                 : 0 %
        Min                                 : 0 %
        Avg                                 : 0 %
    ENC Utilization Samples
        Duration                            : 19.89 sec
        Number of Samples                   : 99
        Max                                 : 0 %
        Min                                 : 0 %
        Avg                                 : 0 %
    DEC Utilization Samples
        Duration                            : 19.89 sec
        Number of Samples                   : 99
        Max                                 : 0 %
        Min                                 : 0 %
        Avg                                 : 0 %

root@2b92dd1ad851:~# ppu-smi -q -d ECC
...
PPU 00000000:3E:00.0
    Ecc Mode
        Current                             : Enabled
        Pending                             : Enabled
    ECC Errors
        Volatile
            SRAM Correctable                : 0
            SRAM Uncorrectable              : 0
            DRAM Correctable                : 0
            DRAM Uncorrectable              : 0
        Aggregate
            SRAM Correctable                : 0
            SRAM Uncorrectable              : 0
            DRAM Correctable                : 0
            DRAM Uncorrectable              : 0

root@2b92dd1ad851:~# ppu-smi -q -d TEMPERATURE
...
PPU 00000000:3E:00.0
    Temperature
        PPU Current Temp                    : 42 C
        PPU Shutdown Temp                   : 95 C
        PPU Slowdown Temp                   : 92 C
        PPU Max Operating Temp              : 85 C
        PPU Target Temperature              : N/A
        Memory Current Temp                 : 42 C
        Memory Max Operating Temp           : 95 C

root@2b92dd1ad851:~# ppu-smi -q -d POWER
...
PPU 00000000:3E:00.0
    Power Readings
        Power Management                    : Supported
        Power Draw                          : 51.99 W
        Power Limit                         : 250.00 W
        Default Power Limit                 : 250.00 W
        Enforced Power Limit                : 250.00 W
        Min Power Limit                     : 150.00 W
        Max Power Limit                     : 250.00 W
    Power Samples
        Duration                            : 2.39 sec
        Number of Samples                   : 119
        Max                                 : 58.68 W
        Min                                 : 51.40 W
        Avg                                 : 55.23 W

root@2b92dd1ad851:~# ppu-smi -q -d CLOCK
...
PPU 00000000:08:00.0                        
    Clocks                                  
        CU                                  : 200 MHz
        Memory                              : 1800 MHz
        Video                               : 1000 MHz
    Applications Clocks                     
        CU                                  : 1700 MHz
        Memory                              : 1800 MHz
    Default Applications Clocks             
        CU                                  : 1700 MHz
        Memory                              : 1800 MHz
    Max Clocks                              
        CU                                  : 1700 MHz
        Memory                              : 1800 MHz
        Video                               : 1000 MHz

root@2b92dd1ad851:~# ppu-smi -q -d COMPUTE
...
PPU 00000000:3E:00.0
    Compute Mode                            : Default

root@2b92dd1ad851:~# ppu-smi -q -d PIDS
...
PPU 00000000:3E:00.0
    Processes
        Compute instance ID                 : N/A
        Process ID                          : 85637
            Type                            : Compute
            Name                            : ppu_test
            Used PPU Memory                 : 413 MiB
        Compute instance ID                 : N/A
        Process ID                          : 87771
            Type                            : Compute
            Name                            : ppu_test_threads
            Used PPU Memory                 : 413 MiB

root@122d8d7a7e37:~# ppu-smi -q -d SUPPORTED_CLOCKS
...
PPU 00000000:01:00.0
    Supported Clocks
        Memory                              : 7000 MHz
            CU                              : 2100 MHz
            CU                              : 2085 MHz
            CU                              : 2070 MHz
            CU                              : 2055 MHz
            CU                              : 2040 MHz
            ...
        Memory                              : 6800 MHz
            CU                              : 2100 MHz
            CU                              : 2085 MHz
            CU                              : 2070 MHz
            CU                              : 2055 MHz
            CU                              : 2040 MHz
            ...

root@122d8d7a7e37:~# ppu-smi -q -d PAGE_RETIREMENT
...
PPU 00000000:10:00.0
    Retired Pages
        Single Bit ECC                      : 0
        Double Bit ECC                      : 0
        Pending Page Blacklist              : No

root@122d8d7a7e37:~# ppu-smi -q -d ROW_REMAPPER
...
PPU 00000000:10:00.0
    Remapped Rows
        Correctable Error                   : 0
        Uncorrectable Error                 : 0
        Pending                             : No
        Remapping Failure Occurred          : No
        Bank Remap Availability Histogram
            Max                             : 3072 bank(s)
            High                            : 0 bank(s)
            Partial                         : 0 bank(s)
            Low                             : 0 bank(s)
            None                            : 0 bank(s)

root@2b92dd1ad851:~# ppu-smi --query-ppu=timestamp,index,name,compute_mode,memory.total,memory.used --format=csv
timestamp, index, name, compute_mode, memory.total [MiB], memory.used [MiB]
2022/09/08 09:24:11.132, 0, alixpu, Default, 40960 MiB, 606 MiB
2022/09/08 09:24:11.139, 1, alixpu, Default, 24576 MiB, 508 MiB

root@2b92dd1ad851:~# ppu-smi --help-query-ppu
List of valid properties to query for the switch "--query-ppu=":

"timestamp"
The timestamp of when the query was made in format "YYYY/MM/DD HH:MM:SS.msec".

"driver_version"
The version of the installed driver. This is an alphanumeric string.

"count"
The number of PPUs in the system.

"name" or "ppu_name"
The official product name of the PPU. This is an alphanumeric string. For all products.

"serial" or "ppu_serial"
This globally unique, immutable, alphanumeric value matches the serial number physically printed on the board.

"uuid" or "ppu_uuid"
A globally unique, immutable, alphanumeric identifier for the PPU that does not correspond to any physical label on the board.

"pci.bus_id" or "ppu_bus_id"
PCI bus id as "domain:bus:device.function", in hex.
...

root@2b92dd1ad851:~# ppu-smi --query-ppu=timestamp,count,name,serial,index,uuid --format=csv
timestamp, count, name, serial, index, uuid
2022/09/08 09:42:34.913, 2, alixpu, 1320421013145, 0, PPU-0cdd7938-b576-2411-a408-3ad81dfc1a78
2022/09/08 09:42:34.920, 2, alixpu, 1323921045367, 1, PPU-16c4c41f-9214-5e29-3b86-7a26ab011d3e

root@2b92dd1ad851:~# ppu-smi --help-query-supported-clocks
List of valid properties to query for the switch "--query-supported-clocks=":

[Section about Supported Clocks properties]
Lists possible combinations of memory and processor clocks at which the PPU can operate (not taking into account HW brake reduced clocks).

"timestamp"
The timestamp of when the query was made in format "YYYY/MM/DD HH:MM:SS.msec".

"name" or "ppu_name"
The official product name of the PPU. This is an alphanumeric string. For all products.

...

"memory" or "mem"
Memory clock.

"processor" or "sm"
Streaming multiprocessor clock.

root@2b92dd1ad851:~# ppu-smi --query-supported-clocks=timestamp,ppu_bus_id,memory,processor --format=csv
timestamp, pci.bus_id, memory [MHz], processor [MHz]
2022/09/08 09:50:26.289, 00000000:3E:00.0, 1215 MHz, 1410 MHz
2022/09/08 09:50:26.289, 00000000:3E:00.0, 1215 MHz, 1395 MHz
2022/09/08 09:50:26.289, 00000000:3E:00.0, 1215 MHz, 1380 MHz
2022/09/08 09:50:26.289, 00000000:3E:00.0, 1215 MHz, 1365 MHz
2022/09/08 09:50:26.289, 00000000:3E:00.0, 1215 MHz, 1350 MHz
2022/09/08 09:50:26.289, 00000000:3E:00.0, 1215 MHz, 1335 MHz
2022/09/08 09:50:26.289, 00000000:3E:00.0, 1215 MHz, 1320 MHz
2022/09/08 09:50:26.289, 00000000:3E:00.0, 1215 MHz, 1305 MHz
2022/09/08 09:50:26.289, 00000000:3E:00.0, 1215 MHz, 1290 MHz
...

root@2b92dd1ad851:~# ppu-smi --help-query-compute-apps
List of valid properties to query for the switch "--query-compute-apps=":

[Section about Active Compute Processes properties]
Lists processes that have a compute context on the device.

"timestamp"
The timestamp of when the query was made in format "YYYY/MM/DD HH:MM:SS.msec".

"name" or "ppu_name"
The official product name of the PPU. This is an alphanumeric string. For all products.

...

"pid"
Process ID of the compute application.

"process_name"
Process Name.

"used_ppu_memory" or "used_memory"
The amount of memory on the device used by the context.

root@2b92dd1ad851:~# ppu-smi --query-compute-apps=timestamp,uuid,pid,process_name,used_ppu_memory --format=csv
timestamp, uuid, pid, process_name, used_ppu_memory [MiB]
2022/09/08 09:57:57.660, PPU-0cdd7938-b576-2411-a408-3ad81dfc1a78, 61785, ppu_test, 413 MiB
2022/09/08 09:57:57.660, PPU-0cdd7938-b576-2411-a408-3ad81dfc1a78, 62103, ppu_test_threads, 413 MiB

root@0549cf16bb85:~# ppu-smi --help-query-retired-pages
List of valid properties to query for the switch "--query-retired-pages=":

[Section about Retired Pages properties]
Lists pages that have been retired or are pending retirement.

"timestamp"
The timestamp of when the query was made in format "YYYY/MM/DD HH:MM:SS.msec".

"name" or "ppu_name"
The official product name of the PPU. This is an alphanumeric string. For all products.
...

"retired_pages.address"
Address of a retired page. Address might be different when ECC is Enabled or Disabled.

"retired_pages.timestamp"
Timestamp at which the page was retired.

"retired_pages.cause"
The reason the page was retired. Can be one of the following values:
 - Double Bit ECC: Indicates the page was retired due to a double-bit ECC error.
 - Single Bit ECC: Indicates the page was retired due to multiple single-bit ECC errors.

root@2b92dd1ad851:~# ppu-smi --query-retired-pages=timestamp,uuid,retired_pages.address,retired_pages.timestamp,retired_pages.cause --format=csv
timestamp, uuid, retired_pages.address, retired_pages.timestamp, retired_pages.cause
12:32:15.329, PPU-3f53d39f-ce6e-dc78-c3d4-4c18653c19c0, 0x0000000073234722, 1663256354, Single Bit ECC

root@0549cf16bb85:~# ppu-smi --help-query-remapped-rows
List of valid properties to query for the switch "--query-remapped-rows=":

"timestamp"
The timestamp of when the query was made in format "YYYY/MM/DD HH:MM:SS.msec".

"name" or "ppu_name"
The official product name of the PPU. This is an alphanumeric string. For all products.
...

"remapped_rows.correctable"
The number of rows that have been remapped due to correctable ECC errors.

"remapped_rows.uncorrectable"
The number of rows that have been remapped due to uncorrectable ECC errors.

"remapped_rows.pending"
Indicates whether row remappings are pending.

"remapped_rows.failure"
Indicates if a row remapping has ever failed.

"remap_availability.bank_histogram.max"
The number of banks that have max remap availability (all reserved rows are available).

"remap_availability.bank_histogram.high"
The number of banks that have high remap availability.

"remap_availability.bank_histogram.partial"
The number of banks that have partial remap availability.
...

root@2b92dd1ad851:~# ppu-smi --query-remapped-rows=timestamp,uuid,remapped_rows.correctable,remapped_rows.uncorrectable,remap_availability.bank_histogram.max --format=csv
timestamp, uuid, remapped_rows.correctable, remapped_rows.uncorrectable, remap_availability.bank_histogram.max
2022/09/08 16:24:40.435, PPU-099ea108-0181-0230-0000-000060f19f20, 0, 0, 3072
2022/09/08 16:24:40.435, PPU-019ea108-01a1-0222-0000-000040dff62d, 0, 0, 3072
2022/09/08 16:24:40.435, PPU-019ea108-0121-0222-0000-0000605e6417, 0, 0, 3072

5. Modify device configuration

5.1 Device configuration options

PPU-SMI lets you modify the device configuration. For example, run ppu-smi -lpc 1410:

root@122d8d7a7e37:~# ppu-smi -lpc 1410
Set PPU clock to (min clock 1410MHz, max clock 1410MHz) for PPU 00000000:01:00.0.
All done.

The following options are supported. Each ppu-smi command can modify only one type of device configuration at a time.

5.1.1 Reset a PPU device

ppu-smi -r resets all PPU devices. You can use the -i option to reset a specific PPU device. For example, ppu-smi -r -i 0 resets PPU device 0. To reset a PPU device, the following prerequisites apply:

1. No HGGC-related applications (compute applications) are running on any PPU device.

2. No PPU-related monitoring or tool applications, such as ppu-smi, ppudbg, or PPU DCGM, are running on any PPU device.

If the reset fails because one of the applications listed above is running, use the following commands to identify them:

# Query for HGGC-related applications
ppu-smi pmon -c 1

# Query for PPU-related monitoring and tool applications
lsof /dev/alixpu

5.2 Other options

6. Collect device statistics

The PPU-SMI stats subcommand collects device sample and event information, printing the data in CSV format for operational analysis. To view the subcommand's help information, run ppu-smi stats -h:

root@2b92dd1ad851:~# ppu-smi stats -h
Generates PPU statistics such as power samples,
utilization samples, xid events, clock change events
and performance capping events.

ppu-smi stats [OPTION1 [ARG1]] [OPTION2 [ARG2]] ...
    -i,   --id
        Enumeration index, Serial number, PCI bus ID or UUID.
        Provide comma separated values for more than one device
    -f,   --filename
        Log to a specified file, rather than to stdout
    -d,   --display
        Display specific metric:
            pwrDraw,temp,memUtil,ppuUtil,
            encUtil,decUtil,memClk,procClk,
            violPwr,violThm,xidEvent,sbEcc,
            dbEcc,pState,clkChg,pwrChg,migChg
        Metric can be combined with comma e.g. pwrDraw,temp
    -c,   --count
        Run for specified number of monitoring cycles and exit
    -h,   --help
        Display help information

Stats in following CSV format:
Device, Power Drawn (pwrDraw), Timestamp (us), Value (Watts)
Device, PPU Temperature (temp), Timestamp (us), Value (C)
Device, PPU Util (ppuUtil), Timestamp (us), Value (%)
...

6.1 Output format

The command outputs query results in CSV format, as shown in the following example:

root@122d8d7a7e37:~# ppu-smi stats
0, violPwr , 1662606252538553, 0
0, violThm , 1662606252537681, 0
0, temp    , 1662606252536736, 51
0, pState  , 1662606252559897, 0
0, clkChg  , 1662606252559964, 0
0, pState  , 1662606252568416, 0
0, clkChg  , 1662606252568499, 0
...
0, temp    , 1662606253541531, 52
0, pwrDraw , 1662606252543916, 45
0, pwrDraw , 1662606252567432, 54
0, pwrDraw , 1662606252584972, 66
0, pwrDraw , 1662606252605044, 66
...

The CSV table has no header. The columns are as follows:

1. The first column is the device index: a zero-based enumeration index assigned to the device at power-on.

2. The second column is the abbreviation for the metric name, such as pwrDraw or temp.

3. The third column is the timestamp in microseconds, representing the number of microseconds from the system clock epoch to the time of the sample.

4. The fourth column contains the unitless sampled value. Refer to the metric descriptions in the next section for an explanation of each value's meaning.

6.2 Metrics

You can specify the metrics to collect with the -d option. Separate multiple metrics with a comma (,), for example, ppu-smi stats -d pwrDraw,temp:

root@122d8d7a7e37:~# ppu-smi stats -d pwrDraw,temp
0, temp    , 1662606935107702, 51
0, pwrDraw , 1662606935059582, 38
0, pwrDraw , 1662606935079459, 39
0, pwrDraw , 1662606935099668, 39
...

If the -d option is omitted, all supported metrics are collected by default. The following table describes the supported metrics:

For event-based metrics (such as xidEvent), PPU-SMI prints output when an event occurs. For sample-based metrics (such as pwrDraw), it aggregates samples and prints the results once per second.

By default, PPU-SMI runs continuously and collects data until you interrupt it by pressing Ctrl+C.

6.3 Other options

7. Monitor device status

Use the PPU-SMI dmon subcommand to display device statistics in a scrolling format. It prints one line of statistics per device per sampling interval, with each metric in a separate column. For example, run ppu-smi dmon:

root@2b92dd1ad851:~# ppu-smi dmon
# ppu   pwr ptemp mtemp    cu  core   mem   enc   dec  mclk  pclk
# idx     W     C     C     %     %     %     %     %   MHz   MHz
    0   109    30    32     0     2     0     0     0  1800  1500
    1   119    33    34     0     0     0     0     0  1800  1500
    0   109    30    32     0     0     0     0     0  1800  1500
    1   121    33    34     0     0     0     0     0  1800  1500
...

Run ppu-smi dmon -h for help on this feature:

root@2b92dd1ad851:~# ppu-smi dmon -h
PPU statistics are displayed in scrolling format with one line
per sampling interval. Metrics to be monitored can be adjusted
based on the width of terminal window.

ppu-smi dmon [OPTION1 [ARG1]] [OPTION2 [ARG2]] ...
    -i,   --id=
        Enumeration index, Serial number, PCI bus ID or UUID.
        Provide comma separated values for more than one device
    -d,   --delay=
        Collection delay/interval in seconds [default=1sec]
    -c,   --count=
        Collect specified number of samples and exit
    -s,   --select=
        One or more metrics [default=puc]
        Can be any of the following:
            p - Power Usage and Temperature
            u - Utilization
            c - Proc and Mem Clocks
            v - Power and Thermal Violations
            m - SM Memory
            e - ECC Errors and PCIe Replay errors
            t - PCIe Rx and Tx Throughput
    -o,   --options=
        One or more from the following:
            D - Include Date (YYYYMMDD) in scrolling output
            T - Include Time (HH:MM:SS) in scrolling output
...

7.1 Monitored metrics

Use the -s option to specify the metrics you want to monitor. Each metric group is represented by a single character. You can combine these characters to monitor multiple metric groups. For example, running ppu-smi dmon -s pc displays metrics related to power (p) and clocks (c).

root@2b92dd1ad851:~# ppu-smi dmon -s pc
# ppu   pwr ptemp mtemp  mclk  pclk
# idx     W     C     C   MHz   MHz
    0    41    45    45  1215   765
    1    30    36    36  1215   930
...

The following table describes the supported metrics:

The following list describes the output columns:

• Basic information

  • Date: The current date.

  • Time: The current time.

  • ppu: The zero-based enumeration index assigned to the device at power-on.

• Metrics for type p

  • pwr: The current power draw in watts (W).

  • ptemp: The current processor temperature.

  • mtemp: The current memory temperature.

• Metrics for type u

  • sm: Streaming Multiprocessor utilization in percent (%).

  • core: Core utilization in percent (%).

  • mem: Memory utilization in percent (%).

  • enc: Encoder utilization in percent (%).

  • dec: Decoder utilization in percent (%).

• Metrics for type c

  • pclk: The processor clock domain frequency in MHz.

  • mclk: The memory clock domain frequency in MHz.

• Metrics for type v

  • pviol: The percentage of time the device was throttled due to power limits during the sampling interval (%).

  • tviol: The percentage of time the device was throttled due to temperature limits during the sampling interval (%).

• Metrics for type m

• mem: The amount of used memory in MiB.

• Metrics for type e

  • sbecc: Number of single-bit ECC errors since the driver was loaded.

  • dbecc: Number of double-bit ECC errors since the driver was loaded.

  • pci: Number of PCIe replay errors.

• Metrics for type t

  • rxpci: PCIe receive throughput in MB/s.

  • txpci: PCIe transmit throughput in MB/s.

7.2 Other options

8. Monitor process status

Use the pmon subcommand to monitor processes on your devices. This command displays process information in a scrolling format, printing one line per process with each metric in a separate column. For example, running ppu-smi pmon:

root@122d8d7a7e37:~# ppu-smi pmon
# ppu     pid  type    sm   mem   enc   dec   command
# idx       #   C/G     %     %     %     %   name
    0    4563     C     0     0     0     0   ppu_test
    0    4991     C     0     0     0     0   ppu_test_thread
...

Run ppu-smi pmon -h to view help information for this subcommand:

root@122d8d7a7e37:~# ppu-smi pmon -h
Process statistics are displayed in scrolling format per sampling
interval. This tool lists the statistics for all the compute
processes running on each device. Metrics to be monitored
can be adjusted based on the width of terminal window.

ppu-smi pmon [OPTION1 [ARG1]] [OPTION2 [ARG2]] ...
    -i,   --id=
        Enumeration index, Serial number, PCI bus ID or UUID.
        Provide comma separated values for more than one device
    -d,   --delay=
        Collection delay/interval in seconds [default=1sec]
    -c,   --count=
        Collect specified number of samples and exit
    -s,   --select=
        One or more metrics [default=u]
        Can be any of the following:
            u - Utilization
            m - Memory usage
    -o,   --options=
        One or more from the following:
            D - Include Date (YYYYMMDD) in scrolling output
            T - Include Time (HH:MM:SS) in scrolling output
...

8.1 Metrics

Use the -s option to specify which metrics to monitor. Each metric group is identified by a single character. To monitor multiple groups, pass a concatenated string of their characters to the -s option. For example, running ppu-smi pmon -s um displays metrics related to process utilization (u) and memory usage (m).

root@122d8d7a7e37:~# ppu-smi pmon -s um
# ppu     pid  type    sm   mem   enc   dec   mem   command
# idx       #   C/G     %     %     %     %    MB   name
    0    4563     C     0     0     0     0   151   ppu_test
    0    4991     C     0     0     0     0   151   ppu_test_thread

The following table describes the supported metrics:

The following list describes the columns in the PPU-SMI scrolling output:

• Basic information

  • Date: The current date.

  • Time: The current time.

  • ppu: The zero-based enumeration index assigned to the device upon power-on.

  • pid: The system process ID (PID).

  • type: The process type. C indicates a compute process.

• Metric group u

  • sm: Stream processor utilization, in percent (%).

  • mem: Memory utilization, in percent (%).

  • enc: Encoder utilization, in percent (%).

  • dec: Decoder utilization, in percent (%).

• Metric group m

  • mem: Process memory usage, in MiB.

8.2 Other options

9. Query ICN link information

PPU-SMI provides the icn subcommand to query ICN link information. Results are grouped by device and ICN link. For example, run ppu-smi icn -s.

root@dfc623e46a90:~# ppu-smi icn -s
PPU 0: PPU (UUID: GPU-019ea108-c180-040a-0000-000000000000)
    Link 0: 50 GB/s
    Link 1: <inactive>
    Link 2: <inactive>
    Link 3: 50 GB/s
    Link 4: 50 GB/s
    Link 5: 50 GB/s
    Link 6: 50 GB/s
PPU 1: PPU (UUID: GPU-019ea108-c180-060c-0000-000000000000)
    Link 0: 50 GB/s
    Link 1: <inactive>
    Link 2: <inactive>
    Link 3: 50 GB/s
...

Run ppu-smi icn -h to view the help information for the ICN feature. You can use the -i and -l options to limit the query to specific devices and links; If you omit these options, the command queries all devices and links. You must specify exactly one query sub-option (for example, -s) in each command:

root@dfc623e46a90:~# ppu-smi icn -h
icn -- Display ICN link information.

ppu-smi icn [OPTION1 [ARG1]] [OPTION2 [ARG2]] ...
    -h,   --help
        Display help information
    -i,   --id=
        Enumeration index, Serial number, PCI bus ID or UUID.
        Provide comma separated values for more than one device.
    -l,   --link=
        Specify a target link ID (0-based link index),
        Without this parameter, all links information are displayed.

    [any one of]

    -s,   --status
        Display link state (active/inactive).
    -c,   --capabilities
        Display link capabilities.
    -p,   --pcibusid
        Display remote node PCI bus ID for a link.
    -r,   --remotelinkinfo
        Display remote device PCI bus ID and ICN link ID for a link.

9.1 Query ICN link status

Run ppu-smi icn -s to query the status and bandwidth of ICN links. You can use the -i option to limit the query to a specific device. For example, run ppu-smi icn -s -i 0.

root@dfc623e46a90:~# ppu-smi icn -s -i 0
PPU 0: PPU (UUID: GPU-019ea108-c180-040a-0000-000000000000)
    Link 0: 50 GB/s
    Link 1: <inactive>
    Link 2: <inactive>
    Link 3: 50 GB/s
    Link 4: 50 GB/s
    Link 5: 50 GB/s
    Link 6: 50 GB/s

• Link 0: 50 GB/s: Indicates that ICN link 0 on PPU 0 is connected to another device with a 50 GB/s bandwidth.

• Link 1: <inactive>: ICN link 1 on PPU 0 is not connected to any device.

9.2 Query ICN link capabilities

Run ppu-smi icn -c to query the capabilities of each ICN link. Only information for active ICN links is displayed.

root@dfc623e46a90:~# ppu-smi icn -c
PPU 0: PPU (UUID: GPU-019ea108-c180-040a-0000-000000000000)
    Link 0, P2P is supported: true
    Link 0, Access to system memory is supported: false
    Link 0, P2P atomics is supported: true
    Link 0, System memory atomics is supported: false
    Link 0, SLI is supported: false
    Link 0, Link is supported: true
    Link 3, P2P is supported: true
    Link 3, Access to system memory is supported: false
    Link 3, P2P atomics is supported: true
    Link 3, System memory atomics is supported: false
    Link 3, SLI is supported: false
    Link 3, Link is supported: true
...

9.3 Query remote PCI bus ID

Run ppu-smi icn -p to query the PCI bus ID of the remote device connected via an ICN link. Only information for active ICN links is displayed.

root@dfc623e46a90:~# ppu-smi icn -p
PPU 0: PPU (UUID: GPU-019ea108-c180-040a-0000-000000000000)
    Link 0: 00000001:CE:00.0
    Link 3: 00000000:89:00.0
    Link 4: 00000000:89:00.0
    Link 5: 00000000:CC:00.0
    Link 6: 00000001:CE:00.0
PPU 1: PPU (UUID: GPU-019ea108-c180-060c-0000-000000000000)
    Link 0: 00000001:D1:00.0
    Link 3: 00000000:C9:00.0
    Link 4: 00000000:86:00.0
    Link 5: 00000000:86:00.0
    Link 6: 00000001:D1:00.0
...

9.4 Query remote link information

Run ppu-smi icn -r to query information about the remote endpoint of an ICN link, including the PCI bus ID of the remote device and its corresponding ICN link index.

root@dfc623e46a90:~# ppu-smi icn -r
PPU 0: PPU (UUID: GPU-019ea108-c180-040a-0000-000000000000)
    Link 0: Remote Device 00000001:CE:00.0: Link 6
    Link 3: Remote Device 00000000:89:00.0: Link 1
    Link 4: Remote Device 00000000:89:00.0: Link 1
    Link 5: Remote Device 00000000:CC:00.0: Link 3
    Link 6: Remote Device 00000001:CE:00.0: Link 6
PPU 1: PPU (UUID: GPU-019ea108-c180-060c-0000-000000000000)
    Link 0: Remote Device 00000001:D1:00.0: Link 7
    Link 3: Remote Device 00000000:C9:00.0: Link 2
    Link 4: Remote Device 00000000:86:00.0: Link 0
    Link 5: Remote Device 00000000:86:00.0: Link 0
    Link 6: Remote Device 00000001:D1:00.0: Link 7
...

9.5 Query cable presence status

Run ppu-smi icn -cs to query the cable presence status for each ICN link port.

root@dfc623e46a90:~# ppu-smi icn -cs
PPU 0: t-head ppu 0 (UUID: GPU-3f53d39f-ce6e-dc78-c3d4-4c18653c19c0)
    Link 0: Connected
    Link 1: Disconnected
    Link 2: Connected
...
PPU 1: t-head ppu 1 (UUID: GPU-3f53d39f-ce6e-dc78-c3d4-4c18653c19c1)
    Link 0: Connected
    Link 1: Connected
    Link 2: Connected
...

9.6 Query link bit width

Run ppu-smi icn -lw to query the ICN link bit width. Only information for active ICN links is displayed.

root@dfc623e46a90:~# ppu-smi icn -lw
PPU 0: PPU-ZW810 (UUID: GPU-019ea108-4111-0220-0000-0000006ef62f)
    Link 0: 16x
    Link 3: 16x
    Link 4: 16x
    Link 5: 16x
    Link 6: 8x
PPU 1: PPU-ZW810 (UUID: GPU-019ea108-8191-042a-0000-0000c0d6cc1d)
    Link 0: 16x
    Link 3: 16x
    Link 4: 16x
    Link 5: 16x
    Link 6: 8x
...

• 16x: The link bit width is 16 bits.

• 8x: The link bit width is 8 bits.

9.7 Query ICN link to physical port mapping

Run ppu-smi icn -lm to query the mapping between ICN links and physical ports.

root@dfc623e46a90:~# ppu-smi icn -lm
PPU 0: PPU-ZW810 (UUID: GPU-019ea108-4111-0220-0000-0000006ef62f)
    Link 0: Physical Port 2
    Link 1: N/A
    Link 2: Physical Port 5
...
PPU 1: PPU-ZW810 (UUID: GPU-019ea108-8191-042a-0000-0000c0d6cc1d)
    Link 0: Physical Port 2
    Link 1: N/A
    Link 2: Physical Port 5
...

• The command displays N/A for links that do not have a physical port mapping.

9.8 Query link statistics and error counts

Run ppu-smi icn -e to query statistics and error counts for ICN links. Only information for active ICN links is displayed.

PPU 0: PPU-ZW810 (UUID: GPU-019ea108-4111-0220-0000-0000006ef62f)
    Link 0: 2 link up times
    Link 0: 1 link down times
    Link 0: 0 FEC correctable errors
    Link 0: 0 FEC uncorrectable errors
    Link 0: 0 TX packet errors
    Link 0: 0 RX packet errors
    Link 0: 13 total TX packets
    Link 0: 11 total RX packets

    Link 3: 2 link up times
    Link 3: 1 link down times
    Link 3: 0 FEC correctable errors
    Link 3: 0 FEC uncorrectable errors
    Link 3: 0 TX packet errors
    Link 3: 0 RX packet errors
    Link 3: 16 total TX packets
    Link 3: 8 total RX packets
...

PPU 1: PPU-ZW810 (UUID: GPU-019ea108-8191-042a-0000-0000c0d6cc1d)
    Link 0: 2 link up times
    Link 0: 1 link down times
    Link 0: 0 FEC correctable errors
    Link 0: 0 FEC uncorrectable errors
    Link 0: 0 TX packet errors
    Link 0: 0 RX packet errors
    Link 0: 9 total TX packets
    Link 0: 4 total RX packets
...

• link up times: The number of times the link has transitioned to the active state.

• link down times: The number of times the link has transitioned to the inactive state.

• FEC correctable errors: The number of correctable errors detected by Forward Error Correction (FEC).

• FEC uncorrectable errors: The number of uncorrectable errors detected by Forward Error Correction (FEC).

• TX packet errors: The number of packet transmission errors.

• RX packet errors: The number of packet reception errors.

• total TX packets: The total number of packets transmitted over the link.

• total RX packets: The total number of packets received over the link.

9.9 Query link throughput

Run ppu-smi icn -gt r to query the current aggregated data throughput for ICN links.

root@0549cf16bb85:~# ppu-smi icn -gt r
PPU 0: PPU (UUID: GPU-019ea108-c110-0828-0000-000000000000)
    Link 0: Raw Tx: 1618498333 KiB
    Link 0: Raw Rx: 1657515730 KiB
    Link 3: Raw Tx: 1653010223 KiB
    Link 3: Raw Rx: 1693049419 KiB
    Link 4: Raw Tx: 1622645860 KiB
    Link 4: Raw Rx: 1662045442 KiB
...
PPU 1: PPU (UUID: GPU-019ea108-c120-040c-0000-000000000000)
    Link 0: Raw Tx: 1618498333 KiB
    Link 0: Raw Rx: 1657515730 KiB
    Link 3: Raw Tx: 1653010223 KiB
    Link 3: Raw Rx: 1693049419 KiB
    Link 4: Raw Tx: 1622645860 KiB
    Link 4: Raw Rx: 1662045442 KiB
...

10. Query device topology

To query topological information between devices, use the PPU-SMI topo subcommand. For example, run ppu-smi topo -m:

root@dfc623e46a90:~# ppu-smi topo -m
         PPU0    PPU1    PPU2    PPU3    PPU4    PPU5    PPU6    PPU7    CPU Affinity    NUMA Affinity
 PPU0    X       ICN2    SYS     ICN1    SYS     SYS     ICN2    SYS     0-47,96-143     0
 PPU1    ICN2    X       ICN1    SYS     SYS     SYS     SYS     ICN2    0-47,96-143     0
 PPU2    SYS     ICN1    X       ICN2    ICN1    ICN1    SYS     SYS     0-47,96-143     0
 PPU3    ICN1    SYS     ICN2    X       ICN1    ICN1    SYS     SYS     0-47,96-143     0
 PPU4    SYS     SYS     ICN1    ICN1    X       ICN2    SYS     ICN1    48-95,144-191   1
 PPU5    SYS     SYS     ICN1    ICN1    ICN2    X       ICN1    SYS     48-95,144-191   1
 PPU6    ICN2    SYS     SYS     SYS     SYS     ICN1    X       ICN2    48-95,144-191   1
 PPU7    SYS     ICN2    SYS     SYS     ICN1    SYS     ICN2    X       48-95,144-191   1
...

Run ppu-smi topo -h to view help for the topo subcommand. You can specify one topo query option, such as -m, per command. When you use the -n or -p options, you must also use the -i option to specify the target devices:

root@dfc623e46a90:~# ppu-smi topo -h
topo -- Display topological information about the system.

ppu-smi topo [OPTION1 [ARG1]] [OPTION2 [ARG2]] ...
    -h,   --help
        Display help information
    -i,   --id=
        Enumeration index, Serial number, PCI bus ID or UUID.
        Provide comma separated values for more than one device.
        Must be used in conjunction with -n or -p.
    -ri,  --rear-id=
        When used with the option to display matrix (-m or -mp), 
        Show PPU devices that match the specified rear id.
    -po,  --ppu-only
        When used with the option to display matrix (-m or -mp),
        Show PPU devices only.
    -rg,  --rear-group
        When used with the option to display matrix (-m or -mp), 
        Group PPU device by rear id.

    [any one of]

    -m,   --matrix
        Display the PPUDirect communication matrix for the system.
    -mp,  --matrix_pci
        Display the PPUDirect communication matrix for the system (PCI Only).
    -lni, --list-network-interface
        Display a list of RDMA network interface controller(NIC) connected to the system.
    -c,   --cpu=
        Specify a CPU number, Display all PPUs with an affinity.
    -n,   --nearest_ppus=
        Display the nearest PPUs for a given traversal path.
        Could be one of the following:
            0 = a single PCIe switch on a dual PPU board
            1 = a single PCIe switch
            2 = multiple PCIe switches
            3 = a PCIe host bridge
            4 = an on-CPU interconnect link between PCIe host bridges
            5 = an SMP interconnect link between NUMA nodes
        Used in conjunction with -i which must be a single device ID.
    -p,   --ppu_path
        Display the most direct path traversal for a pair of PPUs.
        Used in conjunction with -i which must be a pair of device IDs.
    -p2p, --p2pstatus=
        Displays the p2p status between the PPUs of a given p2p capability.
        Could be one of the following:
            r - p2p read capabiity
            w - p2p write capability
            n - p2p ICN link capability
            a - p2p atomics capability
            p - p2p prop capability

10.1 Query the communication matrix

Run ppu-smi topo -m to query the connection status between pairs of devices, and between PPU devices and RDMA NICs. If an ICN link is active, the matrix displays its status instead of the underlying PCIe bus connection. The following describes the output:

root@dfc623e46a90:/# ppu-smi topo -m
         PPU0    PPU1    PPU2    PPU3    PPU4    PPU5    PPU6    PPU7    NIC0    NIC1    NIC2    NIC3    NIC4    NIC5    NIC6    NIC7    CPU Affinity    NUMA Affinity
 PPU0    X       ICN2    SYS     ICN1    SYS     SYS     ICN2    SYS     PXB     SYS     SYS     SYS     SYS     SYS     PXB     SYS     0-47,96-143     0
 PPU1    ICN2    X       ICN1    SYS     SYS     SYS     SYS     ICN2    PIX     SYS     SYS     SYS     SYS     SYS     PIX     SYS     0-47,96-143     0
 PPU2    SYS     ICN1    X       ICN2    ICN1    ICN1    SYS     SYS     SYS     PXB     SYS     SYS     SYS     SYS     SYS     PXB     0-47,96-143     0
 PPU3    ICN1    SYS     ICN2    X       ICN1    ICN1    SYS     SYS     SYS     PXB     SYS     SYS     SYS     SYS     SYS     PXB     0-47,96-143     0
 PPU4    SYS     SYS     ICN1    ICN1    X       ICN2    SYS     ICN1    SYS     SYS     SYS     PXB     SYS     SYS     SYS     SYS     48-95,144-191   1
 PPU5    SYS     SYS     ICN1    ICN1    ICN2    X       ICN1    SYS     SYS     SYS     SYS     PXB     SYS     SYS     SYS     SYS     48-95,144-191   1
 PPU6    ICN2    SYS     SYS     SYS     SYS     ICN1    X       ICN2    SYS     SYS     PXB     SYS     SYS     PXB     SYS     SYS     48-95,144-191   1
 PPU7    SYS     ICN2    SYS     SYS     ICN1    SYS     ICN2    X       SYS     SYS     PIX     SYS     SYS     PIX     SYS     SYS     48-95,144-191   1
 NIC0    PXB     PIX     SYS     SYS     SYS     SYS     SYS     SYS     X       SYS     SYS     SYS     SYS     SYS     PIX     SYS     0-47,96-143     0
 NIC1    SYS     SYS     PXB     PXB     SYS     SYS     SYS     SYS     SYS     X       SYS     SYS     SYS     SYS     SYS     PIX     0-47,96-143     0
 NIC3    SYS     SYS     SYS     SYS     SYS     SYS     PXB     PIX     SYS     SYS     X       SYS     SYS     PIX     SYS     SYS     48-95,144-191   1
 NIC4    SYS     SYS     SYS     SYS     PXB     PXB     SYS     SYS     SYS     SYS     SYS     X       SYS     SYS     SYS     SYS     48-95,144-191   1
 NIC5    SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     X       SYS     SYS     SYS     0-47,96-143     0
 NIC6    SYS     SYS     SYS     SYS     SYS     SYS     PXB     PIX     SYS     SYS     PIX     SYS     SYS     X       SYS     SYS     48-95,144-191   1
 NIC7    PXB     PIX     SYS     SYS     SYS     SYS     SYS     SYS     PIX     SYS     SYS     SYS     SYS     SYS     X       SYS     0-47,96-143     0

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  ICN# = Connection traversing a bonded set of # ICN links

NIC Legend:

  NIC0: mlx5_bond_0
  NIC1: mlx5_bond_1
  NIC2: mlx5_bond_2
  NIC3: mlx5_bond_3
  NIC4: mlx5_bond_4
  NIC5: mlx5_bond_5
  NIC6: mlx5_bond_6
  NIC7: mlx5_bond_7

PPU Rear Group:

  Rear ID 0: PPU 0,1,2,3,4,5,6,7

• ICN2: For example, if there are two active ICN links between PPU0 and PPU1, the matrix shows ICN2.

• SYS: For example, if there are no active ICN links between PPU0 and PPU2, the matrix shows the connection status over the PCIe bus. For a detailed explanation, see the Legend.

• CPU Affinity 0-47,96-143: This column shows the CPU core affinity for each PPU device. For example, PPU0 has an affinity for CPU cores 0 through 47 and 96 through 143.

• NUMA Affinity 0: This column shows the NUMA node affinity for each PPU device. For example, PPU0 has an affinity for NUMA node 0.

• NIC0: An RDMA NIC connected to the system. See the NIC Legend for the corresponding device name.

• PPU Rear Group: This section lists PPU devices grouped by their rear id. Each row shows the devices for one rear.

Use the -po option to display only PPU devices in the communication matrix, excluding RDMA NICs. Use the -mp option to display only PCIe bus connections, excluding the status of ICN links.

For example, run ppu-smi topo -mp -po to query the PCIe bus connections between PPU devices. The following example shows the output:

root@dfc623e46a90:/# ppu-smi topo -mp -po
         PPU0    PPU1    PPU2    PPU3    PPU4    PPU5    PPU6    PPU7    CPU Affinity    NUMA Affinity
 PPU0    X       PXB     SYS     SYS     SYS     SYS     SYS     SYS     0-47,96-143     0
 PPU1    PXB     X       SYS     SYS     SYS     SYS     SYS     SYS     0-47,96-143     0
 PPU2    SYS     SYS     X       PXB     SYS     SYS     SYS     SYS     0-47,96-143     0
 PPU3    SYS     SYS     PXB     X       SYS     SYS     SYS     SYS     0-47,96-143     0
 PPU4    SYS     SYS     SYS     SYS     X       PXB     SYS     SYS     48-95,144-191   1
 PPU5    SYS     SYS     SYS     SYS     PXB     X       SYS     SYS     48-95,144-191   1
 PPU6    SYS     SYS     SYS     SYS     SYS     SYS     X       PXB     48-95,144-191   1
 PPU7    SYS     SYS     SYS     SYS     SYS     SYS     PXB     X       48-95,144-191   1

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge

10.1.1 Filter and group by rear id

You can filter the communication matrix to show only PPU devices on a specific rear. Use the -ri option to specify the rear id. You can find the rear id for a PPU by running commands such as ppu-smi -q or ppu-smi --query-ppu, or by checking the PPU Rear Group section in the full matrix output.

For example, run ppu-smi topo -m -po -ri 1 to query the topological information for PPU devices with rear id 1. The output shows only PPU devices with that rear id: PPU 1, 3, 5, 7, 9, 11, 13, and 15. PPU devices connected to other rears are not displayed:

root@dfc623e46a90:/# ppu-smi topo -m -po -ri 1
         PPU1    PPU3    PPU5    PPU7    PPU9    PPU11   PPU13   PPU15  CPU Affinity    NUMA Affinity
 PPU1    X       ICN2    SYS     ICN1    SYS     SYS     ICN2    SYS    0-47,96-143     0
 PPU3    ICN2    X       ICN1    SYS     SYS     SYS     SYS     ICN2   0-47,96-143     0
 PPU5    SYS     ICN1    X       ICN2    ICN1    ICN1    SYS     SYS    0-47,96-143     0
 PPU7    ICN1    SYS     ICN2    X       ICN1    ICN1    SYS     SYS    0-47,96-143     0
 PPU9    SYS     SYS     ICN1    ICN1    X       ICN2    SYS     ICN1   48-95,144-191   1
 PPU11   SYS     SYS     ICN1    ICN1    ICN2    X       ICN1    SYS    48-95,144-191   1
 PPU13   ICN2    SYS     SYS     SYS     SYS     ICN1    X       ICN2   48-95,144-191   1
 PPU15   SYS     ICN2    SYS     SYS     ICN1    SYS     ICN2    X      48-95,144-191   1

...

PPU Rear Group:

  Rear ID 1: PPU 1,3,5,7,9,11,13,15

PPU-SMI can also group the devices in the communication matrix by their rear id. The PPU devices in the matrix are first sorted by rear id and then by PPU index. For example, all PPU devices with rear id 0 are displayed together at the beginning of the matrix.

Use the -rg option to enable grouping by rear id. For example, run ppu-smi topo -mp -po -rg. The PPU devices in the output are sorted first by their rear id:

root@dfc623e46a90:/# ppu-smi topo -mp -po -rg
         PPU0    PPU2    PPU4    PPU6    PPU8    PPU10   PPU12   PPU14   PPU1    PPU3    PPU5    PPU7    PPU9    PPU11   PPU13   PPU15   CPU Affinity    NUMA Affinity
 PPU0    X       PHB     SYS     PXB     NODE    PIX     PHB     SYS     PXB     NODE    PIX     PHB     SYS     PXB     NODE    PIX     0-1,64-65       0
 PPU2    PHB     X       PXB     NODE    PIX     PHB     SYS     PXB     NODE    PIX     PHB     SYS     PXB     NODE    PIX     PHB     0-1,64-65       0             
 PPU4    SYS     PXB     X       PIX     PHB     SYS     PXB     NODE    PIX     PHB     SYS     PXB     NODE    PIX     PHB     SYS     0-1,64-65       0  
 PPU6    PXB     NODE    PIX     X       SYS     PXB     NODE    PIX     PHB     SYS     PXB     NODE    PIX     PHB     SYS     PXB     0-1,64-65       0             
 PPU8    NODE    PIX     PHB     SYS     X       NODE    PIX     PHB     SYS     PXB     NODE    PIX     PHB     SYS     PXB     NODE    0-1,64-65       0             
 PPU10   PIX     PHB     SYS     PXB     NODE    X       PHB     SYS     PXB     NODE    PIX     PHB     SYS     PXB     NODE    PIX     0-1,64-65       0             
 PPU12   PHB     SYS     PXB     NODE    PIX     PHB     X       PXB     NODE    PIX     PHB     SYS     PXB     NODE    PIX     PHB     0-1,64-65       0             
 PPU14   SYS     PXB     NODE    PIX     PHB     SYS     PXB     X       PIX     PHB     SYS     PXB     NODE    PIX     PHB     SYS     0-1,64-65       0             
 PPU1    PXB     NODE    PIX     PHB     SYS     PXB     NODE    PIX     X       SYS     PXB     NODE    PIX     PHB     SYS     PXB     0-1,64-65       0             
 PPU3    NODE    PIX     PHB     SYS     PXB     NODE    PIX     PHB     SYS     X       NODE    PIX     PHB     SYS     PXB     NODE    0-1,64-65       0             
 PPU5    PIX     PHB     SYS     PXB     NODE    PIX     PHB     SYS     PXB     NODE    X       PHB     SYS     PXB     NODE    PIX     0-1,64-65       0             
 PPU7    PHB     SYS     PXB     NODE    PIX     PHB     SYS     PXB     NODE    PIX     PHB     X       PXB     NODE    PIX     PHB     0-1,64-65       0             
 PPU9    SYS     PXB     NODE    PIX     PHB     SYS     PXB     NODE    PIX     PHB     SYS     PXB     X       PIX     PHB     SYS     0-1,64-65       0             
 PPU11   PXB     NODE    PIX     PHB     SYS     PXB     NODE    PIX     PHB     SYS     PXB     NODE    PIX     X       SYS     PXB     0-1,64-65       0             
 PPU13   NODE    PIX     PHB     SYS     PXB     NODE    PIX     PHB     SYS     PXB     NODE    PIX     PHB     SYS     X       NODE    0-1,64-65       0             
 PPU15   PIX     PHB     SYS     PXB     NODE    PIX     PHB     SYS     PXB     NODE    PIX     PHB     SYS     PXB     NODE    X       0-1,64-65       0

 ...

 PPU Rear Group:

  Rear ID 0: PPU 0,2,4,6,8,10,12,14
  Rear ID 1: PPU 1,3,5,7,9,11,13,15

10.2 List RDMA NICs

Run ppu-smi topo -lni to list the RDMA NICs on the system. The output includes their names and PCI bus IDs. For example:

root@dfc623e46a90:/# ppu-smi topo -lni
NIC 0: mlx5_bond_0 (PCI Bus Id: 00000000:8A:00.1)
NIC 1: mlx5_bond_1 (PCI Bus Id: 00000000:CF:00.1)
NIC 2: mlx5_bond_2 (PCI Bus Id: 00000001:D2:00.0)
NIC 3: mlx5_bond_3 (PCI Bus Id: 00000001:87:00.0)
NIC 4: mlx5_bond_4 (PCI Bus Id: 00000000:2A:00.1)
NIC 5: mlx5_bond_5 (PCI Bus Id: 00000001:D2:00.1)
NIC 6: mlx5_bond_6 (PCI Bus Id: 00000000:8A:00.0)
NIC 7: mlx5_bond_7 (PCI Bus Id: 00000000:CF:00.0)

10.3 List devices by CPU affinity

To list devices that have an affinity for a specific CPU, use the -c option to specify the CPU index. For example, run ppu-smi topo -c 0 to list the devices with an affinity for CPU 0. The following example shows the output:

root@dfc623e46a90:/# ppu-smi topo -c 0
The PPUs that have an affinity with CPU 0 are: 0, 1, 2, 3

10.4 List the nearest PPU devices

To list the PPU devices nearest to a specified PPU, use the -n option to specify the traversal path and the -i option to specify the target PPU device. For a definition of the traversal path values, see the help information. For example, running ppu-smi topo -n 5 -i 0 produces the following output:

root@dfc623e46a90:/# ppu-smi topo -n 5 -i 0
Device 0 is connected by way of a SMP interconnect link between NUMA nodes to device(s): 1, 2, 3, 4, 5, 6, 7

10.5 Find the shortest path between PPUs

To find the shortest path between two PPU devices, use the -p option, and specify the two PPU devices with the -i option. For example, running ppu-smi topo -p -i 0,1 produces the following output:

root@dfc623e46a90:/# ppu-smi topo -p -i 0,1
Device 0 is connected to device 1 by way of potentially multiple PCIe switches.

10.6 Query p2p capability between devices

Use the -p2p option to specify a p2p capability and display the p2p support status between devices. For a definition of the available capabilities, see the help information. For example, to query the p2p read capability between devices, run ppu-smi topo -p2p r. The following example shows the output:

root@dfc623e46a90:/# ppu-smi topo -p2p r
         PPU0    PPU1    PPU2    PPU3    PPU4    PPU5    PPU6    PPU7
 PPU0    X       OK      OK      OK      OK      OK      OK      OK
 PPU1    OK      X       OK      OK      OK      OK      OK      OK
 PPU2    OK      OK      X       OK      OK      OK      OK      OK
 PPU3    OK      OK      OK      X       OK      OK      OK      OK
 PPU4    OK      OK      OK      OK      X       OK      OK      OK
 PPU5    OK      OK      OK      OK      OK      X       OK      OK
 PPU6    OK      OK      OK      OK      OK      OK      X       OK
 PPU7    OK      OK      OK      OK      OK      OK      OK      X

Legend:

  X    = Self
  OK   = Status Ok
  CNS  = Chipset not supported
  PNS  = PPU not supported
  TNS  = Topology not supported
  NS   = Not supported
  U    = Unknown

11. Multi-Instance GPU (MIG) management

You can use PPU-SMI to query information about instances in MIG mode and to create or destroy them. MIG has the following key concepts:

• GPU instance: A partition of PPU hardware resources. Each GPU instance is isolated and runs independently.

• GPU instance profile: A template that defines how PPU hardware resources are partitioned. For example, a profile specifies the number of dedicated CUs and the amount of memory for an instance.

• compute instance: A partition of a GPU instance. A compute instance has dedicated CU resources but shares other resources. You can specify a compute instance for an application to run on.

• compute instance profile: A template that defines how a GPU instance is partitioned. For example, a profile specifies the number of dedicated CUs and the shared resources for an instance.

To enable MIG mode on a PPU, runppu-smi -mig 1 (for more information, see 5. Modify device configuration). You can use themig subcommand to query MIG-related information. For example, runppu-smi mig -lgip to view the supported GPU instance profiles:

root@0549cf16bb85:~# ppu-smi mig -lgip
+---------------------------------------------------------------------------------+
| GPU instance profiles:                                                          |
| PPU  Name                 Profile  Instances   Memory  P2P  CU      DEC   ENC   |
|                             ID     Free/Total   GiB         CpyEng  JPEG  OFA   |
+=================================================================================+
| 1    MIG 8g48gb              3        0/1      48.00   No   64      4     4     |
|                                                             2       4     0     |
+---------------------------------------------------------------------------------+
| 1    MIG 4g24gb              2        1/2      24.00   No   32      2     2     |
|                                                             2       2     0     |
+---------------------------------------------------------------------------------+
| 1    MIG 2g12gb              1        2/4      12.00   No   16      1     1     |
|                                                             2       1     0     |
+---------------------------------------------------------------------------------+
| 1    MIG 1g6gb               0        4/8      6.00    No   8       0     0     |
|                                                             2       0     0     |
+---------------------------------------------------------------------------------+

Runppu-smi mig -h to view help information for MIG. Each command accepts only one mig sub-option. You can use the-i,-gi, or-ci options to restrict the scope of some sub-options. You can use these options individually or in combination. For example,-i 0 -gi 1 restricts the scope to GPU instance 1 on PPU 0.

root@dfc623e46a90:/# ppu-smi mig -h
mig -- Multi Instance GPU management.

ppu-smi mig [OPTION1 [ARG1]] [OPTION2 [ARG2]] ...
    -h,   --help
        Display help information
    -i,   --id=
        Enumeration index, Serial number, PCI bus ID or UUID.
        Provide comma separated values for more than one device.
    -gi,  --gpu-instance-id=
        GPU instance ID.
        Provide comma separated values for more than one GPU instance.
    -ci,  --compute-instance-id=
        Compute instance ID.
        Provide comma separated values for more than one compute instance.
    -C,   --default-compute-instance
        When used with the option to create a GPU instance (-cgi),
        Create compute instance with the default profile.

    [any one of]

    -lgip,--list-gpu-instance-profiles
        List supported GPU instance profiles.
        Option -i can be used to restrict the command to run on a specific PPU.
    -lgipp,--list-gpu-instance-possible-placements
        List possible GPU instance placements in the following format:
          {Start,Start...}:Size
        Option -i can be used to restrict the command to run on a specific PPU.
    -cgi, --create-gpu-instance=
        Create GPU instances for the given profile tuples.
        A profile tuple consists of a profile name or ID and an optional placement specifier,
        which consists of a colon and a placement start index.
        Provide comma separated values for more than one profile tuple(e.g. 1:0,4:2).
        Option -i can be used to restrict the command to run on a specific PPU.
    -dgi, --destroy-gpu-instance
        Destroy GPU instances.
        Options -i and -gi can be used individually or combined
        to restrict the command to run on a specific PPU or GPU instance.
    -lgi, --list-gpu-instances
        List GPU instances.
        Option -i can be used to restrict the command to run on a specific PPU.
    -r,   --reset-gpu-instance
        Trigger reset of the GPU instance.
        Options -i and -gi can be used individually or combined
        to restrict the command to run on a specific PPU or GPU instance.
    -lcip,--list-compute-instance-profiles
        List supported compute instance profiles.
        Options -i and -gi can be used individually or combined
        to restrict the command to run on a specific PPU or GPU instance.
    -lcipp,--list-compute-instance-possible-placements
        List possible compute instance placements in the following format:
          {Start,Start...}:Size
        Options -i and -gi can be used individually or combined
        to restrict the command to run on a specific PPU or GPU instance.
    -cci, --create-compute-instance=
        Create compute instance for the given profile tuples.
        A profile tuple consists of a profile name or ID and an optional placement specifier,
        which consists of a colon and a placement start index.
        Provide comma separated values for more than one profile tuple(e.g. 1:0,4:2).
        If no profile name or ID is given, then the default*
        compute instance profile ID will be used.
        Options -i and -gi can be used individually or combined
        to restrict the command to run on a specific PPU or GPU instance.
    -dci, --destroy-compute-instance
        Destroy compute instances.
        Options -i, -gi and -ci can be used individually or combined
        to restrict the command to run on a specific PPU or GPU instance or compute instance.
    -lci, --list-compute-instances
        List compute instances.
        Options -i and -gi can be used individually or combined
        to restrict the command to run on a specific PPU or GPU instance.

11.1 List GPU instance profiles

Use the-lgip option to list the supported GPU instance profiles. You can then use the-cgi option to create a GPU instance from one of these profiles. Use the-i option to restrict the query to a specific device. For example, runppu-smi mig -i 1 -lgip.

root@a475cc8d4c49:/# ppu-smi mig -i 1 -lgip
+---------------------------------------------------------------------------------+
| GPU instance profiles:                                                          |
| PPU  Name                 Profile  Instances   Memory  P2P  CU      DEC   ENC   |
|                             ID     Free/Total   GiB         CpyEng  JPEG  OFA   |
+=================================================================================+
| 1    MIG 8g96gb              3        1/1      96.00   No   64      4     4     |
|                                                             2       4     0     |
+---------------------------------------------------------------------------------+
| 1    MIG 4g48gb              2        2/2      48.00   No   32      2     2     |
|                                                             2       2     0     |
+---------------------------------------------------------------------------------+
| 1    MIG 2g24gb              1        4/4      24.00   No   16      1     1     |
|                                                             2       1     0     |
+---------------------------------------------------------------------------------+
| 1    MIG 1g12gb              0        8/8      12.00   No   8       0     0     |
|                                                             2       0     0     |
+---------------------------------------------------------------------------------+

• PPU: The device's zero-based index, assigned at power-on.

• Name: The name of the GPU instance profile. You can use this name with the-cgi option to create a GPU instance.

• Profile ID: The ID of the GPU instance profile. You can use this ID with the-cgi option to create a GPU instance.

• Instances Free/Total: The number of available (Free) and total supported (Total) instances for this profile.

• Memory GiB: The device memory allocated to this profile, in GiB.

• P2P: Whether the profile supports peer-to-peer.

• CU: The number of dedicated CU resources.

• DEC: The number of dedicated decoder resources.

• ENC: The number of dedicated encoder resources.

• CpyEng: The number of dedicated copy engine resources.

• JPEG: The number of dedicated JPEG processing units.

• OFA: The number of dedicated OFA processing units.

11.2 List possible GPU instance placements

Use the-lgipp option to list the possible placements for a GPU instance profile. You can then use the-cgi option to create a GPU instance at a specific placement.

Possible placements are displayed in the format{start0, start1, start2}:size, which indicates one or more possible start indices (start0 / start1 / start2) for a givensize. For example, the following output indicates that you can create an instance of size 4 at either start index 0 or start index 4.

{0,4}:4

For example, runppu-smi mig -i 1 -lgipp. The following is a sample result:

root@a475cc8d4c49:/# ppu-smi mig -i 1 -lgipp
PPU 1 profile ID 0 placements: {0,1,2,3,4,5,6,7}:1
PPU 1 profile ID 1 placements: {0,2,4,6}:2
PPU 1 profile ID 2 placements: {0,4}:4
PPU 1 profile ID 3 placements: {0,4}:4

11.3 Create GPU instances

Use the-cgi option to create one or more GPU instances. Separate multiple instance specifications with a comma (,). For each GPU instance you want to create:

1. Specify the GPU instance profile to use by its name or ID.

   1. You can use either the full or short name for the profile. For a profile namedMIG 1g12gb, you can specify eitherMIG 1g12gb or1g12gb.

2. Append a colon (:) and a start index to the profile information to specify the placement.

   1. This is optional.

To create one GPU instance using GPU instance profile ID3 without specifying a placement:

-cgi 3

To create two GPU instances using GPU instance profile IDs0 and1 without specifying a placement:

-cgi 0,1

To create one GPU instance using GPU instance profile ID1 at placement start index4:

-cgi 1:4

To create one GPU instance using the profile short name 1g12gb without specifying a placement:

-cgi 1g12gb

To create two GPU instances using the profile namesMIG 1g12gb andMIG 2g24gb at specific placements:

-cgi "MIG 1g12gb:0,MIG 2g24gb:4"

Use the-i option to specify the PPU on which to create the GPU instance. If you do not specify a device, the command attempts to create the instance on every PPU. For example, runppu-smi mig -i 1 -cgi 1. The output is as follows:

root@a475cc8d4c49:~# ppu-smi mig -i 1 -cgi 1
Successfully created GPU instance ID 0 on PPU 1 using profile MIG 2g24gb (Profile ID 1)

11.4 List GPU instances

Use the-lgi option to list existing GPU instances. Use the-i option to restrict the query to a specific device. For example, runppu-smi mig -i 1 -lgi. The following list describes the output:

root@0549cf16bb85:~# ppu-smi mig -i 1 -lgi
+---------------------------------------------------------+
| GPU instances:                                          |
| PPU  Name                 Profile  Instance  Placement  |
|                             ID        ID     Start:Size |
+=========================================================+
| 1    MIG 2g12gb              1        0         0:2     |
+---------------------------------------------------------+
| 1    MIG 2g12gb              1        2         2:2     |
+---------------------------------------------------------+

• PPU: The device's zero-based index, assigned at power-on.

• Name: The name of the GPU instance.

• Profile ID: The ID of the GPU instance profile that was used to create this GPU instance.

• Instance ID: The ID of the GPU instance. You can use this ID with the-gi option to specify this GPU instance.

• Placement: The placement information for this GPU instance.

11.5 Destroy GPU instances

Use the-dgi option to destroy a GPU instance. Use the-i option to restrict the operation to a specific device and the-gi option to restrict it to a specific GPU instance. You can use the-i and-gi options individually or together.

For example, to destroy GPU instance 1 on PPU 1, runppu-smi mig -dgi -i 1 -gi 1. The following is a sample command:

root@0549cf16bb85:~# ppu-smi mig -dgi -i 1 -gi 1
Successfully destroyed GPU instance ID 1 from PPU 1

11.6 Reset a GPU instance

Use the-r option to reset a GPU instance. Resetting a GPU instance does not affect others. Use the-i option to restrict the operation to a specific device and the-gi option to restrict it to a specific GPU instance. You can use the-i and-gi options individually or together.

For example, to reset GPU instance 0 on PPU 1, runppu-smi mig -i 1 -gi 0 -r. The following is a sample command:

root@0549cf16bb85:~# ppu-smi mig -i 1 -gi 0 -r
Successfully triggered reset of GPU instance ID 0 from PPU 1.

11.7 List compute instance profiles

Use the-lcip option to list the compute instance profiles for a GPU instance. Use the-i option to restrict the query to a specific device and the-gi option to restrict it to a specific GPU instance. You can use the-i and-gi options individually or together. For example, to list the supported compute instance partitions for GPU instance 0 on PPU 1, runppu-smi mig -lcip -i 1 -gi 0. The following list describes the output:

root@0549cf16bb85:~# ppu-smi mig -lcip -i 1 -gi 0
+--------------------------------------------------------------------------------------+
| Compute instance profiles:                                                           |
| PPU    GPU     Name                 Profile  Instances   Exclusive       Shared      |
|      Instance                         ID     Free/Total     CU       DEC   ENC  OFA  |
|         ID                                                          CpyEng JPEG      |
+======================================================================================+
| 1       0      MIG 1u.2g12gb           0       16/16         1         0    0    0   |
|                                                                        2    0        |
+--------------------------------------------------------------------------------------+
| 1       0      MIG 2u.2g12gb           1        8/8          2         0    0    0   |
|                                                                        2    0        |
+--------------------------------------------------------------------------------------+
| 1       0      MIG 3u.2g12gb           2        4/4          3         0    0    0   |
|                                                                        2    0        |
+--------------------------------------------------------------------------------------+
| 1       0      MIG 4u.2g12gb           3        4/4          4         0    0    0   |
|                                                                        2    0        |
+--------------------------------------------------------------------------------------+
| 1       0      MIG 8u.2g12gb           4        2/2          8         0    0    0   |
|                                                                        2    0        |
+--------------------------------------------------------------------------------------+
| 1       0      MIG 12u.2g12gb          5        1/1         12         0    0    0   |
|                                                                        2    0        |
+--------------------------------------------------------------------------------------+
| 1       0      MIG 16u.2g12gb         6*        1/1         16         1    1    0   |
|                                                                        2    1        |
+--------------------------------------------------------------------------------------+

• PPU: The device's zero-based index, assigned at power-on.

• GPU Instance ID: The ID of the GPU instance.

• Name: The name of the compute instance profile. You can use this name with the-cci option to create a compute instance.

• Profile ID: The ID of the compute instance profile. You can use this ID with the-cci option to create a compute instance.

  • An asterisk (*) indicates the default compute instance profile ID. When you create a default compute instance by using the-C or-cci option, this is the profile used.

• Instances Free/Total: The number of available and total compute instances that can be created with this compute instance profile.

• Exclusive CU: The number of exclusive CUs in a compute instance created with this profile.

• DEC: The number of shared decoder resources for this profile.

• ENC: The number of shared encoder resources for this profile.

• CpyEng: The number of shared copy engine resources for this profile.

• JPEG: The number of shared JPEG processing units for this profile.

• OFA: The number of shared OFA processing units for this profile.

11.8 List possible compute instance placements

Use the-lcipp option to list possible placements for a compute instance profile. You can then use the-cci option to create a compute instance at a specific placement. Use the-i option to restrict the operation to a specific device and the-gi option to restrict it to a specific GPU instance. You can use the-i and-gi options individually or together. Possible placements are displayed in the format{start0, start1, start2}:size, which indicates one or more possible start indices (start0 / start1 / start2) for a givensize. For example, the following output indicates that you can create an instance of size 4 at either start index 0 or start index 4.

{0,4}:4

For example, runppu-smi mig -i 3 -gi 0 -lcipp. The following is a sample result:

root@0549cf16bb85:~# ppu-smi mig -i 3 -gi 0 -lcipp
PPU 3 GPU instance 0 profile ID 0 placements: {0,1,2,3,4,5,6,7}:1
PPU 3 GPU instance 0 profile ID 1 placements: {0,2,4,6}:2
PPU 3 GPU instance 0 profile ID 2 placements: {0,4}:3
PPU 3 GPU instance 0 profile ID 3 placements: {0,4}:4
PPU 3 GPU instance 0 profile ID 4 placements: {0}:8

11.9 Create compute instances

Use the-cci option to create one or more compute instances. You can specify the compute instance profile by its name or ID. If you do not specify a profile, the default compute instance profile is used. Separate multiple instance specifications with a comma (,). For each compute instance you want to create:

1. Specify the compute instance profile to use by its name or ID.

   1. You can use either the full or short name for the profile. For a profile namedMIG 1u.1g6gb, you can specify eitherMIG 1u.1g6gb or1u.1g6gb.

2. Append a colon (:) and a start index to the profile information to specify the placement.

   1. This is optional.

Use the-i option to restrict the operation to a specific device and the-gi option to restrict it to a specific GPU instance. You can use the-i and-gi options individually or together. You can also create a default compute instance when creating a GPU instance. For example, runppu-smi mig -i 1 -cgi 1 -C:

root@0549cf16bb85:~# ppu-smi mig -i 1 -cgi 1 -C
Successfully created GPU instance ID 4 on PPU 1 using profile MIG 2g12gb (Profile ID 1)
Successfully created compute instance ID 0 on PPU 1 GPU instance ID 4 using profile MIG 16u.2g12gb (Profile ID 6)

To create one compute instance using compute instance profile ID3:

-cci 3

To create two compute instances using compute instance profile IDs3,4:

-cci 3,4

To create one compute instance using compute instance profile ID1 at placement start index4:

-cci 1:4

To create one compute instance using the profile short name1u.2g12gb:

-cci 1u.2g12gb

To create two compute instances using the profile namesMIG 1u.2g12gb andMIG 2u.2g12gb:

-cci "MIG 1u.2g12gb,MIG 2u.2g12gb"

To create one compute instance using the default compute instance profile:

ppu-smi mig -cci -i 1 -gi 0

You can use the-i and-gi options to specify where to create the compute instance. If you do not specify these options, the command attempts to create the instance on all PPUs and GPU instances. For example, runppu-smi mig -i 1 -gi 0 -cci 3. The following is a sample result:

root@0549cf16bb85:~# ppu-smi mig -i 1 -gi 0 -cci 3
Successfully created compute instance ID 0 on PPU 1 GPU instance ID 0 using profile MIG 4u.2g12gb (Profile ID 3)

11.10 List compute instances

Use the-lci option to list existing compute instances. Use the-i option to restrict the query to a specific device and the-gi option to restrict it to a specific GPU instance. You can use the-i and-gi options individually or together. For example, runppu-smi mig -i 1 -gi 0 -lci. The following list describes the output:

root@0549cf16bb85:~# ppu-smi mig -i 1 -gi 0 -lci
+-------------------------------------------------------------------+
| Compute instances:                                                |
| PPU    GPU     Name                 Profile  Instance  Placement  |
|      Instance                         ID        ID     Start:Size |
|         ID                                                        |
+===================================================================+
| 1       0      MIG 4u.2g12gb           3        0         0:0     |
+-------------------------------------------------------------------+
| 1       0      MIG 8u.2g12gb           4        1         0:1     |
+-------------------------------------------------------------------+

• PPU: The device's zero-based index, assigned at power-on.

• GPU Instance ID: The ID of the GPU instance.

• Name: The name of the compute instance.

• Profile ID: The ID of the compute instance profile that was used to create this compute instance.

• Instance ID: The ID of this compute instance. You can use this ID with the-ci option to specify this compute instance.

• Placement: The placement information for this compute instance.

11.11 Destroy compute instances

Use the-dci option to destroy a compute instance. Use the-i option to restrict the operation to a specific device, the-gi option to restrict it to a specific GPU instance, and the-ci option to restrict it to a specific compute instance. You can use the-i,-gi, and-ci options individually or in combination.

For example, to destroy compute instance 1 on GPU instance 0 of PPU 1, runppu-smi mig -dci -i 1 -gi 0 -ci 1. The following is a sample command:

root@0549cf16bb85:~# ppu-smi mig -dci -i 1 -gi 0 -ci 1
Successfully destroyed compute instance ID 1 from PPU 1 GPU instance 0

12. vGPU

PPU-SMI lets you query information about device virtualization (vGPU), and create and delete vGPU instances. Using vGPU features requires administrator privileges.

The following are key vGPU concepts:

• vGPU type: Defines how PPU resources are partitioned. For example, a vGPU type specifies the number of CEs and the amount of memory allocated to each vGPU instance.

• vGPU instance: A vGPU instance created from a specified vGPU type. You can use a vGPU instance to create a virtual machine (VM).

The following are the states of a vGPU instance:

• active instance: A vGPU instance created and associated with a VM.

• alive instance: A created vGPU instance, including those not associated with a VM.

Run ppu-smi -vm VGPU to enable vGPU mode on a PPU device (for more information, see 5. Modify device configuration). Use the vgpu subcommand to query vGPU information. For example, run ppu-smi vgpu -s to view the vGPU types supported by the device:

root@0549cf16bb85:~# ppu-smi vgpu -s
PPU 0: PPU (UUID: GPU-019ea108-c110-0420-0000-0000e0b7fe3c)
    vGPU Type 0: vGPU-0 (Class: MIG-Backed: 2 CE 5GB Memory 0 ICN 0 DVG 0 EVG)
    vGPU Type 1: vGPU-1 (Class: MIG-Backed: 2 CE 5GB Memory 0 ICN 1 DVG 0 EVG)
    vGPU Type 2: vGPU-2 (Class: MIG-Backed: 2 CE 5GB Memory 0 ICN 0 DVG 1 EVG)
    vGPU Type 3: vGPU-3 (Class: MIG-Backed: 2 CE 5GB Memory 1 ICN 0 DVG 0 EVG)
    vGPU Type 4: vGPU-4 (Class: MIG-Backed: 2 CE 5GB Memory 1 ICN 1 DVG 0 EVG)
    vGPU Type 5: vGPU-5 (Class: MIG-Backed: 2 CE 5GB Memory 1 ICN 0 DVG 1 EVG)
...

Run ppu-smi vgpu -h to get help for vGPU. You can specify one vgpu query or operation option at a time. Use the -i option to target one or more PPU devices. For all query functions of the vgpu subcommand, you can use the -l option to periodically output the query results. The value is in seconds. For example, -l 5 displays the results every 5 seconds until you press Ctrl+C.

root@0549cf16bb85:~# ppu-smi vgpu -h
vgpu -- Virtual GPU management.

ppu-smi vgpu [OPTION1 [ARG1]] [OPTION2 [ARG2]] ...
    -h,   --help
        Display help information
    -i,   --id=
        Enumeration index, Serial number, PCI bus ID or UUID.
        Provide comma separated values for more than one device.
    -v,   --verbose
        Display detailed information about supported vGPU types or vGPU types that can be created.
    -l,   --loop=
        Display information at the specified time interval in seconds until Ctrl-C is pressed.
    -f,   --force
        When used with the option to delete a vGPU instance (-di),
        Delete vGPU instance regardless of VM state.

    [any one of]

    -q,   --query
        Display information about currently running vGPU instances (VM active).
    -a,   --alive
        Display information about currently alive vGPU instances.
    -s,   --supported
        Display supported vGPU types.
    -c,   --creatable
        Display the vGPU types that can currently be created.
    -ci,  --create-instance=
        Create a vGPU instance for the given vGPU type ID.
    -di,  --delete-instance=
        Delete a vGPU instance for the given vGPU instance ID.
        Use option -f to delete vGPU instance regardless of VM state.

12.1 Supported vGPU types

Use the -s option to query the vGPU types supported by a PPU device. Use the -i option to query specific devices. For example, to query device 0, run ppu-smi vgpu -i 0 -s:

root@0549cf16bb85:~# ppu-smi vgpu -i 0 -s
PPU 0: PPU (UUID: GPU-019ea108-c110-0420-0000-0000e0b7fe3c)
    vGPU Type 0: vGPU-0 (Class: MIG-Backed: 2 CE 5GB Memory 0 ICN 0 DVG 0 EVG)
    vGPU Type 1: vGPU-1 (Class: MIG-Backed: 2 CE 5GB Memory 0 ICN 1 DVG 0 EVG)
    vGPU Type 2: vGPU-2 (Class: MIG-Backed: 2 CE 5GB Memory 0 ICN 0 DVG 1 EVG)
    vGPU Type 3: vGPU-3 (Class: MIG-Backed: 2 CE 5GB Memory 1 ICN 0 DVG 0 EVG)
    vGPU Type 4: vGPU-4 (Class: MIG-Backed: 2 CE 5GB Memory 1 ICN 1 DVG 0 EVG)
    vGPU Type 5: vGPU-5 (Class: MIG-Backed: 2 CE 5GB Memory 1 ICN 0 DVG 1 EVG)
...

The list for each PPU device includes:

• vGPU type ID

• Name of the vGPU type

• Detailed classification information for the vGPU type

Use the -s and -v options together to query more detailed vGPU type information. For example, run ppu-smi vgpu -i 0 -s -v. The output fields are described below:

root@0549cf16bb85:~# ppu-smi vgpu -i 0 -s -v
+-------------------------------------------------------------------+
| Supported vGPU types:                                             |
| PPU  Type   Name                 Instances     GPU       Memory   |
|       ID                         Free/Total  Instance     GiB     |
|                                                 ID                |
+===================================================================+
| 0      0    vGPU-0                  4/8         N/A      5.00     |
+-------------------------------------------------------------------+
| 0      1    vGPU-1                  0/1         N/A      5.00     |
+-------------------------------------------------------------------+
| 0      2    vGPU-2                  2/4         N/A      11.00    |
+-------------------------------------------------------------------+

• PPU: The device's enumeration index, assigned at power-on and starting from 0.

• Type ID: The ID of the vGPU type.

• Name: The name of the vGPU type.

• Instances Free/Total: The number of creatable instances / The total number of supported instances of this type.

• GPU instance ID: The MIG GPU instance ID mapped to the vGPU type. N/A indicates no mapping.

• Memory GiB: The memory size allocated to each vGPU instance of this type.

12.2 Creatable vGPU types

Use the -c option to query which vGPU types you can create on a PPU device. Use the -i option to query specific devices. For example, run ppu-smi vgpu -i 0 -c. The output lists only the vGPU types that you can create with available PPU resources:

root@0549cf16bb85:~# ppu-smi vgpu -i 0 -c
PPU 0: PPU (UUID: GPU-019ea108-c110-0420-0000-0000e0b7fe3c)
    vGPU Type 0: vGPU-0 (Class: MIG-Backed: 2 CE 5GB Memory 0 ICN 0 DVG 0 EVG)
    vGPU Type 3: vGPU-3 (Class: MIG-Backed: 2 CE 5GB Memory 1 ICN 0 DVG 0 EVG)
...

The list for each PPU device includes:

• vGPU type ID

• Name of the vGPU type

• Detailed classification information for the vGPU type

Use the -c and -v options together to query detailed information about creatable vGPU types. For example, run ppu-smi vgpu -i 0 -c -v. The output fields are described below:

root@0549cf16bb85:~# ppu-smi vgpu -i 0 -c -v
+-------------------------------------------------------------------+
| Supported vGPU types:                                             |
| PPU  Type   Name                 Instances     GPU       Memory   |
|       ID                         Free/Total  Instance     GiB     |
|                                                 ID                |
+===================================================================+
| 0      0    vGPU-0                  4/8         N/A      5.00     |
+-------------------------------------------------------------------+
| 0      2    vGPU-2                  2/4         N/A      11.00    |
+-------------------------------------------------------------------+

• PPU: The enumeration index assigned to the device upon power-on, starting from 0.

• Type ID: The ID of the vGPU type.

• Name: The name of the vGPU type.

• Instances Free/Total: The number of vGPU instances that can currently be created / The total number of supported instances of this type.

• GPU instance ID: The MIG GPU instance ID mapped to the vGPU type. N/A indicates no mapping.

• Memory GiB: The memory size allocated to each vGPU instance of this type.

12.3 Create a vGPU instance

Use the -ci option to create a vGPU instance of a specified vGPU type.

Use the -i option to target a specific device. For example, to create a vGPU instance of vGPU type ID 0 on PPU device 0, run ppu-smi vgpu -i 0 -ci 0.

root@0549cf16bb85:~# ppu-smi vgpu -ci 0 -i 0
Successfully created vGPU instance 0 (PCI Bus ID 00000000:5E:00.0) on PPU 0 using vGPU type 0.

12.4 Created vGPU instances

Use the -a option to query created vGPU instances (in the alive state). Use the -i option to query specific devices. For example, run ppu-smi vgpu -i 0 -a. The output fields are described below:

root@0549cf16bb85:~# ppu-smi vgpu -i 0 -a
PPU 00000000:10:00.0
    Alive vGPUs                             : 2
    vGPU Instance 1
        vGPU Instance ID                    : 1
        vGPU Name                           : vGPU-10
        vGPU Type                           : 10
        vGPU UUID                           : VGPU-3f53d39f-ce6e-dc78-c3d4-4c18653c19c0
        vGPU PCI Bus ID                     : 00000000:10:01.0
        MDEV UUID                           : N/A
        GPU Instance ID                     : N/A
        ECC Mode                            : Enabled
        Memory Size                         : 28672 MiB
    vGPU Instance 5
        vGPU Instance ID                    : 5
        vGPU Name                           : vGPU-1
        vGPU Type                           : 1
        vGPU UUID                           : VGPU-3f53d39f-ce6e-dc78-c3d4-4c18653c19c1
        vGPU PCI Bus ID                     : 00000000:10:01.4
        MDEV UUID                           : N/A
        GPU Instance ID                     : N/A
        ECC Mode                            : Enabled
        Memory Size                         : 7168 MiB

For each PPU device:

• Alive vGPUs: The number of vGPU instances created on the PPU device.

For each vGPU instance:

• vGPU instance ID: The ID of the vGPU instance. Use this ID with the -di option to delete the vGPU instance.

• vGPU Name: The name of this vGPU instance.

• vGPU type: The ID of the vGPU type.

• vGPU UUID: The UUID of the vGPU instance.

• vGPU PCI bus ID: The PCI bus ID of the vGPU instance, in Bus:Device.Function format.

• MDEV UUID: The UUID of the MIG device mapped to the vGPU instance. N/A indicates no mapping.

• GPU instance ID: The MIG GPU instance ID mapped to the vGPU instance. N/A indicates no mapping.

• ECC mode: The ECC mode of the vGPU instance.

• Memory Size: The total memory allocated to the vGPU instance.

12.5 vGPU instances associated with VMs

Use the -q option to query vGPU instances associated with a VM (in the active state). Use the -i option to query specific devices. For example, run ppu-smi vgpu -i 0 -q. The output fields are described below:

root@0549cf16bb85:~# ppu-smi vgpu -i 0 -q
PPU 00000000:10:00.0
    Active vGPUs                            : 1
    vGPU Instance 1
        vGPU Instance ID                    : 1
        VM UUID                             : ee7b7a4b-388a-4357-a425-5318b2c65b30
        vGPU Name                           : vGPU-10
        vGPU Type                           : 10
        vGPU UUID                           : VGPU-3f53d39f-ce6e-dc78-c3d4-4c18653c19c0
        vGPU PCI Bus ID                     : 00000000:10:01.0
        MDEV UUID                           : N/A
        Guest Driver Version                : 0.8.0
        GPU Instance ID                     : N/A
        ECC Mode                            : Enabled
        Memory Usage
            Total                           : 28672 MiB
            Used                            : 3 MiB
            Free                            : 28669 MiB

For each PPU device:

• Active vGPUs: The number of vGPU instances created and associated with a VM on the PPU device.

For each vGPU instance:

• vGPU instance ID: The ID of the vGPU instance. Use this ID with the -di option to delete the vGPU instance.

• VM UUID: The UUID of the virtual machine.

• VM Domain ID: The domain ID of the virtual machine.

• vGPU Name: The name of this vGPU instance.

• vGPU type: The ID of the vGPU type.

• vGPU UUID: The UUID of the vGPU instance.

• vGPU PCI bus ID: The PCI bus ID of the vGPU instance, in Bus:Device.Function format.

• MDEV UUID: The UUID of the MIG device mapped to the vGPU instance. N/A indicates no mapping.

• guest driver version: The version of the driver installed in the virtual machine.

• GPU instance ID: The MIG GPU instance ID mapped to the vGPU instance. N/A indicates no mapping.

• ECC mode: The ECC mode of the vGPU instance.

• memory usage: The memory usage of the vGPU instance.

  • total: The total memory allocated to the vGPU instance.

  • used: The amount of memory used by the vGPU instance.

  • free: The amount of free memory in the vGPU instance.

12.6 Delete a vGPU instance

Use the -di option to delete a vGPU instance by specifying its ID. Use the -i option to target a specific device. By default, you cannot delete a vGPU instance that is in use by a VM. Use the -f option to force the deletion. For example, to delete the vGPU instance with ID 1 on PPU device 0, run ppu-smi vgpu -i 0 -di 1:

root@0549cf16bb85:~# ppu-smi vgpu -i 0 -di 1
Successfully deleted vGPU instance 1 from PPU 0.

13. Drain a PPU device

You can use PPU-SMI to set and query the drain state of a PPU device. This feature is useful for isolating a faulty device. A drained PPU device has the following characteristics:

• Existing tasks running on the PPU device are not affected.

• The PPU device no longer accepts new compute tasks.

• The PPU device becomes invisible to tools such as PPU-SMI and PPUDBG.

For example, run ppu-smi drain -p 0001:AA:00.0 -m 1 to set the PPU device 0001:AA:00.0 to the draining state:

root@dfc623e46a90:~# ppu-smi drain -p 0001:AA:00.0 -m 1
Successfully set PPU 0001:AA:00.0 drain state to: draining.

Run ppu-smi drain -h to view the help information for the drain command. The -p option is used to specify the PPU device by its PCI bus ID.

root@dfc623e46a90:~# ppu-smi drain -h
drain -- Displays/modifies PPU drain states for power idling.

ppu-smi drain [OPTION1 [ARG1]] [OPTION2 [ARG2]] ...
    -h,   --help
        Display help information
    -p,   --pciid=
        PPU PCI ID in the format XXXX:YY:Z.a
            XXXX = domain
            YY   = bus
            Z    = device
            a    = function
    
    [any one of]

    -m,   --modify=
        Modify the drain state of a PPU specified by -p.
            0 = not draining
            1 = draining
    -q,   --query
        Query the drain state of a PPU specified by -p.

13.1 Set the drain state

Use the -m option to set or unset the drain state of a PPU device. You must use the -p option to specify the PPU device by its PCI bus ID. For example, to drain a PPU device, run ppu-smi drain -p 0001:AA:00.0 -m 1:

root@dfc623e46a90:~# ppu-smi drain -p 0001:AA:00.0 -m 1
Successfully set PPU 0001:AA:00.0 drain state to: draining.

13.2 Query the drain state

Use the -q option to query the drain state of a PPU device. You must use the -p option to specify the PPU device by its PCI bus ID. For example, run ppu-smi drain -p 0001:AA:00.0 -q to query the drain state:

root@dfc623e46a90:~# ppu-smi drain -p 0001:AA:00.0 -q
The current drain state of PPU 0001:AA:00.0 is: draining.

14. Manage GPU Performance Monitoring (gpm)

PPU-SMI lets you manage the stream state of the GPU Performance Monitoring (GPM) feature. When GPM is enabled, it uses the PPU device's performance counter to collect data. This usage prevents other applications that also require the performance counter from running. Use PPU-SMI to query the GPM stream state, pause the GPM feature to let other applications collect performance counter data, and resume it when they finish.

For example, run ppu-smi gpm --get-stream-state to query the GPM stream state of the PPU devices:

root@dfc623e46a90:~# ppu-smi gpm --get-stream-state
PPU 0 GPM stream state: Enabled.
PPU 1 GPM stream state: Disabled.

Run ppu-smi gpm -h to view the help for the gpm feature. Use the -i option to specify one or more PPU devices. Separate multiple device IDs with a comma ,.

root@dfc623e46a90:~# ppu-smi gpm -h
gpm -- GPU Performance Monitoring management.

ppu-smi gpm [OPTION1 [ARG1]] [OPTION2 [ARG2]] ...
    -h,   --help
        Display help information
    -i,   --id=
        Enumeration index, Serial number, PCI bus ID or UUID. 
        Provide comma separated values for more than one device.

    [any one of]

    -s,   --set-stream-state=
        Set GPU Performance Monitoring Stream State:
        0/DISABLED, 1/ENABLED
    -g,   --get-stream-state
        Get GPU Performance Monitoring Stream State
    --get-sample-state
        Get GPU Performance Monitoring Sample State

14.1 Set GPM stream state

Use the -s option to enable or disable the GPM stream state. When the GPM feature is disabled:

• Applications that subscribe to GPM data receive N/A as the performance result.

• The GPM feature no longer uses the PPU device's performance counter.

For example, run ppu-smi gpm -s DISABLED -i 0 to disable the GPM stream state for device PPU 0:

root@dfc623e46a90:~# ppu-smi gpm -s DISABLED -i 0
Set GPM stream state to DISABLED for PPU 00000000:5E:00.0.

14.2 Query GPM stream state

Use the -g option to query the GPM stream state. You can use the -i option to query specific devices. For example, run ppu-smi gpm -g -i 0 to query the GPM stream state for device PPU 0:

root@dfc623e46a90:~# ppu-smi gpm -g -i 0
PPU 0 GPM stream state: Disabled.

14.3 Query GPM sample state

Use the --get-sample-state option to query the GPM sample state. This state indicates whether any application is using GPM to collect performance metrics. For example, run ppu-smi gpm --get-sample-state to query the GPM sample state for all PPU devices:

root@dfc623e46a90:~# ppu-smi gpm --get-sample-state
PPU 0 GPM sample state: Enabled.
PPU 1 GPM sample state: Disabled.

15. Known issues

• SDK v1.00: Queries for the application clock frequency may underreport the actual value.

• SDK v1.00: In the mig subcommand, queries without an instance ID may fail.

• SDK v1.4.0: Queries to the PPU process list may return incorrect process information.

• The ppu-smi -q -d UTILIZATION command does not currently support sampling.

• The stats subcommand does not currently support sampling.

• The ppu-smi icn -gt d command does not currently support querying ICN link data throughput.

• The ppu-smi icn -gt r command may underreport ICN link data volume.

• The system does not currently report the percentage of time spent throttling due to temperature or power consumption limits.

• The system does not currently support querying PCIe replay and error counters.