PPU-SMI (v1.7)

1. Overview

PPU System Management Interface (PPU-SMI) is a command-line tool based on the HanGuang Management Library (HGML) for managing and monitoring PPU devices. You can use it to:

• Modify device configuration and feature switches.

• Query runtime parameters and the feature enable status for a specific device.

• Collect runtime data and specific events, and export them to a table for further analysis.

• Analyze device resource usage by each application.

• Query the topology information for multiple PPU devices.

2. Installation and usage

PPU-SMI is included in the PPU_SDK package. After you install PPU_SDK, navigate to the PPU_SDK directory and run the following script to configure the required environment variables:

source envsetup.sh

After you configure the environment variables, run ppu-smi -v to check the version and confirm the installation:

root@122d8d7a7e37:~# ppu-smi -v
T-Head ppu-smi version v1.0
Build date: Sep  7 2022, 10:43:22
Code node: 4a1989d

Run ppu-smi -h to view help information:

root@122d8d7a7e37:~# ppu-smi -h
PPU System Management Interface -- v1.0

PPU-SMI provides monitoring information for T-head PPU devices.
The data is presented in either a plain text or an XML format, via stdout or a file.
PPU-SMI also provides several management operations for changing the device state.
Note that the functionality of PPU-SMI is exposed through the HGML C-based library.

ppu-smi [OPTION1 [ARG1]] [OPTION2 [ARG2]] ...
    -h,   --help
        Print usage information and exit.
    -v,   --version
        Print version information and exit.

  LIST OPTIONS:

    -L,   --list-ppus
        Display a list of PPUs connected to the system.
...

3. Device summary

3.1 Device list

Run the ppu-smi -L command to view the list of PPU devices in the system.

root@0549cf16bb85:~# ppu-smi -L
PPU 0: PPU (UUID: GPU-019ea108-c110-0828-0000-0000c07e1a46)
PPU 1: PPU (UUID: GPU-019ea108-c120-040c-0000-0000c0267f1e)

The list includes the following information:

• Device index: A zero-based enumeration index assigned at power-on.

• Device name

• Device UUID

3.1.1 MIG devices

When MIG mode is enabled, ppu-smi -L also displays information about MIG devices under each PPU device, as shown in the following example:

root@0549cf16bb85:~# ppu-smi -L
PPU 0: PPU (UUID: GPU-019ea108-c110-0828-0000-0000c07e1a46)
PPU 1: PPU (UUID: GPU-019ea108-c120-040c-0000-0000c0267f1e)
    MIG g1.c1.i1, Device 0 (UUID: MIG-79c62632-04cc-574b-af7b-cb2e307121c85)
    MIG g1.c0.i0, Device 1 (UUID: MIG-79c62632-04cc-574b-af7b-cb2e307121c84)

3.2 Summary

Run ppu-smi without any parameters to view the list of PPU devices in the system.

root@122d8d7a7e37:~# ppu-smi
Wed Sep  7 17:04:22 2022
+-------------------------------------------------------------------------------+
| PPU-SMI 1.2           Driver Version: 543.21.01     HGGC Version: 11.5        |
+---------------------------------+----------------------+----------------------+
| PPU  Name        Persistence M. | Bus-Id               | Volatile Uncorr. ECC |
| Fan  Temp  Perf   Pwr:Usage/Cap | Memory-Usage         | PPU-Util  Compute M. |
|                                 |                      |               MIG M. |
+=================================+======================+======================+
| 0  alixpu                N/A    | 00000000:5E:00.0     |                    0 |
| N/A  35C   P0       100W / 200W | 114MiB / 134473MiB   | 10%          Default |
|                                 |                      |             Disabled |
+---------------------------------+----------------------+----------------------+
| 1  alixpu                N/A    | 00000000:5E:00.1     |                    0 |
| N/A  34C   P0       100W / 200W | 4MiB / 134473MiB     | 10%          Default |
|                                 |                      |             Disabled |
+---------------------------------+----------------------+----------------------+

+-------------------------------------------------------------------------------+
| Processes:                                                                    |
| PPU    GI   CI   PID   Type   Process name                         PPU Memory |
|        ID   ID                                                     Usage      |
+===============================================================================+
| 0      N/A  N/A  25759 C      cuda_test1                               151MiB |
| 0      N/A  N/A  25677 C      cuda_test2                               151MiB |
+-------------------------------------------------------------------------------+

The query results are presented in two tables: the PPU device list and the PPU process list. Information that is not supported by the current device is displayed as N/A.

The PPU devices table includes the following information:

• PPU: The zero-based enumeration index assigned to the device at power-on.

• Name: The device name.

• Persistence M.: Indicates whether persistence mode is enabled.

• Fan: The target fan speed as a percentage of its maximum speed.

• Temp: The current device temperature.

• Perf: The performance state of the device. P0 is the highest performance. P1, P2, and so on indicate sequentially lower performance. If not supported, N/A is displayed.

• Pwr:Usage/Cap: Current power draw (W) / Power limit (W) of the device.

• Bus-Id: The PCI Bus ID of the device.

• Memory-Usage: Used device memory (MiB) / Total device memory (MiB).

• Volatile Uncorr. ECC: The total number of uncorrectable ECC errors that have occurred on the device since the driver was loaded.

• PPU-Util: The current PPU stream processor utilization.

• Compute M.: The current compute mode, such as Default, Prohibited, or Exclusive Process (displayed as E. Process).

• MIG M.: The current MIG mode.

The processes table includes the following information:

• PPU: The zero-based enumeration index assigned to the device at power-on.

• CI ID: The ID of the compute instance that the process uses.

• PID: The system process ID.

• Type: The compute type of the process, where C stands for Compute.

• Process name: The name of the process.

• PPU Memory Usage: The amount of PPU memory (MiB) that this process uses.

3.2.1 MIG devices

If a PPU device has the MIG feature enabled, you can run ppu-smi to view a table of existing MIG device information:

+-----------------------------------------------------------------------------+
| MIG devices:                                                                |
+------------------+---------------------+----------+-------------------------+
| PPU  GI  CI  MIG |        Memory-Usage |      Vol |         Shared          |
|      ID  ID  DEV |                     |  CU  Unc | CpyEng ENC DEC OFA JPEG |
|                  |                     |      ECC |                         |
+==================+=====================+==========+=========================+
|  0   0   0    5  |   106MiB / 33618MiB |  64    0 |   4     2   2   0   0   |
+------------------+---------------------+----------+-------------------------+
|  1   1   2    3  |     6MiB / 16809MiB |  32    0 |   2     0   0   0   0   |
+------------------+---------------------+----------+-------------------------+
|  1   1   3    2  |     6MiB / 16809MiB |  32    0 |   2     0   0   0   0   |
+------------------+---------------------+----------+-------------------------+

The information in the MIG device list table is described as follows:

• PPU: The zero-based enumeration index assigned to the device at power-on.

• GI ID: The GPU instance ID associated with the MIG device.

• CI ID: The compute instance ID associated with the MIG device.

• MIG DEV: The enumeration index of the MIG device.

• Memory-Usage: Used device memory (MiB) / Total device memory (MiB) for this MIG device.

• CU: The number of dedicated CU resources for this MIG device.

• Vol Unc ECC: The total number of uncorrectable ECC errors that have occurred on the device since the driver was loaded.

• Shared: This MIG device shares the resources listed below with other MIG devices on the same GPU instance.

  • CpyEng: The number of shared copy engines.

  • ENC: The number of shared encoders.

  • DEC: The number of shared decoders.

  • OFA: The number of shared OFA processing units.

  • JPEG: The number of shared JPEG processing units.

3.2.2 Other options

4. Query device information

root@122d8d7a7e37:~# ppu-smi -q

==============PPUSMI LOG==============

Timestamp                                   : Wed Sep  7 19:13:36 2022
Driver Version                              : 510.39.01
HGGC Version                                : 11.6
SDK Version                                 : 1.4.45-5188b0

Attached PPUs                               : 1
PPU 00000000:01:00.0
    Product Name                            : alixpu
    Persistence Mode                        : Disabled
    MIG Mode
        Current                             : N/A
        Pending                             : N/A
    Serial Number                           : N/A
    PPU UUID                                : PPU-7f53d39f-ce6e-dc78-c3d4-4c18653c19c0
    PCI
        Bus                                 : 0x01
        Device                              : 0x00
        Domain                              : 0x0000
        Device Id                           : 0x1E0410DE
...

root@122d8d7a7e37:~# ppu-smi -q -d MEMORY
...
PPU 00000000:01:00.0
    HBM Vendor                              : Samsung
    Memory Usage
        Total                               : 11264 MiB
        Used                                : 248 MiB
        Free                                : 11016 MiB

root@2b92dd1ad851:~# ppu-smi -q -d UTILIZATION
...
PPU 00000000:3E:00.0
    Utilization
        Ppu                                 : 38 %
        Core                                : 30 %
        Memory                              : 0 %
        Encoder                             : 0 %
        Decoder                             : 0 %
    PPU Utilization Samples
        Duration                            : 19.89 sec
        Number of Samples                   : 99
        Max                                 : 56 %
        Min                                 : 1 %
        Avg                                 : 31 %
    Memory Utilization Samples
        Duration                            : 19.89 sec
        Number of Samples                   : 99
        Max                                 : 0 %
        Min                                 : 0 %
        Avg                                 : 0 %
    ENC Utilization Samples
        Duration                            : 19.89 sec
        Number of Samples                   : 99
        Max                                 : 0 %
        Min                                 : 0 %
        Avg                                 : 0 %
    DEC Utilization Samples
        Duration                            : 19.89 sec
        Number of Samples                   : 99
        Max                                 : 0 %
        Min                                 : 0 %
        Avg                                 : 0 %

root@2b92dd1ad851:~# ppu-smi -q -d ECC
...
PPU 00000000:3E:00.0
    Ecc Mode
        Current                             : Enabled
        Pending                             : Enabled
    ECC Errors
        Volatile
            SRAM Correctable                : 0
            SRAM Uncorrectable              : 0
            DRAM Correctable                : 0
            DRAM Uncorrectable              : 0
        Aggregate
            SRAM Correctable                : 0
            SRAM Uncorrectable              : 0
            DRAM Correctable                : 0
            DRAM Uncorrectable              : 0

root@2b92dd1ad851:~# ppu-smi -q -d TEMPERATURE
...
PPU 00000000:3E:00.0
    Temperature
        PPU Current Temp                    : 42 C
        PPU Shutdown Temp                   : 95 C
        PPU Slowdown Temp                   : 92 C
        PPU Max Operating Temp              : 85 C
        PPU Target Temperature              : N/A
        Memory Current Temp                 : 42 C
        Memory Max Operating Temp           : 95 C

root@2b92dd1ad851:~# ppu-smi -q -d POWER
...
PPU 00000000:3E:00.0
    Power Readings
        Power Management                    : Supported
        Power Draw                          : 51.99 W
        Power Limit                         : 250.00 W
        Default Power Limit                 : 250.00 W
        Enforced Power Limit                : 250.00 W
        Min Power Limit                     : 150.00 W
        Max Power Limit                     : 250.00 W
    Power Samples
        Duration                            : 2.39 sec
        Number of Samples                   : 119
        Max                                 : 58.68 W
        Min                                 : 51.40 W
        Avg                                 : 55.23 W

root@2b92dd1ad851:~# ppu-smi -q -d CLOCK
...
PPU 00000000:08:00.0                        
    Clocks                                  
        CU                                  : 200 MHz
        Memory                              : 1800 MHz
        Video                               : 1000 MHz
    Applications Clocks                     
        CU                                  : 1700 MHz
        Memory                              : 1800 MHz
    Default Applications Clocks             
        CU                                  : 1700 MHz
        Memory                              : 1800 MHz
    Max Clocks                              
        CU                                  : 1700 MHz
        Memory                              : 1800 MHz
        Video                               : 1000 MHz

root@2b92dd1ad851:~# ppu-smi -q -d COMPUTE
...
PPU 00000000:3E:00.0
    Compute Mode                            : Default

root@2b92dd1ad851:~# ppu-smi -q -d PIDS
...
PPU 00000000:3E:00.0
    Processes
        Compute instance ID                 : N/A
        Process ID                          : 85637
            Type                            : Compute
            Name                            : ppu_test
            Used PPU Memory                 : 413 MiB
        Compute instance ID                 : N/A
        Process ID                          : 87771
            Type                            : Compute
            Name                            : ppu_test_threads
            Used PPU Memory                 : 413 MiB

root@122d8d7a7e37:~# ppu-smi -q -d SUPPORTED_CLOCKS
...
PPU 00000000:01:00.0
    Supported Clocks
        Memory                              : 7000 MHz
            CU                              : 2100 MHz
            CU                              : 2085 MHz
            CU                              : 2070 MHz
            CU                              : 2055 MHz
            CU                              : 2040 MHz
            ...
        Memory                              : 6800 MHz
            CU                              : 2100 MHz
            CU                              : 2085 MHz
            CU                              : 2070 MHz
            CU                              : 2055 MHz
            CU                              : 2040 MHz
            ...

root@122d8d7a7e37:~# ppu-smi -q -d PAGE_RETIREMENT
...
PPU 00000000:10:00.0
    Retired Pages
        Single Bit ECC                      : 0
        Double Bit ECC                      : 0
        Pending Page Blacklist              : No

root@122d8d7a7e37:~# ppu-smi -q -d ROW_REMAPPER
...
PPU 00000000:10:00.0
    Remapped Rows
        Correctable Error                   : 0
        Uncorrectable Error                 : 0
        Pending                             : No
        Remapping Failure Occurred          : No
        Bank Remap Availability Histogram
            Max                             : 3072 bank(s)
            High                            : 0 bank(s)
            Partial                         : 0 bank(s)
            Low                             : 0 bank(s)
            None                            : 0 bank(s)

root@2b92dd1ad851:~# ppu-smi --query-ppu=timestamp,index,name,compute_mode,memory.total,memory.used --format=csv
timestamp, index, name, compute_mode, memory.total [MiB], memory.used [MiB]
2022/09/08 09:24:11.132, 0, alixpu, Default, 40960 MiB, 606 MiB
2022/09/08 09:24:11.139, 1, alixpu, Default, 24576 MiB, 508 MiB

root@2b92dd1ad851:~# ppu-smi --help-query-ppu
List of valid properties to query for the switch "--query-ppu=":

"timestamp"
The timestamp of when the query was made in format "YYYY/MM/DD HH:MM:SS.msec".

"driver_version"
The version of the installed driver. This is an alphanumeric string.

"count"
The number of PPUs in the system.

"name" or "ppu_name"
The official product name of the PPU. This is an alphanumeric string. For all products.

"serial" or "ppu_serial"
This number matches the serial number physically printed on each board. It is a globally unique immutable alphanumeric value.

"uuid" or "ppu_uuid"
This value is the globally unique immutable alphanumeric identifier of the PPU. It does not correspond to any physical label on the board.

"pci.bus_id" or "ppu_bus_id"
PCI bus id as "domain:bus:device.function", in hex.
...

root@2b92dd1ad851:~# ppu-smi --query-ppu=timestamp,count,name,serial,index,uuid --format=csv
timestamp, count, name, serial, index, uuid
2022/09/08 09:42:34.913, 2, alixpu, 1320421013145, 0, PPU-0cdd7938-b576-2411-a408-3ad81dfc1a78
2022/09/08 09:42:34.920, 2, alixpu, 1323921045367, 1, PPU-16c4c41f-9214-5e29-3b86-7a26ab011d3e

root@2b92dd1ad851:~# ppu-smi --help-query-supported-clocks
List of valid properties to query for the switch "--query-supported-clocks=":

[Section about Supported Clocks properties]
Lists the possible combinations of memory and processor clocks that the PPU supports (not taking into account HW brake reduced clocks).

"timestamp"
The timestamp of when the query was made in format "YYYY/MM/DD HH:MM:SS.msec".

"name" or "ppu_name"
The official product name of the PPU. This is an alphanumeric string. For all products.

...

"memory" or "mem"
Memory clock.

"processor" or "sm"
Streaming multiprocessors clock.

root@2b92dd1ad851:~# ppu-smi --query-supported-clocks=timestamp,ppu_bus_id,memory,processor --format=csv
timestamp, pci.bus_id, memory [MHz], processor [MHz]
2022/09/08 09:50:26.289, 00000000:3E:00.0, 1215 MHz, 1410 MHz
2022/09/08 09:50:26.289, 00000000:3E:00.0, 1215 MHz, 1395 MHz
2022/09/08 09:50:26.289, 00000000:3E:00.0, 1215 MHz, 1380 MHz
2022/09/08 09:50:26.289, 00000000:3E:00.0, 1215 MHz, 1365 MHz
2022/09/08 09:50:26.289, 00000000:3E:00.0, 1215 MHz, 1350 MHz
2022/09/08 09:50:26.289, 00000000:3E:00.0, 1215 MHz, 1335 MHz
2022/09/08 09:50:26.289, 00000000:3E:00.0, 1215 MHz, 1320 MHz
2022/09/08 09:50:26.289, 00000000:3E:00.0, 1215 MHz, 1305 MHz
2022/09/08 09:50:26.289, 00000000:3E:00.0, 1215 MHz, 1290 MHz
...

root@2b92dd1ad851:~# ppu-smi --help-query-compute-apps
List of valid properties to query for the switch "--query-compute-apps=":

[Section about Active Compute Processes properties]
Lists processes that have a compute context on the device.

"timestamp"
The timestamp of when the query was made in format "YYYY/MM/DD HH:MM:SS.msec".

"name" or "ppu_name"
The official product name of the PPU. This is an alphanumeric string. For all products.

...

"pid"
Process ID of the compute application.

"process_name"
Process Name.

"used_ppu_memory" or "used_memory"
Amount of memory used on the device by the context.

root@2b92dd1ad851:~# ppu-smi --query-compute-apps=timestamp,uuid,pid,process_name,used_ppu_memory --format=csv
timestamp, uuid, pid, process_name, used_ppu_memory [MiB]
2022/09/08 09:57:57.660, PPU-0cdd7938-b576-2411-a408-3ad81dfc1a78, 61785, ppu_test, 413 MiB
2022/09/08 09:57:57.660, PPU-0cdd7938-b576-2411-a408-3ad81dfc1a78, 62103, ppu_test_threads, 413 MiB

root@0549cf16bb85:~# ppu-smi --help-query-retired-pages
List of valid properties to query for the switch "--query-retired-pages=":

[Section about Retired Pages properties]
Lists pages that have been retired or are pending retirement.

"timestamp"
The timestamp of when the query was made in format "YYYY/MM/DD HH:MM:SS.msec".

"name" or "ppu_name"
The official product name of the PPU. This is an alphanumeric string. For all products.
...

"retired_pages.address"
Address of a retired page. Address might be different when ECC is Enabled or Disabled.

"retired_pages.timestamp"
Timestamp at which the page was retired.

"retired_pages.cause"
Reason that describes why the page was retired. Can take one of two values:
 - Double Bit ECC: The number of PPU device memory pages that have been retired due to a double bit ECC error.
 - Single Bit ECC: The number of PPU device memory pages that have been retired due to multiple single bit ECC errors.

root@2b92dd1ad851:~# ppu-smi --query-retired-pages=timestamp,uuid,retired_pages.address,retired_pages.timestamp,retired_pages.cause --format=csv
timestamp, uuid, retired_pages.address, retired_pages.timestamp, retired_pages.cause
12:32:15.329, PPU-3f53d39f-ce6e-dc78-c3d4-4c18653c19c0, 0x0000000073234722, 1663256354, Single Bit ECC

root@0549cf16bb85:~# ppu-smi --help-query-remapped-rows
List of valid properties to query for the switch "--query-remapped-rows=":

"timestamp"
The timestamp of when the query was made in format "YYYY/MM/DD HH:MM:SS.msec".

"name" or "ppu_name"
The official product name of the PPU. This is an alphanumeric string. For all products.
...

"remapped_rows.correctable"
The number of rows that have been remapped due to correctable ECC errors.

"remapped_rows.uncorrectable"
The number of rows that have been remapped due to uncorrectable ECC errors.

"remapped_rows.pending"
Whether or not there are pending row-remappings.

"remapped_rows.failure"
Whether or not a row remapping has failed in the past.

"remap_availability.bank_histogram.max"
The number of banks that have max remap availability(all reserved rows are available).

"remap_availability.bank_histogram.high"
The number of banks that have high remap availability.

"remap_availability.bank_histogram.partial"
The number of banks that have partial remap availability.
...

root@2b92dd1ad851:~# ppu-smi --query-remapped-rows=timestamp,uuid,remapped_rows.correctable,remapped_rows.uncorrectable,remap_availability.bank_histogram.max --format=csv
timestamp, uuid, remapped_rows.correctable, remapped_rows.uncorrectable, remap_availability.bank_histogram.max
2022/09/08 16:24:40.435, PPU-099ea108-0181-0230-0000-000060f19f20, 0, 0, 3072
2022/09/08 16:24:40.435, PPU-019ea108-01a1-0222-0000-000040dff62d, 0, 0, 3072
2022/09/08 16:24:40.435, PPU-019ea108-0121-0222-0000-0000605e6417, 0, 0, 3072

5. Modify device configuration

5.1 Device configuration options

PPU-SMI allows you to modify device configurations, such as the PPU processor clock. For example, run the following command to set the clock to 1410 MHz: ppu-smi -lpc 1410.

root@122d8d7a7e37:~# ppu-smi -lpc 1410
Set PPU clock to (min clock 1410MHz, max clock 1410MHz) for PPU 00000000:01:00.0.
All done.

Modify only one device configuration per command.

5.1.1 Reset a PPU device

To reset all PPU devices, run ppu-smi -r. To target a specific PPU device, use the -i option. For example, ppu-smi -r -i 0 resets PPU device 0. Before resetting a device, ensure that the following prerequisites are met:

1. No HGGC-related compute applications are running on any PPU device.

2. No PPU-related monitoring or utility applications, such as ppu-smi, ppudbg, or PPU DCGM, are running on any PPU device.

If a reset fails because such applications are running, you can use the following commands to identify them:

# Query for HGGC-related applications
ppu-smi pmon -c 1

# Query for PPU-related monitoring and utility applications
lsof /dev/alixpu

5.2 Other options

6. Collect device statistics

PPU-SMI collects device samples and events, printing them in CSV format to help you analyze the device's operational state. Use the stats subcommand to access device statistics. Run ppu-smi stats -h to view the help information for this subcommand:

root@2b92dd1ad851:~# ppu-smi stats -h
Generates PPU statistics such as power samples,
utilization samples, xid events, clock change events
and performance capping events.

ppu-smi stats [OPTION1 [ARG1]] [OPTION2 [ARG2]] ...
    -i,   --id
        Enumeration index, Serial number, PCI bus ID or UUID.
        Provide comma separated values for more than one device
    -f,   --filename
        Log to a specified file, rather than to stdout
    -d,   --display
        Display specific metric:
            pwrDraw,temp,memUtil,ppuUtil,
            encUtil,decUtil,memClk,procClk,
            violPwr,violThm,xidEvent,sbEcc,
            dbEcc,pState,clkChg,pwrChg,migChg
        Metric can be combined with comma e.g. pwrDraw,temp
    -c,   --count
        Run for specified number of monitoring cycles and exit
    -h,   --help
        Display help information

Stats in following CSV format:
Device, Power Drawn (pwrDraw), Timestamp (us), Value (Watts)
Device, PPU Temperature (temp), Timestamp (us), Value (C)
Device, PPU Util (ppuUtil), Timestamp (us), Value (%)
...

6.1 Output format

The query results are in CSV format. The following example shows the output of running ppu-smi stats:

root@122d8d7a7e37:~# ppu-smi stats
0, violPwr , 1662606252538553, 0
0, violThm , 1662606252537681, 0
0, temp    , 1662606252536736, 51
0, pState  , 1662606252559897, 0
0, clkChg  , 1662606252559964, 0
0, pState  , 1662606252568416, 0
0, clkChg  , 1662606252568499, 0
...
0, temp    , 1662606253541531, 52
0, pwrDraw , 1662606252543916, 45
0, pwrDraw , 1662606252567432, 54
0, pwrDraw , 1662606252584972, 66
0, pwrDraw , 1662606252605044, 66
...

The CSV output does not have a header row. Each column has the following meaning:

1. Column 1: Device index. A zero-based enumeration number assigned to the device when it is powered on.

2. Column 2: The metric abbreviation, such as pwrDraw or temp.

3. Column 3: The timestamp of data collection, in microseconds (us). This value is the elapsed time from the start of the system_clock to the time of sampling.

4. Column 4: The sampled value. This is a unitless value. Its specific meaning and unit are detailed in the metrics reference below.

6.2 Metrics reference

Use the -d option to specify which metrics to collect. Separate multiple metrics with a comma (,), for example, ppu-smi stats -d pwrDraw,temp:

root@122d8d7a7e37:~# ppu-smi stats -d pwrDraw,temp
0, temp    , 1662606935107702, 51
0, pwrDraw , 1662606935059582, 38
0, pwrDraw , 1662606935079459, 39
0, pwrDraw , 1662606935099668, 39
...

If you do not specify the -d option, PPU-SMI collects all supported metrics by default. The following table describes the supported metrics:

For event-based metrics, such as xidEvent, PPU-SMI prints output when an event occurs. For non-event-based metrics, such as pwrDraw, PPU-SMI aggregates the sampling results and prints the output once per second.

By default, PPU-SMI runs continuously and collects data until you press Ctrl + C to stop it.

6.3 Other options

Monitor device status

PPU-SMI supports a scrolling display of device monitoring information through the dmon subcommand. This command prints the monitoring information for one device per line, with each type of information displayed as a scrolling column. For example, run ppu-smi dmon:

root@2b92dd1ad851:~# ppu-smi dmon
# ppu   pwr ptemp mtemp    cu  core   mem   enc   dec  mclk  pclk
# idx     W     C     C     %     %     %     %     %   MHz   MHz
    0   109    30    32     0     2     0     0     0  1800  1500
    1   119    33    34     0     0     0     0     0  1800  1500
    0   109    30    32     0     0     0     0     0  1800  1500
    1   121    33    34     0     0     0     0     0  1800  1500
...

Run ppu-smi dmon -h to view the help information for the device monitoring feature:

root@2b92dd1ad851:~# ppu-smi dmon -h
PPU statistics are displayed in scrolling format with one line
per sampling interval. Metrics to be monitored can be adjusted
based on the width of terminal window.

ppu-smi dmon [OPTION1 [ARG1]] [OPTION2 [ARG2]] ...
    -i,   --id=
        Enumeration index, Serial number, PCI bus ID or UUID.
        Provide comma separated values for more than one device
    -d,   --delay=
        Collection delay/interval in seconds [default=1sec]
    -c,   --count=
        Collect specified number of samples and exit
    -s,   --select=
        One or more metrics [default=puc]
        Can be any of the following:
            p - Power Usage and Temperature
            u - Utilization
            c - Proc and Mem Clocks
            v - Power and Thermal Violations
            m - SM Memory
            e - ECC Errors and PCIe Replay errors
            t - PCIe Rx and Tx Throughput
    -o,   --options=
        One or more from the following:
            D - Include Date (YYYYMMDD) in scrolling output
            T - Include Time (HH:MM:SS) in scrolling output
...

Monitoring metrics

You can use the -s option to specify the metrics you want to monitor. To monitor multiple metrics, provide a string of their corresponding characters to the -s option. For example, ppu-smi dmon -s pc displays metrics related to power (p) and clocks (c).

root@2b92dd1ad851:~# ppu-smi dmon -s pc
# ppu   pwr ptemp mtemp  mclk  pclk
# idx     W     C     C   MHz   MHz
    0    41    45    45  1215   765
    1    30    36    36  1215   930
...

The following table describes the supported metrics.

The following list describes the columns in the PPU-SMI scrolling output:

• Basic information

  • Date: The current date.

  • Time: The current time.

  • ppu: The zero-based enumeration index of the device, assigned at power-on.

• Metric type p

  • pwr: The current power draw in watts (W).

  • ptemp: The current processor temperature.

  • mtemp: The current memory temperature.

• Metric type u

  • sm: The stream processor utilization in percent (%).

  • core: The core utilization in percent (%).

  • mem: The memory utilization in percent (%).

  • enc: The encoder utilization in percent (%).

  • dec: The decoder utilization in percent (%).

• Metric type c

  • pclk: The processor clock frequency in MHz.

  • mclk: The memory clock frequency in MHz.

• Metric type v

  • pviol: The percentage of the sampling interval during which the device was throttled due to power limits (%).

  • tviol: The percentage of the sampling interval during which the device was throttled due to temperature limits (%).

• Metric type m

• mem: The current memory usage in MiB.

• Metric type e

  • sbecc: The single-bit ECC error count since the driver was loaded.

  • dbecc: The double-bit ECC error count since the driver was loaded.

  • pci: The PCI replay count.

• Metric type t

  • rxpci: The PCI receive throughput in MB/s.

  • txpci: The PCI transmit throughput in MB/s.

Other options

8. Monitor process status

PPU-SMI supports scrolling output for process monitoring information by using the pmon subcommand. Each line prints the monitoring information for one process on one device, and each type of information is displayed as a scrolling column. For example, run ppu-smi pmon:

root@122d8d7a7e37:~# ppu-smi pmon
# ppu     pid  type    sm   mem   enc   dec   command
# idx       #   C/G     %     %     %     %   name
    0    4563     C     0     0     0     0   ppu_test
    0    4991     C     0     0     0     0   ppu_test_thread
...

You can run the ppu-smi pmon -h command to view the help information for the process monitoring feature:

root@122d8d7a7e37:~# ppu-smi pmon -h
Process statistics are displayed in scrolling format per sampling
interval. This tool lists the statistics for all the compute
processes running on each device. Metrics to be monitored
can be adjusted based on the width of terminal window.

ppu-smi pmon [OPTION1 [ARG1]] [OPTION2 [ARG2]] ...
    -i,   --id=
        Enumeration index, Serial number, PCI bus ID or UUID.
        Provide comma separated values for more than one device
    -d,   --delay=
        Collection delay/interval in seconds [default=1sec]
    -c,   --count=
        Collect specified number of samples and exit
    -s,   --select=
        One or more metrics [default=u]
        Can be any of the following:
            u - Utilization
            m - Memory usage
    -o,   --options=
        One or more from the following:
            D - Include Date (YYYYMMDD) in scrolling output
            T - Include Time (HH:MM:SS) in scrolling output
...

8.1 Monitoring metrics

You can use the -s option to specify the metrics that you want to monitor. Each metric is identified by a single character, and you can concatenate the characters for multiple metrics and pass them to the -s option. For example, ppu-smi pmon -s um specifies metrics related to process utilization (u) and memory usage (m).

root@122d8d7a7e37:~# ppu-smi pmon -s um
# ppu     pid  type    sm   mem   enc   dec   mem   command
# idx       #   C/G     %     %     %     %    MB   name
    0    4563     C     0     0     0     0   151   ppu_test
    0    4991     C     0     0     0     0   151   ppu_test_thread

The following table describes the supported metrics.

The following list describes the columns in the PPU-SMI scrolling output:

• Basic information

  • Date: The current date.

  • Time: The current time.

  • ppu: The zero-based enumeration index of the device, assigned at power-on.

  • pid: The system process ID.

  • type: The type of process. C indicates a compute type process.

• Metric type u

  • sm: The stream processor utilization, in percent (%).

  • mem: The memory utilization, in percent (%).

  • enc: The encoder utilization, in percent (%).

  • dec: The decoder utilization, in percent (%).

• Metric type p

  • mem: The memory usage of the process, in MiB.

8.2 Other options

9. Query ICN links

PPU-SMI queries ICN link information using the icn subcommand. The results are grouped by device. For example, run ppu-smi icn -s.

root@dfc623e46a90:~# ppu-smi icn -s
PPU 0: PPU (UUID: GPU-019ea108-c180-040a-0000-000000000000)
    Link 0: 50 GB/s
    Link 1: <inactive>
    Link 2: <inactive>
    Link 3: 50 GB/s
    Link 4: 50 GB/s
    Link 5: 50 GB/s
    Link 6: 50 GB/s
PPU 1: PPU (UUID: GPU-019ea108-c180-060c-0000-000000000000)
    Link 0: 50 GB/s
    Link 1: <inactive>
    Link 2: <inactive>
    Link 3: 50 GB/s
...

Run ppu-smi icn -h to view the help information. You can use the -i and -l options to restrict the query to specific devices and links; omitting these options queries all devices and links. You must specify exactly one query option, such as -s.

root@dfc623e46a90:~# ppu-smi icn -h
icn -- Display ICN link information.

ppu-smi icn [OPTION1 [ARG1]] [OPTION2 [ARG2]] ...
    -h,   --help
        Display help information
    -i,   --id=
        Enumeration index, Serial number, PCI bus ID or UUID.
        Provide comma separated values for more than one device.
    -l,   --link=
        Specify a target link ID (0-based link index),
        Without this parameter, all links information are displayed.

    [any one of]

    -s,   --status
        Display link state (active/inactive).
    -c,   --capabilities
        Display link capabilities.
    -p,   --pcibusid
        Display remote node PCI bus ID for a link.
    -r,   --remotelinkinfo
        Display remote device PCI bus ID and ICN link ID for a link.

9.1 Link state

Use ppu-smi icn -s to query the link state and bandwidth. You can use the -i option to target a specific device. For example, run ppu-smi icn -s -i 0. The output is described below:

root@dfc623e46a90:~# ppu-smi icn -s -i 0
PPU 0: PPU (UUID: GPU-019ea108-c180-040a-0000-000000000000)
    Link 0: 50 GB/s
    Link 1: <inactive>
    Link 2: <inactive>
    Link 3: 50 GB/s
    Link 4: 50 GB/s
    Link 5: 50 GB/s
    Link 6: 50 GB/s

• Link 0: 50 GB/s: ICN link 0 on PPU 0 is connected to another device with a bandwidth of 50 GB/s.

• Link 1: <inactive>: ICN link 1 on PPU 0 is not connected to another device.

9.2 Link capabilities

Use ppu-smi icn -c to query ICN link capabilities. The command displays information only for active ICN links. The following is an example of the command output:

root@dfc623e46a90:~# ppu-smi icn -c
PPU 0: PPU (UUID: GPU-019ea108-c180-040a-0000-000000000000)
    Link 0, P2P is supported: true
    Link 0, Access to system memory is supported: false
    Link 0, P2P atomics is supported: true
    Link 0, System memory atomics is supported: false
    Link 0, SLI is supported: false
    Link 0, Link is supported: true
    Link 3, P2P is supported: true
    Link 3, Access to system memory is supported: false
    Link 3, P2P atomics is supported: true
    Link 3, System memory atomics is supported: false
    Link 3, SLI is supported: false
    Link 3, Link is supported: true
...

9.3 Remote PCI bus ID

Use ppu-smi icn -p to query the remote PCI bus ID for each ICN link. The command displays information only for active ICN links. The following is an example of the command output:

root@dfc623e46a90:~# ppu-smi icn -p
PPU 0: PPU (UUID: GPU-019ea108-c180-040a-0000-000000000000)
    Link 0: 00000001:CE:00.0
    Link 3: 00000000:89:00.0
    Link 4: 00000000:89:00.0
    Link 5: 00000000:CC:00.0
    Link 6: 00000001:CE:00.0
PPU 1: PPU (UUID: GPU-019ea108-c180-060c-0000-000000000000)
    Link 0: 00000001:D1:00.0
    Link 3: 00000000:C9:00.0
    Link 4: 00000000:86:00.0
    Link 5: 00000000:86:00.0
    Link 6: 00000001:D1:00.0
...

9.4 Remote link info

Use ppu-smi icn -r to query remote link info. The output includes the PCI bus ID of the remote device and the remote ICN link ID. The command displays information only for active ICN links. The following is an example of the command output:

root@dfc623e46a90:~# ppu-smi icn -r
PPU 0: PPU (UUID: GPU-019ea108-c180-040a-0000-000000000000)
    Link 0: Remote Device 00000001:CE:00.0: Link 6
    Link 3: Remote Device 00000000:89:00.0: Link 1
    Link 4: Remote Device 00000000:89:00.0: Link 1
    Link 5: Remote Device 00000000:CC:00.0: Link 3
    Link 6: Remote Device 00000001:CE:00.0: Link 6
PPU 1: PPU (UUID: GPU-019ea108-c180-060c-0000-000000000000)
    Link 0: Remote Device 00000001:D1:00.0: Link 7
    Link 3: Remote Device 00000000:C9:00.0: Link 2
    Link 4: Remote Device 00000000:86:00.0: Link 0
    Link 5: Remote Device 00000000:86:00.0: Link 0
    Link 6: Remote Device 00000001:D1:00.0: Link 7
...

9.5 Cable presence status

Use ppu-smi icn -cs to query the cable presence status for each ICN link port. The following is an example of the command output:

root@dfc623e46a90:~# ppu-smi icn -cs
PPU 0: t-head ppu 0 (UUID: GPU-3f53d39f-ce6e-dc78-c3d4-4c18653c19c0)
    Link 0: Connected
    Link 1: Disconnected
    Link 2: Connected
...
PPU 1: t-head ppu 1 (UUID: GPU-3f53d39f-ce6e-dc78-c3d4-4c18653c19c1)
    Link 0: Connected
    Link 1: Connected
    Link 2: Connected
...

9.6 Link width

Use ppu-smi icn -lw to query the ICN link width. The command displays information only for active ICN links. The following is an example of the command output:

root@dfc623e46a90:~# ppu-smi icn -lw
PPU 0: PPU-ZW810 (UUID: GPU-019ea108-4111-0220-0000-0000006ef62f)
    Link 0: 16x
    Link 3: 16x
    Link 4: 16x
    Link 5: 16x
    Link 6: 8x
PPU 1: PPU-ZW810 (UUID: GPU-019ea108-8191-042a-0000-0000c0d6cc1d)
    Link 0: 16x
    Link 3: 16x
    Link 4: 16x
    Link 5: 16x
    Link 6: 8x
...

• 16x: The link width is 16 lanes.

• 8x: The link width is 8 lanes.

9.7 Link to physical port mapping

Use ppu-smi icn -lm to query the mapping between ICN links and physical ports. The following is an example of the command output:

root@dfc623e46a90:~# ppu-smi icn -lm
PPU 0: PPU-ZW810 (UUID: GPU-019ea108-4111-0220-0000-0000006ef62f)
    Link 0: Physical Port 2
    Link 1: N/A
    Link 2: Physical Port 5
...
PPU 1: PPU-ZW810 (UUID: GPU-019ea108-8191-042a-0000-0000c0d6cc1d)
    Link 0: Physical Port 2
    Link 1: N/A
    Link 2: Physical Port 5
...

• Links not mapped to a physical port appear as N/A.

9.8 Link statistics and error counts

Use ppu-smi icn -e to query ICN link statistics and error counts. The command displays information only for active ICN links. The following is an example of the command output:

PPU 0: PPU-ZW810 (UUID: GPU-019ea108-4111-0220-0000-0000006ef62f)
    Link 0: 2 link up times
    Link 0: 1 link down times
    Link 0: 0 FEC correctable errors
    Link 0: 0 FEC uncorrectable errors
    Link 0: 0 TX packet errors
    Link 0: 0 RX packet errors
    Link 0: 13 total TX packets
    Link 0: 11 total RX packets

    Link 3: 2 link up times
    Link 3: 1 link down times
    Link 3: 0 FEC correctable errors
    Link 3: 0 FEC uncorrectable errors
    Link 3: 0 TX packet errors
    Link 3: 0 RX packet errors
    Link 3: 16 total TX packets
    Link 3: 8 total RX packets
...

PPU 1: PPU-ZW810 (UUID: GPU-019ea108-8191-042a-0000-0000c0d6cc1d)
    Link 0: 2 link up times
    Link 0: 1 link down times
    Link 0: 0 FEC correctable errors
    Link 0: 0 FEC uncorrectable errors
    Link 0: 0 TX packet errors
    Link 0: 0 RX packet errors
    Link 0: 9 total TX packets
    Link 0: 4 total RX packets
...

• link up times: The number of times the link has become active.

• link down times: The number of times the link has become inactive.

• FEC correctable errors: The number of corrected Forward Error Correction (FEC) errors.

• FEC uncorrectable errors: The number of uncorrectable FEC errors.

• TX packet errors: The number of packet errors during transmission.

• RX packet errors: The number of packet errors during reception.

• total TX packets: The total number of transmitted packets.

• total RX packets: The total number of received packets.

9.9 Link throughput

Use ppu-smi icn -gt r to query the total raw data throughput of ICN links. The following is an example of the command output:

root@0549cf16bb85:~# ppu-smi icn -gt r
PPU 0: PPU (UUID: GPU-019ea108-c110-0828-0000-000000000000)
    Link 0: Raw Tx: 1618498333 KiB
    Link 0: Raw Rx: 1657515730 KiB
    Link 3: Raw Tx: 1653010223 KiB
    Link 3: Raw Rx: 1693049419 KiB
    Link 4: Raw Tx: 1622645860 KiB
    Link 4: Raw Rx: 1662045442 KiB
...
PPU 1: PPU (UUID: GPU-019ea108-c120-040c-0000-000000000000)
    Link 0: Raw Tx: 1618498333 KiB
    Link 0: Raw Rx: 1657515730 KiB
    Link 3: Raw Tx: 1653010223 KiB
    Link 3: Raw Rx: 1693049419 KiB
    Link 4: Raw Tx: 1622645860 KiB
    Link 4: Raw Rx: 1662045442 KiB
...

10. Querying device topology

PPU-SMI allows you to query topology information between devices using the topo subcommand. For example, run ppu-smi topo -m:

root@dfc623e46a90:~# ppu-smi topo -m
         PPU0    PPU1    PPU2    PPU3    PPU4    PPU5    PPU6    PPU7    CPU Affinity    NUMA Affinity
 PPU0    X       ICN2    SYS     ICN1    SYS     SYS     ICN2    SYS     0-47,96-143     0
 PPU1    ICN2    X       ICN1    SYS     SYS     SYS     SYS     ICN2    0-47,96-143     0
 PPU2    SYS     ICN1    X       ICN2    ICN1    ICN1    SYS     SYS     0-47,96-143     0
 PPU3    ICN1    SYS     ICN2    X       ICN1    ICN1    SYS     SYS     0-47,96-143     0
 PPU4    SYS     SYS     ICN1    ICN1    X       ICN2    SYS     ICN1    48-95,144-191   1
 PPU5    SYS     SYS     ICN1    ICN1    ICN2    X       ICN1    SYS     48-95,144-191   1
 PPU6    ICN2    SYS     SYS     SYS     SYS     ICN1    X       ICN2    48-95,144-191   1
 PPU7    SYS     ICN2    SYS     SYS     ICN1    SYS     ICN2    X       48-95,144-191   1
...

Run ppu-smi topo -h to view help for topology queries. You can specify one topo query option (for example, -m) at a time. When using the -n or -p options, you must use the -i option to specify the target devices:

root@dfc623e46a90:~# ppu-smi topo -h
topo -- Display topological information about the system.

ppu-smi topo [OPTION1 [ARG1]] [OPTION2 [ARG2]] ...
    -h,   --help
        Display help information
    -i,   --id=
        Enumeration index, Serial number, PCI bus ID or UUID.
        Provide comma separated values for more than one device.
        Must be used in conjunction with -n or -p.
    -ri,  --rear-id=
        When used with the option to display matrix (-m or -mp), 
        Show PPU devices that match the specified rear id.
    -po,  --ppu-only
        When used with the option to display matrix (-m or -mp),
        Show PPU devices only.
    -rg,  --rear-group
        When used with the option to display matrix (-m or -mp), 
        Group PPU device by rear id.

    [any one of]

    -m,   --matrix
        Display the PPUDirect communication matrix for the system.
    -mp,  --matrix_pci
        Display the PPUDirect communication matrix for the system (PCI Only).
    -lni, --list-network-interface
        Display a list of RDMA network interface controller(NIC) connected to the system.
    -c,   --cpu=
        Specify a CPU number, Display all PPUs with an affinity.
    -n,   --nearest_ppus=
        Display the nearest PPUs for a given traversal path.
        Could be one of the following:
            0 = a single PCIe switch on a dual PPU board
            1 = a single PCIe switch
            2 = multiple PCIe switches
            3 = a PCIe host bridge
            4 = an on-CPU interconnect link between PCIe host bridges
            5 = an SMP interconnect link between NUMA nodes
        Used in conjunction with -i which must be a single device ID.
    -p,   --ppu_path
        Display the most direct path traversal for a pair of PPUs.
        Used in conjunction with -i which must be a pair of device IDs.
    -p2p, --p2pstatus=
        Displays the p2p status between the PPUs of a given p2p capability.
        Could be one of the following:
            r - p2p read capabiity
            w - p2p write capability
            n - p2p ICN link capability
            a - p2p atomics capability
            p - p2p prop capability

10.1 Querying the topology matrix

Run ppu-smi topo -m to query the connection status between device pairs, as well as between PPU devices and RDMA network interface controllers (NICs). If an ICN link is active, the matrix shows its connectivity information first. The output is described below:

root@dfc623e46a90:/# ppu-smi topo -m
         PPU0    PPU1    PPU2    PPU3    PPU4    PPU5    PPU6    PPU7    NIC0    NIC1    NIC2    NIC3    NIC4    NIC5    NIC6    NIC7    CPU Affinity    NUMA Affinity
 PPU0    X       ICN2    SYS     ICN1    SYS     SYS     ICN2    SYS     PXB     SYS     SYS     SYS     SYS     SYS     PXB     SYS     0-47,96-143     0
 PPU1    ICN2    X       ICN1    SYS     SYS     SYS     SYS     ICN2    PIX     SYS     SYS     SYS     SYS     SYS     PIX     SYS     0-47,96-143     0
 PPU2    SYS     ICN1    X       ICN2    ICN1    ICN1    SYS     SYS     SYS     PXB     SYS     SYS     SYS     SYS     SYS     PXB     0-47,96-143     0
 PPU3    ICN1    SYS     ICN2    X       ICN1    ICN1    SYS     SYS     SYS     PXB     SYS     SYS     SYS     SYS     SYS     PXB     0-47,96-143     0
 PPU4    SYS     SYS     ICN1    ICN1    X       ICN2    SYS     ICN1    SYS     SYS     SYS     PXB     SYS     SYS     SYS     SYS     48-95,144-191   1
 PPU5    SYS     SYS     ICN1    ICN1    ICN2    X       ICN1    SYS     SYS     SYS     SYS     PXB     SYS     SYS     SYS     SYS     48-95,144-191   1
 PPU6    ICN2    SYS     SYS     SYS     SYS     ICN1    X       ICN2    SYS     SYS     PXB     SYS     SYS     PXB     SYS     SYS     48-95,144-191   1
 PPU7    SYS     ICN2    SYS     SYS     ICN1    SYS     ICN2    X       SYS     SYS     PIX     SYS     SYS     PIX     SYS     SYS     48-95,144-191   1
 NIC0    PXB     PIX     SYS     SYS     SYS     SYS     SYS     SYS     X       SYS     SYS     SYS     SYS     SYS     PIX     SYS     0-47,96-143     0
 NIC1    SYS     SYS     PXB     PXB     SYS     SYS     SYS     SYS     SYS     X       SYS     SYS     SYS     SYS     SYS     PIX     0-47,96-143     0
 NIC3    SYS     SYS     SYS     SYS     SYS     SYS     PXB     PIX     SYS     SYS     X       SYS     SYS     PIX     SYS     SYS     48-95,144-191   1
 NIC4    SYS     SYS     SYS     SYS     PXB     PXB     SYS     SYS     SYS     SYS     SYS     X       SYS     SYS     SYS     SYS     48-95,144-191   1
 NIC5    SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     X       SYS     SYS     SYS     0-47,96-143     0
 NIC6    SYS     SYS     SYS     SYS     SYS     SYS     PXB     PIX     SYS     SYS     PIX     SYS     SYS     X       SYS     SYS     48-95,144-191   1
 NIC7    PXB     PIX     SYS     SYS     SYS     SYS     SYS     SYS     PIX     SYS     SYS     SYS     SYS     SYS     X       SYS     0-47,96-143     0

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  ICN# = Connection traversing a bonded set of # ICN links

NIC Legend:

  NIC0: mlx5_bond_0
  NIC1: mlx5_bond_1
  NIC2: mlx5_bond_2
  NIC3: mlx5_bond_3
  NIC4: mlx5_bond_4
  NIC5: mlx5_bond_5
  NIC6: mlx5_bond_6
  NIC7: mlx5_bond_7

PPU Rear Group:

  Rear ID 0: PPU 0,1,2,3,4,5,6,7

• ICN2: For example, there are 2 active ICN links between PPU0 and PPU1, which is displayed as ICN2.

• SYS: Indicates no active ICN link exists between the devices. The matrix instead shows the connection status over the PCIe bus. For details, see the Legend.

• CPU Affinity 0-47,96-143: Describes the CPU affinity of the PPU device. For example, PPU0 has an affinity with CPU cores 0 through 47 and 96 through 143.

• NUMA Affinity 0: Describes the NUMA affinity of the PPU device. For example, PPU0 has an affinity with NUMA node 0.

• NIC0: Represents an RDMA network interface controller (NIC) connected to the system. Refer to the NIC Legend for the actual device name.

• PPU Rear Group: Lists the PPU devices grouped by rear ID. Each row shows the PPUs connected to that rear.

Use the -po option to show only PPU devices in the topology matrix, excluding RDMA NICs. Use the -mp option to show only PCIe bus connections, excluding ICN link status.

For example, run ppu-smi topo -mp -po to query the PCIe bus connections between PPU device pairs. The following example shows the output:

root@dfc623e46a90:/# ppu-smi topo -mp -po
         PPU0    PPU1    PPU2    PPU3    PPU4    PPU5    PPU6    PPU7    CPU Affinity    NUMA Affinity
 PPU0    X       PXB     SYS     SYS     SYS     SYS     SYS     SYS     0-47,96-143     0
 PPU1    PXB     X       SYS     SYS     SYS     SYS     SYS     SYS     0-47,96-143     0
 PPU2    SYS     SYS     X       PXB     SYS     SYS     SYS     SYS     0-47,96-143     0
 PPU3    SYS     SYS     PXB     X       SYS     SYS     SYS     SYS     0-47,96-143     0
 PPU4    SYS     SYS     SYS     SYS     X       PXB     SYS     SYS     48-95,144-191   1
 PPU5    SYS     SYS     SYS     SYS     PXB     X       SYS     SYS     48-95,144-191   1
 PPU6    SYS     SYS     SYS     SYS     SYS     SYS     X       PXB     48-95,144-191   1
 PPU7    SYS     SYS     SYS     SYS     SYS     SYS     PXB     X       48-95,144-191   1

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge

10.1.1 Filtering and grouping by rear ID

PPU-SMI lets you filter the topology matrix by rear ID to show only the PPU devices connected to a specific rear. Use the -ri option to specify the rear ID. You can find the rear ID for a PPU by running a command like ppu-smi -q or ppu-smi --query-ppu, or by checking the PPU Rear Group information in the matrix output.

For example, run ppu-smi topo -m -po -ri 1 to query the topology information for PPUs with rear ID 1. The output shows only PPUs with that rear ID, such as PPU 1, 3, and 5, while PPUs connected to other rears are not displayed:

root@dfc623e46a90:/# ppu-smi topo -m -po -ri 1
         PPU1    PPU3    PPU5    PPU7    PPU9    PPU11   PPU13   PPU15  CPU Affinity    NUMA Affinity
 PPU1    X       ICN2    SYS     ICN1    SYS     SYS     ICN2    SYS    0-47,96-143     0
 PPU3    ICN2    X       ICN1    SYS     SYS     SYS     SYS     ICN2   0-47,96-143     0
 PPU5    SYS     ICN1    X       ICN2    ICN1    ICN1    SYS     SYS    0-47,96-143     0
 PPU7    ICN1    SYS     ICN2    X       ICN1    ICN1    SYS     SYS    0-47,96-143     0
 PPU9    SYS     SYS     ICN1    ICN1    X       ICN2    SYS     ICN1   48-95,144-191   1
 PPU11   SYS     SYS     ICN1    ICN1    ICN2    X       ICN1    SYS    48-95,144-191   1
 PPU13   ICN2    SYS     SYS     SYS     SYS     ICN1    X       ICN2   48-95,144-191   1
 PPU15   SYS     ICN2    SYS     SYS     ICN1    SYS     ICN2    X      48-95,144-191   1

...

PPU Rear Group:

  Rear ID 1: PPU 1,3,5,7,9,11,13,15

PPU-SMI also lets you group the topology matrix by rear ID. When grouped, the PPU devices in the matrix are sorted first by rear ID and then by PPU index. For example, PPUs with rear ID 0 will be grouped together and displayed first.

Use the -rg option to enable grouping by rear ID. For example, run ppu-smi topo -mp -po -rg. In the output, the PPUs are sorted by rear ID first:

root@dfc623e46a90:/# ppu-smi topo -mp -po -rg
         PPU0    PPU2    PPU4    PPU6    PPU8    PPU10   PPU12   PPU14   PPU1    PPU3    PPU5    PPU7    PPU9    PPU11   PPU13   PPU15   CPU Affinity    NUMA Affinity
 PPU0    X       PHB     SYS     PXB     NODE    PIX     PHB     SYS     PXB     NODE    PIX     PHB     SYS     PXB     NODE    PIX     0-1,64-65       0
 PPU2    PHB     X       PXB     NODE    PIX     PHB     SYS     PXB     NODE    PIX     PHB     SYS     PXB     NODE    PIX     PHB     0-1,64-65       0             
 PPU4    SYS     PXB     X       PIX     PHB     SYS     PXB     NODE    PIX     PHB     SYS     PXB     NODE    PIX     PHB     SYS     0-1,64-65       0  
 PPU6    PXB     NODE    PIX     X       SYS     PXB     NODE    PIX     PHB     SYS     PXB     NODE    PIX     PHB     SYS     PXB     0-1,64-65       0             
 PPU8    NODE    PIX     PHB     SYS     X       NODE    PIX     PHB     SYS     PXB     NODE    PIX     PHB     SYS     PXB     NODE    0-1,64-65       0             
 PPU10   PIX     PHB     SYS     PXB     NODE    X       PHB     SYS     PXB     NODE    PIX     PHB     SYS     PXB     NODE    PIX     0-1,64-65       0             
 PPU12   PHB     SYS     PXB     NODE    PIX     PHB     X       PXB     NODE    PIX     PHB     SYS     PXB     NODE    PIX     PHB     0-1,64-65       0             
 PPU14   SYS     PXB     NODE    PIX     PHB     SYS     PXB     X       PIX     PHB     SYS     PXB     NODE    PIX     PHB     SYS     0-1,64-65       0             
 PPU1    PXB     NODE    PIX     PHB     SYS     PXB     NODE    PIX     X       SYS     PXB     NODE    PIX     PHB     SYS     PXB     0-1,64-65       0             
 PPU3    NODE    PIX     PHB     SYS     PXB     NODE    PIX     PHB     SYS     X       NODE    PIX     PHB     SYS     PXB     NODE    0-1,64-65       0             
 PPU5    PIX     PHB     SYS     PXB     NODE    PIX     PHB     SYS     PXB     NODE    X       PHB     SYS     PXB     NODE    PIX     0-1,64-65       0             
 PPU7    PHB     SYS     PXB     NODE    PIX     PHB     SYS     PXB     NODE    PIX     PHB     X       PXB     NODE    PIX     PHB     0-1,64-65       0             
 PPU9    SYS     PXB     NODE    PIX     PHB     SYS     PXB     NODE    PIX     PHB     SYS     PXB     X       PIX     PHB     SYS     0-1,64-65       0             
 PPU11   PXB     NODE    PIX     PHB     SYS     PXB     NODE    PIX     PHB     SYS     PXB     NODE    PIX     X       SYS     PXB     0-1,64-65       0             
 PPU13   NODE    PIX     PHB     SYS     PXB     NODE    PIX     PHB     SYS     PXB     NODE    PIX     PHB     SYS     X       NODE    0-1,64-65       0             
 PPU15   PIX     PHB     SYS     PXB     NODE    PIX     PHB     SYS     PXB     NODE    PIX     PHB     SYS     PXB     NODE    X       0-1,64-65       0

 ...

 PPU Rear Group:

  Rear ID 0: PPU 0,2,4,6,8,10,12,14
  Rear ID 1: PPU 1,3,5,7,9,11,13,15

10.2 Listing RDMA NICs

You can use the -lni option to list the RDMA NIC devices in the system. For example, run ppu-smi topo -lni to display a list of NICs including their names and PCI bus IDs:

root@dfc623e46a90:/# ppu-smi topo -lni
NIC 0: mlx5_bond_0 (PCI Bus Id: 00000000:8A:00.1)
NIC 1: mlx5_bond_1 (PCI Bus Id: 00000000:CF:00.1)
NIC 2: mlx5_bond_2 (PCI Bus Id: 00000001:D2:00.0)
NIC 3: mlx5_bond_3 (PCI Bus Id: 00000001:87:00.0)
NIC 4: mlx5_bond_4 (PCI Bus Id: 00000000:2A:00.1)
NIC 5: mlx5_bond_5 (PCI Bus Id: 00000001:D2:00.1)
NIC 6: mlx5_bond_6 (PCI Bus Id: 00000000:8A:00.0)
NIC 7: mlx5_bond_7 (PCI Bus Id: 00000000:CF:00.0)

10.3 Querying devices by CPU affinity

PPU-SMI can list the devices that have an affinity for a specific CPU. Use the -c option to specify the CPU index. For example, to list the devices with an affinity for CPU 0, run ppu-smi topo -c 0:

root@dfc623e46a90:/# ppu-smi topo -c 0
The PPUs that have an affinity with CPU 0 are: 0, 1, 2, 3

10.4 Querying nearest PPU devices

PPU-SMI supports querying the list of neighboring devices for a specific PPU. Use the -n option to specify the topology scope of the query. For the definition of topology scope, see the help information. Use the -i option to specify the target PPU device. For example, run ppu-smi topo -n 5 -i 0. The following shows an example of the query result:

root@dfc623e46a90:/# ppu-smi topo -n 5 -i 0
Device 0 is connected by way of a SMP interconnect link between NUMA nodes to device(s): 1, 2, 3, 4, 5, 6, 7

10.5 Querying the most direct PPU path

PPU-SMI can show the most direct path between two PPU devices. Use the -i option to specify the pair of device IDs. For example, run ppu-smi topo -p -i 0,1:

root@dfc623e46a90:/# ppu-smi topo -p -i 0,1
Device 0 is connected to device 1 by way of potentially multiple PCIe switches.

10.6 Querying P2P capability

You can use the -p2p option to specify a capability type, and PPU-SMI will display the support status for P2P capabilities between devices. For the definition of capability type, see the help information. For example, run ppu-smi topo -p2p r to query the P2P read capability between devices. An example of the query result is as follows:

root@dfc623e46a90:/# ppu-smi topo -p2p r
         PPU0    PPU1    PPU2    PPU3    PPU4    PPU5    PPU6    PPU7
 PPU0    X       OK      OK      OK      OK      OK      OK      OK
 PPU1    OK      X       OK      OK      OK      OK      OK      OK
 PPU2    OK      OK      X       OK      OK      OK      OK      OK
 PPU3    OK      OK      OK      X       OK      OK      OK      OK
 PPU4    OK      OK      OK      OK      X       OK      OK      OK
 PPU5    OK      OK      OK      OK      OK      X       OK      OK
 PPU6    OK      OK      OK      OK      OK      OK      X       OK
 PPU7    OK      OK      OK      OK      OK      OK      OK      X

Legend:

  X    = Self
  OK   = Status Ok
  CNS  = Chipset not supported
  PNS  = PPU not supported
  TNS  = Topology not supported
  NS   = Not supported
  U    = Unknown

11. Multi-instance GPU (MIG)

PPU-SMI lets you query, create, and destroy instances in Multi-Instance GPU (MIG) mode.

• GPU instance: A PPU hardware resource can be partitioned into multiple instances. Each partition is a GPU instance that runs in isolation from others.

• GPU instance profile: Defines how a PPU's hardware resources can be partitioned. For example, a profile specifies the number of dedicated CUs and the amount of device memory for each resulting GPU instance.

• compute instance: A GPU instance can be further partitioned into multiple instances. Each partition is a compute instance. A compute instance has dedicated CU resources but shares other resources. You can configure applications to run on specific compute instances.

• compute instance profile: Defines how a GPU instance can be partitioned. For example, a profile specifies the number of dedicated CUs and the shared resources for each resulting compute instance.

You can enable MIG mode on a PPU device by running ppu-smi -mig 1 (for details, see 5. Modify device configuration). Use the mig subcommand to query MIG information. For example, run ppu-smi mig -lgip to list the supported GPU instance profiles:

root@0549cf16bb85:~# ppu-smi mig -lgip
+---------------------------------------------------------------------------------+
| GPU instance profiles:                                                          |
| PPU  Name                 Profile  Instances   Memory  P2P  CU      DEC   ENC   |
|                             ID     Free/Total   GiB         CpyEng  JPEG  OFA   |
+=================================================================================+
| 1    MIG 8g48gb              3        0/1      48.00   No   64      4     4     |
|                                                             2       4     0     |
+---------------------------------------------------------------------------------+
| 1    MIG 4g24gb              2        1/2      24.00   No   32      2     2     |
|                                                             2       2     0     |
+---------------------------------------------------------------------------------+
| 1    MIG 2g12gb              1        2/4      12.00   No   16      1     1     |
|                                                             2       1     0     |
+---------------------------------------------------------------------------------+
| 1    MIG 1g6gb               0        4/8      6.00    No   8       0     0     |
|                                                             2       0     0     |
+---------------------------------------------------------------------------------+

Run ppu-smi mig -h for help with MIG commands. You can specify one mig query sub-option at a time. You can restrict some sub-options to a specific scope by using the -i, -gi, or -ci options. You can use these options individually or in combination. For example, -i 0 -gi 1 targets gpu instance 1 on PPU 0.

root@dfc623e46a90:/# ppu-smi mig -h
mig -- Multi Instance GPU management.

ppu-smi mig [OPTION1 [ARG1]] [OPTION2 [ARG2]] ...
    -h,   --help
        Display help information
    -i,   --id=
        Enumeration index, Serial number, PCI bus ID or UUID.
        Provide comma separated values for more than one device.
    -gi,  --gpu-instance-id=
        GPU instance ID.
        Provide comma separated values for more than one GPU instance.
    -ci,  --compute-instance-id=
        Compute instance ID.
        Provide comma separated values for more than one compute instance.
    -C,   --default-compute-instance
        When used with the option to create a GPU instance (-cgi),
        Create compute instance with the default profile.

    [any one of]

    -lgip,--list-gpu-instance-profiles
        List supported GPU instance profiles.
        Option -i can be used to restrict the command to run on a specific PPU.
    -lgipp,--list-gpu-instance-possible-placements
        List possible GPU instance placements in the following format:
          {Start,Start...}:Size
        Option -i can be used to restrict the command to run on a specific PPU.
    -cgi, --create-gpu-instance=
        Create GPU instances for the given profile tuples.
        A profile tuple consists of a profile name or ID and an optional placement specifier,
        which consists of a colon and a placement start index.
        Provide comma separated values for more than one profile tuple(e.g. 1:0,4:2).
        Option -i can be used to restrict the command to run on a specific PPU.
    -dgi, --destroy-gpu-instance
        Destroy GPU instances.
        Options -i and -gi can be used individually or combined
        to restrict the command to run on a specific PPU or GPU instance.
    -lgi, --list-gpu-instances
        List GPU instances.
        Option -i can be used to restrict the command to run on a specific PPU.
    -r,   --reset-gpu-instance
        Trigger reset of the GPU instance.
        Options -i and -gi can be used individually or combined
        to restrict the command to run on a specific PPU or GPU instance.
    -lcip,--list-compute-instance-profiles
        List supported compute instance profiles.
        Options -i and -gi can be used individually or combined
        to restrict the command to run on a specific PPU or GPU instance.
    -lcipp,--list-compute-instance-possible-placements
        List possible compute instance placements in the following format:
          {Start,Start...}:Size
        Options -i and -gi can be used individually or combined
        to restrict the command to run on a specific PPU or GPU instance.
    -cci, --create-compute-instance=
        Create compute instance for the given profile tuples.
        A profile tuple consists of a profile name or ID and an optional placement specifier,
        which consists of a colon and a placement start index.
        Provide comma separated values for more than one profile tuple(e.g. 1:0,4:2).
        If no profile name or ID is given, then the default*
        compute instance profile ID will be used.
        Options -i and -gi can be used individually or combined
        to restrict the command to run on a specific PPU or GPU instance.
    -dci, --destroy-compute-instance
        Destroy compute instances.
        Options -i, -gi and -ci can be used individually or combined
        to restrict the command to run on a specific PPU or GPU instance or compute instance.
    -lci, --list-compute-instances
        List compute instances.
        Options -i and -gi can be used individually or combined
        to restrict the command to run on a specific PPU or GPU instance.

11.1 Supported GPU instance profiles

Use the -lgip option to list supported GPU instance profiles. You can then use the -cgi option with a profile from this list to create a GPU instance. Use the -i option to restrict the query to a specific device. For example, run ppu-smi mig -i 1 -lgip. The following describes the output fields:

root@a475cc8d4c49:/# ppu-smi mig -i 1 -lgip
+---------------------------------------------------------------------------------+
| GPU instance profiles:                                                          |
| PPU  Name                 Profile  Instances   Memory  P2P  CU      DEC   ENC   |
|                             ID     Free/Total   GiB         CpyEng  JPEG  OFA   |
+=================================================================================+
| 1    MIG 8g96gb              3        1/1      96.00   No   64      4     4     |
|                                                             2       4     0     |
+---------------------------------------------------------------------------------+
| 1    MIG 4g48gb              2        2/2      48.00   No   32      2     2     |
|                                                             2       2     0     |
+---------------------------------------------------------------------------------+
| 1    MIG 2g24gb              1        4/4      24.00   No   16      1     1     |
|                                                             2       1     0     |
+---------------------------------------------------------------------------------+
| 1    MIG 1g12gb              0        8/8      12.00   No   8       0     0     |
|                                                             2       0     0     |
+---------------------------------------------------------------------------------+

• PPU: The zero-based enumeration index assigned to the device when it powers on.

• Name: The name of the GPU instance profile. You can use this name with the -cgi option to create a GPU instance.

• profile ID: The ID of the GPU instance profile. You can use this ID with the -cgi option to create a GPU instance.

• Instances Free/Total: The number of available GPU instances out of the total number supported by this profile.

• Memory GiB: The device memory allocated by this profile, in GiB.

• P2P: Indicates whether this profile supports peer-to-peer capability.

• CU: The number of dedicated CU resources for this profile.

• DEC: The number of dedicated decoder resources for this profile.

• ENC: The number of dedicated encoder resources for this profile.

• CpyEng: The number of dedicated copy engine resources for this profile.

• JPEG: The number of dedicated JPEG processing units for this profile.

• OFA: The number of dedicated OFA processing units for this profile.

11.2 Possible GPU instance placements

Use the -lgipp option to list possible placements for GPU instance profiles. You can then use the -cgi option to create a GPU instance at a specific placement.

Possible placements are displayed in the format {start0, start1, start2}:size, where start0, start1, start2 are possible start indices, and size is the size of each placement. For example, the output in the following code block indicates that you can create an instance of size 4 at either start index 0 or 4.

{0,4}:4

For example, run ppu-smi mig -i 1 -lgipp. The following is an example of the output:

root@a475cc8d4c49:/# ppu-smi mig -i 1 -lgipp
PPU 1 profile ID 0 placements: {0,1,2,3,4,5,6,7}:1
PPU 1 profile ID 1 placements: {0,2,4,6}:2
PPU 1 profile ID 2 placements: {0,4}:4
PPU 1 profile ID 3 placements: {0,4}:4

11.3 Creating GPU instances

Use the -cgi option to create one or more GPU instances. Separate multiple GPU instance specifications with a comma (,). For each GPU instance you want to create:

1. Specify which GPU instance profile to use by providing its name or ID.

   1. You can use either the full or short name of the profile. For example, for a profile named MIG 1g12gb, you can specify either MIG 1g12gb or 1g12gb.

2. Optionally, specify a start index for the GPU instance by appending a colon (:) and the index to the profile information.

   1. The placement specifier can be omitted.

Example: Create one GPU instance by specifying profile ID 3, without a placement specifier.

-cgi 3

Example: Create two GPU instances by specifying profile IDs 0 and 1, without placement specifiers.

-cgi 0,1

Example: Create one GPU instance by specifying profile ID 1 and a start index of 4.

-cgi 1:4

Example: Create one GPU instance by specifying the short profile name 1g12gb, without a placement specifier.

-cgi 1g12gb

Example: Create two GPU instances by specifying the profile names MIG 1g12gb and MIG 2g24gb, with placement specifiers for each.

-cgi "MIG 1g12gb:0,MIG 2g24gb:4"

You can use the -i option to create the GPU instance on a specific PPU device. If you omit this option, the command attempts to create the instance on every PPU device. For example, run ppu-smi mig -i 1 -cgi 1. The following is an example of the output:

root@a475cc8d4c49:~# ppu-smi mig -i 1 -cgi 1
Successfully created GPU instance ID 0 on PPU 1 using profile MIG 2g24gb (Profile ID 1)

11.4 Listing GPU instances

Use the -lgi option to list existing GPU instances. Use the -i option to restrict the query to a specific device. For example, run ppu-smi mig -i 1 -lgi. The following describes the output fields:

root@0549cf16bb85:~# ppu-smi mig -i 1 -lgi
+---------------------------------------------------------+
| GPU instances:                                          |
| PPU  Name                 Profile  Instance  Placement  |
|                             ID        ID     Start:Size |
+=========================================================+
| 1    MIG 2g12gb              1        0         0:2     |
+---------------------------------------------------------+
| 1    MIG 2g12gb              1        2         2:2     |
+---------------------------------------------------------+

• PPU: The zero-based enumeration index assigned to the device when it powers on.

• Name: The name of the GPU instance.

• profile ID: The ID of the GPU instance profile that was used to create this GPU instance.

• instance ID: The ID of the GPU instance. You can use this ID with the -gi option to target this GPU instance.

• placement: The placement information for this GPU instance.

11.5 Destroying GPU instances

You can use the -dgi option to delete a GPU instance, the -i option to specify the target devices, and the -gi option to specify the target GPU instances. The -i and -gi options can be used separately or together.

For example, to destroy gpu instance 1 on PPU 1, run ppu-smi mig -dgi -i 1 -gi 1. The following is an example of the output:

root@0549cf16bb85:~# ppu-smi mig -dgi -i 1 -gi 1
Successfully destroyed GPU instance ID 1 from PPU 1

11.6 Resetting a GPU instance

The -r option resets a GPU instance without affecting the operation of other GPU instances. The -i option constrains the operation to specific devices, and the -gi option constrains the operation to specific GPU instances. The -i and -gi options can be used individually or together.

For example, to reset gpu instance 0 on PPU 1, run ppu-smi mig -i 1 -gi 0 -r. The following is an example of the output:

root@0549cf16bb85:~# ppu-smi mig -i 1 -gi 0 -r
Successfully trigger reset of GPU instance ID 0 from PPU 1.

11.7 Supported compute instance profiles

You can use the -lcip option to query the compute instance profiles that a GPU instance can be partitioned into, the -i option to limit the query to a specific device, and the -gi option to limit the query to a specific GPU instance. The -i and -gi options can be used separately or together. For example, run ppu-smi mig -lcip -i 1 -gi 0 to query the supported compute instance partitions for gpu instance 0 on PPU 1. The following describes the results:

root@0549cf16bb85:~# ppu-smi mig -lcip -i 1 -gi 0
+--------------------------------------------------------------------------------------+
| Compute instance profiles:                                                           |
| PPU    GPU     Name                 Profile  Instances   Exclusive       Shared      |
|      Instance                         ID     Free/Total     CU       DEC   ENC  OFA  |
|         ID                                                          CpyEng JPEG      |
+======================================================================================+
| 1       0      MIG 1u.2g12gb           0       16/16         1         0    0    0   |
|                                                                        2    0        |
+--------------------------------------------------------------------------------------+
| 1       0      MIG 2u.2g12gb           1        8/8          2         0    0    0   |
|                                                                        2    0        |
+--------------------------------------------------------------------------------------+
| 1       0      MIG 3u.2g12gb           2        4/4          3         0    0    0   |
|                                                                        2    0        |
+--------------------------------------------------------------------------------------+
| 1       0      MIG 4u.2g12gb           3        4/4          4         0    0    0   |
|                                                                        2    0        |
+--------------------------------------------------------------------------------------+
| 1       0      MIG 8u.2g12gb           4        2/2          8         0    0    0   |
|                                                                        2    0        |
+--------------------------------------------------------------------------------------+
| 1       0      MIG 12u.2g12gb          5        1/1         12         0    0    0   |
|                                                                        2    0        |
+--------------------------------------------------------------------------------------+
| 1       0      MIG 16u.2g12gb         6*        1/1         16         1    1    0   |
|                                                                        2    1        |
+--------------------------------------------------------------------------------------+

• PPU: The zero-based enumeration index assigned to the device when it powers on.

• GPU Instance ID: The ID of the GPU instance.

• Name: The name of the compute instance profile. You can use this name with the -cci option to create a compute instance.

• profile ID: The ID of the compute instance profile. You can use this ID with the -cci option to create a compute instance.

  • An asterisk (*) marks the default profile ID, which is used when you create a default compute instance with the -C or -cci option.

• Instances Free/Total: The number of available compute instances out of the total number supported by this profile.

• Exclusive CU: The number of dedicated CUs for compute instances created with this profile.

• DEC: The number of shared decoder resources for this profile.

• ENC: The number of shared encoder resources for this profile.

• CpyEng: The number of shared copy engine resources for this profile.

• JPEG: The number of shared JPEG processing units for this profile.

• OFA: The number of shared OFA processing units for this profile.

11.8 Possible compute instance placements

You can use the -lcipp option to query the possible placements for a compute instance profile. You can then use the -cci option to create a compute instance at a specific placement. Use the -i option to restrict the operation to specific devices and the -gi option to restrict the operation to specific GPU instances. The -i and -gi options can be used individually or in combination. The possible placements are displayed in the following format: {start0, start1, start2}:size. This format indicates that there can be multiple start positions, such as start0 / start1 / start2, and that the size is the same for each placement. For example, this indicates that you can create a compute instance profile at placement 0 and 4, and the size of each instance is 4.

{0,4}:4

For example, run ppu-smi mig -i 3 -gi 0 -lcipp. The following is an example of the output:

root@0549cf16bb85:~# ppu-smi mig -i 3 -gi 0 -lcipp
PPU 3 GPU instance 0 profile ID 0 placements: {0,1,2,3,4,5,6,7}:1
PPU 3 GPU instance 0 profile ID 1 placements: {0,2,4,6}:2
PPU 3 GPU instance 0 profile ID 2 placements: {0,4}:3
PPU 3 GPU instance 0 profile ID 3 placements: {0,4}:4
PPU 3 GPU instance 0 profile ID 4 placements: {0}:8

11.9 Creating compute instances

Use the -cci option to create one or more compute instances. You can specify the name or ID of the compute instance profile. If you do not provide an argument, the command uses the default compute instance profile. Separate multiple compute instance specifications with a comma (,). For each compute instance you want to create:

1. Specify which compute instance profile to use by providing its name or ID.

   1. You can use either the full or short name of the profile. For example, for a profile named MIG 1u.1g6gb, you can specify either MIG 1u.1g6gb or 1u.1g6gb.

2. Optionally, specify a start index for the compute instance by appending a colon (:) and the index to the profile information.

   1. The placement specifier can be omitted.

You can use the -i option to limit an operation to specific devices and the -gi option to limit the operation to specific GPU instances. The -i and -gi options can be used individually or together. In addition, you can create a default Compute instance when you create a GPU instance. For example, run ppu-smi mig -i 1 -cgi 1 -C:

root@0549cf16bb85:~# ppu-smi mig -i 1 -cgi 1 -C
Successfully created GPU instance ID 4 on PPU 1 using profile MIG 2g12gb (Profile ID 1)
Successfully created compute instance ID 0 on PPU 1 GPU instance ID 4 using profile MIG 16u.2g12gb (Profile ID 6)

Example: Create one compute instance by specifying profile ID 3.

-cci 3

Example: Create two compute instances by specifying profile IDs 3,4.

-cci 3,4

Example: Create one compute instance by specifying profile ID 1 and a start index of 4.

-cci 1:4

Example: Create one compute instance by specifying the short profile name 1u.2g12gb.

-cci 1u.2g12gb

Example: Create two compute instances by specifying the profile names MIG 1u.2g12gb and MIG 2u.2g12gb.

-cci "MIG 1u.2g12gb,MIG 2u.2g12gb"

Example: Create one compute instance by using the default profile, without specifying a profile name or ID.

ppu-smi mig -cci -i 1 -gi 0

You can use the -i and -gi options to specify the PPU device and GPU instance for the new compute instances. If you omit these options, the command attempts to create the instances on all PPU devices and GPU instances. For example, run ppu-smi mig -i 1 -gi 0 -cci 3. The following is an example of the output:

root@0549cf16bb85:~# ppu-smi mig -i 1 -gi 0 -cci 3
Successfully created compute instance ID 0 on PPU 1 GPU instance ID 0 using profile MIG 4u.2g12gb (Profile ID 3)

11.10 Listing compute instances

Use the -lci option to query information about existing Compute instances, the -i option to limit the query scope to specific devices, and the -gi option to limit the query scope to specific GPU instances. The -i and -gi options can be used individually or together. For example, when you run ppu-smi mig -i 1 -gi 0 -lci, the query results are described as follows:

root@0549cf16bb85:~# ppu-smi mig -i 1 -gi 0 -lci
+-------------------------------------------------------------------+
| Compute instances:                                                |
| PPU    GPU     Name                 Profile  Instance  Placement  |
|      Instance                         ID        ID     Start:Size |
|         ID                                                        |
+===================================================================+
| 1       0      MIG 4u.2g12gb           3        0         0:0     |
+-------------------------------------------------------------------+
| 1       0      MIG 8u.2g12gb           4        1         0:1     |
+-------------------------------------------------------------------+

• PPU: The zero-based enumeration index assigned to the device when it powers on.

• GPU Instance ID: The ID of the GPU instance.

• Name: The name of the compute instance.

• profile ID: The ID of the compute instance profile that was used to create this compute instance.

• instance ID: The ID of this compute instance. You can use this ID with the -ci option to target this compute instance.

• placement: The placement information for this compute instance.

11.11 Destroying compute instances

You can use the -dci option to delete a compute instance, the -i option to limit the operation to specific devices, the -gi option to limit the operation to specific GPU instances, and the -ci option to limit the operation to specific compute instances. The -i, -gi, and -ci options can be used alone or in combination.

For example, to destroy compute instance 1 in gpu instance 0 on PPU 1, run ppu-smi mig -dci -i 1 -gi 0 -ci 1. The following is an example of the output:

root@0549cf16bb85:~# ppu-smi mig -dci -i 1 -gi 0 -ci 1
Successfully destroyed compute instance ID 1 from PPU 1 GPU instance 0

12. vGPU

PPU-SMI supports querying vGPU information, as well as creating and deleting vGPU instances. Using vGPU features requires administrator permissions.

The following key concepts apply to vGPU:

• vGPU type: Defines the types of vGPU that a PPU device supports. Each type specifies a resource partitioning scheme, such as the number of CEs and the amount of memory allocated to each vGPU instance.

• vGPU instance: An instance of a vGPU created from a specific vGPU type. You can use vGPU instances to create virtual machines (VMs).

The status of a vGPU instance can be one of the following:

• active instance: A vGPU instance that has been created and is associated with a VM.

• alive instance: A vGPU instance that has been created, regardless of whether it is associated with a VM.

You can enable vGPU mode on a PPU device by running ppu-smi -vm VGPU (for details, see 5. Modify device configuration). Use the vgpu subcommand to query vGPU-related information. For example, run ppu-smi vgpu -s to view the vGPU types that the device supports:

root@0549cf16bb85:~# ppu-smi vgpu -s
PPU 0: PPU (UUID: GPU-019ea108-c110-0420-0000-0000e0b7fe3c)
    vGPU Type 0: vGPU-0 (Class: MIG-Backed: 2 CE 5GB Memory 0 ICN 0 DVG 0 EVG)
    vGPU Type 1: vGPU-1 (Class: MIG-Backed: 2 CE 5GB Memory 0 ICN 1 DVG 0 EVG)
    vGPU Type 2: vGPU-2 (Class: MIG-Backed: 2 CE 5GB Memory 0 ICN 0 DVG 1 EVG)
    vGPU Type 3: vGPU-3 (Class: MIG-Backed: 2 CE 5GB Memory 1 ICN 0 DVG 0 EVG)
    vGPU Type 4: vGPU-4 (Class: MIG-Backed: 2 CE 5GB Memory 1 ICN 1 DVG 0 EVG)
    vGPU Type 5: vGPU-5 (Class: MIG-Backed: 2 CE 5GB Memory 1 ICN 0 DVG 1 EVG)
...

You can run ppu-smi vgpu -h to view help information about the vGPU subcommand. You can specify one query or operation option for the vgpu subcommand at a time. You can use the -i option to specify one or more PPU devices. For all query functions of the vgpu subcommand, you can use the -l option to periodically output the query results. The interval is specified in seconds. For example, -l 5 means that the results are output every 5 seconds until you press ctrl +c to cancel the query.

root@0549cf16bb85:~# ppu-smi vgpu -h
vgpu -- Virtual GPU management.

ppu-smi vgpu [OPTION1 [ARG1]] [OPTION2 [ARG2]] ...
    -h,   --help
        Display help information
    -i,   --id=
        Enumeration index, Serial number, PCI bus ID or UUID.
        Provide comma separated values for more than one device.
    -v,   --verbose
        Display detailed information about supported vGPU types or vGPU types that can be created.
    -l,   --loop=
        Display information at the specified time interval in seconds until Ctrl-C is pressed.
    -f,   --force
        When used with the option to delete a vGPU instance (-di),
        Delete vGPU instance regardless of VM state.

    [any one of]

    -q,   --query
        Display information about currently running vGPU instances (VM active).
    -a,   --alive
        Display information about currently alive vGPU instances.
    -s,   --supported
        Display supported vGPU types.
    -c,   --creatable
        Display the vGPU types that can currently be created.
    -ci,  --create-instance=
        Create a vGPU instance for the given vGPU type ID.
    -di,  --delete-instance=
        Delete a vGPU instance for the given vGPU instance ID.
        Use option -f to delete vGPU instance regardless of VM state.

12.1 Supported vGPU types

Use the -s option to list the vGPU types that a PPU device supports. You can use the -i option to restrict the query to specific devices. For example, run ppu-smi vgpu -i 0 -s. The output is described below:

root@0549cf16bb85:~# ppu-smi vgpu -i 0 -s
PPU 0: PPU (UUID: GPU-019ea108-c110-0420-0000-0000e0b7fe3c)
    vGPU Type 0: vGPU-0 (Class: MIG-Backed: 2 CE 5GB Memory 0 ICN 0 DVG 0 EVG)
    vGPU Type 1: vGPU-1 (Class: MIG-Backed: 2 CE 5GB Memory 0 ICN 1 DVG 0 EVG)
    vGPU Type 2: vGPU-2 (Class: MIG-Backed: 2 CE 5GB Memory 0 ICN 0 DVG 1 EVG)
    vGPU Type 3: vGPU-3 (Class: MIG-Backed: 2 CE 5GB Memory 1 ICN 0 DVG 0 EVG)
    vGPU Type 4: vGPU-4 (Class: MIG-Backed: 2 CE 5GB Memory 1 ICN 1 DVG 0 EVG)
    vGPU Type 5: vGPU-5 (Class: MIG-Backed: 2 CE 5GB Memory 1 ICN 0 DVG 1 EVG)
...

The list for each PPU device includes the following information:

• vGPU type ID

• Name of the vGPU type

• Detailed classification of the vGPU type

You can combine the -s and -v options to get more detailed vGPU type information. For example, run ppu-smi vgpu -i 0 -s -v. The following list describes the output fields:

root@0549cf16bb85:~# ppu-smi vgpu -i 0 -s -v
+-------------------------------------------------------------------+
| Supported vGPU types:                                             |
| PPU  Type   Name                 Instances     GPU       Memory   |
|       ID                         Free/Total  Instance     GiB     |
|                                                 ID                |
+===================================================================+
| 0      0    vGPU-0                  4/8         N/A      5.00     |
+-------------------------------------------------------------------+
| 0      1    vGPU-1                  0/1         N/A      5.00     |
+-------------------------------------------------------------------+
| 0      2    vGPU-2                  2/4         N/A      11.00    |
+-------------------------------------------------------------------+

• PPU: The zero-based enumeration index assigned to the device after it is powered on.

• Type ID: The ID of the vGPU type.

• Name: The name of the vGPU type.

• Instances Free/Total: The number of available instances of this type, versus the total number of supported instances.

• GPU instance ID: The MIG GPU instance ID mapped to the vGPU type. N/A indicates that no mapping exists.

• Memory GiB: The amount of memory allocated to each vGPU instance.

12.2 Creatable vGPU types

Use the -c option to list the vGPU types that can currently be created on a PPU device. You can use the -i option to restrict the query to specific devices. For example, run ppu-smi vgpu -i 0 -c. The output includes only the vGPU types that can be created with the PPU's current resources:

root@0549cf16bb85:~# ppu-smi vgpu -i 0 -c
PPU 0: PPU (UUID: GPU-019ea108-c110-0420-0000-0000e0b7fe3c)
    vGPU Type 0: vGPU-0 (Class: MIG-Backed: 2 CE 5GB Memory 0 ICN 0 DVG 0 EVG)
    vGPU Type 3: vGPU-3 (Class: MIG-Backed: 2 CE 5GB Memory 1 ICN 0 DVG 0 EVG)
...

The list for each PPU device includes the following information:

• vGPU type ID

• Name of the vGPU type

• Detailed classification of the vGPU type

You can combine the -c and -v options to get more detailed information about creatable vGPU types. For example, run ppu-smi vgpu -i 0 -c -v. The following list describes the output fields:

root@0549cf16bb85:~# ppu-smi vgpu -i 0 -c -v
+-------------------------------------------------------------------+
| Supported vGPU types:                                             |
| PPU  Type   Name                 Instances     GPU       Memory   |
|       ID                         Free/Total  Instance     GiB     |
|                                                 ID                |
+===================================================================+
| 0      0    vGPU-0                  4/8         N/A      5.00     |
+-------------------------------------------------------------------+
| 0      2    vGPU-2                  2/4         N/A      11.00    |
+-------------------------------------------------------------------+

• PPU: The zero-based enumeration index assigned to the device after it is powered on.

• Type ID: The ID of the vGPU type.

• Name: The name of the vGPU type.

• Instances Free/Total: The number of available instances of this type, versus the total number of supported instances.

• GPU instance ID: The MIG GPU instance ID mapped to the vGPU type. N/A indicates that no mapping exists.

• Memory GiB: The amount of memory allocated to each vGPU instance.

12.3 Create vGPU instance

Use the -ci option to create a vGPU instance from a specified vGPU type.

You can use the -i option to restrict the operation to a specific device. For example, run ppu-smi vgpu -i 0 -ci 0 to create a vGPU instance from vGPU type 0 on PPU device 0. The following is sample output:

root@0549cf16bb85:~# ppu-smi vgpu -ci 0 -i 0
Successfully created vGPU instance 0 (PCI Bus ID 00000000:5E:00.0) on PPU 0 using vGPU type 0.

12.4 Alive vGPU instances

Use the -a option to list all created (alive) vGPU instances. You can use the -i option to restrict the query to a specific device. For example, run ppu-smi vgpu -i 0 -a. The output is described below:

root@0549cf16bb85:~# ppu-smi vgpu -i 0 -a
PPU 00000000:10:00.0
    Alive vGPUs                             : 2
    vGPU Instance 1
        vGPU Instance ID                    : 1
        vGPU Name                           : vGPU-10
        vGPU Type                           : 10
        vGPU UUID                           : VGPU-3f53d39f-ce6e-dc78-c3d4-4c18653c19c0
        vGPU PCI Bus ID                     : 00000000:10:01.0
        MDEV UUID                           : N/A
        GPU Instance ID                     : N/A
        ECC Mode                            : Enabled
        Memory Size                         : 28672 MiB
    vGPU Instance 5
        vGPU Instance ID                    : 5
        vGPU Name                           : vGPU-1
        vGPU Type                           : 1
        vGPU UUID                           : VGPU-3f53d39f-ce6e-dc78-c3d4-4c18653c19c1
        vGPU PCI Bus ID                     : 00000000:10:01.4
        MDEV UUID                           : N/A
        GPU Instance ID                     : N/A
        ECC Mode                            : Enabled
        Memory Size                         : 7168 MiB

For each PPU device:

• Alive vGPUs: The number of vGPU instances that have been created on the PPU device.

For each vGPU instance:

• vGPU Instance ID: The ID of the vGPU instance. You can use this ID with the -di option to delete this instance.

• vGPU Name: The name of this vGPU instance.

• vGPU Type: The ID of the corresponding vGPU type.

• vGPU UUID: The UUID of the vGPU instance.

• vGPU PCI bus ID: The PCI bus ID of the vGPU instance, which is Bus:Device.Function.

• MDEV UUID: The UUID of the mediated device (mdev) corresponding to the vGPU instance. N/A indicates that it is not mapped to an mdev.

• GPU Instance ID: The MIG GPU instance ID corresponding to the vGPU instance. N/A indicates that it is not mapped to a MIG GPU instance.

• ECC mode: The ECC mode of the vGPU instance.

• Memory Size: The total amount of memory allocated to the vGPU instance.

12.5 Active vGPU instances

Use the -q option to list vGPU instances that are associated with a VM (active). You can use the -i option to restrict the query to a specific device. For example, run ppu-smi vgpu -i 0 -q. The output is described below:

root@0549cf16bb85:~# ppu-smi vgpu -i 0 -q
PPU 00000000:10:00.0
    Active vGPUs                            : 1
    vGPU Instance 1
        vGPU Instance ID                    : 1
        VM UUID                             : ee7b7a4b-388a-4357-a425-5318b2c65b30
        vGPU Name                           : vGPU-10
        vGPU Type                           : 10
        vGPU UUID                           : VGPU-3f53d39f-ce6e-dc78-c3d4-4c18653c19c0
        vGPU PCI Bus ID                     : 00000000:10:01.0
        MDEV UUID                           : N/A
        Guest Driver Version                : 0.8.0
        GPU Instance ID                     : N/A
        ECC Mode                            : Enabled
        Memory Usage
            Total                           : 28672 MiB
            Used                            : 3 MiB
            Free                            : 28669 MiB

For each PPU device:

• Active vGPUs: The number of vGPU instances that have been created and are associated with a VM on the PPU device.

For each vGPU instance:

• vGPU Instance ID: The ID of the vGPU instance. You can use this ID with the -di option to delete this instance.

• VM UUID: The UUID of the virtual machine.

• VM Domain ID: The domain ID of the virtual machine.

• vGPU Name: The name of this vGPU instance.

• vGPU Type: The ID of the corresponding vGPU type.

• vGPU UUID: The UUID of the vGPU instance.

• vGPU PCI bus ID: The PCI bus ID of the vGPU instance, the Bus:Device.Function information.

• MDEV UUID: The UUID of the mediated device (mdev) corresponding to the vGPU instance. N/A indicates that the vGPU instance is not mapped to an mdev.

• Guest Driver Version: The version of the driver inside the virtual machine.

• GPU Instance ID: The MIG GPU instance ID corresponding to the vGPU instance. N/A indicates that the vGPU instance is not mapped to a MIG GPU instance.

• ECC mode: The ECC mode of the vGPU instance.

• Memory Usage: Details about the memory allocated to and used by the vGPU instance.

  • Total: The total amount of memory allocated to the vGPU instance.

  • Used: The amount of memory used by the vGPU instance.

  • Free: The amount of free memory available to the vGPU instance.

12.6 Delete vGPU instance

Use the -di option to delete a vGPU instance by specifying its ID. You can use the -i option to restrict the operation to a specific device. By default, you cannot delete a vGPU instance that is currently used by a VM. To override this, use the -f option to force the deletion. For example, to delete the vGPU instance with ID 1 on PPU device 0, run ppu-smi vgpu -i 0 -di 1. The following is sample output:

root@0549cf16bb85:~# ppu-smi vgpu -i 0 -di 1
Successfully deleted vGPU instance 1 from PPU 0.

13. Drain PPU devices

PPU-SMI lets you set and query the drain state of a PPU device. This allows you to take a faulty PPU device out of service without disrupting ongoing tasks. When a PPU device is in a drain state:

• The PPU device continues to process its current tasks.

• The PPU device will not accept new compute tasks.

• The PPU device does not appear in tools such as PPU-SMI and PPUDBG.

For example, run ppu-smi drain -p 0001:AA:00.0 -m 1 to set the PPU device with the PCI bus ID 0001:AA:00.0 to the draining state:

root@dfc623e46a90:~# ppu-smi drain -p 0001:AA:00.0 -m 1
Successfully set PPU 0001:AA:00.0 drain state to: draining.

Run ppu-smi drain -h to view help for the drain feature. You must specify the target PPU device by providing its PCI bus ID with the -p option:

root@dfc623e46a90:~# ppu-smi drain -h
drain -- Displays/modifies PPU drain states for power idling.

ppu-smi drain [OPTION1 [ARG1]] [OPTION2 [ARG2]] ...
    -h,   --help
        Display help information
    -p,   --pciid=
        PPU PCI ID in the format XXXX:YY:Z.a
            XXXX = domain
            YY   = bus
            Z    = device
            a    = function
    
    [any one of]

    -m,   --modify=
        Modify the drain state of a PPU specified by -p.
            0 = not draining
            1 = draining
    -q,   --query
        Query the drain state of a PPU specified by -p.

13.1 Set the drain state

Use the -m option to set or clear the drain state of a PPU device. You must specify the target PPU device by its PCI bus ID using the -p option. For example, run ppu-smi drain -p 0001:AA:00.0 -m 1 to drain a PPU device. The command produces the following output:

root@dfc623e46a90:~# ppu-smi drain -p 0001:AA:00.0 -m 1
Successfully set PPU 0001:AA:00.0 drain state to: draining.

13.2 Query the drain state

Use the -q option to query the drain state of a PPU device. You must specify the target PPU device by its PCI bus ID using the -p option. For example, to query the drain state, run ppu-smi drain -p 0001:AA:00.0 -q. The command returns the following output:

root@dfc623e46a90:~# ppu-smi drain -p 0001:AA:00.0 -q
The current drain state of PPU 0001:AA:00.0 is: draining.

14. GPU Performance Monitoring (GPM) management

PPU-SMI allows you to set and query the enabled status of Performance Monitoring (GPM). When the Performance Monitoring (GPM) feature is enabled, it uses the PPU device performance counter to collect data. As a result, other applications that require the PPU device performance counter cannot run. You can use PPU-SMI to query the enabled status of the Performance Monitoring (GPM) feature, pause the Performance Monitoring (GPM) feature to allow other applications to collect performance counter data, and then resume the Performance Monitoring (GPM) feature after they are finished.

For example, run the ppu-smi gpm --get-stream-state command to check whether the Graphics Performance Monitor (GPM) feature of the PPU device is suspended:

root@dfc623e46a90:~# ppu-smi gpm --get-stream-state
PPU 0 GPM stream state: Enabled.
PPU 1 GPM stream state: Disabled.

You can run ppu-smi gpm -h to view the help information for the drain feature. Use the -i option to specify one or more PPU devices, separating multiple device IDs with a comma (,).

root@dfc623e46a90:~# ppu-smi gpm -h
gpm -- GPU Performance Monitoring management.

ppu-smi gpm [OPTION1 [ARG1]] [OPTION2 [ARG2]] ...
    -h,   --help
        Display help information
    -i,   --id=
        Enumeration index, Serial number, PCI bus ID or UUID. 
        Provide comma separated values for more than one device.

    [any one of]

    -s,   --set-stream-state=
        Set GPU Performance Monitoring Stream State:
        0/DISABLED, 1/ENABLED
    -g,   --get-stream-state
        Get GPU Performance Monitoring Stream State
    --get-sample-state
        Get GPU Performance Monitoring Sample State

14.1 Set GPM stream state

You can use the -s option to enable or disable the status output of the Performance Monitoring (GPM) feature. When the GPM feature is disabled:

• The performance result for an application that subscribes to data from Global Performance Monitoring (GPM) is N/A.

• The Performance Monitoring (GPM) feature no longer uses the performance counters of PPU devices.

For example, run the ppu-smi gpm -s DISABLED -i 0 command to pause the PPU 0 device's Performance Monitoring (GPM) output:

root@dfc623e46a90:~# ppu-smi gpm -s DISABLED -i 0
Set GPM stream state to DISABLED for PPU 00000000:5E:00.0.

14.2 Query GPM stream state

Use the -g option to query the output status of the Performance Monitoring (GPM) feature, and the -i option to specify the devices for the query. For example, run ppu-smi gpm -g -i 0 to query the GPM output status of PPU 0:

root@dfc623e46a90:~# ppu-smi gpm -g -i 0
PPU 0 GPM stream state: Disabled.

14.3 Query GPM sample state

You can use the --get-sample-state option to query the sample state of the Performance Monitoring (GPM) feature. The sample state indicates whether any application is using GPM to collect performance metrics. For example, run ppu-smi gpm --get-sample-state to query the sample state of the GPM feature for all PPU devices:

root@dfc623e46a90:~# ppu-smi gpm --get-sample-state
PPU 0 GPM sample state: Enabled.
PPU 1 GPM sample state: Disabled.

Known issues

• SDK v1.00: The reported application clock frequency can be lower than the actual value.

• SDK v1.00: Queries using the mig subcommand may fail if you do not specify an instance ID.

• SDK v1.4.0: The PPU process list can contain incorrect process information.

• The ppu-smi -q -d UTILIZATION command does not currently support sampling-related information.

• The stats subcommand does not currently support sampling-related information.

• The ppu-smi icn -gt d command cannot currently query ICN link data throughput.

• The ppu-smi icn -gt r command underreports ICN link data volume.

• Querying percentage statistics for frequency throttling due to temperature or power limits is not currently supported.

• Querying PCIe replay and error counters is not currently supported.