DCGM monitoring (v1.7)
1. Overview
PPU DCGM is a suite of tools for managing and monitoring PPUs in clusters and data centers, based on the open-source DCGM. It includes the following components:
dcgmi command-line tool
nv-hostengine command-line tool
dcgm shared library
dcgm-exporter tool
2. Download and installation
The PPU DCGM tool is released as a separate package. Click the download link to download it from the PPU artifactory page.
You need an account and password to download the package. Contact your account manager (PDSA) to obtain them.
The package includes:
RPM and DEB packages for PPU DCGM.
hgdcgm-exporter(directory).
Installation prerequisites:
PPU driver
PPU SDK
root permission (some features may be unavailable without it).
2.1 Stop and remove previous versions of PPU DCGM
The required command depends on whether you have root permission:
# Run with root permission to terminate nv-hostengine
sudo nv-hostengine -t
# To run without root permission, ensure the path to nvhostengine.pid is the same as the one used when starting nv-hostengine.
nv-hostengine -t --pid <your path>/nvhostengine.pid
# For the DEB package (Ubuntu)
sudo dpkg -r datacenter-gpu-manager
# For the RPM package (CentOS)
sudo rpm -e datacenter-gpu-manager2.2 Install the PPU DCGM package
# Since version 3.0.8, DCGM is provided as a single package and is no longer split into separate PCIe and OAM packages.
# The following example uses version 3.0.9:
# For Debian-based distributions (Ubuntu)
sudo dpkg -i --force-overwrite datacenter-gpu-manager_3.0.9_amd64.deb
# For RPM-based distributions (CentOS)
sudo rpm --force -ivh --nodeps datacenter-gpu-manager-3.0.9-1-x86_64.rpm2.3 Install dependencies
The PPU DCGM runtime environment requires the following software:
On APT-based Linux distributions, such as Ubuntu or Debian, run the following commands to install
pciutils:sudo apt-get update sudo apt-get install pciutilsOn YUM-based Linux distributions, such as CentOS or Red Hat, run the following commands to install
pciutils:sudo yum makecache sudo yum install pciutils
2.4 Start the PPU DCGM background service
The required command depends on whether you have root permission:
# Run with root permission to start the PPU DCGM background service.
nv-hostengine
# To run without root permission, specify the path for nvhostengine.pid.
# Command: nv-hostengine --pid <your path>/nvhostengine.pid
# Example:
nv-hostengine --pid /work/test/nvhostengine.pidYou can run dcgmi discovery -l to verify that PPU DCGM is running:
root@be1816e3958c:~# dcgmi discovery -l
2 GPUs found.
+--------+----------------------------------------------------------------------+
| GPU ID | Device Information |
+--------+----------------------------------------------------------------------+
| 0 | Name: PPU |
| | PCI Bus ID: 00000000:10:00.0 |
| | Device UUID: GPU-019ea108-c110-0828-0000-00002062b161 |
+--------+----------------------------------------------------------------------+
| 1 | Name: PPU |
| | PCI Bus ID: 00000000:11:00.0 |
| | Device UUID: GPU-019ea108-c120-040c-0000-0000c007e820 |
+--------+----------------------------------------------------------------------+
...3. Dcgmi CLI
dcgmi is a command-line tool for interactive queries. It communicates with nv-hostengine to collect and display data.
Before using the dcgmi tool, start the nv-hostengine service. Note: Running without root permission will disable some features.
# Start the DCGM background service with root privilege.
nv-hostengine
# To run with non-root privilege, specify the path to nvhostengine.pid.
nv-hostengine --pid <your path>/nvhostengine.pidTo view help for dcgmi subcommands, run dcgmi -h:
root@be1816e3958c:~# dcgmi -h
Usage: dcgmi
dcgmi subsystem
dcgmi -v
Flags:
-v vv Get DCGMI version information
subsystem The subsystem to access.
Subsystems available:
topo GPU topology [dcgmi topo -h for more info]
stats process statistics [dcgmi stats -h for more info]
diag system validation/diagnostic [dcgmi diag –h for more info]
policy policy management [dcgmi policy –h for more info]
health health monitoring [dcgmi health –h for more info]
config configuration management [dcgmi config –h for more info]
group GPU group management [dcgmi group –h for more info]
fieldgroup field group management [dcgmi fieldgroup –h for more info]
discovery Discovers GPUs on the system [dcgmi discovery –h for more info]
nvlink Displays NvLink link statuses and error counts [dcgmi nvlink –h for more info]
dmon Monitors GPU statistics [dcgmi dmon –h for more info]
modules Controls and lists DCGM modules
profile Controls and lists DCGM profiling metrics
-- ignore_rest Ignores subsequent labeled arguments.
--version Displays version information and exits.
-h --help Displays usage information and exits.
...3.1 View the device list (discovery)
Use the discovery subcommand to view the PPU device list and their status information.
To view help information, run dcgmi discovery -h:
root@be1816e3958c:~# dcgmi discovery -h
discovery -- Used to discover and identify GPUs and their attributes.
Usage: dcgmi discovery
dcgmi discovery --host <IP/FQDN> -l
dcgmi discovery --host <IP/FQDN> -i <flags> -g <groupId> -v
dcgmi discovery -c
Flags:
-g --group groupId The group ID to query.
--host IP/FQDN Connects to specified IP or fully-qualified domain
name. To connect to a host engine that was
started with -d (unix socket), prefix the unix
socket filename with 'unix://'. [default =
localhost]
-l --list List all GPUs discovered on the host.
-i --info flags Specify which information to return. [default =
atp]
a - device info
p - power limits
t - thermal limits
c - clocks
-c --compute-hierarchy List all of the gpu instances and compute
instances
...3.1.1 View the device list
Run dcgmi discovery -l to view the PPU device list. The output displays information such as the device name, PCI bus ID, and UUID:
root@be1816e3958c:~# dcgmi discovery -l
2 GPUs found.
+--------+----------------------------------------------------------------------+
| GPU ID | Device Information |
+--------+----------------------------------------------------------------------+
| 0 | Name: PPU |
| | PCI Bus ID: 00000000:10:00.0 |
| | Device UUID: GPU-019ea108-c110-0828-0000-00002062b161 |
+--------+----------------------------------------------------------------------+
| 1 | Name: PPU |
| | PCI Bus ID: 00000000:11:00.0 |
| | Device UUID: GPU-019ea108-c120-040c-0000-0000c007e820 |
+--------+----------------------------------------------------------------------+
...3.1.2 View status information
You can use the -i option to view the status information of a device, such as its PCI Bus ID. The -i parameter specifies the information to be displayed:
| Description |
a | device information |
p | power limit |
t | thermal limit |
c | clock information |
For example, run dcgmi discovery -i p to view device power limits. By default, if you do not specify a GPU group with the -g flag, the command displays aggregated status information for all PPUs:
root@be1816e3958c:~# dcgmi discovery -i p
+--------------------------+-------------------------------------------------+
| Group of 2 GPUs | Device Information |
+==========================+=================================================+
| Current Power Limit (W) | 300 |
| Default Power Limit (W) | 300 |
| Max Power Limit (W) | 300 |
| Min Power Limit (W) | 200 |
| Enforced Power Limit (W) | 300 |
+--------------------------+-------------------------------------------------+Add the -v flag to display status information for individual GPUs. For example, run dcgmi discovery -i a -g 0 -v to display basic device information for each GPU in GPU group 0:
root@be1816e3958c:~# dcgmi discovery -i a -g 0 -v
Device info:
+--------------------------+-------------------------------------------------+
| GPU ID: 0 | Device Information |
+==========================+=================================================+
| Device Name | PPU |
| PCI Bus ID | 00000000:10:00.0 |
| UUID | GPU-019ea108-c110-0828-0000-00002062b161 |
| Serial Number | TH7510H07 |
| InfoROM Version | Not Supported |
| VBIOS | 1.4.44 |
+--------------------------+-------------------------------------------------+
+--------------------------+-------------------------------------------------+
| GPU ID: 1 | Device Information |
+==========================+=================================================+
| Device Name | PPU |
| PCI Bus ID | 00000000:11:00.0 |
| UUID | GPU-019ea108-c120-040c-0000-0000c007e820 |
| Serial Number | TH7510H07 |
| InfoROM Version | Not Supported |
| VBIOS | 1.4.44 |
+--------------------------+-------------------------------------------------+3.1.3 View MIG instance information
Use the -c flag to list existing GPU instances and compute instances. For example, running dcgmi discovery -c displays the hierarchy of MIG instances for all PPUs:
root@be1816e3958c:/work/setup# dcgmi discovery -c
+-------------------+--------------------------------------------------------------------+
| Instance Hierarchy |
+===================+====================================================================+
| GPU 1 | GPU GPU-019ea108-c120-040c-0000-0000c007e820 (EntityID: 1) |
| -> I 1/0 | GPU Instance (EntityID: 68) |
| -> CI 1/0/0 | Compute Instance (EntityID: 68) |
| -> I 1/1 | GPU Instance (EntityID: 69) |
| -> CI 1/1/0 | Compute Instance (EntityID: 69) |
+-------------------+--------------------------------------------------------------------+3.2 Monitor device parameters (dmon)
Use the dmon subcommand to view and monitor device parameters. You can query parameters from a specified field group for devices within a specified GPU group. Run dcgmi dmon -h for help:
root@be1816e3958c:~# dcgmi dmon -h
dmon -- Used to monitor GPUs and their stats.
Usage: dcgmi dmon
dcgmi dmon -i <gpuId> -g <groupId> -f <fieldGroupId> -e <fieldId> -d
<delay> -c <count> -l
Flags:
--host IP/FQDN Connects to specified IP or fully-qualified domain
name. To connect to a host engine that was
started with -d (unix socket), prefix the unix
socket filename with 'unix://'. [default =
localhost]
-f --field-group-id fieldGroupId The field group to query on the specified
host.
-e --field-id fieldId Field identifier to view/inject.
-l --list List to look up the long names, short names and
field ids.
-h --help Displays usage information and exits.
-i --gpu-id gpuId The comma separated list of GPU/GPU-I/GPU-CI IDs
to run the dmon on. Default is -1 which runs for
all supported GPU. Run dcgmi discovery -c to
check list of available GPU entities
-g --group-id groupId The group to query on the specified host.
-d --delay delay In milliseconds. Integer representing how often
to query results from DCGM and print them for all
of the entities. [default = 1000 msec, Minimum
value = 1 msec.]
-c --count count Integer representing How many times to loop
before exiting. [default- runs forever.]
...3.2.1 List supported parameters
Use the -l option to list supported parameters. For example, run dcgmi dmon -l. The output lists the long name, short name, and corresponding field id for each parameter:
root@be1816e3958c:~# dcgmi dmon -l
________________________________________________________________________________________________________________________
Long Name Short Name Field ID
________________________________________________________________________________________________________________________
driver_version DRVER 1
nvml_version NVVER 2
process_name PRNAM 3
device_count DVCNT 4
cuda_driver_version CDVER 5
name DVNAM 50
brand DVBRN 51
nvml_index NVIDX 52
serial_number SRNUM 53
uuid UUID# 54
minor_number MNNUM 55
oem_inforom_version OEMVR 56
pci_busid PCBID 57
...3.2.2 Query device parameters
Use the -e option to query specific parameters by their field ids. For example, run dcgmi dmon -e 140,150 to query the memory and core temperature of all GPUs:
root@be1816e3958c:~# dcgmi dmon -e 140,150
#Entity MMTMP TMPTR
ID C C
GPU 1 29 29
GPU 0 30 28
GPU 1 30 29
GPU 0 30 27
...Use the -f option to specify a field group and query all parameters in that group. For example, run dcgmi dmon -f 5 to query all parameters in field group 5. Use the fieldgroup subcommand to manage field groups:
# Create a field group
root@be1816e3958c:~# dcgmi fieldgroup -c test -f 140,150
Successfully created field group "test" with a field group ID of 5
# Query the field group
root@be1816e3958c:~# dcgmi dmon -f 5
#Entity MMTMP TMPTR
ID C C
GPU 1 30 29
GPU 0 30 28
GPU 1 30 29
GPU 0 30 28
...Note: For a list of supported field ids, see Field id support status.
3.2.3 Other control parameters
By default, dmon periodically prints device parameters. Press Ctrl + C to stop the query. Use the -d option to specify the query interval and the -c option to specify the count. For example, dcgmi dmon -e 140,150 -c 1 queries the data only once.
root@be1816e3958c:~# dcgmi dmon -e 140,150 -c 1
#Entity MMTMP TMPTR
ID C C
GPU 1 29 29
GPU 0 30 283.3 Modify device configuration
Use the config subcommand to view and modify the device configuration, including parameters such as application clock frequency and ECC mode. Run dcgmi config -h to view the help information:
root@be1816e3958c:~# dcgmi config -h
config -- Used to configure settings for groups of GPUs.
Usage: dcgmi config
dcgmi config --host <IP/FQDN> -g <groupId> --enforce
dcgmi config --host <IP/FQDN> -g <groupId> --get -v -j
dcgmi config --host <IP/FQDN> -g <groupId> --set -e <0/1> -s <0/1> -a
<mem,proc> -P <limit> -c <mode>
Flags:
-g --group groupId The GPU group to query on the specified host.
--host IP/FQDN Connects to specified IP or fully-qualified domain
name. To connect to a host engine that was
started with -d (unix socket), prefix the unix
socket filename with 'unix://'. [default = localhost]
--set Set configuration.
--get Get configuration. Displays the Target and the
Current Configuration.
------
1.Sync Boost - Current and Target Sync Boost State
2.SM Application Clock - Current and Target SM application clock values
3.Memory Application Clock - Current and Target Memory application clock values
4.ECC Mode - Current and Target ECC Mode
5 Power Limit - Current and Target power limits
6.Compute Mode - Current and Target compute mode
--enforce Enforce configuration.
-h --help Displays usage information and exits.
-v --verbose Display policy information per GPU.
-j --json Print the output in a json format
-e --eccmode 0/1 Configure Ecc mode. (1 to Enable, 0 to Disable)
-s --syncboost 0/1 Configure Syncboost. (1 to Enable, 0 to Disable)
-a --appclocks mem,proc Configure Application Clocks. Must use memory,proc clocks (csv) format(MHz).
-P --powerlimit limit Configure Power Limit (Watts).
-c --compmode mode Configure Compute Mode. Can be any of the
following:
0 - Unrestricted
1 - Prohibited
2 - Exclusive Process
...Unsupported functionality:
The
-s --syncboostoption for setting the sync boost mode.
3.3.1 Query device configuration
Use the --get option to query the current and target configuration values. For example, run dcgmi config --get to view a summary of the configuration for all GPUs, where:
The
Targetcolumn shows the configuration value that issetbut notenforced.The
Currentcolumn shows the configuration value that isenforced.
root@be1816e3958c:~# dcgmi config --get
+------------------------------+------------------------------+------------------------------+
| DCGM_ALL_SUPPORTED_GPUS |
| Group of 2 GPUs |
+==============================+==============================+==============================+
| Field | Target | Current |
+------------------------------+------------------------------+------------------------------+
| Compute Mode | Not Specified | Unrestricted |
| ECC Mode | Enabled | Disabled |
| Sync Boost | Not Specified | Not Supported |
| Memory Application Clock | Not Specified | 1800 |
| SM Application Clock | Not Specified | 1200 |
| Power Limit | Not Specified | 300 |
+------------------------------+------------------------------+------------------------------+
...Use the -v option to view the configuration status of each GPU:
root@be1816e3958c:~# dcgmi config --get -v
+------------------------------+------------------------------+------------------------------+
| GPU ID: 0 |
| GPU |
+==============================+==============================+==============================+
| Field | Target | Current |
+------------------------------+------------------------------+------------------------------+
| Compute Mode | Not Specified | Unrestricted |
| ECC Mode | Enabled | Disabled |
| Sync Boost | Not Specified | Not Supported |
| Memory Application Clock | Not Specified | 1800 |
| SM Application Clock | Not Specified | 1200 |
| Power Limit | Not Specified | 300 |
+------------------------------+------------------------------+------------------------------+
+------------------------------+------------------------------+------------------------------+
| GPU ID: 1 |
| GPU |
+==============================+==============================+==============================+
| Field | Target | Current |
+------------------------------+------------------------------+------------------------------+
| Compute Mode | Not Specified | Unrestricted |
| ECC Mode | Enabled | Disabled |
| Sync Boost | Not Specified | Not Supported |
| Memory Application Clock | Not Specified | 1800 |
| SM Application Clock | Not Specified | 1200 |
| Power Limit | Not Specified | 300 |
+------------------------------+------------------------------+------------------------------+3.3.2 Modify device configuration
Use the --set option with other options, such as -e to modify the ECC mode. For example, to enable ECC mode, run dcgmi config --set -e 1:
root@be1816e3958c:~# dcgmi config --set -e 1
Configuration successfully set.DCGM supports modifying the following device configuration options:
Parameter | Description |
-e --eccmode | Modify ECC mode |
-a --appclocks | Modify application clock frequency |
-P --powerlimit | Modify power limit |
-c --compmode | Modify compute mode |
-s --syncboost | Not supported. This option is deprecated. |
The modified configuration is cached by DCGM. You can use the --enforce option to apply the cached configuration to the corresponding devices. For example, after a device is reset, you can use the --enforce option to restore the previous configuration by running dcgmi config --enforce:
root@be1816e3958c:~# dcgmi config --enforce
Configuration successfully enforced.3.4 Diagnosing GPU device status (diag)
Thediag subcommand runs a diagnostic to check the status of GPU devices. DCGM supports multiple diagnostic test levels. To view help information, rundcgmi diag -h:
root@be1816e3958c:~# dcgmi diag -h
diag -- Used to run diagnostics on the system.
Usage: dcgmi diag
dcgmi diag --host <IP/FQDN> -g <groupId> -r <diag> -p
<test_name.variable_name=variable_value> -c
</full/path/to/config/file> -f <fakeGpuList> -i <gpuList> -v
--statsonfail --debugLogFile <debug file> --statspath <plugin
statistics path> -j --throttle-mask <> --fail-early
--check-interval <failure check interval> --iterations <iterations>
Flags:
...
-r --run diag Run a diagnostic. (Note: higher numbered tests
include all beneath.)
1 - Quick (System Validation ~ seconds)
2 - Medium (Extended System Validation ~ 2 minutes)
3 - Long (System HW Diagnostics ~ 15 minutes)
4 - Extended (Longer-running System HW Diagnostics)
Specific tests to run may be specified by name,
and multiple tests may be specified as a comma
separated list. For example, the command:
dcgmi diag -r "sm stress,diagnostic"
would run the SM Stress and Diagnostic tests together.
-p --parameters test_name.variable_name=variable_value Test parameters to set for this run.
-c --configfile /full/path/to/config/file Path to the configuration file.
-i --gpuList gpuList A comma-separated list of the gpus on which the
diagnostic should run. Cannot be used with -g.
-v --verbose Show information and warnings for each test.
...
--throttle-mask Specify which throttling reasons should be
ignored. You can provide a comma separated list
of reasons. For example, specifying 'HW_SLOWDOWN
,SW_THERMAL' would ignore the HW_SLOWDOWN and
SW_THERMAL throttling reasons. Alternatively, you
can specify the integer value of the ignore
bitmask. For the bitmask, multiple reasons may be
specified by the sum of their bit masks.
...
--fail-early Enable early failure checks for the Targeted Power
, Targeted Stress, SM Stress, and Diagnostic
tests. When enabled, these tests check for a
failure once every 5 seconds (can be modified by
the --check-interval parameter) while the test is
running instead of a single check performed after
the test is complete. Disabled by default.
--check-intervalfailure check interval Specify the interval (in seconds)
at which the early failure checks should occur
for the Targeted Power, Targeted Stress, SM
Stress, and Diagnostic tests when early failure
checks are enabled. Default is once every 5
seconds. Interval must be between 1 and 300
--iterations iterations Specify a number of iterations of the diagnostic
to run consecutively. (Must be greater than 0.)
...Unsupported Functionality:
--train: Automatically tunes parameters based on the results.
Thediag subcommand depends on components from the GPU SDK. Before using thediag subcommand, ensure the GPU SDK is installed correctly and that its toolchain versions, such as GCC, meet the GPU SDK's runtime requirements.
3.4.1 Running a diagnostic
The -r option runs the diagnostic test program. You can use the -r option to specify the set of diagnostic test programs to run. The supported sets of programs are as follows:
Test name | Description | Level 1: Quick | Level 2: < 2 min | Level 3: ~ 15 min | Level 4: 1-2 hours |
software | Basic environment configuration Feature support status | Yes | Yes | Yes | Yes |
pcie | PCIe test ICN link test | Yes | Yes | Yes | |
memory | Memory allocation test Memory read/write test | Yes | Yes | Yes | |
memory_bandwidth | Memory bandwidth test | Yes | Yes | Yes | |
diagnostic | Compute capability test | Yes | Yes | ||
sm_stress | Compute capability stress test | Yes | Yes | ||
targeted_stress | Controlled compute capability test | Yes | Yes | ||
targeted_power | Controlled power test | Yes | Yes | ||
memtest | Memory stress test | Yes |
Thememory,memory_bandwidth, andmemtest diagnostic tests require the ECC feature to be enabled. Use the following commands to enable the ECC feature for all devices and then reset the GPU devices to apply this configuration:
dcgmi config --set -e 1
dcgmi config --enforce
Reset the GPU devices
For example, rundcgmi diag -r 1 to rundiagnostic test level 1:
root@be1816e3958c:/work/deploy# dcgmi diag -r 1
Successfully ran diagnostic for group.
+---------------------------+------------------------------------------------+
| Diagnostic | Result |
+===========================+================================================+
|----- Deployment --------+------------------------------------------------|
| Denylist | Pass |
| NVML Library | Pass |
| CUDA Main Library | Pass |
| Permissions and OS Blocks | Pass |
| Persistence Mode | Skip |
| Environment Variables | Pass |
| Page Retirement/Row Remap | Pass |
| Graphics Processes | N/A |
| Inforom | Skip |
+---------------------------+------------------------------------------------+You can run a specific test program by passing the diagnostic_test_name with the -r option. To pass parameters to the program, use the -p option in the test_name.variable_name=variable_value format. For example, to control the runtime duration of sm_stress, execute dcgmi diag -r "sm_stress" -p sm_stress.test_duration=5:
root@be1816e3958c:/work/deploy# dcgmi diag -r "sm_stress" -p sm_stress.test_duration=5
Successfully ran diagnostic for group.
+---------------------------+------------------------------------------------+
| Diagnostic | Result |
+===========================+================================================+
|----- Deployment --------+------------------------------------------------|
| Denylist | Pass |
| NVML Library | Pass |
| CUDA Main Library | Pass |
| Permissions and OS Blocks | Pass |
| Persistence Mode | Pass |
| Environment Variables | Pass |
| Page Retirement/Row Remap | Pass |
| Graphics Processes | Pass |
| Inforom | Skip |
+----- Integration -------+------------------------------------------------+
+----- Hardware ----------+------------------------------------------------+
+----- Stress ------------+------------------------------------------------+
| SM Stress | Pass - All |
+---------------------------+------------------------------------------------+Do not use the diagnostic tests from thediag subcommand as criteria for a Return Merchandise Authorization (RMA). To check for product quality issues, use tools such asField Diag andBug Report.
3.4.2 Other control options
Use the
-goption to specify the GPU group for the diagnostic, or use the-ioption to specify GPU indices.Use the
--throttle-maskoption to ignore specific throttling reasons, such as thermal protection, that might otherwise cause a test to fail.Use the
--fail-earlyand--check-intervaloptions to periodically check for failure conditions during the test, rather than only at the end. This allows the diagnostic to exit early if a failure is detected.
Thediag subcommand does not support running diagnostics on a mix of GPU device models. If your system contains multiple GPU models, use the-i or-g option to select devices of the same model.
3.5 Topology (topo)
The topo subcommand shows the topology between GPUs. For help, run dcgmi topo -h:
root@be1816e3958c:~# dcgmi topo -h
topo -- Used to find the topology of GPUs on the system.
Usage: dcgmi topo
dcgmi topo --host <IP/FQDN> -g <groupId> -j
dcgmi topo --host <IP/FQDN> --gpuid <gpuId> -j
Flags:
-g --group groupId The group ID to query.
--host IP/FQDN Connects to specified IP or fully-qualified domain
name. To connect to a host engine that was
started with -d (unix socket), prefix the unix
socket filename with 'unix://'. [default = localhost]
-h --help Displays usage information and exits.
--gpuid gpuId The GPU ID to query.
-j --json Print the output in a json format
...3.5.1 Device topology
The --gpuid option shows the topology between a specific GPU and other devices, while the -g option shows a summary for an entire GPU group. For example, run dcgmi topo --gpuid 0 to see how GPU 0 connects to other devices:
root@be1816e3958c:~# dcgmi topo --gpuid 0
+-------------------+------------------------------------------------------------------------------+
| Topology Information |
| GPU ID: 0 |
+===================+==============================================================================+
| CPU Core Affinity | 0 - 47, 96 - 143 |
| To GPU 1 | Connected via a single PCIe switch |
| | Connected via one NVLINK (Link: 5) |
| To GPU 2 | Connected via a CPU-level link |
| | Connected via one NVLINK (Link: 3) |
| To GPU 3 | Connected via a CPU-level link |
| | Connected via one NVLINK (Link: 2) |
| To GPU 4 | Connected via a CPU-level link |
| To GPU 5 | Connected via a CPU-level link |
| To GPU 6 | Connected via a CPU-level link |
| To GPU 7 | Connected via a CPU-level link |
| | Connected via one NVLINK (Link: 0) |
+-------------------+------------------------------------------------------------------------------+3.6 ICN link status (nvlink)
Use the nvlink subcommand to view the ICN link status. Run dcgmi nvlink -h to view help:
root@be1816e3958c:/work/deploy# dcgmi nvlink -h
nvlink -- Used to get NvLink link status or error counts for GPUs and
NvSwitches in the system
NVLINK Error description
=========================
CRC FLIT Error => Data link receive flow control digit CRC error.
CRC Data Error => Data link receive data CRC error.
Replay Error => Data link transmit replay error.
Recovery Error => Data link transmit recovery error.
Usage: dcgmi nvlink
dcgmi nvlink --host <IP/FQDN> -g <gpuId> -e -j
dcgmi nvlink --host <IP/FQDN> -s
Flags:
--host IP/FQDN Connects to specified IP or fully-qualified domain
name. To connect to a host engine that was
started with -d (unix socket), prefix the unix
socket filename with 'unix://'. [default = localhost]
-e --errors Print NvLink errors for a given gpuId (-g).
-s --link-status Print NvLink link status for all GPUs and
NvSwitches in the system.
-h --help Displays usage information and exits.
-g --gpuid gpuId The GPU ID to query. Required for -e
-j --json Print the output in a json format
...Unsupported features:
Viewing the ICN error counter with the
-e --errorsflag.
3.6.1 Link status
Use the -s flag to query the ICN link status. The following example shows the output from dcgmi nvlink -s, where U indicates the link is Up and D indicates the link is Down:
root@b5ab3167ed51:~# dcgmi nvlink -s
+----------------------+
| NvLink Link Status |
+----------------------+
GPUs:
gpuId 0:
U D U U U U D _ _ _ _ _ _ _ _ _ _ _
gpuId 1:
U D U U U U D _ _ _ _ _ _ _ _ _ _ _
gpuId 2:
U D U U U U D _ _ _ _ _ _ _ _ _ _ _
gpuId 3:
U D U U U U D _ _ _ _ _ _ _ _ _ _ _
gpuId 4:
U D U U U U D _ _ _ _ _ _ _ _ _ _ _
gpuId 5:
U D U U U U D _ _ _ _ _ _ _ _ _ _ _
gpuId 6:
U D U U U U D _ _ _ _ _ _ _ _ _ _ _
gpuId 7:
U D U U U U D _ _ _ _ _ _ _ _ _ _ _
NvSwitches:
No NvSwitches found.
Key: Up=U, Down=D, Disabled=X, Not Supported=_3.7 Manage device policies
Use the policy subcommand to set and view PPU management policies, such as automatically resetting a device when an ECC error or overheating occurs.
Run dcgmi policy -h to view the help information:
root@be1816e3958c:~# dcgmi policy -h
policy -- Used to control policies for groups of GPUs. Policies control actions
which are triggered by specific events.
Usage: dcgmi policy
dcgmi policy --host <IP/FQDN> -g <groupId> --get -j
dcgmi policy --host <IP/FQDN> -g <groupId> --reg
dcgmi policy --host <IP/FQDN> -g <groupId> --set <actn,val> -M <max> -T
<max> -P <max> -e -p -n -x
Flags:
-g --group groupId The GPU group to query on the specified host.
...
--get Get the current violation policy.
--reg Register this process for policy updates. This
process will sit in an infinite loop waiting for
updates from the policy manager.
--set actn,val Set the current violation policy. Use csv action ,validation (ie. 1,2)
-----
Action to take when any of the violations
specified occur.
0 - None
1 - GPU Reset
-----
Validation to take after the violation action has been performed.
0 - None
1 - System Validation (short)
2 - System Validation (medium)
3 - System Validation (long)
--clear Clear the current violation policy.
-h --help Displays usage information and exits.
-v --verbose Display policy information per GPU.
-M --maxpages max Specify the maximum number of retired pages that
will trigger a violation.
-T --maxtemp max Specify the maximum temperature a group's GPUs can
reach before triggering a violation.
-P --maxpower max Specify the maximum power a group's GPUs can reach
before triggering a violation.
-e --eccerrors Add ECC double bit errors to the policy
conditions.
-p --pcierrors Add PCIe replay errors to the policy conditions.
-n --nvlinkerrors Add NVLink errors to the policy conditions.
-x --xiderrors Add XID errors to the policy conditions.
...Unsupported functionality:
The
-p --pcierrorsoption to set the policy threshold for PCIe errors.The
-n --nvlinkerrorsoption to set the policy threshold for NVLink errors.
3.7.1 View the current policy
Use the --get option to view the current policy configuration. The following example shows the output of running dcgmi policy --get:
root@be1816e3958c:~# dcgmi policy --get
Policy information
+-----------------------------+------------------------------------------------+
| Policy Information |
| DCGM_ALL_SUPPORTED_GPUS |
+=============================+================================================+
| Violation conditions | Double-bit ECC errors |
| | PCI errors and replays |
| | Max temperature threshold - 90 |
| Isolation mode | Manual |
| Action on violation | None |
| Validation after action | System Validation (Long) |
| Validation failure action | None |
+-----------------------------+------------------------------------------------+Violation conditions: Conditions that trigger a policy violation, such as an ECC error, a PCIe error, or exceeding the temperature threshold.
Action on violation: The action taken for a policy violation, such as resetting the PPU device.
Validation after action: The validation performed after the action.
3.7.2 Set and apply a policy
Setting and applying a policy is a two-step process:
Use the
--setoption with other options like-Mand-Tto set the policy, including conditions like a temperature threshold.Use the
--regoption to instruct the currentdcgmiprocess to start polling for policy violations.
The --set option specifies the action for a policy violation, such as resetting a PPU, and the validation to perform afterward. You must use the --set option with one or more condition options. The supported condition options are:
Condition option | Description |
-M --maxpages | The number of retired pages reaches the specified threshold. |
-T --maxtemp | The device temperature exceeds the specified threshold. |
-P --maxpower | The device power exceeds the specified threshold. |
-e --eccerrors | A double-bit ECC error occurs. |
-p --pcierrors | A PCIe error occurs (Unsupported). |
-x --xiderrors | An XID error occurs in the driver. |
For example, run dcgmi policy --set 0,3 -T 90 -p -e to set policies for temperature, PCIe errors, and ECC errors. Then, run dcgmi policy --reg to start monitoring for violations. If a violation occurs, the process performs the specified action, such as resetting the affected device:
root@be1816e3958c:~# dcgmi policy --set 0,3 -T 90 -p -e
Policy successfully set.
root@be1816e3958c:~# dcgmi policy --reg
Listening for violations.
...3.8 Device health
Use the health subcommand to monitor and view device health, such as checking for ECC errors and whether thermal or power thresholds have been exceeded.
Run dcgmi health -h to view help information:
root@be1816e3958c:~# dcgmi health -h
health -- Used to manage the health watches of a group. The health of the GPUs
in a group can then be monitored during a process.
Usage: dcgmi health
dcgmi health --host <IP/FQDN> -g <groupId> -c -j
dcgmi health --host <IP/FQDN> -g <groupId> -f -j
dcgmi health --host <IP/FQDN> -g <groupId> -s <flags> -j -m <seconds> -u <seconds>
Flags:
-g --group groupId The GPU group to query on the specified host.
...
-f --fetch Fetch the current watch status.
-s --set flags Set the watches to be monitored. [default = pm]
a - all watches
p - PCIe watches (*)
m - memory watches (*)
i - infoROM watches
t - thermal and power watches (*)
n - NVLink watches (*)
(*) watch requires 60 sec before first query
--clear Disable all watches being monitored.
-c --check Check to see if any errors or warnings have
occurred in the currently monitored watches.
...
-m --max-keep-age seconds How long DCGM should cache the samples in seconds.
-u --update-interval seconds How often DCGM should retrieve health from the driver in seconds.
...3.8.1 Enabled watches
Use the -f option to view the currently enabled watches. For example, running dcgmi health -f shows that the Memory and PCIe watches are enabled.
root@be1816e3958c:~# dcgmi health -f
Health monitor systems report
+-----------------+--------------------------------------------------------------------+
| PCIe | On |
| NVLink | On |
| Memory | On |
| SM | Off |
| InfoROM | On |
| Thermal | On |
| Power | On |
| Driver | Off |
| NVSwitch NF | Off |
| NVSwitch F | Off |
+-----------------+--------------------------------------------------------------------+3.8.2 Setting watches
Use the -s option to set the watches to monitor, and use the --clear option to disable all monitored watches.
For subscribed watches, use the -m option to set how long to cache data and the -u option to set how often to query device status. For example, run dcgmi health -s a -m 30 -u 1 to subscribe to all watches, cache data for 30 seconds, and query device status every second:
root@be1816e3958c:~# dcgmi health -s a -m 30 -u 1
Health monitor systems set successfully.3.8.3 Watch errors
Use the -c option to check for errors in subscribed watches. For example, run dcgmi health -c:
root@be1816e3958c:~# dcgmi health -c
+---------------------------+----------------------------------------------------------+
| Health Monitor Report |
+===========================+==========================================================+
| Overall Health | Failure |
| GPU | |
| -> 0 | Failure |
| -> Errors | |
| -> NVLink system | Failure |
| | GPU 0's NVLink link 0 is currently down Run a field |
| | diagnostic on the GPU. |
...3.9 Device groups
Use the group subcommand to manage groups of devices or modules. For example, you can add multiple PPU devices to a group and then specify that group in other dcgmi subcommands with the -g option. PPU DCGM supports multiple group levels, such as PPU-level, ICN link-level, and compute instance-level groups.
PPU DCGM includes a default device group with group ID 0, containing all supported PPU devices. If you omit the -g option, other dcgmi subcommands default to group 0 (all PPU devices).
Run dcgmi group -h to view help information:
root@be1816e3958c:~# dcgmi group -h
group -- Used to create and maintain groups of GPUs. Groups of GPUs can then be
uniformly controlled through other DCGMI subsystems.
Usage: dcgmi group
dcgmi group --host <IP/FQDN> -l -j
dcgmi group --host <IP/FQDN> -c <groupName> --default --defaultnvswitches
dcgmi group --host <IP/FQDN> -c <groupName> -a <entityId>
dcgmi group --host <IP/FQDN> -d <groupId>
dcgmi group --host <IP/FQDN> -g <groupId> -i -j
dcgmi group --host <IP/FQDN> -g <groupId> -a <entityId>
dcgmi group --host <IP/FQDN> -g <groupId> -r <entityId>
Flags:
...
-l --list List the groups that currently exist for a host.
-d --delete groupId Delete a group on the remote host.
-c --create groupName Create a group on the remote host.
-h --help Displays usage information and exits.
-i --info Display the information for the specified group ID.
-r --remove entityId Remove device(s) from group. (csv gpuIds, or
entityIds like gpu:0,nvswitch:994)
-a --add entityId Add device(s) to group. (csv gpuIds or entityIds
simlar to gpu:0, instance:1, compute_instance:2,
nvswitch:994)
--default Adds all available GPUs to the group being created.
--defaultnvswitches Adds all available NvSwitches to the group being created.3.9.1 View groups
Use the -l option to list existing groups. For example, run dcgmi group -l:
root@be1816e3958c:~# dcgmi group -l
+-------------------+----------------------------------------------------------+
| GROUPS |
| 2 groups found. |
+===================+==========================================================+
| Groups | |
| -> 0 | |
| -> Group ID | 0 |
| -> Group Name | DCGM_ALL_SUPPORTED_GPUS |
| -> Entities | GPU 0, GPU 1 |
| -> 1 | |
| -> Group ID | 1 |
| -> Group Name | DCGM_ALL_SUPPORTED_NVSWITCHES |
| -> Entities | None |
+-------------------+----------------------------------------------------------+Use the -i option to view the details of a group. Specify the group ID with the -g option. For example, run dcgmi group -g 0 -i:
root@be1816e3958c:~# dcgmi group -g 0 -i
+-------------------+----------------------------------------------------------+
| GROUP INFO |
+===================+==========================================================+
| 0 | |
| -> Group ID | 0 |
| -> Group Name | DCGM_ALL_SUPPORTED_GPUS |
| -> Entities | GPU 0, GPU 1 |
+-------------------+----------------------------------------------------------+3.9.2 Manage groups
Use the -c option to create a group and the -a option to add devices to it. For example, run dcgmi group -c test -a gpu:0,1 to create a group named test and add PPU 0 and PPU 1 to it. Use the --default or --defaultnvswitches option to add all available devices to the group.
root@be1816e3958c:~# dcgmi group -c test -a gpu:0,1
Successfully created group "test" with a group ID of 2
Add to group operation successful.Use the -r option to remove a device from a group. For example, run dcgmi group -g 2 -r 1 to remove PPU 1 from group 2.
root@be1816e3958c:~# dcgmi group -g 2 -r 1
Remove from group operation successful.Use the -d option to delete a group. For example, run dcgmi group -d 2 to delete group 2:
root@be1816e3958c:~# dcgmi group -d 2
Successfully removed group 23.10 Manage field groups (fieldgroup)
Use the fieldgroup subcommand to manage field groups, which are collections of field IDs. You can then specify a field group to query all parameters in the group at once. For example, in the dmon subcommand, use the -f option to specify a field group.
Run dcgmi fieldgroup -h to view its help:
root@be1816e3958c:~# dcgmi fieldgroup -h
fieldgroup -- Used to create and maintain groups of field IDs. Groups of field
IDs can then be uniformly controlled through other DCGMI subsystems.
Usage: dcgmi fieldgroup
dcgmi fieldgroup --host <IP/FQDN> -l -j
dcgmi fieldgroup --host <IP/FQDN> -c <fieldGroupName> -f <fieldIds>
dcgmi fieldgroup --host <IP/FQDN> -i -g <fieldGroupId> -j
dcgmi fieldgroup --host <IP/FQDN> -d -g <fieldGroupId>
Flags:
...
-l --list List the field groups that currently exist for a host.
-i --info Display the information for the specified field group ID.
-d --delete Delete a field group on the remote host.
-c --create fieldGroupName Create a field group on the remote host.
-h --help Displays usage information and exits.
-g --fieldgroup fieldGroupId The field group to query on the specified host.
-f --fieldids fieldIds Comma-separated list of the field ids to add to a
field group when creating a new one.
...3.10.1 List field groups
Use the -l option to list field groups. For example, run dcgmi fieldgroup -l:
root@be1816e3958c:~# dcgmi fieldgroup -l
3 field groups found.
+-------------------+----------------------------------------------------------+
| FIELD GROUPS |
+===================+==========================================================+
| ID | 1 |
| Name | DCGM_INTERNAL_30SEC |
| Field IDs | 300 |
+-------------------+----------------------------------------------------------+
+-------------------+----------------------------------------------------------+
| FIELD GROUPS |
+===================+==========================================================+
| ID | 2 |
| Name | DCGM_INTERNAL_HOURLY |
| Field IDs | 501, 509, 510, 511, 512, 513 |
+-------------------+----------------------------------------------------------+
...Use the -i option to view detailed information for a field group, specifying the group with the -g option. For example, run dcgmi fieldgroup -g 2 -i:
root@be1816e3958c:~# dcgmi fieldgroup -g 2 -i
+-------------------+----------------------------------------------------------+
| FIELD GROUPS |
+===================+==========================================================+
| ID | 2 |
| Name | DCGM_INTERNAL_HOURLY |
| Field IDs | 501, 509, 510, 511, 512, 513 |
+-------------------+----------------------------------------------------------+3.10.2 Manage field groups
Use the -c option to create a field group and the -f option to specify its field IDs. For example, run dcgmi fieldgroup -c temperature -f 140,150 to create a field group named temperature that includes the field IDs for memory and PPU core temperature:
root@be1816e3958c:~# dcgmi fieldgroup -c temperature -f 140,150
Successfully created field group "temperature" with a field group ID of 4Use the -d option to delete a field group, specifying the group with the -g option. For example, run dcgmi fieldgroup -d -g 4:
root@be1816e3958c:~# dcgmi fieldgroup -d -g 4
Successfully removed field group 43.11 Profiling metrics (profile)
Use the profile subcommand to view and manage profiling metrics. Run dcgmi profile -h to view the help information:
root@457471bb3c5e:~# dcgmi profile -h
profile -- View available profiling metrics for GPUs
Usage: dcgmi profile
Flags:
--host IP/FQDN Connects to specified IP or fully-qualified domain
name. To connect to a host engine that was
started with -d (unix socket), prefix the unix
socket filename with 'unix://'. [default =
localhost]
-l --list List available profiling metrics for a GPU or
group of GPUs
--pause Pause DCGM profiling in order to run NVIDIA
developer tools like nvprof, nsight compute, or
nsight systems.
--resume Resume DCGM profiling that was paused previously
with --pause.
-h --help Displays usage information and exits.
-j --json Print the output in a json format
-i --gpu-id gpuId The comma seperated list of GPU IDs to query.
Default is supported GPUs on the system. Run
dcgmi discovery -l to check list of GPUs
available
-g --group-id groupId The group of GPUs to query on the specified host.
...3.11.1 List available profiling metrics
Use the dcgmi profile -l option to view available profiling metrics. You can use the -i or -g options to constrain the query to specific GPUs. For example:
root@457471bb3c5e:~# dcgmi profile -l
+----------------+----------+------------------------------------------------------+
| Group.Subgroup | Field ID | Field Tag |
+----------------+----------+------------------------------------------------------+
| A.0 | 1002 | sm_active |
| A.0 | 1003 | sm_occupancy |
...To query these profiling metrics, specify their Field IDs with the dcgmi dmon -e option. For example:
dcgmi dmon -e 1002,10033.11.2 Pause and resume profiling
Subscribing to profiling metrics (Field ID > 1000) consumes GPU performance data collection resources. This prevents other DCGM-based tools, such as dcgm-exporter and the Asight tool, from running. Use dcgmi profile --pause to pause profiling. Afterwards, use dcgmi profile --resume to resume it. While paused, profiling metrics appear as N/A. For example:
# Pause profiling
dcgmi profile --pause
# Collect acu tracing data
acu -o test_report -f python test_linear.py
# Resume profiling
dcgmi profile --resume4. DCGM API support status
PPU DCGM provides the DCGM shared library libdcgm.so. The dcgmi command-line tool, the nv-hostengine background service, and hgdcgm-exporter are all built on this dynamic library. For details on the interfaces exposed by libdcgm.so, see the API documentation in dcgm_agent.h.
The support status for the DCGM APIs in libdcgm.so is as follows:
API name | Description | Supported |
dcgmInit | Initializes DCGM within the current process. | Yes |
dcgmStartEmbedded | Starts an embedded host engine agent within the current process. | Yes |
dcgmProfGetSupportedMetricGroups | Gets all profiling metric groups for a specified GPU group. | Yes |
dcgmGroupCreate | Creates an entity group to store one or more entity IDs and returns an opaque handle to it in pDcgmGrpId. | Yes |
dcgmFieldGroupCreate | Creates a group of fields and returns its handle in dcgmFieldGroupId. | Yes |
dcgmWatchFields | Instructs DCGM to start recording updates for a specified field collection. | Yes |
dcgmUpdateAllFields | Instructs the DCGM module to update all watched fields. | Yes |
dcgmEntityGetLatestValues | Gets the latest cached values for a field group for a specific entity. | Yes |
dcgmFieldGroupDestroy | Removes a field group created with dcgmFieldGroupCreate. | Yes |
DcgmFieldsTerm | Terminates the DcgmFields module. This function should be called once within your program. | Yes |
dcgmGetGpuInstanceHierarchy | Retrieves the hierarchy of GPUs, GPU Instances, and Compute Instances by populating a list of entities, each with a reference to its parent. | Yes |
dcgmGroupAddDevice | Adds the specified GPU ID to the group represented by groupId. | Yes |
dcgmGetAllSupportedDevices | Gets the identifiers for all DCGM-supported devices on the system. | Yes |
dcgmGetAllDevices | Gets the identifiers for all devices on the system. | Yes |
dcgmGetDeviceTopology | Gets the device topology for the specified GPU. | Yes |
dcgmGetDeviceAttributes | Gets the attributes for the specified GPU. | Yes |
dcgmHealthSet | Enables the DCGM health check system for the systems specified in dcgmHealthSystems_t. | Yes |
dcgmHealthCheck | Checks the configured health checks for any errors, failures, or warnings that have occurred since the last check. | Yes |
5. Field ID support
The PPU supports the following DCGM field IDs:
Parameter | Field ID | Description | Supported |
DCGM_FI_DRIVER_VERSION | 1 | The driver version. | Yes |
DCGM_FI_NVML_VERSION | 2 | The underlying NVML version. | Yes |
DCGM_FI_PROCESS_NAME | 3 | The process name. | Yes |
DCGM_FI_DEV_COUNT | 4 | Number of devices on the node. | Yes |
DCGM_FI_CUDA_DRIVER_VERSION | 5 | The CUDA driver version. | Yes |
DCGM_FI_DEV_NAME | 50 | Name of the GPU device. | Yes |
DCGM_FI_DEV_NVML_INDEX | 52 | NVML index of the GPU. | Yes |
DCGM_FI_DEV_SERIAL | 53 | Device serial number. | Yes |
DCGM_FI_DEV_UUID | 54 | UUID of the device. | Yes |
DCGM_FI_GPU_TOPOLOGY_PCI | 60 | Static PCI topology of all GPUs on the system. | Yes |
DCGM_FI_DEV_MIG_MODE | 67 | The current MIG mode. | Yes |
DCGM_FI_DEV_VBIOS_VERSION | 85 | VBIOS version of the device. | Yes |
DCGM_FI_DEV_MEM_AFFINITY_0 | 86 | Device memory node affinity, 0-63. | Yes |
DCGM_FI_DEV_MEM_AFFINITY_1 | 87 | Device memory node affinity, 64-127. | Yes |
DCGM_FI_DEV_MEM_AFFINITY_2 | 88 | Device memory node affinity, 128-191. | Yes |
DCGM_FI_DEV_MEM_AFFINITY_3 | 89 | Device memory node affinity, 192-255. | Yes |
DCGM_FI_DEV_GPM_SUPPORT | 94 | GPM support for the device. | Yes |
DCGM_FI_DEV_SM_CLOCK | 100 | SM clock frequency in MHz. | Yes |
DCGM_FI_DEV_MEM_CLOCK | 101 | Memory clock frequency in MHz. | Yes |
DCGM_FI_DEV_VIDEO_CLOCK | 102 | Video encoder/decoder clock in MHz. | Yes |
DCGM_FI_DEV_APP_SM_CLOCK | 110 | SM application clocks in MHz. | Yes |
DCGM_FI_DEV_APP_MEM_CLOCK | 111 | Memory application clocks in MHz. | Yes |
DCGM_FI_DEV_CLOCK_THROTTLE_REASONS | 112 | Current clock throttle reasons. | Yes |
DCGM_FI_DEV_MAX_SM_CLOCK | 113 | Maximum supported SM clock in MHz. | Yes |
DCGM_FI_DEV_MAX_MEM_CLOCK | 114 | Maximum supported memory clock in MHz. | Yes |
DCGM_FI_DEV_MAX_VIDEO_CLOCK | 115 | Maximum supported video encoder/decoder clock in MHz. | Yes |
DCGM_FI_DEV_MEMORY_TEMP | 140 | Memory temperature in °C. | Yes |
DCGM_FI_DEV_GPU_TEMP | 150 | GPU temperature in °C. | Yes |
DCGM_FI_DEV_MEM_MAX_OP_TEMP | 151 | Maximum operating temperature for the GPU memory in °C. | Yes |
DCGM_FI_DEV_GPU_MAX_OP_TEMP | 152 | Maximum operating temperature for the GPU in °C. | Yes |
DCGM_FI_DEV_POWER_USAGE | 155 | Power draw in W. | Yes |
DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION | 156 | Total energy consumption since boot in mJ. | No |
DCGM_FI_DEV_POWER_USAGE_INSTANT | 157 | Instantaneous power draw of the device in W. | No |
DCGM_FI_DEV_SLOWDOWN_TEMP | 158 | Slowdown temperature in °C. | Yes |
DCGM_FI_DEV_SHUTDOWN_TEMP | 159 | Shutdown temperature in °C. | Yes |
DCGM_FI_DEV_POWER_MGMT_LIMIT | 160 | Current power limit in W. | Yes |
DCGM_FI_DEV_POWER_MGMT_LIMIT_MIN | 161 | Minimum power management limit in W. | Yes |
DCGM_FI_DEV_POWER_MGMT_LIMIT_MAX | 162 | Maximum power management limit in W. | Yes |
DCGM_FI_DEV_POWER_MGMT_LIMIT_DEF | 163 | Default power management limit in W. | Yes |
DCGM_FI_DEV_ENFORCED_POWER_LIMIT | 164 | The effective power limit (in W) that the driver enforces after accounting for all limiters. | Yes |
DCGM_FI_DEV_PCIE_TX_THROUGHPUT | 200 | PCIe TX throughput in KB, as reported by NVML. | No |
DCGM_FI_DEV_PCIE_RX_THROUGHPUT | 201 | PCIe RX throughput in KB, as reported by NVML. | No |
DCGM_FI_DEV_PCIE_REPLAY_COUNTER | 202 | PCIe replay counter. | No |
DCGM_FI_DEV_GPU_UTIL | 203 | GPU utilization in %. | Yes |
DCGM_FI_DEV_MEM_COPY_UTIL | 204 | Memory copy utilization in %. | Yes |
DCGM_FI_DEV_ENC_UTIL | 206 | Encoder utilization in %. | Yes |
DCGM_FI_DEV_DEC_UTIL | 207 | Decoder utilization in %. | Yes |
DCGM_FI_DEV_PCIE_MAX_LINK_GEN | 235 | Maximum PCIe link generation. | Yes |
DCGM_FI_DEV_PCIE_MAX_LINK_WIDTH | 236 | Maximum PCIe link width. | Yes |
DCGM_FI_DEV_PCIE_LINK_GEN | 237 | Current PCIe link generation. | Yes |
DCGM_FI_DEV_PCIE_LINK_WIDTH | 238 | Current PCIe link width. | Yes |
DCGM_FI_DEV_FB_TOTAL | 250 | Total GPU frame buffer in MB. | Yes |
DCGM_FI_DEV_FB_FREE | 251 | Free frame buffer in MB. | Yes |
DCGM_FI_DEV_FB_USED | 252 | Used frame buffer in MB. | Yes |
DCGM_FI_DEV_ECC_CURRENT | 300 | The current ECC mode. | Yes |
DCGM_FI_DEV_ECC_PENDING | 301 | The pending ECC mode. | Yes |
DCGM_FI_DEV_ECC_SBE_VOL_TOTAL | 310 | Total single-bit volatile ECC errors. | Yes |
DCGM_FI_DEV_ECC_DBE_VOL_TOTAL | 311 | Total double-bit volatile ECC errors. | Yes |
DCGM_FI_DEV_ECC_SBE_AGG_TOTAL | 312 | Total single-bit persistent ECC errors. | Yes |
DCGM_FI_DEV_ECC_DBE_AGG_TOTAL | 313 | Total double-bit persistent ECC errors. | Yes |
DCGM_FI_DEV_ECC_SBE_VOL_DEV | 318 | Device memory single-bit volatile ECC errors. | Yes |
DCGM_FI_DEV_ECC_DBE_VOL_DEV | 319 | Device memory double-bit volatile ECC errors. | Yes |
DCGM_FI_DEV_ECC_SBE_AGG_DEV | 328 | Single-bit aggregate (persistent) ECC errors in device memory. | Yes |
DCGM_FI_DEV_ECC_DBE_AGG_DEV | 329 | Double-bit aggregate (persistent) ECC errors in device memory. | Yes |
DCGM_FI_DEV_BANKS_REMAP_ROWS_AVAIL_MAX | 385 | Historical max available spare memory rows per memory bank. | Yes |
DCGM_FI_DEV_BANKS_REMAP_ROWS_AVAIL_HIGH | 386 | Historical high mark of available spare memory rows per memory bank. | Yes |
DCGM_FI_DEV_BANKS_REMAP_ROWS_AVAIL_PARTIAL | 387 | Historical mark of partial available spare memory rows per memory bank. | Yes |
DCGM_FI_DEV_BANKS_REMAP_ROWS_AVAIL_LOW | 388 | Historical low mark of available spare memory rows per memory bank. | Yes |
DCGM_FI_DEV_BANKS_REMAP_ROWS_AVAIL_NONE | 389 | Historical marker of memory banks with no available spare memory rows. | Yes |
DCGM_FI_DEV_RETIRED_SBE | 390 | Total pages retired due to single-bit errors. | Yes |
DCGM_FI_DEV_RETIRED_DBE | 391 | Total pages retired due to double-bit errors. | Yes |
DCGM_FI_DEV_RETIRED_PENDING | 392 | Total pages pending retirement. | Yes |
DCGM_FI_DEV_UNCORRECTABLE_REMAPPED_ROWS | 393 | Number of remapped rows for uncorrectable errors. | Yes |
DCGM_FI_DEV_CORRECTABLE_REMAPPED_ROWS | 394 | Number of remapped rows for correctable errors. | Yes |
DCGM_FI_DEV_ROW_REMAP_FAILURE | 395 | Indicates whether row remapping has failed. | Yes |
DCGM_FI_DEV_ROW_REMAP_PENDING | 396 | Indicates whether row remapping is pending. | Yes |
DCGM_FI_DEV_NVLINK_CRC_FLIT_ERROR_COUNT_TOTAL | 409 | Total NVLink flow-control CRC errors. | No |
DCGM_FI_DEV_NVLINK_CRC_DATA_ERROR_COUNT_TOTAL | 419 | Total NVLink data CRC errors. | No |
DCGM_FI_DEV_NVLINK_REPLAY_ERROR_COUNT_TOTAL | 429 | Total NVLink retries. | No |
DCGM_FI_DEV_NVLINK_BANDWIDTH_L0 | 440 | NVLink bandwidth counter for lane 0. | Yes |
DCGM_FI_DEV_NVLINK_BANDWIDTH_L1 | 441 | NVLink bandwidth counter for lane 1. | Yes |
DCGM_FI_DEV_NVLINK_BANDWIDTH_L2 | 442 | NVLink bandwidth counter for lane 2. | Yes |
DCGM_FI_DEV_NVLINK_BANDWIDTH_L3 | 443 | NVLink bandwidth counter for lane 3. | Yes |
DCGM_FI_DEV_NVLINK_BANDWIDTH_L4 | 444 | NVLink bandwidth counter for lane 4. | Yes |
DCGM_FI_DEV_NVLINK_BANDWIDTH_L5 | 445 | NVLink bandwidth counter for lane 5. | Yes |
DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL | 449 | Total NVLink bandwidth counter for all lanes. | Yes |
DCGM_FI_DEV_NVLINK_BANDWIDTH_L6 | 475 | NVLink bandwidth counter for lane 6. | Yes |
DCGM_FI_DEV_NVLINK_TX_BANDWIDTH_L0 | 825 | NVLink TX bandwidth counter for lane 0. | Yes |
DCGM_FI_DEV_NVLINK_TX_BANDWIDTH_L1 | 826 | NVLink TX bandwidth counter for lane 1. | Yes |
DCGM_FI_DEV_NVLINK_TX_BANDWIDTH_L2 | 827 | NVLink TX bandwidth counter for lane 2. | Yes |
DCGM_FI_DEV_NVLINK_TX_BANDWIDTH_L3 | 828 | NVLink TX bandwidth counter for lane 3. | Yes |
DCGM_FI_DEV_NVLINK_TX_BANDWIDTH_L4 | 829 | NVLink TX bandwidth counter for lane 4. | Yes |
DCGM_FI_DEV_NVLINK_TX_BANDWIDTH_L5 | 830 | NVLink TX bandwidth counter for lane 5. | Yes |
DCGM_FI_DEV_NVLINK_TX_BANDWIDTH_L6 | 831 | NVLink TX bandwidth counter for lane 6. | Yes |
DCGM_FI_DEV_NVLINK_TX_BANDWIDTH_L7 | 832 | NVLink TX bandwidth counter for lane 7. | Yes |
DCGM_FI_DEV_NVLINK_TX_BANDWIDTH_L8 | 833 | NVLink TX bandwidth counter for lane 8. | Yes |
DCGM_FI_DEV_NVLINK_TX_BANDWIDTH_L9 | 834 | NVLink TX bandwidth counter for lane 9. | Yes |
DCGM_FI_DEV_NVLINK_TX_BANDWIDTH_L10 | 835 | NVLink TX bandwidth counter for lane 10. | Yes |
DCGM_FI_DEV_NVLINK_TX_BANDWIDTH_L11 | 836 | NVLink TX bandwidth counter for lane 11. | Yes |
DCGM_FI_DEV_NVLINK_TX_BANDWIDTH_L12 | 837 | NVLink TX bandwidth counter for lane 12. | Yes |
DCGM_FI_DEV_NVLINK_TX_BANDWIDTH_L13 | 838 | NVLink TX bandwidth counter for lane 13. | Yes |
DCGM_FI_DEV_NVLINK_TX_BANDWIDTH_L14 | 839 | NVLink TX bandwidth counter for lane 14. | Yes |
DCGM_FI_DEV_NVLINK_TX_BANDWIDTH_L15 | 840 | NVLink TX bandwidth counter for lane 15. | Yes |
DCGM_FI_DEV_NVLINK_TX_BANDWIDTH_L16 | 841 | NVLink TX bandwidth counter for lane 16. | Yes |
DCGM_FI_DEV_NVLINK_TX_BANDWIDTH_L17 | 842 | NVLink TX bandwidth counter for lane 17. | Yes |
DCGM_FI_DEV_NVLINK_TX_BANDWIDTH_TOTAL | 843 | Total NVLink TX bandwidth counter for all TX lanes. | Yes |
DCGM_FI_DEV_NVLINK_RX_BANDWIDTH_L0 | 879 | NVLink RX bandwidth counter for lane 0. | Yes |
DCGM_FI_DEV_NVLINK_RX_BANDWIDTH_L1 | 880 | NVLink RX bandwidth counter for lane 1. | Yes |
DCGM_FI_DEV_NVLINK_RX_BANDWIDTH_L2 | 881 | NVLink RX bandwidth counter for lane 2. | Yes |
DCGM_FI_DEV_NVLINK_RX_BANDWIDTH_L3 | 882 | NVLink RX bandwidth counter for lane 3. | Yes |
DCGM_FI_DEV_NVLINK_RX_BANDWIDTH_L4 | 883 | NVLink RX bandwidth counter for lane 4. | Yes |
DCGM_FI_DEV_NVLINK_RX_BANDWIDTH_L5 | 884 | NVLink RX bandwidth counter for lane 5. | Yes |
DCGM_FI_DEV_NVLINK_RX_BANDWIDTH_L6 | 885 | NVLink RX bandwidth counter for lane 6. | Yes |
DCGM_FI_DEV_NVLINK_RX_BANDWIDTH_L7 | 886 | NVLink RX bandwidth counter for lane 7. | Yes |
DCGM_FI_DEV_NVLINK_RX_BANDWIDTH_L8 | 887 | NVLink RX bandwidth counter for lane 8. | Yes |
DCGM_FI_DEV_NVLINK_RX_BANDWIDTH_L9 | 888 | NVLink RX bandwidth counter for lane 9. | Yes |
DCGM_FI_DEV_NVLINK_RX_BANDWIDTH_L10 | 889 | NVLink RX bandwidth counter for lane 10. | Yes |
DCGM_FI_DEV_NVLINK_RX_BANDWIDTH_L11 | 890 | NVLink RX bandwidth counter for lane 11. | Yes |
DCGM_FI_DEV_NVLINK_RX_BANDWIDTH_L12 | 891 | NVLink RX bandwidth counter for lane 12. | Yes |
DCGM_FI_DEV_NVLINK_RX_BANDWIDTH_L13 | 892 | NVLink RX bandwidth counter for lane 13. | Yes |
DCGM_FI_DEV_NVLINK_RX_BANDWIDTH_L14 | 893 | NVLink RX bandwidth counter for lane 14. | Yes |
DCGM_FI_DEV_NVLINK_RX_BANDWIDTH_L15 | 894 | NVLink RX bandwidth counter for lane 15. | Yes |
DCGM_FI_DEV_NVLINK_RX_BANDWIDTH_L16 | 895 | NVLink RX bandwidth counter for lane 16. | Yes |
DCGM_FI_DEV_NVLINK_RX_BANDWIDTH_L17 | 896 | NVLink RX bandwidth counter for lane 17. | Yes |
DCGM_FI_DEV_NVLINK_RX_BANDWIDTH_TOTAL | 897 | Total NVLink RX bandwidth counter for all RX lanes. | Yes |
DCGM_FI_PROF_GR_ENGINE_ACTIVE | 1001 | Percentage of time the graphics engine is active. | No |
DCGM_FI_PROF_SM_ACTIVE | 1002 | The percentage of cycles that an SM has at least one assigned warp. | Yes |
DCGM_FI_PROF_SM_OCCUPANCY | 1003 | The ratio of resident warps to the maximum supported on an SM, in %. | Yes |
DCGM_FI_PROF_PIPE_TENSOR_ACTIVE | 1004 | Percentage of cycles the tensor (HMMA) pipe is active. | Yes |
DCGM_FI_PROF_DRAM_ACTIVE | 1005 | Percentage of cycles the device memory interface is active transferring data. | Yes |
DCGM_FI_PROF_PIPE_FP64_ACTIVE | 1006 | Percentage of cycles the FP64 pipes are active. | No |
DCGM_FI_PROF_PIPE_FP32_ACTIVE | 1007 | Percentage of cycles the FP32 pipes are active. | No |
DCGM_FI_PROF_PIPE_FP16_ACTIVE | 1008 | Percentage of cycles the FP16 pipes are active. | No |
DCGM_FI_PROF_PCIE_TX_BYTES | 1009 | The number of active PCIe TX bytes, including header and payload. | Yes |
DCGM_FI_PROF_PCIE_RX_BYTES | 1010 | The number of active PCIe RX bytes, including header and payload. | Yes |
DCGM_FI_PROF_NVLINK_TX_BYTES | 1011 | The total number of active NVLink TX bytes, including header and payload. | Yes |
DCGM_FI_PROF_NVLINK_RX_BYTES | 1012 | The total number of active NVLink RX bytes, including header and payload. | Yes |
DCGM_FI_PROF_PIPE_INT_ACTIVE | 1016 | Percentage of cycles the integer pipe is active. | Yes |
DCGM_FI_PROF_NVLINK_L0_TX_BYTES | 1040 | Number of active NVLink TX bytes on lane 0, including header and payload. | Yes |
DCGM_FI_PROF_NVLINK_L0_RX_BYTES | 1041 | Number of active NVLink RX bytes on lane 0, including header and payload. | Yes |
DCGM_FI_PROF_NVLINK_L1_TX_BYTES | 1042 | Number of active NVLink TX bytes on lane 1, including header and payload. | Yes |
DCGM_FI_PROF_NVLINK_L1_RX_BYTES | 1043 | Number of active NVLink RX bytes on lane 1, including header and payload. | Yes |
DCGM_FI_PROF_NVLINK_L2_TX_BYTES | 1044 | Number of active NVLink TX bytes on lane 2, including header and payload. | Yes |
DCGM_FI_PROF_NVLINK_L2_RX_BYTES | 1045 | Number of active NVLink RX bytes on lane 2, including header and payload. | Yes |
DCGM_FI_PROF_NVLINK_L3_TX_BYTES | 1046 | Number of active NVLink TX bytes on lane 3, including header and payload. | Yes |
DCGM_FI_PROF_NVLINK_L3_RX_BYTES | 1047 | Number of active NVLink RX bytes on lane 3, including header and payload. | Yes |
DCGM_FI_PROF_NVLINK_L4_TX_BYTES | 1048 | Number of active NVLink TX bytes on lane 4, including header and payload. | Yes |
DCGM_FI_PROF_NVLINK_L4_RX_BYTES | 1049 | Number of active NVLink RX bytes on lane 4, including header and payload. | Yes |
DCGM_FI_PROF_NVLINK_L5_TX_BYTES | 1050 | Number of active NVLink TX bytes on lane 5, including header and payload. | Yes |
DCGM_FI_PROF_NVLINK_L5_RX_BYTES | 1051 | Number of active NVLink RX bytes on lane 5, including header and payload. | Yes |
DCGM_FI_PROF_NVLINK_L6_TX_BYTES | 1052 | Number of active NVLink TX bytes on lane 6, including header and payload. | Yes |
DCGM_FI_PROF_NVLINK_L6_RX_BYTES | 1053 | Number of active NVLink RX bytes on lane 6, including header and payload. | Yes |
DCGM_FI_PROF_KSD_HIT_RATE | 6001 | KSD cache hit rate in %. | Yes |
DCGM_FI_PROF_KVD_HIT_RATE | 6002 | KVD cache hit rate in %. | Yes |
DCGM_FI_PROF_L2_HIT_RATE | 6003 | L2 cache hit rate in %. | Yes |
DCGM_FI_PROF_LLC_HIT_RATE | 6004 | LLC cache hit rate in %. | No |
6. dcgm-exporter
PPU DCGM provides the dcgm-exporter tool to simplify integration with Kubernetes environments, allowing external systems to query and manage PPU devices.
6.1 Procedure
As described in Obtain and Install, after you decompress the PPU DCGM installation package, you will obtain the hgdcgm-exporter folder. This folder contains the dcgm-exporter executable file and default-counters.csv. You can modify default-counters.csv to control the information that is provided. The following is an example of the content in default-counters.csv:
# Format
# If line starts with a '#' it is considered a comment
# DCGM FIELD, Prometheus metric type, help message
# Clocks
DCGM_FI_DEV_SM_CLOCK, gauge, SM clock frequency (in MHz).
DCGM_FI_DEV_MEM_CLOCK, gauge, Memory clock frequency (in MHz).
# Temperature
DCGM_FI_DEV_MEMORY_TEMP, gauge, Memory temperature (in C).
DCGM_FI_DEV_GPU_TEMP, gauge, GPU temperature (in C).
# Power
DCGM_FI_DEV_POWER_USAGE, gauge, Power draw (in W).
# PCIE
DCGM_FI_DEV_PCIE_TX_THROUGHPUT, counter, Total number of bytes transmitted through PCIe TX (in KB) via NVML.
DCGM_FI_DEV_PCIE_RX_THROUGHPUT, counter, Total number of bytes received through PCIe RX (in KB) via NVML.
# Utilization (the sample period varies depending on the product)
DCGM_FI_DEV_GPU_UTIL, gauge, GPU utilization (in %).
DCGM_FI_DEV_ENC_UTIL, gauge, Encoder utilization (in %).
DCGM_FI_DEV_DEC_UTIL , gauge, Decoder utilization (in %).
# Memory usage
DCGM_FI_DEV_FB_FREE, gauge, Framebuffer memory free (in MiB).
DCGM_FI_DEV_FB_USED, gauge, Framebuffer memory used (in MiB).
DCGM_FI_DEV_FB_TOTAL, gauge, Framebuffer memory total (in MiB)
...Run ./dcgm-exporter to start dcgm-exporter. The output is similar to the following, where 9400 is the HTTP interface for the exporter:
# Start the dcgm-exporter service
./dcgm-exporter &
# Query the metrics endpoint
curl localhost:9400/metricsThe command produces output similar to the following:
# HELP DCGM_FI_DEV_SM_CLOCK SM clock frequency (in MHz).
# TYPE DCGM_FI_DEV_SM_CLOCK gauge
DCGM_FI_DEV_SM_CLOCK{gpu="0",UUID="GPU-019ea108-c170-021e-0000-00002069567c",device="ppu0",modelName="PPU",Hostname="ubuntu",DCGM_FI_DRIVER_VERSION="0.8.0"} 1500
# HELP DCGM_FI_DEV_MEM_CLOCK Memory clock frequency (in MHz).
# TYPE DCGM_FI_DEV_MEM_CLOCK gauge
DCGM_FI_DEV_MEM_CLOCK{gpu="0",UUID="GPU-019ea108-c170-021e-0000-00002069567c",device="ppu0",modelName="PPU",Hostname="ubuntu",DCGM_FI_DRIVER_VERSION="0.8.0"} 1800
# HELP DCGM_FI_DEV_MEMORY_TEMP Memory temperature (in C).
# TYPE DCGM_FI_DEV_MEMORY_TEMP gauge
DCGM_FI_DEV_MEMORY_TEMP{gpu="0",UUID="GPU-019ea108-c170-021e-0000-00002069567c",device="ppu0",modelName="PPU",Hostname="ubuntu",DCGM_FI_DRIVER_VERSION="0.8.0"} 36
# HELP DCGM_FI_DEV_GPU_TEMP GPU temperature (in C).
# TYPE DCGM_FI_DEV_GPU_TEMP gauge
DCGM_FI_DEV_GPU_TEMP{gpu="0",UUID="GPU-019ea108-c170-021e-0000-00002069567c",device="ppu0",modelName="PPU",Hostname="ubuntu",DCGM_FI_DRIVER_VERSION="0.8.0"} 38
# HELP DCGM_FI_DEV_POWER_USAGE Power draw (in W).
# TYPE DCGM_FI_DEV_POWER_USAGE gauge
DCGM_FI_DEV_POWER_USAGE{gpu="0",UUID="GPU-019ea108-c170-021e-0000-00002069567c",device="ppu0",modelName="PPU",Hostname="ubuntu",DCGM_FI_DRIVER_VERSION="0.8.0"} 130.730000
# HELP DCGM_FI_DEV_PCIE_TX_THROUGHPUT Total number of bytes transmitted through PCIe TX (in KB) via NVML.
# TYPE DCGM_FI_DEV_PCIE_TX_THROUGHPUT counter
DCGM_FI_DEV_PCIE_TX_THROUGHPUT{gpu="0",UUID="GPU-019ea108-c170-021e-0000-00002069567c",device="ppu0",modelName="PPU",Hostname="ubuntu",DCGM_FI_DRIVER_VERSION="0.8.0"} 0.000000
# HELP DCGM_FI_DEV_PCIE_RX_THROUGHPUT Total number of bytes received through PCIe RX (in KB) via NVML.
# TYPE DCGM_FI_DEV_PCIE_RX_THROUGHPUT counter
DCGM_FI_DEV_PCIE_RX_THROUGHPUT{gpu="0",UUID="GPU-019ea108-c170-021e-0000-00002069567c",device="ppu0",modelName="PPU",Hostname="ubuntu",DCGM_FI_DRIVER_VERSION="0.8.0"} 0.000000
... 6.2 Using with the dcgmi CLI
When the dcgmi command-line tool and dcgm-exporter run on the same system, multiple subscribers subscribe to PPU monitoring data because dcgm-exporter uses an embedded DCGM server by default. This can cause issues such as increased monitoring performance overhead and failed subscriptions to performance analysis metrics (Field ID > 1000).
You can use the -r parameter of dcgm-exporter to specify the port address of the nv-hostengine dcgmi background service so that dcgm-exporter can retrieve monitoring data through nv-hostengine. For example:
./dcgm-exporter -r localhost:55557. FAQ
7.1 DCGM tool in Docker
When starting a container with docker run, omit the --gpus option. You can give the container access to PPU devices on the host by either creating a privileged container with the --privileged flag or mapping the devices to an unprivileged container with the --device flag.
To start an image with access to all cards, use the --privileged flag as shown in the following command:
docker run --privileged --ipc=host --shm-size=4g --ulimit memlock=-1 --ulimit stack=67108864 --init -it -v $HOME:/mnt -w /workspace --name test_$USER <DOCKER_NAME> /bin/bashTo start an image with a specific PPU device, refer to the following command. The example below grants access to only the PPU 0 device with the --device=/dev/alixpu_ppu0 --device=/dev/alixpu --device=/dev/alixpu_ctl flags:
docker run --device=/dev/alixpu_ppu0 --device=/dev/alixpu --device=/dev/alixpu_ctl --ipc=host --shm-size=4g --ulimit memlock=-1 --ulimit stack=67108864 --init -it -v $HOME:/mnt -w /workspace --name tf_$USER <DOCKER_NAME> /bin/bash8. Known issues
SDK v1.0.0: DCGM is not supported on vGPU-enabled systems.
SDK v1.0.0: The reported application clock frequency can be lower than the actual value.
SDK v1.0.0: Query results for field IDs greater than 1000 may be inaccurate.
SDK v1.2.0: The
dcgmitool and DCGM API do not currently support queries for utilization metrics such as Tensor Core or Pipe.Features that modify the device configuration are not available when running with non-root permissions.
The
dcgmitool does not currently support thestatssubcommand.The
dcgmi nvlinksubcommand cannot currently query ICN errors.The
dcgmi policysubcommand does not support setting error policy thresholds for PCIe or ICN.The
dcgmi healthsubcommand does not support subscribing to infoROM, thermal and power, or NVLink.The ICN link throughput rate represents the raw data throughput and includes overhead such as message headers.
Setting or querying policy violations is not supported.
Querying sample data, such as periodic temperature or power samples, is not supported.
Querying vGPU utilization or process information is not supported.
Persistence mode is not supported.
Managing performance states is not supported.
Adjusting the memory clock frequency is not supported.
The
memory_bandwidthdiagnostic test fails if the PPU frequency does not reach its maximum.In rare cases, starting
nv-hostenginecan trigger the error "DCGM Failed to find any GPUs on the node". This occurs when residual files from a previous KMD or PPU SDK version remain on the server.