Use Nsight Systems to analyze and optimize the performance of AI applications

Benefits

• Cross-platform visualization:

  Nsight Systems captures and visualizes system-wide activity in real time, including CPU, GPU, network interface controller (NIC), storage, and other accelerator execution and resource usage. This helps developers better understand interactions and dependencies between hardware components.

• Deep hardware insights:

  Nsight Systems provides device-level profiling metrics to reveal GPU workload details. For example:

  • It shows GPU utilization to identify idle periods or overloads.

  • It analyzes kernel scheduling and execution, including whether grid dimensions are set appropriately and whether stream concurrency fully uses GPU resources.

  • It checks the occupancy of each streaming multiprocessor (SM) and warp scheduling efficiency.

  • It confirms whether the application effectively uses hardware units such as Tensor Cores, which are designed specifically for machine learning and deep learning tasks.

  • It calculates the GPU instruction execution rate to assess performance bottlenecks and potential optimization opportunities.

• Automatic optimization suggestions:

  Nsight Systems provides Expert Systems Analysis. If the tool detects potential optimization points in your program, it offers optimization suggestions.

Limitations

• Modified launch commands affect program performance:

  Profiling requires you to launch the target application through its command line interface (CLI), such as nsys profile. This operation may increase the startup time or alter the process environment, which can affect the original application's performance.

• Lacks continuous profiling capability:

  For long-running applications, especially those with intermittent GPU usage such as online inference services, Nsight Systems may collect massive amounts of data, which makes the results hard to read and analyze.

Step 1: Create a sample model

After you build a deep learning network, you can use NVIDIA Nsight Systems for baseline performance analysis to identify optimization opportunities. This topic uses the sample model from the TensorBoard-plugin tutorial to demonstrate how to find these opportunities. Follow these steps:

1. Inside the PyTorch container, create a main.py file and copy the following content into it.

   The main.py file contains the Python script logic for Nsight Systems to analyze.

   import nvtx  # Import the nvtx package to observe logical relationships between functions in Nsight Systems.
   import torch
   import torch.nn
   import torch.optim
   import torch.profiler
   import torch.utils.data
   import torchvision.datasets
   import torchvision.models
   import torchvision.transforms as T
   
   transform = T.Compose([
       T.Resize(224),
       T.ToTensor(),
       T.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])
   
   train_set = torchvision.datasets.CIFAR10(root='./data', train=True, download=True, transform=transform)
   train_loader = torch.utils.data.DataLoader(train_set, batch_size=32, shuffle=True)
   
   
   device = torch.device("cuda:0")
   model = torchvision.models.resnet18(weights='IMAGENET1K_V1').cuda(device)
   criterion = torch.nn.CrossEntropyLoss().cuda(device)
   optimizer = torch.optim.SGD(model.parameters(), lr=0.001, momentum=0.9)
   model.train()
   
   
   def train(data, batch_idx):
       # Data transmission
       nvtx.push_range("copy data " + str(batch_idx), color="rapids")
       inputs, labels = data[0].to(device=device), data[1].to(device=device)
       nvtx.pop_range()
       # Forward propagation
       nvtx.push_range("forward " + str(batch_idx), color="yellow")
       outputs = model(inputs)
       loss = criterion(outputs, labels)
       nvtx.pop_range()
       # Backward propagation
       nvtx.push_range("backward " + str(batch_idx), color="green")
       optimizer.zero_grad()
       loss.backward()
       optimizer.step()
       nvtx.pop_range()
   
   # enumerate(train_loader) cannot insert nvtx to collect data loading time. Replace the for loop with equivalent code below.
   dl = iter(train_loader)
   batch_idx = 0
   while True:
       try:
           # Collect total duration of the last 3 batches.
           if batch_idx == 5:
               tet = nvtx.start_range(message="Total Elapsed Time(3 batchs)", color="orange")
           # Observe only the first 8 batches. Use the first 5 for wait and warmup; focus on the last 3.
           if batch_idx >= 8:
               nvtx.end_range(tet)
               break
           # Data loading.
           nvtx.push_range("__next__ " + str(batch_idx), color="orange")
           batch_data = next(dl)
           nvtx.pop_range()
           # Batch processing, including data transmission and GPU computing.
           nvtx.push_range("batch " + str(batch_idx), color="cyan")
           train(batch_data, batch_idx)
           nvtx.pop_range()
           batch_idx += 1
       except StopIteration:
           nvtx.pop_range()
           break

2. Inside the PyTorch container, run the following command to generate a file named baseline.nsys-rep in the current directory.

   nsys profile \
   	-w true \
   	--cuda-memory-usage=true \
   	--python-backtrace=cuda \
   	-s cpu \
   	-f true \
   	-x true \
   	-o baseline \
   	python ./main.py

   Expected output:

   Files already downloaded and verified
   Generating '/tmp/nsys-report-6673.qdstrm'
   [1/1] [========================100%] baseline.nsys-rep
   Generated:
       /root/baseline.nsys-rep

3. Import the baseline.nsys-rep file into the Nsight Systems UI to analyze the model. The following figure shows the initial view:

   Note

   If you have not installed Nsight Systems, go to the NVIDIA Developer website and install the appropriate version for your system.

   

4. This topic references only the last 3 batches. The first 3 batches may be affected by initialization and warmup, which can lead to an inaccurate performance assessment. The following figure zooms in on the last 3 batches.

   

   • The figure shows two NVTX recordings of batch time: one in the CUDA stream (Default Stream, labeled 1) and one in the Python thread (labeled 2). Their batch time statistics, including copy data, forward, and backward, differ significantly. When you call CUDA APIs and run kernels on the GPU, rely on the NVTX data from the CUDA stream (labeled 1). Because batch transmission and GPU computing are asynchronous from the CPU's perspective, the Python thread statistics may be inaccurate.

   • Batch data loading and batch computing, including batch transmission to the GPU, overlap. As shown in the figure, when loading Batch 6, the GPU is still computing Batch 5. Therefore, when you calculate the total duration of the 3 batches, do not count the overlapping time.

   • The baseline stage durations are as follows:

     Stage duration	Time (ms)
     Average data loading time (marked as __next__ in the figure)	(37.276 + 35.794 + 35.790) / 3 = 36.287
     Average data transmission time (collapsed in the figure)	(4.39 + 4.363 + 4.317) / 3 = 4.357
     Average batch processing time (includes data transmission, forward, and backward)	(38.738 + 37.78 + 37.754) / 3 = 38.091
     Total duration of 3 batches (from loading Batch 5 to completing Batch 7 computation)	176.878 ms
     Average samples processed per second (samples/s)	32 (batch size) * 3 / (176.878 / 1000) = 542.746

     

Step 2: Optimize and analyze the model

Model optimization procedure

A single-machine deep learning training job consists of the following stages:

• Data loading: Load data from a disk or other network storage to the host memory and preprocess it, such as by removing noise. This is the data loading stage.

• Data transmission: Transfer data from the host memory to the GPU memory.

• Training: GPU compute units, such as the CUDA Core and Tensor Core, use this data for training.

The following sections provide examples for each part.

Optimization 1: Reduce data loading time

Reducing data loading time significantly improves training performance. If data loading takes longer than GPU training, the GPU may sit idle while waiting for data.

As shown in the following figure, the baseline, with no optimizations, has large gaps between batch computations. The root cause is the long batch data loading time. The GPU waits a long time for data before it can process each batch. Therefore, data loading becomes the training bottleneck.

The PyTorch DataLoader supports multiple workers (multi-process) to load mini-batch data simultaneously. You can modify the train_loader parameter in main.py to use 8 workers for data loading.

train_loader = torch.utils.data.DataLoader(train_set,num_workers=8, batch_size=32, shuffle=True)

After you rerun the script, the gaps between batches disappear, and the data loading time drops significantly. The average loading time for three batches is (2.879 ms + 104.995 μs + 180.066 μs) / 3 = 1.055 ms. The number of samples processed per second (samples/s) is 32 * 3 / (120.328 / 1000) = 797.819. You can ignore the data loading time because it overlaps with GPU computing. The performance improves by (797.819 - 542.746) / 542.746 * 100% = 47.00% compared to the baseline.

Optimization 2: Reduce data transmission time

With 8 workers set, you can continue to optimize the sample model. The previous analysis showed that the average data transmission time was 4.357 ms. You can enable Pin Memory to reduce this time.

PyTorch supports storing loaded data directly in Pin Memory. You can modify the train_loader parameter in main.py to enable Pin Memory.

train_loader = torch.utils.data.DataLoader(train_set,num_workers=8, batch_size=32,pin_memory=True, shuffle=True)

After you rerun the script, the average data transmission time is (1.956 + 1.979 + 1.989) / 3 = 1.975 ms, which reduces the baseline time by 4.357 ms - 1.975 ms = 2.382 ms. The total duration drops to 99.535 ms. The average number of samples processed per second (samples/s) is:

32 * 3 /(99.535 / 1000) = 964.485. The performance improves by (964.485 - 797.819) / 797.819 * 100% = 20.89%.

Optimization 3: Fully overlap data loading and data computing

After reducing the data transmission time, you can seek further optimization. Analyzing the Pin Memory-enabled timeline shows that batch loading and batch computing, including transmission, overlap, but each batch load overlaps with only one batch compute. As shown in the figure, Batch 5 loading overlaps with Batch 4 computing, Batch 6 with Batch 5, and Batch 7 with Batch 6. Cases where Batch 6 and 7 loading overlap with Batch 5 computing do not occur. In addition, after each data loading operation, cudaStreamSynchronize forces a CPU-GPU sync, which is unnecessary for batch computing.

This sync comes from the following line of code:

inputs, labels = data[0].to(device=device), data[1].to(device=device)

By default, the to function executes cudaStreamSynchronize after each host-to-GPU data transfer. However, the to function also supports a non-blocking mode (asynchronous operation) using the non_blocking=True parameter:

inputs, labels = data[0].to(device=device,non_blocking=True), data[1].to(device=device,non_blocking=True)

In non-blocking mode, the GPU data transmission is queued at the end of the CUDA stream without blocking the CPU. The CPU can then continue with subsequent tasks without waiting for the transmission to complete.

After you modify the code and rerun the script, the results are as follows.

The figure shows the following:

• When the GPU computes Batch 3, the CPU has already finished loading the data for Batch 5, 6, and 7. This achieves a full overlap between batch loading and computing.

• The loading for Batch 5, 6, and 7 is fully hidden within the computing for Batch 2, 3, and 4. Their loading time is negligible.

The number of samples processed per second (samples/s) is 32 × 3 / (94.164 / 1000) = 1019.498. The performance improves by (1019.498 - 964.485) / 964.485 × 100% = 5.7% compared to Optimization 2.

Optimization 4: Automatic Mixed Precision

The previous steps optimized data loading and transmission. This section focuses on reducing the GPU batch computing time.

PyTorch supports mixed-precision training through Automatic Mixed Precision (AMP). In AMP mode, some tensors are automatically converted to low-precision 16-bit floating-point numbers and run on GPU Tensor Cores. This reduces GPU memory usage and computing time.

You can modify the code as shown below to enable AMP. In a production environment, a full AMP implementation may require gradient scaling, which this demo omits. For more information about correct usage, see the AMP documentation.

# train step
def train(data):
    # Omit other code
    # Enable amp
    with torch.autocast(device_type='cuda', dtype=torch.float16):
        outputs = model(inputs)
        loss = criterion(outputs, labels)
    # Omit other code   

With AMP enabled, the batch computing time drops to 58.938 ms. The average number of samples processed per second is 32 × 3 / (58.938 /1000) = 1628.83. The performance improves by (1628.83 - 1019.498) / 1019.498 × 100% = 59.77%.

Inspecting the forward and backward passes of a batch shows the kernel composition. Before AMP was enabled, operations mostly used float32 data. If you sort the kernels by descending execution time, the top kernels range from 300 μs to 1 ms.

After AMP is enabled, most operations use float16 data, and the top kernels range from 100 μs to 300 μs.

Optimization 5: Increase batch size

Increasing the batch size can boost the number of samples processed per second. However, you must consider GPU memory availability and the impact of large batches on loss values.

You can modify the code as shown below to adjust the batch size from 32 to 512.

train_loader = torch.utils.data.DataLoader(train_set,num_workers=8, batch_size=512,pin_memory=True, shuffle=True)

After running the script, the total duration is 697.666 ms. The average number of samples processed per second (samples/s) is 512 × 3 / (697.666 / 1000) = 2201.627. The performance improves by (2201.627 - 1628.83) / 1628.83 × 100% = 35.17%.

Optimization 6: Model compilation

By default, PyTorch uses just-in-time compilation. You can use the PyTorch compile API to compile the model into graph mode.

You can modify the code as follows:

model = torchvision.models.resnet18(weights='IMAGENET1K_V1').cuda(device)
model = torch.compile(model)

After running the script, the total duration is 567.971 ms. The average number of samples processed per second is 512 × 3 / (567.971 / 1000) = 2704.36. The performance improves by (2704.36 - 2201.627) / 2201.627 × 100% = 22.83%.

Note that model compilation may not help in small-batch scenarios.

Optimization 7: Overlap batch transmission and batch computing

Batch transmission and computing execute sequentially. Batch computing starts only after the transmission is complete. During transmission, the GPU is idle. With larger batch sizes, the transmission time becomes significant compared to the computing time.

Normally, batch computing requires the data to be completely transmitted to the GPU first. Starting the computation during transmission yields incorrect results. However, using a pipeline approach, if the batch computing time exceeds the transmission time, you can start transmitting Batch 2 while computing Batch 1, and transmit Batch 3 while computing Batch 2. This approach hides the transmission time within the computing time.

Single-stream execution cannot achieve this overlap. However, PyTorch lets you create multiple streams in addition to the default stream. The full code is as follows:

import nvtx  # Import nvtx package
import torch
import torch.nn
import torch.optim
import torch.profiler
import torch.utils.data
import torchvision.datasets
import torchvision.models
import torchvision.transforms as T

transform = T.Compose([
    T.Resize(224),
    T.ToTensor(),
    T.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])
train_set = torchvision.datasets.CIFAR10(root='./data', train=True, download=True, transform=transform)
train_loader = torch.utils.data.DataLoader(train_set, num_workers=8, batch_size=512, pin_memory=True, shuffle=True)

device = torch.device("cuda:0")
model = torchvision.models.resnet18(weights='IMAGENET1K_V1').cuda(device)
model = torch.compile(model)
criterion = torch.nn.CrossEntropyLoss().cuda(device)
optimizer = torch.optim.SGD(model.parameters(), lr=0.001, momentum=0.9)

def train(model, device, train_loader, optimizer, criterion):
	model.train()
	dl = iter(train_loader)
	transfered_data = []
	# Create a new stream
	s = torch.cuda.Stream()

	def prepare_batch(batch_idx):
		nvtx.push_range("__next__ " + str(batch_idx), color="orange")
		(inputs, labels) = next(dl)
		# Transmit data in the new stream
		nvtx.pop_range()
		nvtx.push_range("cpy " + str(batch_idx), color="rapids")
		with torch.cuda.stream(s):
			inputs, labels = inputs.to(device=device, non_blocking=True), labels.to(device=device, non_blocking=True)
		nvtx.pop_range()
		return (inputs, labels)

	def batch_compute(batch_idx, data):
		nvtx.push_range("batch " + str(batch_idx), color="cyan")
		nvtx.push_range("forward " + str(batch_idx), color="yellow")
		with torch.autocast(device_type='cuda', dtype=torch.float16):
			outputs = model(data[0])
			loss = criterion(outputs, data[1])
		nvtx.pop_range()
		nvtx.push_range("backward " + str(batch_idx), color="green")
		optimizer.zero_grad()
		loss.backward()
		optimizer.step()
		nvtx.pop_range()
		torch.cuda.current_stream().wait_stream(s)
		nvtx.pop_range()

	batch_idx = 0
	tet = None
	while True:
		try:
			if batch_idx == 6:
				tet = nvtx.start_range(message="Total Elapsed Time(3 batchs)", color="orange")
			if batch_idx >= 8:
				break
			data = prepare_batch(batch_idx)
			transfered_data.append(data)
			# Skip computation when batch_idx is 0
			# Each loop requires the default stream to wait for the new stream's data transmission,
			# ensuring data readiness for the next batch computation.
			if batch_idx > 0:
				batch_compute(batch_idx - 1, transfered_data[batch_idx - 1])
			else:
				torch.cuda.current_stream().wait_stream(s)
			batch_idx += 1
		except StopIteration:
			nvtx.pop_range()
			break
	batch_compute(batch_idx - 1, transfered_data[batch_idx - 1])
	nvtx.end_range(tet)


train(model, device, train_loader, optimizer, criterion)

The following figure shows the results of the run. The transmission for Batches 2 to 7 overlaps with the computation for Batches 1 to 4. The transmission for Batches 0 and 1 is not shown due to space limitations. To ensure correct results, the data transmission for each batch is completed before its computation begins. For example, the transmission for Batch 4 occurs during the computation for Batch 2.

The total duration is 484.016 ms. The average number of samples processed per second is 3173.449 (512 × 3 ÷ (484.016 ÷ 1000)), which is a 17.35% performance improvement ((3173.449 - 2704.36) ÷ 2704.36 × 100%).

Note that overlapping computation and data transmission increases GPU memory usage. In this example, when computing Batch 5, all batches reside in GPU memory.

Summary

Performance gains from each optimization: