Video Codec/Image使用指南 (v1.6)-ZHENWU PPU Cloud Service(ppu)-阿里云帮助中心

1. Overview

PPU is compatible with NVIDIA's hardware acceleration libraries for video and image codecs, pre-processing, and post-processing. These libraries include:

Video Codec SDK: Provides hardware acceleration support for cuvid decode and nvenc encode. The current version is 12.2.
nvJPEG: Provides hardware acceleration support for JPEG decode and JPEG encode. The current version is 12.3.
npp: Provides hardware acceleration support for 2D image post-processing. The current version is 11.6.

2. Ecosystem

The following applications and frameworks can use the hardware acceleration capabilities.

FFmpeg: You can enable hardware acceleration by enabling cuvid, nvenc, and libnpp during compilation. When you encode, decode, or transcode videos, you can specify cuvid, nvenc, or npp filters. FFmpeg is widely used, and many video encoding or decoding applications, including PyAV (the backend for torchvision), depend on it.
DALI: A parallel acceleration framework for data loaders from NVIDIA. It supports hardware-accelerated JPEG and video codecs.
- Official Website: https://docs.nvidia.com/deeplearning/dali/user-guide/docs/index.html
- Sample Code: https://github.com/NVIDIA/DALI/blob/main/docs/examples/use_cases/

In addition to these popular large-scale frameworks, you can also use the following wrapper libraries to access the codec/JPEG APIs:

nvImageCodec: A wrapper library from NVIDIA that reads various image formats and converts them into tensor data. It also supports hardware-accelerated JPEG encode and provides a Python interface.
- Official Website: https://docs.nvidia.com/cuda/nvimagecodec/
- Source Code: https://github.com/NVIDIA/nvImageCodec
VPF: NVIDIA's Video Processing Framework, a wrapper library that encapsulates the capabilities of the Video Codec SDK and provides a Python interface.
- Official Website: https://developer.nvidia.com/blog/vpf-hardware-accelerated-video-processing-framework-in-python/
- Source Code: https://github.com/NVIDIA/VideoProcessingFramework
- VPF may have compilation compatibility issues with the latest version of FFmpeg. If so, refer to the following patch:
  Diff
```
diff --git a/src/TC/src/FFmpegDemuxer.cpp b/src/TC/src/FFmpegDemuxer.cpp
index a711aa1..e671189 100644
index a711aa1..e671189 100644
--- a/src/TC/src/FFmpegDemuxer.cpp
+++ b/src/TC/src/FFmpegDemuxer.cpp
@@ -583,5 +583,9 @@ FFmpegDemuxer::FFmpegDemuxer(AVFormatContext *fmtcx) : fmtc(fmtcx) {

   /* Some inputs doesn't allow seek functionality.
    * Check this ahead of time. */
+#if LIBAVFORMAT_VERSION_MAJOR >= 61
+  is_seekable = fmtc->iformat->flags & AVFMT_NOTIMESTAMPS;
+#else
   is_seekable = fmtc->iformat->read_seek || fmtc->iformat->read_seek2;
+#endif
```
  On some Docker containers, running pip install . -v in the VideoProcessingFramework directory may cause the process to hang. If this occurs, try uninstalling and reinstalling ninja by running pip uninstall ninja. Note: NVIDIA has frozen updates for VPF and now supports PyNvCodec instead. The PPU video codec supports PyNvCodec.

3. Build and run

Video Codec SDK

The PPU video codec is compatible with the NVIDIA Video Codec SDK. Official website: https://developer.nvidia.com/nvidia-video-codec-sdk/download

Download the Video Codec SDK version that matches the CUDA version compatible with your PPU SDK. The known version mapping is as follows:

Prerequisites

Because the PPU does not support OpenGL or Vulkan, you must modify the Samples/CMakeLists.txt file before compilation to remove unsupported projects. Keep only the following common codec projects:

# Keep the following and comment out the rest
add_subdirectory(AppEncode/AppEncCuda)
add_subdirectory(AppEncode/AppEncPerf)
add_subdirectory(AppEncode/AppEncMultiInstance)

add_subdirectory(AppEncode/AppEncDec)
add_subdirectory(AppTranscode/AppTrans)
add_subdirectory(AppTranscode/AppTransPerf)

add_subdirectory(AppDecode/AppDec)
add_subdirectory(AppDecode/AppDecMultiFiles)
add_subdirectory(AppDecode/AppDecPerf)

The Video Codec SDK depends on FFmpeg to provide demuxing capabilities, so you must install FFmpeg first. Although only demuxing is required, compile and install a hardware-accelerated version of FFmpeg to avoid confusion (refer to the Video FFMPEG User Guide).
Assume the FFmpeg installation directory is FFMPEG_PATH.
If you do not specify a directory during compilation, it defaults to /usr/local. Otherwise, the include, lib, and bin contents of FFmpeg are copied to $FFMPEG_PATH/include, $FFMPEG_PATH/lib, and $FFMPEG_PATH/bin.
Run pkg-config --modversion libavcodec to display the version number. If the version does not appear, first run export PKG_CONFIG_PATH=$FFMPEG_PATH/lib/pkgconfig:$PKG_CONFIG_PATH, and then try the pkg-config command again.
Set the path for the CUDA libraries.
export CMAKE_LIBRARY_PATH=${CUDA_HOME}/lib64
Here, CUDA_HOME points to the CUDA_SDK within the PPU SDK directory.
If the downloaded Video Codec SDK version does not match the CUDA version declared as compatible by the PPU SDK, the build may fail. If this happens, run the following command:
cp Interface/* ${CUDA_HOME}/include
Here, Interface refers to the Interface directory within the downloaded Video Codec SDK.

Build

cd Samples  # Sample folder of the video-codec-sdk download path
mkdir build
cd build
cmake ..
make

Run

# Decode. Running 12 threads concurrently can achieve peak performance.
./AppDecode/AppDecPerf/AppDecPerf -thread 12 -i test_1920x1080_yuv420p_50fps_h264.mp4

# Encode. Running 4 threads concurrently can achieve peak performance.
./AppEncode/AppEncPerf/AppEncPerf -i crowd_run_1920x1088_nv12_8bit_50.0fps_n500.yuv -s 1920x1088 -if nv12 -fps 30  -preset p4 -gop 60 -rc cbr -bitrate 10000000 -codec h265 -thread 4

Note

The PPU has 12 video decode cores (the Zhenwu 810E has 10). Optimal performance is achieved with 12 threads. Increasing the thread count further may yield a slight performance boost, but it will not be significant. Performance scales up to 7 PPU processes and does not increase further.
The PPU has 4 video encode cores, so 4 threads achieve optimal performance.

nvJPEG:

The PPU SDK is compatible with nvJPEG. Test JPEG functionality using the nvJPEG samples. nvJPEG official website: https://developer.nvidia.com/nvjpeg

nvJPEG samples: https://github.com/NVIDIA/CUDALibrarySamples/tree/master/nvJPEG/

Build:

The decoder is in the nvJPEG-Decoder directory, and the encoder is in the nvJPEG-Encoder directory.

$ mkdir build
$ cd build
$ cmake ..
$ make

Run:

# Decode. Assumes the JPEG image set is in the 1080p/ directory and the batch size is 128.
./nvjpegDecoder -i 1080p/ -b 128

# Encode (the sample actually decodes first, then encodes).
./nvJPEG_encoder -i input.jpg  -o output.jpg

Note:

The PPU has 4 JPEG encode cores, so 4 threads achieve optimal performance.

The PPU has 8 JPEG decode cores. Batch mode automatically enables 8 internal threads for optimal performance.

4. Specifications

Video decode

Supports the cuvid decoder.
Codecs:
- HEVC (H.265) - ITU-T Rec. H.265 (04/2013), ISO/IEC 23008-2
  - Main Profile, Level 5.1, High Tier
  - Main10 Profile, Level 5.1, High Tier
  - Main Still Profile
- VP9 - vp9-bitstream-specification-v0.6-20160331-draft
  - Profile 0, 8-bit
  - Profile 2, 10-bit
- AVC (H.264) - ITU-T Rec. H.264 (03/2010) / ISO/IEC 14496-10
  - Main Profile, levels 1 - 5.2
  - High Profile, levels 1 - 5.2
  - High 10 Profile, levels 1 - 5.2
  - Baseline Profile, levels 1 - 5.2
- AV1 Bitstream & Decoding Process Specification Version 1.0.0 with Errata 1
  - Main Profile, Level 5.1
- AVS2
Supports resolutions up to 8192x8192.
Supports up to 192 concurrent FHD streams.

Video encode

Supports the nvenc encoder.
Codecs:
- AVC (H.264): Spec Version 12:ISO/IEC 14496-10 / ITU-T Rec. H.264 (03/2010)
  - Baseline Profile, levels 1 – 5.2
  - Main Profile, levels 1 - 5.2
  - High Profile, levels 1 – 5.2
  - High 10 Profile, levels 1 - 5.2
- HEVC (H.265): ITU-T Rec. H.265 (04/2013), ISO/IEC 23008-2
  - Main Profile, Level 5.1, High Tier
  - Main10 profile, Level 5.1, High Tier
  - Main Still Profile
- AV1 Bitstream Specification Version 1.0.0 with Errata 1
  - Main Profile, Level 5.1
Supports resolutions up to 4K
Supports RGB input format (converted to YUV420 via inlinePP)
Supports crop, scale, and rotate with inlinePP
Supports up to 32 concurrent FHD streams.

JPEG

Supports the nvJPEG decoder and encoder.
Supports resolutions up to 32Kx32K.
Supports RGB format input and output with inlinePP
Supports crop, scale, and rotate with inlinePP
Supports up to 960 FPS at UHD resolution.

Image processing

Supports the npp library for 2D image processing.

5. Known issues

Video decode

Supports cuvid decode mode but does not support raw nvdec mode. Both modes offer the same hardware acceleration capabilities, but cuvid mode is simpler to use.
Legacy formats such as MPEG1, MPEG2, MPEG4, VC1, and VP8 are not supported.

Video encode

YUV444 12-bit is not supported.

JPEG

Lossless JPEG is not supported.
JPEG2000 is not supported.

2D image

npp supports only the Image Process API and does not support the Signal Process API.