HgDeepStream User Guide (v2.1)

2.1 Supported configurations and dependencies

2.2 Install Gstreamer dependencies

Ubuntu:

apt-get update &&
apt-get install -y autoconf automake libtool autopoint pkg-config \
                 bison nasm meson python3-pip flex libxml2-dev \
                 libglib2.0-dev libjson-glib-dev \
                 libgirepository1.0-dev \
                 python3-gi python-gi-dev

pip install --upgrade pip==26.0.1
pip3 install --upgrade setuptools==69.5.0
PYGOBJECT_WITHOUT_PYCAIRO=1 pip3 install pygobject==3.44.2 --no-build-isolation

2.3 Build and install Gstreamer

Ubuntu users can install Gstreamer from source or with apt.

git clone -b 1.20.3 --depth 1 https://github.com/GStreamer/gstreamer.git

GENERAL_OPTIONS="-Dbase=enabled \
                    -Dgood=enabled \
                    -Dlibav=disabled \
                    -Dges=disabled \
                    -Drtsp_server=enabled \
                    -Dugly=enabled \
                    -Dgtk_doc=disabled \
                    -Dgst-examples=disabled \
                    -Dtests=disabled \
                    -Dexamples=disabled \
                    -Dgpl=enabled \
                    -Dqt5=disabled \
                    -Dlibnice=disabled \
                    -Dtls=disabled \
                    -Dpython=enabled \
                    "
BASE_OPTIONS="-Dgst-plugins-base:gl=disabled \
                    -Dgst-plugins-base:x11=disabled \
                    -Dgst-plugins-base:examples=disabled \
                    -Dgst-plugins-base:tests=disabled \
                    -Dgst-plugins-base:orc=disabled \
                    -Dgst-plugins-base:ogg=disabled \
                    -Dgst-plugins-base:opus=disabled \
                    -Dgst-plugins-base:vorbis=disabled \
                    "

GOOD_OPTIONS="-Dgst-plugins-good:doc=disabled \
                    -Dgst-plugins-good:examples=disabled \
                    -Dgst-plugins-good:tests=disabled \
                    -Dgst-plugins-good:alpha=disabled \
                    -Dgst-plugins-good:goom=disabled \
                    -Dgst-plugins-good:goom2k1=disabled \
                    -Dgst-plugins-good:oss=disabled \
                    -Dgst-plugins-good:oss4=disabled \
                    -Dgst-plugins-good:v4l2=disabled \
                    -Dgst-plugins-good:cairo=disabled \
                    "

BAD_OPTIONS="-Dgst-plugins-bad:doc=disabled  \
                    -Dgst-plugins-bad:examples=disabled \
                    -Dgst-plugins-bad:tests=disabled \
                    -Dgst-plugins-bad:orc=disabled \
                    -Dgst-plugins-bad:openh264=disabled \
                    -Dgst-plugins-bad:openjpeg=disabled \
                    -Dgst-plugins-bad:openmpt=disabled \
                    -Dgst-plugins-bad:gl=disabled \
                    -Dgst-plugins-bad:x11=disabled \
                    -Dgst-plugins-bad:opencv=disabled \
                    -Dgst-plugins-bad:openal=disabled \
                    -Dgst-plugins-bad:openexr=disabled \
                    -Dgst-plugins-bad:wayland=disabled \
                    -Dgst-plugins-bad:vulkan=disabled \
                    -Dgst-plugins-bad:nvcodec=disabled \
                    -Dgst-plugins-bad:videoparsers=enabled \
                    -Dgst-plugins-bad:ivfparse=enabled \
                    -Dgst-plugins-bad:x265=disabled \
                    "

UGLY_OPTIONS="-Dgst-plugins-ugly:orc=disabled \
                    -Dgst-plugins-ugly:doc=disabled \
                    -Dgst-plugins-ugly:x264=disabled \
                    "

RTSP_SERVER_OPTIONS="-Dgst-rtsp-server:doc=disabled \
                        -Dgst-rtsp-server:examples=enabled \
                        -Dgst-rtsp-server:tests=enabled"

PYTHON_OPTIONS="-Dgst-python:python=python3 \
                        -Dgst-python:plugin=enabled"

meson setup  ${GENERAL_OPTIONS} \
            ${BASE_OPTIONS} \
            ${GOOD_OPTIONS} \
            ${BAD_OPTIONS} \
            ${RTSP_SERVER_OPTIONS} \
            ${UGLY_OPTIONS} \
            ${PYTHON_OPTIONS} \
            build \
            gstreamer

ninja -C build install

apt-get update && apt-get install -y  libgstreamer1.0-dev \
                    gstreamer1.0-plugins-good \
                    gstreamer1.0-plugins-bad \
                    gstreamer1.0-plugins-ugly  \
                    gstreamer1.0-tools \
                    libgstreamer-plugins-bad1.0-dev \
                    gstreamer1.0-rtsp \
                    gir1.2-gst-rtsp-server-1.0 \
                    python3-gst-1.0

2.4 Install HgDeepStream with pip

To use HgDeepStream, install the pyhgds package from the T-Head pip repository.

2.5 Configure runtime environment variables

Ubuntu:

# Get the Python version number, 3.10 or 3.12
PY_VERSION=$(python3 -c "import sys; print('.'.join(map(str, sys.version_info[:2])))")

# Plugin search path
export GST_PLUGIN_PATH=/usr/local/lib/x86_64-linux-gnu/:/usr/local/lib/python${PY_VERSION}/site-packages/pyhgds_extlibs/gst-plugins:$GST_PLUGIN_PATH

export LD_LIBRARY_PATH=/usr/local/lib/x86_64-linux-gnu:$LD_LIBRARY_PATH

#python gir search path
export GI_TYPELIB_PATH=/usr/lib/x86_64-linux-gnu/girepository-1.0:/usr/local/lib/x86_64-linux-gnu/girepository-1.0/:/usr/local/lib/girepository-1.0:$GI_TYPELIB_PATH

3.1 Hardware video decoding

3.2 Hardware video encoding

3.3 Memory/GPU memory transfer

3.4 Hardware-based image transformation

4.1 Transcoding + format conversion

This example takes an h264 mp4 file as input. It uses hardware decoding, converts the image format to P010_10LE, and then uses hardware encoding to produce a 10-bit h265 mp4 file.

import sys
import os

import gi
import argparse
gi.require_version('Gst', '1.0')
gi.require_version('GLib', '2.0')
gi.require_version('GObject', '2.0')
from gi.repository import GLib, GObject, Gst

python_version = f"{sys.version_info.major}.{sys.version_info.minor}"
gst_env_value = f"/usr/local/lib/python{python_version}/site-packages/pyhgds_extlibs/gst-plugins"

if 'GST_PLUGIN_PATH' in os.environ:
    os.environ['GST_PLUGIN_PATH'] = gst_env_value + ':'+ os.environ['GST_PLUGIN_PATH']
else:
    os.environ['GST_PLUGIN_PATH'] = gst_env_value

def bus_call(bus, message, loop):
    t = message.type
    if t == Gst.MessageType.EOS:
        sys.stdout.write("End-of-stream\n")
        loop.quit()
    elif t == Gst.MessageType.ERROR:
        err, debug = message.parse_error()
        sys.stderr.write("Error: %s: %s\n" % (err, debug))
        loop.quit()
    return True

def transcode_main(args):
    # Parse command line arguments
    parser = argparse.ArgumentParser(description='Transcode MP4 file from H.264 to H.265')
    parser.add_argument('-i', '--input', required=True, help='Input MP4 file path')
    parser.add_argument('-o', '--output', required=True, help='Output MP4 file path')
    parsed_args = parser.parse_args(args[1:])

     # Check if input file exists
    if not os.path.exists(parsed_args.input):
        print(f"Error: Input file '{parsed_args.input}' does not exist.")
        return -1

    # Initialize GStreamer
    GObject.threads_init()
    Gst.init(None)

# Create the pipeline
    pipeline = Gst.Pipeline.new('video-transcoder')

    # Create elements
    filesrc = Gst.ElementFactory.make('filesrc', 'file-source')
    qtdemux = Gst.ElementFactory.make('qtdemux', 'qtdemux')
    h264parse = Gst.ElementFactory.make('h264parse', 'h264-parser')
    h264_hgdec = Gst.ElementFactory.make('h264_hgdec', 'h264-decoder')
    hgvideoconvert = Gst.ElementFactory.make('hgvideoconvert', 'video-converter')
    filter1 = Gst.ElementFactory.make("capsfilter", "filter1")
    h265_hgenc = Gst.ElementFactory.make('h265_hgenc', 'h265-encoder')
    h265_parse = Gst.ElementFactory.make('h265parse', 'h265-parse')

    mux = Gst.ElementFactory.make('mp4mux', 'mp4-mux')
    filesink = Gst.ElementFactory.make('filesink', 'file-sink')

    # Check if all elements were created successfully
    if not all([pipeline, filesrc, qtdemux, h264parse, h264_hgdec, hgvideoconvert, h265_hgenc, mux, h265_parse, filesink]):
        sys.stderr.write("Failed to create one or more elements\n")
        return -1

 # Add elements to the pipeline
    pipeline.add(filesrc)
    pipeline.add(qtdemux)
    pipeline.add(h264parse)
    pipeline.add(h264_hgdec)
    pipeline.add(hgvideoconvert)
    pipeline.add(filter1)
    pipeline.add(h265_hgenc)
    pipeline.add(mux)
    pipeline.add(h265_parse)
    pipeline.add(filesink)

    # Set properties
    filesrc.set_property('location', parsed_args.input)
    caps1 = Gst.Caps.from_string('video/x-raw(memory:NVMM), format=P010_10LE')
    filter1.set_property("caps", caps1)

    def on_pad_added(demuxer, pad):
        print("on_pad_added")
        if pad.get_property("template").name_template == "video_%u":
            pad.link(h264parse.get_static_pad("sink"))
        elif pad.get_property("template").name_template == "audio_%u":
            # ignore audio stream
            pass
    qtdemux.connect("pad-added", on_pad_added)

    # Set the output location
    filesink.set_property('location', parsed_args.output)

    filesrc.link(qtdemux)
    h264parse.link(h264_hgdec)
    h264_hgdec.link(hgvideoconvert)
    hgvideoconvert.link(filter1)
    filter1.link(h265_hgenc)
    h265_hgenc.link(h265_parse)
    h265_parse.link(mux)
    mux.link(filesink)

     # Create a GLib loop to handle events
    loop = GLib.MainLoop()

    # Add a message handler to the bus
    bus = pipeline.get_bus()
    bus.add_signal_watch()
    bus.connect("message", bus_call, loop)

     # Start playing the pipeline
    ret = pipeline.set_state(Gst.State.PLAYING)
    if ret == Gst.StateChangeReturn.FAILURE:
        sys.stderr.write("Unable to set the pipeline to the playing state.\n")
        return -1

 # Wait until error or EOS
    try:
        loop.run()
    except KeyboardInterrupt:
        pass
 # Clean up
    pipeline.set_state(Gst.State.NULL)

    # Print output file location
    print(f"Transcoding completed. Output file saved to: {os.path.abspath(parsed_args.output)}")

if __name__ == '__main__':
    sys.exit(transcode_main(sys.argv))

Run the command python3 transcode_mp4.py -i input.mp4 -o output.mp4.

5.1. Transcoding + format conversion

This example decodes an h264 mp4 file, crops the decoded images, and rotates them 90° clockwise. It then converts the images to the P010_10LE data format and finally encodes them into a 10-bit h265 mp4 file.

gst-launch-1.0 filesrc location=input.mp4 ! qtdemux ! h264parse ! h264_hgdec gpu-id=0 ! queue ! hgvideoconvert src-crop=100:100:1280:720 dest-crop=100:100:640:360 flip-method=clockwise  ! 'video/x-raw(memory:NVMM), format=P010_10LE, width=1920, height=1080' ! queue ! h265_hgenc gpu-id=0 ! h265parse !  mp4mux ! filesink location=output.mp4

7.1 Links

GStreamer official documentation

7.2 x11 image display configuration

If a local X server is available, you can use the official GStreamer plugin ximagesink to display images.

Follow these steps:

Step 1: When building Gstreamer as described in [2.3 Build and install Gstreamer], enable the x11 configuration -Dgst-plugins-base:x11=enabled.

Step 2: Configure the DISPLAY environment variable.

Step 3: Convert the image format to BGRx, move the GPU memory data to the host, and then pass it to the ximagesink plugin.

Example command:

gst-launch-1.0 filesrc location=input.mp4 ! qtdemux ! h264parse ! h264_hgdec gpu-id=0 ! hgvideoconvert flip-method=clockwise ! 'video/x-raw(memory:NVMM), format=BGRx, width=1920, height=1080' ! hgcudadownload ! ximagesink

7.3 RTSP stream publishing and playback

GStreamer provides RTSP tools that help developers set up an RTSP server. When you build Gstreamer as described in [2.3 Build and install Gstreamer], you must enable the build option -Drtsp_server=enabled.

#!/usr/bin/python3
"""
The server will stream test video at rtsp://127.0.0.1:8554/test
"""

import sys
import gi
import os

gi.require_version('Gst', '1.0')
gi.require_version('GLib', '2.0')
gi.require_version('GObject', '2.0')

try:
    gi.require_version('GstRtspServer', '1.0')
except ValueError:
    sys.stderr.write("GstRtspServer 1.0 not found. Install gstreamer1.0-rtsp-server.\n")
    sys.exit(1)

from gi.repository import GLib, GObject, Gst, GstRtspServer

python_version = f"{sys.version_info.major}.{sys.version_info.minor}"
gst_env_value = f"/usr/local/lib/python{python_version}/site-packages/pyhgds_extlibs/gst-plugins"

if 'GST_PLUGIN_PATH' in os.environ:
    os.environ['GST_PLUGIN_PATH'] = gst_env_value + ':'+ os.environ['GST_PLUGIN_PATH']
else:
    os.environ['GST_PLUGIN_PATH'] = gst_env_value

def main():
    Gst.init(None)

    loop = GLib.MainLoop()
    server = GstRtspServer.RTSPServer.new()
    mounts = server.get_mount_points()
    factory = GstRtspServer.RTSPMediaFactory.new()
    factory.set_launch(
        "( videotestsrc is-live=1 ! video/x-raw, format=NV12, width=1280, height=720 ! "
        "hgcudaupload ! h264_hgenc ! rtph264pay name=pay0 pt=96 )"
    )

    factory.set_shared(True)

    mounts.add_factory("/test", factory)
    mounts = None

    server.attach(None)

    print("stream ready at rtsp://127.0.0.1:8554/test")

    try:
        loop.run()
    except KeyboardInterrupt:
        print("Interrupted by user")

    loop = None
    server = None

    return 0

if __name__ == '__main__':
    sys.exit(main())

7.4 glib upgrade

# e.g. 2.78.3
git clone -b 2.78.3 --depth 1 https://github.com/GNOME/glib.git
cd glib
meson build --prefix=/usr
ninja -C build install

7.5 FFmpeg-related plugin usage notes

Gstreamer provides FFmpeg-related plugins. When you build Gstreamer as described in [2.3 Build and install Gstreamer], you must enable the build option -Dlibav=enabled. During the build, you must also configure the environment variable export PKG_CONFIG_PATH=/xxx/ffmpeg/lib/pkgconfig:$PKG_CONFIG_PATH.

For details, see [Video FFmpeg User Guide]. To build Gstreamer 1.20.3 with FFmpeg 6.1.2, apply the following patch:

diff --git a/subprojects/gst-libav/ext/libav/gstavviddec.c b/subprojects/gst-libav/ext/libav/gstavviddec.c
index b957d22..cc38398
--- a/subprojects/gst-libav/ext/libav/gstavviddec.c
+++ b/subprojects/gst-libav/ext/libav/gstavviddec.c
@@ -567,7 +567,7 @@ gst_ffmpegviddec_set_format (GstVideoDecoder * decoder,
   if (ffmpegdec->max_threads == 0) {
     /* When thread type is FF_THREAD_FRAME, extra latency is introduced equal
      * to one frame per thread. We thus need to calculate the thread count ourselves */
-    if ((!(oclass->in_plugin->capabilities & AV_CODEC_CAP_AUTO_THREADS)) ||
+    if ((!(oclass->in_plugin->capabilities & AV_CODEC_CAP_OTHER_THREADS)) ||
         (ffmpegdec->context->thread_type & FF_THREAD_FRAME))
       ffmpegdec->context->thread_count =
           MIN (gst_ffmpeg_auto_max_threads (), 16);