Revision control

Copy as Markdown

// Copyright (C) 2024 The Qt Company Ltd.
// SPDX-License-Identifier: LicenseRef-Qt-Commercial OR GFDL-1.3-no-invariants-only
/*!
\page qtmultimedia-gstreamer.html
\title Qt Multimedia GStreamer backend
\brief Platform notes for the GStreamer backend
This page covers limitations and customization points of the GStreamer backend
of Qt Multimedia, the default media backend on embedded Linux.
\section1 Architectural Considerations
GStreamer is the default media backend on embedded Linux mainly because it is
the framework most embedded board vendors prioritize for hardware accelerated
media support. FFmpeg may also be suitable on some platforms, so users are
encouraged to experiment with both media backends to find out which works
best for them.
Qt Multimedia is not a general purpose streaming framework and not necessarily
the architecturally best way to use GStreamer with Qt. Developers, who need a
high degree of control over the GStreamer pipeline, but only want to show the
video output Qt, may want to consider using GStreamer's
\section1 Limitations and Known Issues
GStreamer is not bundled with Qt, but it is typically deployed with the Linux distribution.
\list
\li Certain bugs may be due to the GStreamer version being used. We recommend to use the latest
GStreamer bug fix release on your platform.
\li Certain bugs may also be related to the libraries used by GStreamer, like Pulseaudio. Most
notably Pulseaudio v16 has a known bug that causes the GStreamer pipeline to hang and requires
backports of these two patches:
\list
\endlist
This bug currently affects most mainstream Linux distributions, including Ubuntu 22.04, 23.10 and
24.04, Debian 11 and 12, as well as Fedora 39 and 40.
\li Switching camera devices or cameras during recording is not supported
\li Seeking, playback rates, loop, switching sinks have known bugs.
\li Audio features requires PulseAudio. See \l{Qt Multimedia on Linux}{Linux platform notes}
for details.
\li Some embedded boards needs individual configurations to solve pipeline
negotiation/linking issues and improve performance. We recommend developers to
experiment with the environment variables listed under \l{Customization points}
when facing such problems. Qt's Yocto meta layer files contain recommended
configurations for certain combinations of boards, BSPs and Boot to Qt
versions. Here are links to the dev branch versions:
\list
\endlist
\li Some NXP i.MX boards expose a v4l2 device named imx-capture, which can cause
errors when used with v4l2src in GStreamer. To prevent Qt Multimedia from
adding imx-capture to the list of available camera devices, set the environment
variable QT_GSTREAMER_SKIP_IMXCAPTURE=1. Note that this is considered as
private API which will be removed when NXP fixes the problems with imx-capture.
\li Video playback or camera streaming using system memory video frames or
software conversion can result in dropped frames and unresponsiveness due to
heavy CPU load. If the GStreamer decoder or camera device outputs DMA-BUF- or
GLMemory-backed video frames in one of our supported pixel formats, Qt can
render them without additional conversion or copying. GStreamer selects a
format and memory type based on the capabilities of its pipeline elements.
Different ways of controlling which elements are added to the pipeline are
described under \l{Customization points}. See also
\l{Specifying I/O mode for v4l2 cameras}.
\endlist
\section1 Customization points
Qt Multimedia provides certain customization points to allow access to the
underlying GStreamer pipeline. The entry point is \l QGStreamerInterface.
Additional customizations can be done by setting environment variables.
\warning The customization points and Qt specific environment variables listed
here, are considered as private APIs and may be subject to change. Be aware
that environment variables are by default inherited by child processes and can
have unintended consequences.
\section2 External OpenGL texture rendering
Qt imports DMA-BUF-backed video frames into the OpenGL graphics pipeline by
binding the underlying buffer as an EGL image and exposing it to shaders as an
OpenGL texture. By default, Qt samples these textures using the GL_TEXTURE_2D
texture target and standard shader code. Some embedded boards performs better
with the GL_TEXTURE_EXTERNAL_OES texture target instead, together with a
dedicated fragment shader. This behavior can be enabled by setting the
following environment variable:
\code
QT_MULTIMEDIA_FORCE_GL_TEXTURE_EXTERNAL_OES=1
\endcode
\section2 Specifying I/O mode for v4l2 cameras
When using a v4l2 capture device with a QCamera or QML Camera, the
property of the underlying v4l2src GStreamer element affects whether or not it
outputs DMA-BUF-backed video frames that allows zero-copy rendering. In some
cases the default auto mode chooses mmap mode even though the camera driver
supports dmabuf mode. The io-mode property int value can be set explicitly via
the environment variable below. Here's an example showing how to set io-mode to
dmabuf with the enum GST_V4L2_IO_DMABUF:
\code
QT_GSTREAMER_V4L2SRC_IOMODE=4
\endcode
\section2 Selecting specific GStreamer elements
A GStreamer distribution can include multiple different plugins that can
potentially perform the same task in the media pipeline. GStreamer
automatically selects elements like decoders and demuxers based on their
feature rank, which can be overridden via a GStreamer environment variable.
Here's an example that ranks the v4l2h264dec hardware decoder higher than any
H.264 software decoder, while also making sure the aiurdemux demuxer isn't
selected:
\code
GST_PLUGIN_FEATURE_RANK=v4l2h264dec:MAX,aiurdemux:NONE
\endcode
To inspect which elements are being created during runtime, enable debug logging
level 4 for GstElementFactory:
\code
GST_DEBUG=GST_ELEMENT_FACTORY:4
\endcode
Furthermore, you can visually inspect the whole GStreamer pipeline by
specifying a folder for \c {.dot} graph files that will be dumped at
various events, for example, changes in playback state:
\code
GST_DEBUG_DUMP_DOT_DIR=/root/graphs
\endcode
Read more about these environment variables in the GStreamer documentation:
\list
\endlist
\section3 Specifying hardware conversion elements
If the media source of a GStreamer pipeline cannot provide video buffers in a
pixel format supported by Qt Multimedia, the pipeline will use software
conversion unless it contains a suitable hardware accelerated video conversion
element. To specify a hardware conversion element to be used, set it as a
{GStreamer pipeline description} to the following environment variable:
\code
QT_GSTREAMER_OVERRIDE_VIDEO_CONVERSION_ELEMENT
\endcode
Some vendor specific conversion elements (imxvideoconvert_g2d, nvvidconv) are
added to the pipeline by default if available. This can cause unnecessary
conversion if they aren't needed, and can be disabled by specifying a GStreamer
{identity} element using the same environment variable:
\code
QT_GSTREAMER_OVERRIDE_VIDEO_CONVERSION_ELEMENT=identity
\endcode
GStreamer can also perform format conversion with OpenGL elements, providing Qt
with GLMemory-backed video frames:
\code
QT_GSTREAMER_OVERRIDE_VIDEO_CONVERSION_ELEMENT=glupload ! glcolorconvert
\endcode
\section2 Raw pipeline access
The \c{GstPipeline} underlying the \l{QMediaPlayer} and \l{QMediaCaptureSession} can be accessed.
\warning This is an unsafe API, as the pipeline is still managed by the Qt implementation. Great
care is required when using this API.
\code
#include <QtMultimedia/spi/qgstreamerinterface.h>
[...]
QMediaPlayer player;
GstPipeline *pipeline = QGStreamerInterface::instance()->gstPipeline(&player);
[...]
QMediaCaptureSession session;
GstPipeline *pipeline = QGStreamerInterface::instance()->gstPipeline(&session);
\endcode
\section2 Custom GStreamer elements as sinks and sources
It is possible to create GStreamer elements from a GStreamer pipeline description and wrap them
inside a \l QGStreamerVideoSource or \l QAudioDevice:
\code
#include <QtMultimedia/spi/qgstreamerinterface.h>
#include <QtMultimedia/spi/qgstreamervideosource.h>
[...]
QMediaCaptureSession session;
session.setVideoSink(wid.videoSink());
QGStreamerVideoSource videoSource(QStringLiteral("videotestsrc is-live=true ! gamma gamma=2.0"));
session.setNativeVideoSource(&videoSource);
videoSource.start();
\endcode
\section2 QMediaPlayer: custom sources
The \c{QMediaPlayer} accepts a GStreamer pipeline description as source URI:
\code
QMediaPlayer player;
player.setSource(u"gstreamer-pipeline: videotestsrc name=testsrc"_s);
\endcode
This will try to compile the pipeline description to use as source in the QMediaPlayer and will be
automatically connected to the sinks of the QMediaPlayer.
\warning Hic sunt dracones! Custom pipelines are an experimental feature: the custom pipelines do
not map well to QMediaPlayer APIs, most notably the media status, metadata APIs, and transport state.
Most calls will directly map to the GStreamer pipeline, which can lead to undefined behavior
depending on the pipeline. In most cases, the \c{gstreamer-pipeline:} may not be the right choice
for application code: for arbitrary video sources, the \c{QMediaCaptureSession} with a custom camera
(see above) is the preferred choice. For arbitrarily complex pipelines that only want to draw into a
Qt/QML GUI, GStreamer's \c{qml6glsink} (see below) may be a more robust choice.
*/