FFmpeg: probe encoder pixel formats when none are declared


avcodec_get_supported_config(AV_CODEC_CONFIG_PIX_FORMAT) may report no
pixel formats at all, which means "unknown" rather than "nothing is
supported". Several encoders behave this way, e.g. *_mediacodec or
*_v4l2m2m.

findTargetFormat() then returned the source format unchanged, so a BGRA
source was handed to an encoder that may not it and avcodec_open2
failed. findTargetSWFormat() and findAndScoreTargetSWFormats() yielded
nothing for hw codecs exposing neither frame constraints nor pixel
formats, making VideoFrameEncoder reject them outright. And
findSWFormatScores() ranked such codecs last, so they were tried with
the wrong format anyway.

Determine the formats an encoder accepts by opening it once per
candidate format with a minimal codec context. Only the formats commonly
supported by mpeg2, mpeg4, h264, h265, vp8, vp9 and av1 are probed, at
the resolution the encoder will actually run at.
We cache the results to prevent excessive format probing.

Pick-to: 6.12
Change-Id: I8edec5750c21bfc4a03f6ed1ca3e39789b58225f
Reviewed-by: Artem Dyomin <artem.dyomin@qt.io>
commit9e16b640f5bc86aff245729b1e98a64c82e5e933
parent0b0821da1423842ffcd907ddf97bb57c90919811
authorTim Blechmann <tim.blechmann@qt.io>
committerTim Blechmann <tim.blechmann@qt.io>
commit timeMon, 14 Sep 2026 13:33:54 +0000