[FFmpeg-devel] [PATCH v4 0/6] Implement SEI parsing for QSV decoders

[Xiang, Haihao]

On Sun, 2022-06-26 at 23:41 +0000, ffmpegagent wrote:
> Missing SEI information has always been a major drawback when using the QSV
> decoders. I used to think that there's no chance to get at the data without
> explicit implementation from the MSDK side (or doing something weird like
> parsing in parallel). It turned out that there's a hardly known api method
> that provides access to all SEI (h264/hevc) or user data (mpeg2video).
> 
> This allows to get things like closed captions, frame packing, display
> orientation, HDR data (mastering display, content light level, etc.) without
> having to rely on those data being provided by the MSDK as extended buffers.
> 
> The commit "Implement SEI parsing for QSV decoders" includes some hard-coded
> workarounds for MSDK bugs which I reported:
> 
https://github.com/Intel-Media-SDK/MediaSDK/issues/2597#issuecomment-1072795311
> 
> But that doesn't help. Those bugs exist and I'm sharing my workarounds,
> which are empirically determined by testing a range of files. If someone is
> interested, I can provide private access to a repository where we have been
> testing this. Alternatively, I could also leave those workarounds out, and
> just skip those SEI types.
> 
> In a previous version of this patchset, there was a concern that payload
> data might need to be re-ordered. Meanwhile I have researched this carefully
> and the conclusion is that this is not required.
> 
> My detailed analysis can be found here:
> https://gist.github.com/softworkz/36c49586a8610813a32270ee3947a932
> 
> v3
> 
>  * frame.h: clarify doc text for av_frame_copy_side_data()
> 
> v2
> 
>  * qsvdec: make error handling consistent and clear
>  * qsvdec: remove AV_CODEC_ID_MPEG1VIDEO constants
>  * hevcdec: rename function to ff_hevc_set_side_data(), add doc text
> 
> v3
> 
>  * qsvdec: fix c/p error
> 
> softworkz (6):
>   avutil/frame: Add av_frame_copy_side_data() and
>     av_frame_remove_all_side_data()
>   avcodec/vpp_qsv: Copy side data from input to output frame
>   avcodec/mpeg12dec: make mpeg_decode_user_data() accessible
>   avcodec/hevcdec: make set_side_data() accessible
>   avcodec/h264dec: make h264_export_frame_props() accessible
>   avcodec/qsvdec: Implement SEI parsing for QSV decoders
> 
>  doc/APIchanges               |   4 +
>  libavcodec/h264_slice.c      |  98 ++++++++-------
>  libavcodec/h264dec.h         |   2 +
>  libavcodec/hevcdec.c         | 117 +++++++++---------
>  libavcodec/hevcdec.h         |   9 ++
>  libavcodec/mpeg12.h          |  28 +++++
>  libavcodec/mpeg12dec.c       |  40 +-----
>  libavcodec/qsvdec.c          | 234 +++++++++++++++++++++++++++++++++++
>  libavfilter/qsvvpp.c         |   6 +
>  libavfilter/vf_overlay_qsv.c |  19 ++-
>  libavutil/frame.c            |  67 ++++++----
>  libavutil/frame.h            |  32 +++++
>  libavutil/version.h          |   2 +-
>  13 files changed, 485 insertions(+), 173 deletions(-)
> 
> 
> base-commit: 6a82412bf33108111eb3f63076fd5a51349ae114
> Published-As: 
> https://github.com/ffstaging/FFmpeg/releases/tag/pr-ffstaging-31%2Fsoftworkz%2Fsubmit_qsv_sei-v4
> Fetch-It-Via: git fetch https://github.com/ffstaging/FFmpeg pr-ffstaging-
> 31/softworkz/submit_qsv_sei-v4
> Pull-Request: https://github.com/ffstaging/FFmpeg/pull/31
> 
> Range-diff vs v3:
> 
>  1:  c442597a35 ! 1:  7656477360 avutil/frame: Add av_frame_copy_side_data()
> and av_frame_remove_all_side_data()
>      @@ doc/APIchanges: libavutil:     2021-04-27
>        
>        API changes, most recent first:
>        
>      -+2022-05-26 - xxxxxxxxx - lavu 57.26.100 - frame.h
>      ++2022-05-26 - xxxxxxxxx - lavu 57.28.100 - frame.h
>       +  Add av_frame_remove_all_side_data(), av_frame_copy_side_data(),
>       +  AV_FRAME_TRANSFER_SD_COPY, and AV_FRAME_TRANSFER_SD_FILTER.
>       +
>      - 2022-05-23 - xxxxxxxxx - lavu 57.25.100 - avutil.h
>      -   Deprecate av_fopen_utf8() without replacement.
>      - 
>      + 2022-06-12 - xxxxxxxxxx - lavf 59.25.100 - avio.h
>      +   Add avio_vprintf(), similar to avio_printf() but allow to use it
>      +   from within a function taking a variable argument list as input.
>       
>        ## libavutil/frame.c ##
>       @@ libavutil/frame.c: FF_ENABLE_DEPRECATION_WARNINGS
>      @@ libavutil/frame.h: int av_frame_copy(AVFrame *dst, const AVFrame
> *src);
>       + * @param src a frame from which to copy the side data.
>       + * @param flags a combination of AV_FRAME_TRANSFER_SD_*
>       + *
>      -+ * @return >= 0 on success, a negative AVERROR on error.
>      ++ * @return 0 on success, a negative AVERROR on error.
>       + *
>       + * @note This function will create new references to side data buffers
> in src,
>       + * unless the AV_FRAME_TRANSFER_SD_COPY flag is passed.
>      @@ libavutil/version.h
>         */
>        
>        #define LIBAVUTIL_VERSION_MAJOR  57
>      --#define LIBAVUTIL_VERSION_MINOR  25
>      -+#define LIBAVUTIL_VERSION_MINOR  26
>      +-#define LIBAVUTIL_VERSION_MINOR  27
>      ++#define LIBAVUTIL_VERSION_MINOR  28
>        #define LIBAVUTIL_VERSION_MICRO 100
>        
>        #define LIBAVUTIL_VERSION_INT   AV_VERSION_INT(LIBAVUTIL_VERSION_MAJOR,
> \
>  2:  6f50d0bd57 = 2:  06976606c5 avcodec/vpp_qsv: Copy side data from input to
> output frame
>  3:  f682b1d695 = 3:  320a8a535c avcodec/mpeg12dec: make
> mpeg_decode_user_data() accessible
>  4:  995d835233 = 4:  e58ad6564f avcodec/hevcdec: make set_side_data()
> accessible
>  5:  ac8dc06395 = 5:  a57bfaebb9 avcodec/h264dec: make
> h264_export_frame_props() accessible
>  6:  27c3dded4d = 6:  3f2588563e avcodec/qsvdec: Implement SEI parsing for QSV
> decoders

Patchset LGTM,

-Haihao