[FFmpeg-devel] [PATCH] avcodec/h264, videotoolbox: handle streams with multiple/changing PPS

[Aman Gupta]

On Tue, Feb 14, 2017 at 5:14 PM Aman Gupta <ffmpeg at tmm1.net> wrote:

> From: Aman Gupta <aman at tmm1.net>
>
> The videotoolbox hwaccel only receives SLICE and IDR_SLICE NALUs. This
> works fine most of the time, but some streams fail to decode because
> changes in PPS are not propagated to the VT decoder.


Perhaps SPS can change mid-stream, and should be passed into the VT decoder
as well? I'm not familiar enough with h264 to know.


>
> The failures in this case are incredibly annoying, as VTDecodeFrame()
> still returns noErr. Simiarly the decoder callback is invoked with noErr,
> and a NULL imageBuffer. Even though all the VT apis indicate success, no
> frames are actually decoded.
>
> When running ffmpeg via lldb however, some internal VT errors and
> warnings show up all of a sudden. These suggest that the bitstream is
> failing some internal consistency checks.
>
>     $ ffmpeg -y -loglevel error -threads 1 -hwaccel videotoolbox \
>              -i http://tmm1.s3.amazonaws.com/h264.ts -map v -f null
> /dev/null
>     ...
>     [h264 @ 0x7fdadc000000] hardware accelerator failed to decode picture
>     Error while decoding stream #0:0: Unknown error occurred
>     vt decoder cb: output image buffer is null
>     ...
>
>     $ lldb -- ffmpeg ...
>     ...
>     ffmpeg[49384:2009219] GVA error: AVC_RBSP::parseSliceHeader error
>     ffmpeg[49384:2009219] GVA error: pushPicture parseSliceHeader
>     vt decoder cb: output image buffer is null
>     ffmpeg[49384:2009219] GVA warning: OutputQueueReadyCallback status =
> 1, buffer == 0x0
>     [h264 @ 0x10300a200] hardware accelerator failed to decode picture:
> Unknown error occurred
>     Error while decoding stream #0:0: Unknown error occurred
>     ...
>
> After this patch, there are no more errors and the sample decodes as
> expected.
> ---
>  libavcodec/h264dec.c      | 7 +++++++
>  libavcodec/videotoolbox.c | 8 +++++---
>  2 files changed, 12 insertions(+), 3 deletions(-)
>
> diff --git a/libavcodec/h264dec.c b/libavcodec/h264dec.c
> index 41c0964..af8b256 100644
> --- a/libavcodec/h264dec.c
> +++ b/libavcodec/h264dec.c
> @@ -755,6 +755,13 @@ FF_ENABLE_DEPRECATION_WARNINGS
>                                                         nal->size_bits);
>              if (ret < 0 && (h->avctx->err_recognition & AV_EF_EXPLODE))
>                  goto end;
> +            if (avctx->hwaccel && avctx->hwaccel->pix_fmt ==
> AV_PIX_FMT_VIDEOTOOLBOX) {
> +                ret = avctx->hwaccel->decode_slice(avctx,
> +                                                   nal->raw_data,
> +                                                   nal->raw_size);
> +                if (ret < 0)
> +                    goto end;
> +            }
>              break;
>          case H264_NAL_AUD:
>          case H264_NAL_END_SEQUENCE:
> diff --git a/libavcodec/videotoolbox.c b/libavcodec/videotoolbox.c
> index 1288aa5..1b5dffd 100644
> --- a/libavcodec/videotoolbox.c
> +++ b/libavcodec/videotoolbox.c
> @@ -138,8 +138,6 @@ int ff_videotoolbox_h264_start_frame(AVCodecContext
> *avctx,
>      VTContext *vtctx = avctx->internal->hwaccel_priv_data;
>      H264Context *h  = avctx->priv_data;
>
> -    vtctx->bitstream_size = 0;
> -
>      if (h->is_avc == 1) {
>          return videotoolbox_buffer_copy(vtctx, buffer, size);
>      }
> @@ -373,8 +371,12 @@ static int videotoolbox_h264_end_frame(AVCodecContext
> *avctx)
>  {
>      H264Context *h = avctx->priv_data;
>      AVFrame *frame = h->cur_pic_ptr->f;
> +    VTContext *vtctx = avctx->internal->hwaccel_priv_data;
> +    int ret;
>
> -    return videotoolbox_common_end_frame(avctx, frame);
> +    ret = videotoolbox_common_end_frame(avctx, frame);
> +    vtctx->bitstream_size = 0;
> +    return ret;
>  }
>
>  static int videotoolbox_mpeg_start_frame(AVCodecContext *avctx,
> --
> 2.10.1 (Apple Git-78)
>
>