[FFmpeg-devel] [PATCH] lavc/mediacodec: add hwaccel support

[Matthieu Bouron]

On Fri, Jun 24, 2016 at 06:18:02PM +0200, Michael Niedermayer wrote:
> On Fri, Jun 24, 2016 at 11:17:41AM +0200, Matthieu Bouron wrote:
> > On Thu, Apr 07, 2016 at 02:51:44PM +0200, Matthieu Bouron wrote:
> > > On Wed, Mar 23, 2016 at 6:16 PM, Matthieu Bouron <matthieu.bouron at gmail.com>
> > > wrote:
> > > 
> > > >
> > > >
> > > > On Tue, Mar 22, 2016 at 10:04 AM, Matthieu Bouron <
> > > > matthieu.bouron at gmail.com> wrote:
> > > >
> > > >>
> > > >>
> > > >> On Fri, Mar 18, 2016 at 5:50 PM, Matthieu Bouron <
> > > >> matthieu.bouron at gmail.com> wrote:
> > > >>
> > > >>> From: Matthieu Bouron <matthieu.bouron at stupeflix.com>
> > > >>>
> > > >>> ---
> > > >>>
> > > >>> Hello,
> > > >>>
> > > >>> The following patch add hwaccel support to the mediacodec (h264) decoder
> > > >>> by allowing
> > > >>> the user to render the output frames directly on a surface.
> > > >>>
> > > >>> In order to do so the user needs to initialize the hwaccel through the
> > > >>> use of
> > > >>> av_mediacodec_alloc_context and av_mediacodec_default_init functions.
> > > >>> The later
> > > >>> takes a reference to an android/view/Surface as parameter.
> > > >>>
> > > >>> If the hwaccel successfully initialize, the decoder output frames pix
> > > >>> fmt will be
> > > >>> AV_PIX_FMT_MEDIACODEC. The following snippet of code demonstrate how to
> > > >>> render
> > > >>> the frames on the surface:
> > > >>>
> > > >>>     AVMediaCodecBuffer *buffer = (AVMediaCodecBuffer *)frame->data[3];
> > > >>>     av_mediacodec_release_buffer(buffer, 1);
> > > >>>
> > > >>> The last argument of av_mediacodec_release_buffer enable rendering of the
> > > >>> buffer on the surface (or not if set to 0).
> > > >>>
> > > >>> Regarding the internal changes in the mediacodec decoder:
> > > >>>
> > > >>> MediaCodec.flush() discards both input and output buffers meaning that if
> > > >>> MediaCodec.flush() is called all output buffers the user has a reference
> > > >>> on are
> > > >>> now invalid (and cannot be used).
> > > >>> This behaviour does not fit well in the avcodec API.
> > > >>>
> > > >>> When the decoder is configured to output software buffers, there is no
> > > >>> issue as
> > > >>> the buffers are copied.
> > > >>>
> > > >>> Now when the decoder is configured to output to a surface, the user
> > > >>> might not
> > > >>> want to render all the frames as fast as the decoder can go and might
> > > >>> want to
> > > >>> control *when* the frame are rendered, so we need to make sure that the
> > > >>> MediaCodec.flush() call is delayed until all the frames the user retains
> > > >>> has
> > > >>> been released or rendered.
> > > >>>
> > > >>> Delaying the call to MediaCodec.flush() means buffering any inputs that
> > > >>> come
> > > >>> the decoder until the user has released/renderer the frame he retains.
> > > >>>
> > > >>> This is a limitation of this hwaccel implementation, if the user retains
> > > >>> a
> > > >>> frame (a), then issue a flush command to the decoder, the packets he
> > > >>> feeds to
> > > >>> the decoder at that point will be queued in the internal decoder packet
> > > >>> queue
> > > >>> (until he releases the frame (a)). This scenario leads to a memory usage
> > > >>> increase to say the least.
> > > >>>
> > > >>> Currently there is no limitation on the size of the internal decoder
> > > >>> packet
> > > >>> queue but this is something that can be added easily. Then, if the queue
> > > >>> is
> > > >>> full, what would be the behaviour of the decoder ? Can it block ? Or
> > > >>> should it
> > > >>> returns something like AVERROR(EAGAIN) ?
> > > >>>
> > > >>> About the other internal decoder changes I introduced:
> > > >>>
> > > >>> The MediaCodecDecContext is now refcounted (using the lavu/atomic api)
> > > >>> since
> > > >>> the (hwaccel) frames can be retained by the user, we need to delay the
> > > >>> destruction of the codec until the user has released all the frames he
> > > >>> has a
> > > >>> reference on.
> > > >>> The reference counter of the MediaCodecDecContext is incremented each
> > > >>> time an
> > > >>> (hwaccel) frame is outputted by the decoder and decremented each time a
> > > >>> (hwaccel) frame is released.
> > > >>>
> > > >>> Also, when the decoder is configured to output to a surface the pts that
> > > >>> are
> > > >>> given to the MediaCodec API are now rescaled based on the codec_timebase
> > > >>> as
> > > >>> those timestamps values are propagated to the frames rendered on the
> > > >>> surface
> > > >>> since Android M. Not sure if it's really useful though.
> > > >>>
> > > >>> On the performance side:
> > > >>>
> > > >>> On a nexus 5, decoding an h264 stream (main profile) 1080p at 60fps:
> > > >>>   - software output + rgba conversion goes at 59~60fps
> > > >>>   - surface output + render on a surface goes at 100~110fps
> > > >>>
> > > >>>
> > > >> [...]
> > > >>
> > > >> Patch updated with the following differences:
> > > >>   * the public mediacodec api is now always built (not only when
> > > >> mediacodec is available) (and the build when mediacodec is not available
> > > >> has been fixed)
> > > >>   * the documentation of av_mediacodec_release_buffer has been improved a
> > > >> bit
> > > >>
> > > >
> > > > Patch updated with the following differences:
> > > >   MediaCodecBuffer->released type is now a volatile int (instead of a int*)
> > > >   MediaCodecContext->refcount type is now a volatile int (instead of a
> > > > int*)
> > > >
> > > 
> > > Ping.
> > 
> > Rebased patch attached.
> > 
> > Matthieu
> 
> >  configure                       |    1 
> >  libavcodec/Makefile             |    6 
> >  libavcodec/allcodecs.c          |    1 
> >  libavcodec/mediacodec.c         |  133 ++++++++++++++++++++
> >  libavcodec/mediacodec.h         |   88 +++++++++++++
> >  libavcodec/mediacodec_surface.c |   66 +++++++++
> >  libavcodec/mediacodec_surface.h |   31 ++++
> >  libavcodec/mediacodec_wrapper.c |    5 
> >  libavcodec/mediacodecdec.c      |  265 +++++++++++++++++++++++++++++++++-------
> >  libavcodec/mediacodecdec.h      |   17 ++
> >  libavcodec/mediacodecdec_h264.c |   44 +++++-
> >  libavcodec/version.h            |    2 
> 
> >  libavutil/pixdesc.c             |    4 
> >  libavutil/pixfmt.h              |    2 
> >  libavutil/version.h             |    2 
> 
> libavutil changes look good, i dont know mediacodec so cannot easily
> comment about the rest

Thanks.

I'll wait another 3 days and if there is no objection I will push the
patch.

Matthieu

[...]