[FFmpeg-devel] [PATCH] lavc/mediacodec: add hwaccel support
Michael Niedermayer
michael at niedermayer.cc
Fri Jun 24 18:18:02 CEST 2016
On Fri, Jun 24, 2016 at 11:17:41AM +0200, Matthieu Bouron wrote:
> On Thu, Apr 07, 2016 at 02:51:44PM +0200, Matthieu Bouron wrote:
> > On Wed, Mar 23, 2016 at 6:16 PM, Matthieu Bouron <matthieu.bouron at gmail.com>
> > wrote:
> >
> > >
> > >
> > > On Tue, Mar 22, 2016 at 10:04 AM, Matthieu Bouron <
> > > matthieu.bouron at gmail.com> wrote:
> > >
> > >>
> > >>
> > >> On Fri, Mar 18, 2016 at 5:50 PM, Matthieu Bouron <
> > >> matthieu.bouron at gmail.com> wrote:
> > >>
> > >>> From: Matthieu Bouron <matthieu.bouron at stupeflix.com>
> > >>>
> > >>> ---
> > >>>
> > >>> Hello,
> > >>>
> > >>> The following patch add hwaccel support to the mediacodec (h264) decoder
> > >>> by allowing
> > >>> the user to render the output frames directly on a surface.
> > >>>
> > >>> In order to do so the user needs to initialize the hwaccel through the
> > >>> use of
> > >>> av_mediacodec_alloc_context and av_mediacodec_default_init functions.
> > >>> The later
> > >>> takes a reference to an android/view/Surface as parameter.
> > >>>
> > >>> If the hwaccel successfully initialize, the decoder output frames pix
> > >>> fmt will be
> > >>> AV_PIX_FMT_MEDIACODEC. The following snippet of code demonstrate how to
> > >>> render
> > >>> the frames on the surface:
> > >>>
> > >>> AVMediaCodecBuffer *buffer = (AVMediaCodecBuffer *)frame->data[3];
> > >>> av_mediacodec_release_buffer(buffer, 1);
> > >>>
> > >>> The last argument of av_mediacodec_release_buffer enable rendering of the
> > >>> buffer on the surface (or not if set to 0).
> > >>>
> > >>> Regarding the internal changes in the mediacodec decoder:
> > >>>
> > >>> MediaCodec.flush() discards both input and output buffers meaning that if
> > >>> MediaCodec.flush() is called all output buffers the user has a reference
> > >>> on are
> > >>> now invalid (and cannot be used).
> > >>> This behaviour does not fit well in the avcodec API.
> > >>>
> > >>> When the decoder is configured to output software buffers, there is no
> > >>> issue as
> > >>> the buffers are copied.
> > >>>
> > >>> Now when the decoder is configured to output to a surface, the user
> > >>> might not
> > >>> want to render all the frames as fast as the decoder can go and might
> > >>> want to
> > >>> control *when* the frame are rendered, so we need to make sure that the
> > >>> MediaCodec.flush() call is delayed until all the frames the user retains
> > >>> has
> > >>> been released or rendered.
> > >>>
> > >>> Delaying the call to MediaCodec.flush() means buffering any inputs that
> > >>> come
> > >>> the decoder until the user has released/renderer the frame he retains.
> > >>>
> > >>> This is a limitation of this hwaccel implementation, if the user retains
> > >>> a
> > >>> frame (a), then issue a flush command to the decoder, the packets he
> > >>> feeds to
> > >>> the decoder at that point will be queued in the internal decoder packet
> > >>> queue
> > >>> (until he releases the frame (a)). This scenario leads to a memory usage
> > >>> increase to say the least.
> > >>>
> > >>> Currently there is no limitation on the size of the internal decoder
> > >>> packet
> > >>> queue but this is something that can be added easily. Then, if the queue
> > >>> is
> > >>> full, what would be the behaviour of the decoder ? Can it block ? Or
> > >>> should it
> > >>> returns something like AVERROR(EAGAIN) ?
> > >>>
> > >>> About the other internal decoder changes I introduced:
> > >>>
> > >>> The MediaCodecDecContext is now refcounted (using the lavu/atomic api)
> > >>> since
> > >>> the (hwaccel) frames can be retained by the user, we need to delay the
> > >>> destruction of the codec until the user has released all the frames he
> > >>> has a
> > >>> reference on.
> > >>> The reference counter of the MediaCodecDecContext is incremented each
> > >>> time an
> > >>> (hwaccel) frame is outputted by the decoder and decremented each time a
> > >>> (hwaccel) frame is released.
> > >>>
> > >>> Also, when the decoder is configured to output to a surface the pts that
> > >>> are
> > >>> given to the MediaCodec API are now rescaled based on the codec_timebase
> > >>> as
> > >>> those timestamps values are propagated to the frames rendered on the
> > >>> surface
> > >>> since Android M. Not sure if it's really useful though.
> > >>>
> > >>> On the performance side:
> > >>>
> > >>> On a nexus 5, decoding an h264 stream (main profile) 1080p at 60fps:
> > >>> - software output + rgba conversion goes at 59~60fps
> > >>> - surface output + render on a surface goes at 100~110fps
> > >>>
> > >>>
> > >> [...]
> > >>
> > >> Patch updated with the following differences:
> > >> * the public mediacodec api is now always built (not only when
> > >> mediacodec is available) (and the build when mediacodec is not available
> > >> has been fixed)
> > >> * the documentation of av_mediacodec_release_buffer has been improved a
> > >> bit
> > >>
> > >
> > > Patch updated with the following differences:
> > > MediaCodecBuffer->released type is now a volatile int (instead of a int*)
> > > MediaCodecContext->refcount type is now a volatile int (instead of a
> > > int*)
> > >
> >
> > Ping.
>
> Rebased patch attached.
>
> Matthieu
> configure | 1
> libavcodec/Makefile | 6
> libavcodec/allcodecs.c | 1
> libavcodec/mediacodec.c | 133 ++++++++++++++++++++
> libavcodec/mediacodec.h | 88 +++++++++++++
> libavcodec/mediacodec_surface.c | 66 +++++++++
> libavcodec/mediacodec_surface.h | 31 ++++
> libavcodec/mediacodec_wrapper.c | 5
> libavcodec/mediacodecdec.c | 265 +++++++++++++++++++++++++++++++++-------
> libavcodec/mediacodecdec.h | 17 ++
> libavcodec/mediacodecdec_h264.c | 44 +++++-
> libavcodec/version.h | 2
> libavutil/pixdesc.c | 4
> libavutil/pixfmt.h | 2
> libavutil/version.h | 2
libavutil changes look good, i dont know mediacodec so cannot easily
comment about the rest
[...]
--
Michael GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB
Those who are best at talking, realize last or never when they are wrong.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: Digital signature
URL: <http://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20160624/c4332a39/attachment.sig>
More information about the ffmpeg-devel
mailing list