[FFmpeg-devel] [PATCH] lavc/mediacodec: add hwaccel support

[Matthieu Bouron]

On Thu, Apr 07, 2016 at 02:51:44PM +0200, Matthieu Bouron wrote:
> On Wed, Mar 23, 2016 at 6:16 PM, Matthieu Bouron <matthieu.bouron at gmail.com>
> wrote:
> 
> >
> >
> > On Tue, Mar 22, 2016 at 10:04 AM, Matthieu Bouron <
> > matthieu.bouron at gmail.com> wrote:
> >
> >>
> >>
> >> On Fri, Mar 18, 2016 at 5:50 PM, Matthieu Bouron <
> >> matthieu.bouron at gmail.com> wrote:
> >>
> >>> From: Matthieu Bouron <matthieu.bouron at stupeflix.com>
> >>>
> >>> ---
> >>>
> >>> Hello,
> >>>
> >>> The following patch add hwaccel support to the mediacodec (h264) decoder
> >>> by allowing
> >>> the user to render the output frames directly on a surface.
> >>>
> >>> In order to do so the user needs to initialize the hwaccel through the
> >>> use of
> >>> av_mediacodec_alloc_context and av_mediacodec_default_init functions.
> >>> The later
> >>> takes a reference to an android/view/Surface as parameter.
> >>>
> >>> If the hwaccel successfully initialize, the decoder output frames pix
> >>> fmt will be
> >>> AV_PIX_FMT_MEDIACODEC. The following snippet of code demonstrate how to
> >>> render
> >>> the frames on the surface:
> >>>
> >>>     AVMediaCodecBuffer *buffer = (AVMediaCodecBuffer *)frame->data[3];
> >>>     av_mediacodec_release_buffer(buffer, 1);
> >>>
> >>> The last argument of av_mediacodec_release_buffer enable rendering of the
> >>> buffer on the surface (or not if set to 0).
> >>>
> >>> Regarding the internal changes in the mediacodec decoder:
> >>>
> >>> MediaCodec.flush() discards both input and output buffers meaning that if
> >>> MediaCodec.flush() is called all output buffers the user has a reference
> >>> on are
> >>> now invalid (and cannot be used).
> >>> This behaviour does not fit well in the avcodec API.
> >>>
> >>> When the decoder is configured to output software buffers, there is no
> >>> issue as
> >>> the buffers are copied.
> >>>
> >>> Now when the decoder is configured to output to a surface, the user
> >>> might not
> >>> want to render all the frames as fast as the decoder can go and might
> >>> want to
> >>> control *when* the frame are rendered, so we need to make sure that the
> >>> MediaCodec.flush() call is delayed until all the frames the user retains
> >>> has
> >>> been released or rendered.
> >>>
> >>> Delaying the call to MediaCodec.flush() means buffering any inputs that
> >>> come
> >>> the decoder until the user has released/renderer the frame he retains.
> >>>
> >>> This is a limitation of this hwaccel implementation, if the user retains
> >>> a
> >>> frame (a), then issue a flush command to the decoder, the packets he
> >>> feeds to
> >>> the decoder at that point will be queued in the internal decoder packet
> >>> queue
> >>> (until he releases the frame (a)). This scenario leads to a memory usage
> >>> increase to say the least.
> >>>
> >>> Currently there is no limitation on the size of the internal decoder
> >>> packet
> >>> queue but this is something that can be added easily. Then, if the queue
> >>> is
> >>> full, what would be the behaviour of the decoder ? Can it block ? Or
> >>> should it
> >>> returns something like AVERROR(EAGAIN) ?
> >>>
> >>> About the other internal decoder changes I introduced:
> >>>
> >>> The MediaCodecDecContext is now refcounted (using the lavu/atomic api)
> >>> since
> >>> the (hwaccel) frames can be retained by the user, we need to delay the
> >>> destruction of the codec until the user has released all the frames he
> >>> has a
> >>> reference on.
> >>> The reference counter of the MediaCodecDecContext is incremented each
> >>> time an
> >>> (hwaccel) frame is outputted by the decoder and decremented each time a
> >>> (hwaccel) frame is released.
> >>>
> >>> Also, when the decoder is configured to output to a surface the pts that
> >>> are
> >>> given to the MediaCodec API are now rescaled based on the codec_timebase
> >>> as
> >>> those timestamps values are propagated to the frames rendered on the
> >>> surface
> >>> since Android M. Not sure if it's really useful though.
> >>>
> >>> On the performance side:
> >>>
> >>> On a nexus 5, decoding an h264 stream (main profile) 1080p at 60fps:
> >>>   - software output + rgba conversion goes at 59~60fps
> >>>   - surface output + render on a surface goes at 100~110fps
> >>>
> >>>
> >> [...]
> >>
> >> Patch updated with the following differences:
> >>   * the public mediacodec api is now always built (not only when
> >> mediacodec is available) (and the build when mediacodec is not available
> >> has been fixed)
> >>   * the documentation of av_mediacodec_release_buffer has been improved a
> >> bit
> >>
> >
> > Patch updated with the following differences:
> >   MediaCodecBuffer->released type is now a volatile int (instead of a int*)
> >   MediaCodecContext->refcount type is now a volatile int (instead of a
> > int*)
> >
> 
> Ping.

Rebased patch attached.

Matthieu
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-lavc-add-mediacodec-hwaccel-support.patch
Type: text/x-diff
Size: 35066 bytes
Desc: not available
URL: <http://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20160624/c7711201/attachment.patch>