[FFmpeg-devel] [PATCH] lavc/mediacodec: add hwaccel support

[Matthieu Bouron]

On Tue, Mar 22, 2016 at 10:04 AM, Matthieu Bouron <matthieu.bouron at gmail.com
> wrote:

>
>
> On Fri, Mar 18, 2016 at 5:50 PM, Matthieu Bouron <
> matthieu.bouron at gmail.com> wrote:
>
>> From: Matthieu Bouron <matthieu.bouron at stupeflix.com>
>>
>> ---
>>
>> Hello,
>>
>> The following patch add hwaccel support to the mediacodec (h264) decoder
>> by allowing
>> the user to render the output frames directly on a surface.
>>
>> In order to do so the user needs to initialize the hwaccel through the
>> use of
>> av_mediacodec_alloc_context and av_mediacodec_default_init functions. The
>> later
>> takes a reference to an android/view/Surface as parameter.
>>
>> If the hwaccel successfully initialize, the decoder output frames pix fmt
>> will be
>> AV_PIX_FMT_MEDIACODEC. The following snippet of code demonstrate how to
>> render
>> the frames on the surface:
>>
>>     AVMediaCodecBuffer *buffer = (AVMediaCodecBuffer *)frame->data[3];
>>     av_mediacodec_release_buffer(buffer, 1);
>>
>> The last argument of av_mediacodec_release_buffer enable rendering of the
>> buffer on the surface (or not if set to 0).
>>
>> Regarding the internal changes in the mediacodec decoder:
>>
>> MediaCodec.flush() discards both input and output buffers meaning that if
>> MediaCodec.flush() is called all output buffers the user has a reference
>> on are
>> now invalid (and cannot be used).
>> This behaviour does not fit well in the avcodec API.
>>
>> When the decoder is configured to output software buffers, there is no
>> issue as
>> the buffers are copied.
>>
>> Now when the decoder is configured to output to a surface, the user might
>> not
>> want to render all the frames as fast as the decoder can go and might
>> want to
>> control *when* the frame are rendered, so we need to make sure that the
>> MediaCodec.flush() call is delayed until all the frames the user retains
>> has
>> been released or rendered.
>>
>> Delaying the call to MediaCodec.flush() means buffering any inputs that
>> come
>> the decoder until the user has released/renderer the frame he retains.
>>
>> This is a limitation of this hwaccel implementation, if the user retains a
>> frame (a), then issue a flush command to the decoder, the packets he
>> feeds to
>> the decoder at that point will be queued in the internal decoder packet
>> queue
>> (until he releases the frame (a)). This scenario leads to a memory usage
>> increase to say the least.
>>
>> Currently there is no limitation on the size of the internal decoder
>> packet
>> queue but this is something that can be added easily. Then, if the queue
>> is
>> full, what would be the behaviour of the decoder ? Can it block ? Or
>> should it
>> returns something like AVERROR(EAGAIN) ?
>>
>> About the other internal decoder changes I introduced:
>>
>> The MediaCodecDecContext is now refcounted (using the lavu/atomic api)
>> since
>> the (hwaccel) frames can be retained by the user, we need to delay the
>> destruction of the codec until the user has released all the frames he
>> has a
>> reference on.
>> The reference counter of the MediaCodecDecContext is incremented each
>> time an
>> (hwaccel) frame is outputted by the decoder and decremented each time a
>> (hwaccel) frame is released.
>>
>> Also, when the decoder is configured to output to a surface the pts that
>> are
>> given to the MediaCodec API are now rescaled based on the codec_timebase
>> as
>> those timestamps values are propagated to the frames rendered on the
>> surface
>> since Android M. Not sure if it's really useful though.
>>
>> On the performance side:
>>
>> On a nexus 5, decoding an h264 stream (main profile) 1080p at 60fps:
>>   - software output + rgba conversion goes at 59~60fps
>>   - surface output + render on a surface goes at 100~110fps
>>
>>
> [...]
>
> Patch updated with the following differences:
>   * the public mediacodec api is now always built (not only when
> mediacodec is available) (and the build when mediacodec is not available
> has been fixed)
>   * the documentation of av_mediacodec_release_buffer has been improved a
> bit
>

Patch updated with the following differences:
  MediaCodecBuffer->released type is now a volatile int (instead of a int*)
  MediaCodecContext->refcount type is now a volatile int (instead of a int*)

Matthieu

[...]
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-lavc-add-mediacodec-hwaccel-support.patch
Type: text/x-diff
Size: 31941 bytes
Desc: not available
URL: <http://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20160323/418a9b51/attachment.patch>