[FFmpeg-devel] [PATCH] lavc/mediacodec: add hwaccel support

[Matthieu Bouron]

On Wed, Mar 23, 2016 at 6:16 PM, Matthieu Bouron <matthieu.bouron at gmail.com>
wrote:

>
>
> On Tue, Mar 22, 2016 at 10:04 AM, Matthieu Bouron <
> matthieu.bouron at gmail.com> wrote:
>
>>
>>
>> On Fri, Mar 18, 2016 at 5:50 PM, Matthieu Bouron <
>> matthieu.bouron at gmail.com> wrote:
>>
>>> From: Matthieu Bouron <matthieu.bouron at stupeflix.com>
>>>
>>> ---
>>>
>>> Hello,
>>>
>>> The following patch add hwaccel support to the mediacodec (h264) decoder
>>> by allowing
>>> the user to render the output frames directly on a surface.
>>>
>>> In order to do so the user needs to initialize the hwaccel through the
>>> use of
>>> av_mediacodec_alloc_context and av_mediacodec_default_init functions.
>>> The later
>>> takes a reference to an android/view/Surface as parameter.
>>>
>>> If the hwaccel successfully initialize, the decoder output frames pix
>>> fmt will be
>>> AV_PIX_FMT_MEDIACODEC. The following snippet of code demonstrate how to
>>> render
>>> the frames on the surface:
>>>
>>>     AVMediaCodecBuffer *buffer = (AVMediaCodecBuffer *)frame->data[3];
>>>     av_mediacodec_release_buffer(buffer, 1);
>>>
>>> The last argument of av_mediacodec_release_buffer enable rendering of the
>>> buffer on the surface (or not if set to 0).
>>>
>>> Regarding the internal changes in the mediacodec decoder:
>>>
>>> MediaCodec.flush() discards both input and output buffers meaning that if
>>> MediaCodec.flush() is called all output buffers the user has a reference
>>> on are
>>> now invalid (and cannot be used).
>>> This behaviour does not fit well in the avcodec API.
>>>
>>> When the decoder is configured to output software buffers, there is no
>>> issue as
>>> the buffers are copied.
>>>
>>> Now when the decoder is configured to output to a surface, the user
>>> might not
>>> want to render all the frames as fast as the decoder can go and might
>>> want to
>>> control *when* the frame are rendered, so we need to make sure that the
>>> MediaCodec.flush() call is delayed until all the frames the user retains
>>> has
>>> been released or rendered.
>>>
>>> Delaying the call to MediaCodec.flush() means buffering any inputs that
>>> come
>>> the decoder until the user has released/renderer the frame he retains.
>>>
>>> This is a limitation of this hwaccel implementation, if the user retains
>>> a
>>> frame (a), then issue a flush command to the decoder, the packets he
>>> feeds to
>>> the decoder at that point will be queued in the internal decoder packet
>>> queue
>>> (until he releases the frame (a)). This scenario leads to a memory usage
>>> increase to say the least.
>>>
>>> Currently there is no limitation on the size of the internal decoder
>>> packet
>>> queue but this is something that can be added easily. Then, if the queue
>>> is
>>> full, what would be the behaviour of the decoder ? Can it block ? Or
>>> should it
>>> returns something like AVERROR(EAGAIN) ?
>>>
>>> About the other internal decoder changes I introduced:
>>>
>>> The MediaCodecDecContext is now refcounted (using the lavu/atomic api)
>>> since
>>> the (hwaccel) frames can be retained by the user, we need to delay the
>>> destruction of the codec until the user has released all the frames he
>>> has a
>>> reference on.
>>> The reference counter of the MediaCodecDecContext is incremented each
>>> time an
>>> (hwaccel) frame is outputted by the decoder and decremented each time a
>>> (hwaccel) frame is released.
>>>
>>> Also, when the decoder is configured to output to a surface the pts that
>>> are
>>> given to the MediaCodec API are now rescaled based on the codec_timebase
>>> as
>>> those timestamps values are propagated to the frames rendered on the
>>> surface
>>> since Android M. Not sure if it's really useful though.
>>>
>>> On the performance side:
>>>
>>> On a nexus 5, decoding an h264 stream (main profile) 1080p at 60fps:
>>>   - software output + rgba conversion goes at 59~60fps
>>>   - surface output + render on a surface goes at 100~110fps
>>>
>>>
>> [...]
>>
>> Patch updated with the following differences:
>>   * the public mediacodec api is now always built (not only when
>> mediacodec is available) (and the build when mediacodec is not available
>> has been fixed)
>>   * the documentation of av_mediacodec_release_buffer has been improved a
>> bit
>>
>
> Patch updated with the following differences:
>   MediaCodecBuffer->released type is now a volatile int (instead of a int*)
>   MediaCodecContext->refcount type is now a volatile int (instead of a
> int*)
>

Ping.