[FFmpeg-devel] [PATCH 2/2] lavc: add h264 mediacodec decoder
Matthieu Bouron
matthieu.bouron at gmail.com
Wed Mar 2 15:34:09 CET 2016
On Tue, Mar 01, 2016 at 11:26:45PM +0100, Michael Niedermayer wrote:
> On Tue, Mar 01, 2016 at 08:01:45PM +0100, Matthieu Bouron wrote:
> > On Sat, Feb 27, 2016 at 04:28:43PM +0100, Michael Niedermayer wrote:
> > > On Fri, Feb 26, 2016 at 04:54:47PM +0100, Matthieu Bouron wrote:
> > > > On Tue, Feb 23, 2016 at 11:28:01AM +0100, wm4 wrote:
> > > > > On Tue, 23 Feb 2016 09:53:43 +0100
> > > > > Matthieu Bouron <matthieu.bouron at gmail.com> wrote:
> > > > >
> > > > > > On Mon, Feb 22, 2016 at 01:08:49PM +0100, Michael Niedermayer wrote:
> > > > > > > On Mon, Feb 22, 2016 at 12:20:36PM +0100, Matthieu Bouron wrote:
> > > > > > > > From: Matthieu Bouron <matthieu.bouron at stupeflix.com>
> > > > > > > [...]
> > > > > > > > + codec = (*env)->NewObject(env, jfields.mediacodec_list_class, jfields.init_id, 0);
> > > > > > > > + if (!codec) {
> > > > > > > > + av_log(NULL, AV_LOG_ERROR, "Could not create media codec list\n");
> > > > > > > > + goto done;
> > > > > > > > + }
> > > > > > > > +
> > > > > > > > + tmp = (*env)->CallObjectMethod(env, codec, jfields.find_decoder_for_format_id, format);
> > > > > > > > + if (!tmp) {
> > > > > > > > + av_log(NULL, AV_LOG_ERROR, "Could not find decoder in media codec list\n");
> > > > > > > > + goto done;
> > > > > > > > + }
> > > > > > > > +
> > > > > > > > + name = ff_jni_jstring_to_utf_chars(env, tmp, NULL);
> > > > > > > > + if (!name) {
> > > > > > >
> > > > > > > > + av_log(NULL, AV_LOG_ERROR, "Could not convert jstring to utf chars\n");
> > > > > > >
> > > > > > > some non NULL context would be better, if possible, so the user knows
> > > > > > > where an error came from
> > > > > >
> > > > > > It would require to pass the log_ctx (avctx) as an argument of
> > > > > > ff_AMediaCodecList_getCodecByName.
> > > > > >
> > > > > > All the functions of this wrapper does not take a log_ctx as argument
> > > > > > to be as close as possible to the original NDK MediaCodec API. The
> > > > > > NDK MediaCodec API does not provide any wrapper around MediaCodecList.
> > > > > >
> > > > > > I would like the whole wrapper API to be consistent but also as close as
> > > > > > possible to original one.
> > > > > > If you think it's mandatory I can add a log_ctx argument to every
> > > > > > functions of the wrapper API (so it will be used directly by av_log and
> > > > > > ff_jni_exception_check).
> > > > > >
> > > > > > On another note, I fixed locally this part of the code by adding missing calls
> > > > > > to ff_jni_exception_check.
> > > > > >
> > > > > > [...]
> > > > >
> > > > > Is it possible to store the log_ctx somewhere in the JNI?
> > > > >
> > > > > We might at some point in the future forbid av_log(NULL, ...) calls, and
> > > > > then it'd get complicated.
> > > >
> > > > Patch updated with the following differences:
> > > > * All logging arguments in mediacodec_wrapper functions are now non
> > > > NULL
> > > > * The mediacodec buffer copy functions has been moved to a separate file
> > > > * The Mediacodec enum codec flags are properly retrieved from JNI
> > > >
> > > > The codec extradata conversion to annex b simplification has not been
> > > > addressed in this update. The bitstream filter api does not seem to
> > > > provide a way to do the extradata conversion without feeding an actual
> > > > packet to the api (av_bitstream_filter_filter) and i would like to keep
> > > > the codec initialization in the init function.
> > > >
> > > > The patchset has been pushed to a new branch:
> > > > https://github.com/mbouron/FFmpeg/tree/feature/mediacodec-support-v3
> > > >
> > > > Matthieu
> > >
> > > > configure | 5
> > > > libavcodec/Makefile | 3
> > > > libavcodec/allcodecs.c | 1
> > > > libavcodec/mediacodec_sw_buffer.c | 339 +++++++
> > > > libavcodec/mediacodec_sw_buffer.h | 62 +
> > > > libavcodec/mediacodec_wrapper.c | 1674 ++++++++++++++++++++++++++++++++++++++
> > > > libavcodec/mediacodec_wrapper.h | 122 ++
> > > > libavcodec/mediacodecdec.c | 563 ++++++++++++
> > > > libavcodec/mediacodecdec.h | 82 +
> > > > libavcodec/mediacodecdec_h264.c | 356 ++++++++
> > > > 10 files changed, 3207 insertions(+)
> > > > f545068afece74d27cc04a365d9de7dcf5586a7d 0002-lavc-add-h264-mediacodec-decoder.patch
> > > > From 805c7b95433f1bfbbc5d47fd59a1bbb755edb112 Mon Sep 17 00:00:00 2001
> > > > From: Matthieu Bouron <matthieu.bouron at stupeflix.com>
> > > > Date: Thu, 21 Jan 2016 09:29:39 +0100
> > > > Subject: [PATCH 2/2] lavc: add h264 mediacodec decoder
> > >
> > > breaks fate
> > > swr-resample_lin-fltp-48000-44100
> > > TEST swr-resample_lin-dblp-8000-44100
> > > TEST swr-resample_lin-dblp-8000-48000
> > > --- ./tests/ref/fate/source 2016-02-24 22:42:26.379879152 +0100
> > > +++ tests/data/fate/source 2016-02-27 14:44:09.176735216 +0100
> > > @@ -27,3 +27,4 @@
> > > compat/avisynth/windowsPorts/windows2linux.h
> > > compat/float/float.h
> > > compat/float/limits.h
> > > +libavcodec/mediacodec_sw_buffer.h
> > > Test source failed. Look at tests/data/fate/source.err for details.
> > > make: *** [fate-source] Error 1
> > > make: *** Waiting for unfinished jobs....
> >
> > New patch attached fixing the issue. It also fixes an issue while the
> > binding was trying to access the field MediaCodec.BUFFER_FLAG_KEY_FRAME
> > which is only available in android >= 5 causing the JNI code to try to
> > call a method on a NULL field ID which caused the VM to crash on older
> > devices. This field is now non mandatory.
> >
> > The codec extradata conversion is still not addressed as the bitstream API
> > does not expose a method to do such conversion without feeding an actual
> > packet to it.
> >
> > I think what is left to do at this point is more testing towards corrupted
> > bitstream and see how to the API behaves and see how it impacts the
> > decoder (as I rely on counters of queued and dequeued buffers, which is a
> > bit evil).
> > If you happen to have such h264 samples, it would be very helpful.
>
> its probably most efficient if you run the code under a fuzzer
>
> fuzzed files which triggered other issues are likely not optimal nor
> nummerous enough for hitting many different issues
>
> the fate samples likely are a good source for undamaged h264 to fuzz
>
Patch updated with the following differences:
* The avctx width/height are now retrieved from MediaCodec using the
crop parameters the API returns, making the decoder handle properly
reinit-h264_420_8-to-small_420_8.h264
I'm currently testing the decoder against fuzzed samples. So far, the
impact of a corrupted bitstream is having the decoder waiting 1second and
then timing out while receiving a flush packet. It also outputs an error
message like the following one:
Failed to dequeue output buffer within 1000ms, output will probably lack
%d frames.
Is that acceptable to block for such a long time in this particular case ?
Matthieu
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0002-lavc-add-h264-mediacodec-decoder.patch
Type: text/x-diff
Size: 113232 bytes
Desc: not available
URL: <http://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20160302/0625023a/attachment.patch>
More information about the ffmpeg-devel
mailing list