[FFmpeg-devel] [Internet][PATCH 00/12] Add MediaCodec encoder and NDK MediaCodec support

[Zhao Zhili]

> -----Original Message-----
> From: ffmpeg-devel <ffmpeg-devel-bounces at ffmpeg.org> On Behalf Of Chema Gonzalez
> Sent: 2022年11月12日 0:55
> To: FFmpeg development discussions and patches <ffmpeg-devel at ffmpeg.org>
> Cc: matthieu.bouron at stupeflix.com; aman at tmm1.net
> Subject: Re: [FFmpeg-devel] [Internet][PATCH 00/12] Add MediaCodec encoder and NDK MediaCodec support
> 
> On Thu, Nov 10, 2022 at 7:36 PM "zhilizhao(赵志立)" <quinkblack at foxmail.com> wrote:
> >
> > Ping for review.
> >
> > > On Oct 24, 2022, at 11:16, Zhao Zhili <quinkblack at foxmail.com> wrote:
> > >
> > > From: Zhao Zhili <zhilizhao at tencent.com>
> > >
> > > Firstly, some bugs were fixed (patch 1-4).
> > >
> > > Patch 5 and 6 make mediacodec_wrapper support Java MediaCodec and NDK
> > > MediaCodec. The use case I'm considering is run FFmpeg on cmdline without JVM,
> > > for example, run FFmpeg inside of termux (an Android terminal emulator). It's
> > > well known that NDK MediaCodec missing some important functions, like get the
> > > list of codecs, but still useable.
> > >
> > > Patch 7 add NDK MediaCodec decoder support. It can be enabled via options,
> > > and enabled automatically if no JVM is available.
> > >
> > > Patch 8 add ANativeWindow support to hwcontext_mediacodec. It can be set by
> > > user, and can be created via AMediaCodec_createPersistentInputSurface
> > > automatically. This is a preparation for encoder.
> > >
> > > Patch 9 makes MediaCodec decoder to support ANativeWindow directly. It worth
> > > to note that AVMediaCodecContext has only surface. Although we provided
> > > av_mediacodec_alloc_context(), we didn't strictly prevent users to allocate
> > > AVMediaCodecContext on stack. I'm not sure if it's OK to add new field to
> > > AVMediaCodecContext.
> > >
> > > Patch 10 add MediaCodec encoder support. Frame can be feed to encoder via
> > > buffer, or via Surface/ANativeWindow. If Surface/ANativeWindow is used, and
> > > the frames come from our MediaCodec decoder wrapper, we can control it's
> > > 'render' (send to encoder's surface) via av_mediacodec_release_buffer(). A DTS
> > > generation strategy works in this case. However, if frames comes from other
> > > sources, like a camera, there is no way to control the 'render' yet, so DTS is
> > > missing in this case.
> > >
> > > Finally, we can do mediacodec transcoding with FFmpeg cmdline on Android.
> > > More importantly, we can do MediaCodec decoder to encoder without copy frames,
> > > although it's very limited since most of avfilters doesn't work. For example:
> > >
> > > ./ffmpeg -hwaccel mediacodec -hwaccel_output_format mediacodec -i /sdcard/test.mp4 -an -c:v h264_mediacodec -y
> /sdcard/out.mp4
> 
> How do you build the ffmpeg binary you're using for experiments in the
> android device?

Just configure with --enable-mediacodec and --enable-jni. Here is the build
script I used for the development.

https://github.com/quink-black/ffmpeg-ci/blob/master/android_ffmpeg.sh

Here is the Android Studio project which can be used to test the mediacodec
wrapper implemented via JNI.

> 
> 
> 
> 
> > >
> > > Since there is no real AVHWFrameContext implementation in hwcontext_mediacodec.
> > > there is no hwframe_ctx for mediacodec and av_hwframe_transfer_data() doesn't
> > > work. So if -hwaccel_output_format isn't being specified like:
> > >
> > > ./ffmpeg -hwaccel mediacodec -i /sdcard/test.mp4 -an -c:v h264_mediacodec -y /sdcard/out.mp4
> > >
> > > It will trigger a crash in av_hwframe_transfer_data. Patch 11 add a check on
> > > hwframe_ctx. Patch 12 set hwaccel_output_format automatically to avoid such
> > > case.
> > >
> > >
> > > Zhao Zhili (12):
> > >  avcodec/mediacodec: fix incorrect crop info
> > >  avcodec/mediacodecdec: don't break out if both input and output port
> > >    return try again
> > >  avcodec/mediacodecdec_common: fix misuse av_free/av_freep
> > >  avcodec/mediacodecdec_common: fix useless av_buffer_unref
> > >  avcodec/mediacodec_wrapper: separate implementation from interface
> > >  avcodec/mediacodec: add NDK media codec wrapper
> > >  avcodec/mediacodecdec: enable NDK mediacodec
> > >  avutil/hwcontext_mediacodec: add ANativeWindow support
> > >  avcodec/mediacodec: add ANativeWindow support
> > >  avcodec: add MediaCodec encoder
> > >  avutil/hwcontext: verify hw_frames_ctx in transfer_data_alloc
> > >  fftools/ffmpeg_opt: set default hwaccel_output_format for mediacodec
> > >
> > > Changelog                         |   2 +
> > > configure                         |   6 +
> > > fftools/ffmpeg_opt.c              |   4 +
> > > libavcodec/Makefile               |   2 +
> > > libavcodec/allcodecs.c            |   2 +
> > > libavcodec/mediacodec_surface.c   |  46 +-
> > > libavcodec/mediacodec_surface.h   |   8 +-
> > > libavcodec/mediacodec_wrapper.c   | 942 +++++++++++++++++++++++++++---
> > > libavcodec/mediacodec_wrapper.h   | 275 +++++++--
> > > libavcodec/mediacodecdec.c        |  21 +-
> > > libavcodec/mediacodecdec_common.c |  35 +-
> > > libavcodec/mediacodecdec_common.h |   1 +
> > > libavcodec/mediacodecenc.c        | 495 ++++++++++++++++
> > > libavcodec/version.h              |   2 +-
> > > libavutil/hwcontext.c             |   6 +-
> > > libavutil/hwcontext_mediacodec.c  |  56 +-
> > > libavutil/hwcontext_mediacodec.h  |  11 +
> > > libavutil/version.h               |   4 +-
> > > 18 files changed, 1780 insertions(+), 138 deletions(-)
> > > create mode 100644 libavcodec/mediacodecenc.c
> > >
> > > --
> > > 2.25.1
> > >
> > >
> >
> > _______________________________________________
> > ffmpeg-devel mailing list
> > ffmpeg-devel at ffmpeg.org
> > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> >
> > To unsubscribe, visit link above, or email
> > ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".