[FFmpeg-devel] [RFC] changing resolution via video filter

[Aman Gupta]

From: Aman Gupta <aman at tmm1.net>

This patchset is not intended to be merged, rather to start a discussion about
some limitations of the current filter graph design in ffmpeg.c

My goal here is to create preview images for a video stream with ffmpeg, with
black bars automatically cropped out. Currently, to acheive this you need to run
ffmpeg twice: once with vf_cropdetect to get the crop parameters, then again with
vf_crop to apply the cropping.

After this patchset, the whole thing can be done in a single command like:

  wget https://s3.amazonaws.com/tmm1/4x3-letterbox.mpg
  ./ffmpeg -hide_banner -ss 2 -i 4x3-letterbox.mpg -vf cropdetect -vframes 1 -y -c:v jpeg2000 preview.jpg

The main problem is that video filters cannot change the height/width of a frame. The
resolution is calculated up-front when the filtergraph and encoder are created,
and any attempts by a vf to change the outlink height/width are essentially ignored.
This is because the encoder AVCodecContext has already been initialized, and none of the
codecs in lavc can handle resolution changes on the fly.

With this patchset, I taught the jpeg2000 encoder to reinitialize itself when
it detects a resolution change. Then, instead of using vf_crop (which could easily be
taught to read cropdetect parameters from the AVFrame metadata dict), I changed
vf_cropdetect to export the crop parameters directly into the AVFrame crop fields.
However to make this have an effect, I also needed to add an av_frame_apply_cropping()
call in between the filter graph and encoder. At the moment, av_frame_apply_cropping()
is done right after decoding only, meaning filters cannot influence cropping decisions.

Obviously this is a very blunt change that fixes one very specific use-case.
But I'm curious what (if any) changes could be made to allow this to work more
generally across other existing filters and encoders.

Aman Gupta (3):
  avcodec/j2kenc: reinitialize state if frame size changes
  avfilter/vf_cropdetect: don't ignore frames, export crop parameters
  ffmpeg: apply frame cropping between filter graph and encoder

 fftools/ffmpeg.c            |  4 ++++
 libavcodec/j2kenc.c         | 14 +++++++++++---
 libavfilter/vf_cropdetect.c |  9 +++++++--
 3 files changed, 22 insertions(+), 5 deletions(-)

-- 
2.14.2