[FFmpeg-devel] [PATCH v20 02/20] avutil/frame: Prepare AVFrame\n for subtitle handling

Tue Dec 14 01:30:20 EET 2021

> -----Original Message-----
> From: ffmpeg-devel <ffmpeg-devel-bounces at ffmpeg.org> On Behalf Of Michael
> Niedermayer
> Sent: Sunday, December 12, 2021 11:03 PM
> To: FFmpeg development discussions and patches <ffmpeg-devel at ffmpeg.org>
> Subject: Re: [FFmpeg-devel] [PATCH v20 02/20] avutil/frame: Prepare AVFrame\n
> for subtitle handling
> 
> > You had posted two cases that were failing.
> >
> > 1. > ./ffmpeg -i ~/tickets/1332/Starship_Troopers.vob -scodec xsub -qscale
> 2 -an
> > > file1332.avi
> >
> > ==> Fixed since V18
> >
> >
> > 2. > This breaks:
> > > ./ffmpeg -i ~/tickets/153/bbc_small.ts -filter_complex
> '[0:v][0:s]overlay' -
> > > qscale 2 -t 3 -y file.avi
> >
> > ==> It wasn't actually a regression. It was a bug in dvbdubdec that just
> got
> > covered up earlier by some sub2video hacks.
> >
> > I have submitted this fix for the error:
> >
> >
> https://patchwork.ffmpeg.org/project/ffmpeg/patch/DM8P223MB03655DEE6FF0228743
> 117178BA6A9 at DM8P223MB0365.NAMP223.PROD.OUTLOOK.COM/
> 
> doesnt fix it

Well, it fixes the command line above which you had posted :-)

> the v23_plus set still fails:
> 
> ./ffmpeg -ss 20 -i dvbsubtest.ts  -filter_complex "[0:v][0:s]overlay[v]" -map
> '[v]' -map 0:a -acodec copy -vcodec mpeg4 -t 5 -bitexact /tmp/file.avi

Yup, this one failed. It's fixed in v24.

I've also made a visual comparison to the result of the same command line with 
current ffmpeg.

(The results are not bit-exact due to the different way how things are working now)

> Same failure with a different iinput:
> 
> ./ffmpeg -i ~/tickets/4062/negative_pts_sub.ts -copyts -filter_complex
> '[0:0][0:3]overlay=shortest=1[outv0]' -map 0:1 -map '[outv0]' -bitexact
> /tmp/sadlybroken.ts
> 
> Input #0, mpegts, from 'tickets//4062/negative_pts_sub.ts':
>   Duration: 00:00:04.89, start: -47.631967, bitrate: 6154 kb/s
>   Program 1601
>     Metadata:
>       service_name    : Yle TV1 HD 7M
>       service_provider: Yle
>   Stream #0:0[0x137]: Video: h264 (Main) ([27][0][0][0] / 0x001B),
> yuv420p(tv, bt709, top first), 1920x1080 [SAR 1:1 DAR 16:9], 25 fps, 50 tbr,
> 90k tbn
>   Stream #0:1[0x3b6](fin): Audio: ac3 ([6][0][0][0] / 0x0006), 48000 Hz,
> stereo, fltp, 448 kb/s
>   Stream #0:2[0x3b9](dut): Audio: ac3 ([6][0][0][0] / 0x0006), 48000 Hz,
> stereo, fltp, 192 kb/s (visual impaired) (descriptions)
>   Stream #0:3[0x4cb](fin): Subtitle: dvb_subtitle ([6][0][0][0] / 0x0006)
>   Stream #0:4[0x4e2](fin): Subtitle: dvb_subtitle ([6][0][0][0] / 0x0006)
> (hearing impaired)
>   Stream #0:5[0x1450](fin): Subtitle: dvb_teletext ([6][0][0][0] / 0x0006),
> 492x250
> Stream mapping:
>   Stream #0:0 (h264) -> overlay (graph 0)
>   Stream #0:3 (dvbsub) -> overlay (graph 0)
>   Stream #0:1 -> #0:0 (ac3 (native) -> mp2 (native))
>   overlay:default (graph 0) -> Stream #0:1 (mpeg2video)
> Press [q] to stop, [?] for help
> [h264 @ 0x55f22e15d840] reference picture missing during reorder
> [h264 @ 0x55f22e15d840] Missing reference picture, default is 2147483647
> [h264 @ 0x55f22df57d00] mmco: unref short failure
> [h264 @ 0x55f22df7fb00] reference picture missing during reorder
>     Last message repeated 1 times
> [h264 @ 0x55f22df7fb00] Missing reference picture, default is 65774
>     Last message repeated 1 times
> [h264 @ 0x55f22e0cbb00] mmco: unref short failure
> [h264 @ 0x55f22de22840] mmco: unref short failure
> [h264 @ 0x55f22df89a00] reference picture missing during reorder
> [h264 @ 0x55f22df89a00] Missing reference picture, default is 65782
> [h264 @ 0x55f22e15d840] mmco: unref short failure
> subtitle input filter: decoding size 0x0
> Auto-inserting graphicsub2video filter
> [swscaler @ 0x55f22fabad80] Value 0.000000 for parameter 'srcw' out of range
> [1 - 2.14748e+09]
> [swscaler @ 0x55f22fabad80] Value 0.000000 for parameter 'srch' out of range
> [1 - 2.14748e+09]
> [swscaler @ 0x55f22fabad80] Value 0.000000 for parameter 'dstw' out of range
> [1 - 2.14748e+09]
> [swscaler @ 0x55f22fabad80] Value 0.000000 for parameter 'dsth' out of range
> [1 - 2.14748e+09]
> [graphicsub2video @ 0x55f22fb10b40] [IMGUTILS @ 0x7fff903e52b0] Picture size
> 0x0 is invalid
> Error reinitializing filters!
> Failed to inject frame into filter network: Invalid argument
> Error while processing the decoded data for stream #0:0
> Conversion failed!

The problem here is that subtitles are rare and we don't know the size until 
a subtitle event arrives.

Such issues haven't come up in our own command lines, because I'm always 
inserting the subscale filter there, to make sure the graphical subs are 
resized to the desired size (sometimes the video gets rescaled itself before the
overlay is done).

The current ffmpeg's sub2video implementation employs a hack by using the video
size when there's no size available from the dvbsub decoding context.
That leads to a somewhat weird situation, where the initial (empty) heartbeat
frames have a size of 1920x1080. These are also fixed to RGBA, so a scale
filter is inserted before the overlay filter.
When subtitle frames are actually arriving, the framesize of the dvb subs 
changes to the actual 720x576 which leads to countless warnings like these

> Changing video frame properties on the fly is not supported by all filters.
> filter context - w: 1920 h: 1080 fmt: 28, incoming frame - w: 720 h: 576 fmt: 28 pts_time: 143.467

Nonetheless, the auto-inserted scale filter will scale them up to 1920x1080.
(whether intended of not by sub2video)

For keeping compatibility with sub2video command lines, I'm now always inserting
the subscale filter initialized to the size of the first video stream input to 
the filter graph, in addition to the textsub2video. This preserves previous 
functionality, but the recommended (and better way) is to use the overlaygraphicsubs
filter instead, where the overlay size can be set using expressions (so it can 
be set to the video input size of the filter).

Also, overlaygraphicsubs is more effective as it avoids the intermediate creation 
of full (transparent) frames and rather overlays the individual subtitle rects 
directly (which means it does nothing and passes through the input frame,
when there are no subtitles to overlay).

The textsub2video filter (which is used to mimic the sub2video behaviour) in 
turn, makes primarily sense in cases where you want to burn-in subtitles
via hardware-overlay and you need to prepare overlay frames for hw upload.

Summary: Fixed in V24

> ./ffmpeg -i ~/tickets/4752/dump_dvbsubtitles.mp4 -ss 5 -t 1 -filter_complex '[0:v][0:s]overlay' -bitexact /tmp/withsubtitles.ts

I can't find the file. The link in the ticket doesn't work anymore and I 
can't find it on https://streams.videolan.org/ffmpeg

> Heres one thet generates different output: (i have not checked at all if this
> is a bug just seeing its differnt)
> 
> ./ffmpeg -i ~/tickets/679/oversized_pgs_subtitles.mkv -filter_complex
> '[0:s:1]scale=848x480,[0:v]overlay=shortest=1' -bitexact /tmp/old-pgstest.avi

The subtitle stream reports eof after the last subtitle event is read, which means
the beginning of the last subtitle. For overlay, the parameter "shortest=1" is specified,
so it all ends at the beginning of the last subtitle display time.
When we remove the 'shortest' parameter, we get the same duration and amount of frames.
(also visually equal result) 
This is slightly different but IMO correct behaviour.

> \[a-s\]_full_metal_panic_fumoffu_-_01_-_the_man_from_the_south_-
> _a_hostage_with_no_compromises__rs2_\[1080p_bd-rip\]\[BBB48A25\].mkv  
> Closedcaption_rollup.ts

Couldn't find those files

> ./ffmpeg -i ~/tickets/2397/242_4.mkv -filter_complex '[0:v][0:s:1]overlay' -
> qscale 2 -bitexact /tmp/file2397.avi

In this case, the difference is as follows:

- current ffmpeg produces an erroneous frame at the end
- that frame has a timestamp of more than a second ahead of the previous one
  see screenshot LastFrameError
- this leads to incorrect ffprobe output regarding duration, bitrate etc.
  see screenshot Compare_242_Results:
  The left column is the result generated with the subtitles patchset,
  the right column is the from the regular ffmpeg
  (green means equal values, red indicates differences)

This is a bug in current ffmpeg that is fixed by the subtitles patchset.

I'll post V24 in a moment.
Thanks a lot for doing those tests,

softworkz

-------------- next part --------------
A non-text attachment was scrubbed...
Name: LastFrameError.png
Type: image/png
Size: 88093 bytes
Desc: LastFrameError.png
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20211213/496016e4/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Compare_242_Results.png
Type: image/png
Size: 84963 bytes
Desc: Compare_242_Results.png
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20211213/496016e4/attachment-0001.png>