[FFmpeg-devel] [PATCH 1/7] lavu: add ecinfo sidedata

[Michael Niedermayer]

On Tue, Jul 25, 2023 at 02:05:43PM +0200, J. Dekker wrote:
> 
> Hi Devin,
> 
> Devin Heitmueller <devin.heitmueller at ltnglobal.com> writes:
> > On Fri, Jul 21, 2023 at 9:38 AM J. Dekker <jdek at itanimul.li> wrote:
> >
> > I appreciate the value of stats so I can tell that the stream had
> > errors, but how is this side data "helpful to a renderer attempting to
> > filter or conceal video decoding errors and artifacts" if there is no
> > information relating to the region of the picture where the errors are
> > present?  Is the assumption that an application will simply have some
> > threshold at which it decides to duplicate the previous frame rather
> > than showing the current one?
> 
> Yes, this is primarily the usecase here initially. As some point a
> renderer would rather duplicate previous frame than rely on
> reconstruction. This could be done within EC itself but letting the
> decoder make this choice rather than exporting the information to a
> renderer and doing it there seems incorrect to me.

Error concealment is not merely a choice of copying or not

Some areas may be damaged, surrounding areas may be undamaged
an example could be a row of macroblocks / a slice
To perform error concealment properly you need to know the motion
vectors of the surrounding slices and their macroblock types
for example if all the surrounding macroblocks have a motion vector
pointing left by 7 pixels the lost slice likely needs to be
using the same.
This requires also knowledge of the reference frames the decoder used
and ability to apply similar motion compensation to know what "-7" means
If there are different motion vecstors used in surrounding macroblocks
it becomes even more complex

Also with some decoders you can loose the texture of a slice but have
undamaged motion vectors. Or you may have intra slices with DC undamaged
but AC lost for every macroblock

And if its all intra blocks, some inpainting based on spatial surroundings
may be better than using the previosu frames.
This is all implemented in our error concealment code.

Trying to do this outside is a mistake. Not only is it hard due to the
deep connection to decoder state it is also something every user of
libavcodec needs. The code belongs in a place every libavcodec user
has access to and can reuse it from.

Thx

[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

If a bugfix only changes things apparently unrelated to the bug with no
further explanation, that is a good sign that the bugfix is wrong.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: not available
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20230726/030262dd/attachment.sig>