[FFmpeg-devel] [PATCH v3] lavc/vvc: Stop c->pix_fmt and s->pix_fmt diverging

[Frank Plowman]

When the chroma format changes mid-sequence, c->pix_fmt and s->pix_fmt
can get out-of-sync.  More specifically,
1. export_frame_params is called.
   c->pix_fmt and s->pix_fmt are both set to their new values.
2. set_output_format is called.
   c->pix_fmt is set back to its old value.

The two getting out of sync results in a memory error.  The check in
export_frame_params passes as it only tests the s->pix_fmt, therefore
c->pix_fmt keeps the old value and the buffers are not allocated for the
new pixel format.

Patch fixes this by not setting c->pix_fmt in set_output_format.

Signed-off-by: Frank Plowman <post at frankplowman.com>
---
@Nuo Mi, Fei Wang:
Does this look okay?  I am not really sure why c->pix_fmt was set again
in set_output_format, was this accounting for some specific scenario?
In terms of bitstreams where the pix_fmt changes, I only have fuzzed
bitstreams unfortunately, so while this prevents crashing it is
difficult to know whether the output is correct.
---
 libavcodec/vvc/dec.c | 1 -
 1 file changed, 1 deletion(-)

diff --git a/libavcodec/vvc/dec.c b/libavcodec/vvc/dec.c
index 1cb168de7e..53e3eeec6d 100644
--- a/libavcodec/vvc/dec.c
+++ b/libavcodec/vvc/dec.c
@@ -970,7 +970,6 @@ static int set_output_format(const VVCContext *s, const AVFrame *output)
         if ((ret = ff_set_dimensions(c, output->width, output->height)) < 0)
             return ret;
     }
-    c->pix_fmt = output->format;
     return 0;
 }
 
-- 
2.47.0