[FFmpeg-devel] [PATCH v3] lavf: vvc_probe: enforce SPS -> PPS -> IRAP detection order to fix invalid stream detection

[xiongweixiao]

The current vvc_probe function checks for SPS, PPS, and IRAP NAL units but does not enforce their detection order. This can lead to false positives when PPS or IRAP appear before SPS, causing malformed or non-compliant bitstreams to be incorrectly identified as valid VVC streams.

This patch modifies the logic to use boolean flags and enforces the correct detection sequence: SPS must be detected before PPS, and PPS before IRAP. This ensures more accurate probing and prevents invalid bitstreams from being misidentified.

This change addresses issues reported in:
- https://trac.ffmpeg.org/ticket/11496
- https://trac.ffmpeg.org/ticket/11087

Signed-off-by: xiongweixiao <xiongweixiaoxxw at gmail.com>

[v2] lavf: vvc_probe: fix based on review feedback

Changes since v1:
- Added a flag to force sps, pps, iraq to appear in order;
- Modified score where pps, iraq appeared in the wrong position;
- Set a smaller score for incomplete sps, pps, iraq to deal with situations that may look like vvc format

Signed-off-by: xiongweixiao <xiongweixiaoxxw at gmail.com>

[v3] lavf fix code format
remove trailing whitspace
---
 libavformat/vvcdec.c | 19 ++++++++++++++++---
 1 file changed, 16 insertions(+), 3 deletions(-)

diff --git a/libavformat/vvcdec.c b/libavformat/vvcdec.c
index 0a0e24b71c..1bfa1b5d69 100644
--- a/libavformat/vvcdec.c
+++ b/libavformat/vvcdec.c
@@ -44,6 +44,7 @@ static int vvc_probe(const AVProbeData *p)
 {
     uint32_t code = -1;
     int sps = 0, pps = 0, irap = 0;
+    int valid_pps = 0, valid_irap = 0;
     int i;
 
     for (i = 0; i < p->buf_size - 1; i++) {
@@ -60,17 +61,29 @@ static int vvc_probe(const AVProbeData *p)
 
             switch (type) {
             case VVC_SPS_NUT:       sps++;  break;
-            case VVC_PPS_NUT:       pps++;  break;
+            case VVC_PPS_NUT:
+                pps++;
+                if (sps)
+                    valid_pps++;
+                break;
             case VVC_IDR_N_LP:
             case VVC_IDR_W_RADL:
             case VVC_CRA_NUT:
-            case VVC_GDR_NUT:       irap++; break;
+            case VVC_GDR_NUT:
+                irap++;
+                if (valid_pps)
+                    valid_irap++;
+                break;
             }
         }
     }
 
-    if (sps && pps && irap)
+    if (valid_irap)
         return AVPROBE_SCORE_EXTENSION + 1; // 1 more than .mpg
+    if (sps && pps && irap)
+        return AVPROBE_SCORE_EXTENSION / 2;
+    if (sps || pps || irap)
+        return AVPROBE_SCORE_EXTENSION / 4;
     return 0;
 }
 
-- 
2.34.1