[FFmpeg-devel] [PATCH v2 1/2] lavc/vaapi_decode: Use dynamic frame pool if possible

[Xiang, Haihao]

From: Haihao Xiang <haihao.xiang at intel.com>

libva2 doesn't require a fixed surface-array any more, so we may use
dynamic frame pool for decoding when libva2 is available, which allows a
downstream element stores more frames from VAAPI decoders and fixes the
error below:

$ ffmpeg -hwaccel vaapi -hwaccel_output_format vaapi \
-i input.mp4 -c:v hevc_vaapi -f null -
...
[h264 @ 0x557a075a1400] get_buffer() failed
[h264 @ 0x557a075a1400] thread_get_buffer() failed
[h264 @ 0x557a075a1400] decode_slice_header error
[h264 @ 0x557a075a1400] no frame!

Signed-off-by: Haihao Xiang <haihao.xiang at intel.com>
---
 libavcodec/vaapi_decode.c | 36 ++++++++++++++++++++----------------
 1 file changed, 20 insertions(+), 16 deletions(-)

diff --git a/libavcodec/vaapi_decode.c b/libavcodec/vaapi_decode.c
index 5665639dd7..21b273cd0f 100644
--- a/libavcodec/vaapi_decode.c
+++ b/libavcodec/vaapi_decode.c
@@ -599,22 +599,26 @@ static int vaapi_decode_make_config(AVCodecContext *avctx,
         if (err < 0)
             goto fail;
 
-        frames->initial_pool_size = 1;
-        // Add per-codec number of surfaces used for storing reference frames.
-        switch (avctx->codec_id) {
-        case AV_CODEC_ID_H264:
-        case AV_CODEC_ID_HEVC:
-        case AV_CODEC_ID_AV1:
-            frames->initial_pool_size += 16;
-            break;
-        case AV_CODEC_ID_VP9:
-            frames->initial_pool_size += 8;
-            break;
-        case AV_CODEC_ID_VP8:
-            frames->initial_pool_size += 3;
-            break;
-        default:
-            frames->initial_pool_size += 2;
+        if (CONFIG_VAAPI_1)
+            frames->initial_pool_size = 0;
+        else {
+            frames->initial_pool_size = 1;
+            // Add per-codec number of surfaces used for storing reference frames.
+            switch (avctx->codec_id) {
+            case AV_CODEC_ID_H264:
+            case AV_CODEC_ID_HEVC:
+            case AV_CODEC_ID_AV1:
+                frames->initial_pool_size += 16;
+                break;
+            case AV_CODEC_ID_VP9:
+                frames->initial_pool_size += 8;
+                break;
+            case AV_CODEC_ID_VP8:
+                frames->initial_pool_size += 3;
+                break;
+            default:
+                frames->initial_pool_size += 2;
+            }
         }
     }
 
-- 
2.34.1