[FFmpeg-devel] [PATCH RESEND] avcodec/vaapi_encode: skip AVBR if HRD parameters are set

[Cameron Gutman]

AVBR does not use VAEncMiscParameterTypeHRD, so attempting to set
rc_buffer_size and bit_rate together will cause the rc_buffer_size
to be ignored if the VAAPI driver supports AVBR. We should prefer
regular VBR for that case.

Signed-off-by: Cameron Gutman <aicommander at gmail.com>
---
 libavcodec/vaapi_encode.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/libavcodec/vaapi_encode.c b/libavcodec/vaapi_encode.c
index 8960e6b20a..8bd0701caf 100644
--- a/libavcodec/vaapi_encode.c
+++ b/libavcodec/vaapi_encode.c
@@ -1338,7 +1338,10 @@ static av_cold int vaapi_encode_init_rate_control(AVCodecContext *avctx)
         TRY_RC_MODE(RC_MODE_CBR, 0);
 
     if (avctx->bit_rate > 0) {
-        TRY_RC_MODE(RC_MODE_AVBR, 0);
+        // AVBR does not enforce RC buffer constraints
+        if (!avctx->rc_buffer_size && !avctx->rc_initial_buffer_occupancy)
+            TRY_RC_MODE(RC_MODE_AVBR, 0);
+
         TRY_RC_MODE(RC_MODE_VBR, 0);
         TRY_RC_MODE(RC_MODE_CBR, 0);
     } else {
-- 
2.49.0