[FFmpeg-devel] [PATCH] lavc/vaapi_encode_h265: fix max_transform_hierarchy_depth_inter/intra

[Linjie Fu]

Set the max_transform_hierarchy_depth_inter/intra to 2 by default
based on the Programmer's Reference Manuals (PRM) in [1].

Intel Encoder only supports 2 levels of quad-tree. That is:
    - max_transform_hierarchy_depth_inter/intra <= 2.

[1] <https://01.org/sites/default/files/documentation/intel-gfx-prm-osrc-kbl-vol10-hevc.pdf>

Signed-off-by: Linjie Fu <linjie.fu at intel.com>
---
Fixed value for intel platform, makes more sense on TGL+ platform.
(If conflict with other driver capability, we may add query support
 later)
 libavcodec/vaapi_encode_h265.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/libavcodec/vaapi_encode_h265.c b/libavcodec/vaapi_encode_h265.c
index cd48545..d6cb82a 100644
--- a/libavcodec/vaapi_encode_h265.c
+++ b/libavcodec/vaapi_encode_h265.c
@@ -445,8 +445,9 @@ static int vaapi_encode_h265_init_sequence_params(AVCodecContext *avctx)
     sps->log2_min_luma_transform_block_size_minus2   = 0;
     sps->log2_diff_max_min_luma_transform_block_size = 3;
     // Full transform hierarchy allowed (2-5).
-    sps->max_transform_hierarchy_depth_inter = 3;
-    sps->max_transform_hierarchy_depth_intra = 3;
+    // Default to 2 based on Programmer's Reference Manuals of Intel graphics
+    sps->max_transform_hierarchy_depth_inter = 2;
+    sps->max_transform_hierarchy_depth_intra = 2;
     // AMP works.
     sps->amp_enabled_flag = 1;
     // SAO and temporal MVP do not work.
-- 
2.7.4