[FFmpeg-user] Intel QSV and xstack_qsv: Error while filtering: Internal bug, should not have happened

[Dennis Mungai]

On Wed, 27 Dec 2023 at 18:40, Shane Warren <shanew at innovsys.com> wrote:

>
> I'm testing out an Intel Flex 140 card and I am attempting to transcode 4
> inputs to 1 output using the xstack_qsv filter.  My command takes in 4 udp
> multicast ts files (h264 encoded) and outputs one h264_qsv encoded udp
> transport stream. My command looks like so:
>
> /tmp/ffmpeg_g  -nostats -loglevel verbose -init_hw_device
> qsv=card1:/dev/dri/card1 -hwaccel_device card1 -filter_hw_device card1
> -fflags discardcorrupt  -fflags +genpts -fflags discardcorrupt  \
> -hwaccel qsv -hwaccel_output_format qsv -thread_queue_size 4096
> -async_depth 1 -i "udp://@
> 225.105.0.1:10102?fifo_size=214688&buffer_size=851968&timeout=800000&overrun_nonfatal=1"
> \
> -hwaccel qsv -hwaccel_output_format qsv -thread_queue_size 4096
> -async_depth 1 -i "udp://@
> 225.105.0.14:10102?fifo_size=214688&buffer_size=851968&timeout=800000&overrun_nonfatal=1"
> \
> -hwaccel qsv -hwaccel_output_format qsv -thread_queue_size 4096
> -async_depth 1 -i "udp://@
> 225.105.0.5:10102?fifo_size=214688&buffer_size=851968&timeout=800000&overrun_nonfatal=1"
> \
> -hwaccel qsv -hwaccel_output_format qsv -thread_queue_size 4096
> -async_depth 1 -i "udp://@
> 225.105.0.4:10102?fifo_size=214688&buffer_size=851968&timeout=800000&overrun_nonfatal=1"
> \
> -noautoscale -filter_complex "\
> [0:v]vpp_qsv=deinterlace=2:w=720:h=480:framerate=30000/1001[v1]; \
> [1:v]vpp_qsv=deinterlace=2:w=720:h=480:framerate=30000/1001[v2]; \
> [2:v]vpp_qsv=deinterlace=2:w=720:h=480:framerate=30000/1001[v3]; \
> [3:v]vpp_qsv=deinterlace=2:w=720:h=480:framerate=30000/1001[v4]; \
> [v1][v2][v3][v4] xstack_qsv=inputs=4:layout=0_0|0_h0|w0_0|w0_h0[mosiac]; \
> [mosiac]vpp_qsv=w=1920:h=1080,hwdownload,format=nv12,pad=1920:1080:(ow-iw)/2:(oh-ih)/2,hwupload=extra_hw_frames=64,format=qsv[v]"
> \
> -map "[v]" -map 0:a:0 -map 1:a:0 -map 2:a:0 -map 3:a:0 \
> -c:v h264_qsv -noautoscale -async_depth 1 -scenario livestreaming -r:v
> 30000/1001 -b:v 6000k -minrate:v 6000k -maxrate:v 6000k -bufsize:v 12000k
> -threads 1 -profile:v high -bf:v 0 -g:v 15 \
> -filter:a "aresample=async=1" -c:a aac -ac:a 2 -ar:a 48000 -b:a 192k
> -fps_mode auto \
> -f mpegts -muxrate 7450599 -pes_payload_size 1528 "udp://@
> 229.100.100.44:10102?pkt_size=1316&fifo_size=90000&bitrate=7450599&burst_bits=10528
> "
>
> The issue is eventually stream 1 changes resolution from 1280x720
> (progressive) to 1920x1080 (interlaced). The stream dies with the following
> logs coming out:
>
> [h264_qsv @ 0x55a240e68000] Error submitting the frame for encoding.
> [vost#0:0/h264_qsv @ 0x55a240e30780] Error submitting video frame to the
> encoder
> Error while filtering: Internal bug, should not have happened
>
> I can transcode the single stream that changes resolution on its own and
> it can survive a resolution change using this command:
>
> /tmp/ffmpeg_g  -nostats -loglevel verbose -init_hw_device
> qsv=card1:/dev/dri/card1 -hwaccel_device card1 -filter_hw_device card1
> -fflags discardcorrupt  -fflags +genpts -fflags discardcorrupt  -hwaccel
> qsv -hwaccel_output_format qsv -async_depth 1 -i "udp://@
> 225.105.0.1:10102?fifo_size=214688&buffer_size=851968&timeout=800000&overrun_nonfatal=1"
> -vf "vpp_qsv=deinterlace=2:w=1920:h=1080:framerate=30000/1001" -map "0:v"
> -map 0:a:0 -c:v h264_qsv -noautoscale  -async_depth 1 -scenario
> livestreaming -r:v 30000/1001 -b:v 5000k -minrate:v 5000k -maxrate:v 5000k
> -bufsize:v 10000k -a53cc 1 -forced_idr 1 -threads 1 -profile:v high -bf:v 0
> -g:v 15 -filter:a "aresample=async=1" -c:a aac -ac:a 2 -ar:a 48000 -b:a
> 192k -fps_mode auto -f mpegts -muxrate 6450599 -pes_payload_size 1528
> "udp://@
> 229.100.100.44:10102?pkt_size=1316&fifo_size=90000&bitrate=6450599&burst_bits=10528
> "
>
> When that stream changes resolution I see these come out in the log, and
> the stream carries on without dying:
>
> [h264_qsv @ 0x561bc472f8c0] Video parameter change
> [AVHWDeviceContext @ 0x7f313000d1c0] VAAPI driver: Intel iHD driver for
> Intel(R) Gen Graphics - 23.2.4 ().
> [AVHWDeviceContext @ 0x7f313000d1c0] Driver not found in known nonstandard
> list, using standard behaviour.
> [h264_qsv @ 0x561bc472f8c0] Decoder: output is video memory surface
>
> My quesiton is why does the stream die when using xstack_qsv but works
> fine on its own.
>

This is definitely a bug, something to do with frame pool setting(s) in
that filter chain. You may want to open a ticket for this on ffmpeg trac.
The xstack_qsv may need a dynamic frame pool workaround to address this
anomaly on video parameter changes.
Adding in the Intel guys on this one.