[FFmpeg-devel] [PATCH] avcodec/v4l2_m2m_dec: dequeue frame if input isn't ready

[Cameron Gutman]

On 12/27/21 00:17, Andriy Gelman wrote:
> On Tue, 14. Dec 02:12, Cameron Gutman wrote:
>> The V4L2M2M API operates asynchronously, so multiple packets can
>> be enqueued before getting a batch of frames back. Since it was
>> only possible to receive a frame by submitting another packet,
>> there wasn't a way to drain those excess output frames from when
>> avcodec_receive_frame() returned AVERROR(EAGAIN).
>>
> 
>> In my testing, this change reduced decode latency of a real-time
>> 60 FPS H.264 stream by approximately 10x (200ms -> 20ms) on a
>> Raspberry Pi 4.
> 
> I was doing some more tests today, but didn't have any luck dequeuing a frame
> if ff_decode_get_packet() returned EAGAIN.

Hmm, maybe there is something different about your test harness?
I'm receiving 720p 60 FPS H.264 ES in real-time from the network.

For each H.264 encoded frame I receive off the network, my basic
approach is like this (simplified for brevity):

avcodec_send_packet(&pkt);
do {
    frame = av_frame_alloc();
    if ((err = avcodec_receive_frame(frame)) == 0) {
        render_frame(frame);
    }
} while (err == 0);

I'll usually get EAGAIN immediately for the first few frames I submit
(so no output frame yet), but then I'll get a batch of output frames
back after the first completed decode. That drains the excess latency
from the pipeline to avoid always being behind.

For cases where we want to prioritize latency over throughput, I've
had success with this approach too:

avcodec_send_packet(&pkt);
while (avcodec_receive_frame(frame) == AVERROR(EAGAIN)) {
    msleep(1);
}
render_frame(frame);

In this case, we can retry avcodec_receive_frame() until we get the
frame back that we just submitted for decoding.

The patch here enables both of these use-cases by allowing V4L2M2M
to retry getting a decoded frame without new input data. Both of
these also work with MMAL after the recent decoupled dataflow patch.

> Could you share the dataset?
> 

It is 720p60.h264 from here:
https://onedrive.live.com/?authkey=%21ALoKfcPfFeKyhzs&id=C15BF9770619F56%21165617&cid=0C15BF9770619F56

> Thanks,
> 

Thank you