[FFmpeg-devel] [PATCHv8] libavcodec: v4l2: add support for v4l2 mem2mem codecs
Mark Thompson
sw at jkqxz.net
Tue Sep 5 00:29:18 EEST 2017
On 04/09/17 22:07, Jorge Ramirez wrote:
> On 09/04/2017 10:45 PM, Mark Thompson wrote:
>> On 04/09/17 21:34, Jorge Ramirez wrote:
>>> On 09/04/2017 07:55 PM, Mark Thompson wrote:
>>>> On 04/09/17 18:01, Jorge Ramirez wrote:
>>>>> On 09/04/2017 06:31 PM, Mark Thompson wrote:
>>>>>> On 04/09/17 17:00, Jorge Ramirez wrote:
>>>>>>> On 09/03/2017 08:23 PM, Mark Thompson wrote:
>>>>>>>> On 03/09/17 17:54, Jorge Ramirez wrote:
>>>>>>>>> On 09/03/2017 02:27 AM, Mark Thompson wrote:
>>>>>>>>>>> +/* in ffmpeg there is a single thread could be queueing/dequeuing buffers so a
>>>>>>>>>>> + * timeout is * required when retrieving a frame in case the driver has not received
>>>>>>>>>>> + * enough input * to start generating output.
>>>>>>>>>>> + *
>>>>>>>>>>> + * once decoding starts, the timeout should not be hit.
>>>>>>>>>> This seems like it could introduce a significant delay on startup for no good reason. Can you instead just queue packets until either you run out of input buffers or a nonblocking dequeue succeeds?
>>>>>>>>>>
>>>>>>>>>> (I might need to think more about how the semantics of this work.)
>>>>>>>>>>
>>>>>>>>> if the decoder needs 4 blocks, the delay is 200ms, if it is 10 blocks, that is 500ms which doesn't seem too significant? when I test I barely notice the difference with respect to using the h264 codec (or any of the others in fact)
>>>>>>>>>
>>>>>>>>> the best solution would be to be able to block until the capture queue has frames ready but for that we would need another thread inputting independently on the other queue...does ffmpeg allow for this? separate threads for input and output?
>>>>>>>> Since the API is nonblocking, you can just return EAGAIN from receive_frame if there are any free buffers (to request more input). You would then only block waiting for output if there is no more input (end of stream) or there aren't any free buffers (so no more input could be accepted). Ideally there would then be no timeouts at all except in error cases.
>>>>>>> sure, can do that as well, not a problem.
>>>>>>>
>>>>>>> the encoding API doesnt seem to allow for this though: once it retrieves a valid frame it appears to keep on reading them without inputing others (this causes teh capture queue to block for ever)
>>>>>>>
>>>>>>> is this intentional or is it a bug?
>>>>>> The encode API should be identical to the decode API with frames/packets swapped (see docs in avcodec.h).
>>>>>>
>>>>>> If you have an lavc-using program which calls receive_packet() repeatedly after it has returned EAGAIN and never calls send_packet() then that program is wrong.
>>>>> thanks for the prompt answer.
>>>>>
>>>>> yes I am just using the ffmpeg binary to encode a stream; however once a valid encoded packet is returned, send_frame is not ever called again unless I return EAGAIN from v4l2m2m_receive_packet.
>>>>> But I cant return EAGAIN on receive_packet while I am blocked waiting for data which will never arrive (since send_frame is not executing)
>>>>>
>>>>> seems to me like a bug in ffmeg but I dont like to question baseline code with obvious bugs (this seems to simple to be real)
>>>>>
>>>>> anyway looking at the function do_video_out() the code seems strange but it explains why my encoding blocks unless I timeout and return EAGAIN.
>>>>>
>>>>>
>>>>> ret = avcodec_send_frame(enc, in_picture);
>>>>> if (ret < 0)
>>>>> goto error;
>>>>>
>>>>> while (1) {
>>>>> ret = avcodec_receive_packet(enc, &pkt);
>>>>> update_benchmark("encode_video %d.%d", ost->file_index, ost->index);
>>>>> if (ret == AVERROR(EAGAIN))
>>>>> break;
>>>>> if (ret < 0)
>>>>> goto error;
>>>>>
>>>>> if (debug_ts) {
>>>>> av_log(NULL, AV_LOG_INFO, "encoder -> type:video "
>>>>> "pkt_pts:%s pkt_pts_time:%s pkt_dts:%s pkt_dts_time:%s\n",
>>>>> av_ts2str(pkt.pts), av_ts2timestr(pkt.pts, &enc->time_base),
>>>>> av_ts2str(pkt.dts), av_ts2timestr(pkt.dts, &enc->time_base));
>>>>> }
>>>>>
>>>>> if (pkt.pts == AV_NOPTS_VALUE && !(enc->codec->capabilities & AV_CODEC_CAP_DELAY))
>>>>> pkt.pts = ost->sync_opts;
>>>>>
>>>>> av_packet_rescale_ts(&pkt, enc->time_base, ost->mux_timebase);
>>>>>
>>>>> if (debug_ts) {
>>>>> av_log(NULL, AV_LOG_INFO, "encoder -> type:video "
>>>>> "pkt_pts:%s pkt_pts_time:%s pkt_dts:%s pkt_dts_time:%s\n",
>>>>> av_ts2str(pkt.pts), av_ts2timestr(pkt.pts, &ost->mux_timebase),
>>>>> av_ts2str(pkt.dts), av_ts2timestr(pkt.dts, &ost->mux_timebase));
>>>>> }
>>>>>
>>>>> frame_size = pkt.size;
>>>>> output_packet(of, &pkt, ost, 0);
>>>>>
>>>>> /* if two pass, output log */
>>>>> if (ost->logfile && enc->stats_out) {
>>>>> fprintf(ost->logfile, "%s", enc->stats_out);
>>>>> }
>>>>> }
>>>>> }
>>>>>
>>>>>
>>>>> so if I queue 20 frames in the output queue and the allow frames to be dequeued, all of them are dequeued at once and then the code just blocks waiting for more input...
>>>>>
>>>>>
>>>>>
>>>>> does the above look ok to you?
>>>> Yes: send_frame() is always callable once, and then receive_packet() is callable repeatedly until it returns EAGAIN; once it does, send_frame() is necessarily callable again.
>>>>
>>>> Can you offer a sequence of valid returns from the lavc API which would break it? (Ignoring what the implementation behind it actually is for a moment.)
>>>
>>> yes, just the code I have in the patch I am preparing for v9 for instance where I remove the timeout.
>>> https://github.com/ldts/FFmpeg/blob/patches/v9/libavcodec/v4l2_m2m_enc.c#L267
>>>
>>> encoding just blocks.
>>> After having queued all input buffers it dequeues the first packet ...but then it just and keeps on trying to do that for ever since we block for output ready and never send another frame.
>>>
>>> decoding works as expected.
>>> https://github.com/ldts/FFmpeg/blob/patches/v9/libavcodec/v4l2_m2m_dec.c#L148
>> There is too much code there for me to follow exactly what the sequence is without hardware to run it on. Could you post an ordered list of all of the avcodec_send_frame() and avcodec_receive_packet() calls and results which result in this deadlock? If it is an error in ffmpeg.c then those calls must be violating the API contact somehow independent of the actual implementation being used, and that could be verified and fixed without the v4l2 encoder.
>
> sorry for the rather long trace (I thought it was better than missing info or using http://haste.bin or similar)
>
>
> linaro at db820 raw]$ cat encode.h264.sh
> ffmpeg -loglevel debug -f rawvideo -pix_fmt nv12 -s:v 1280:720 -r 25 -i ~/Videos/raw/freeway.yuv -c:v h264_v4l2m2m out/out.h264.mp4
>
>
> linaro at db820 raw]$ ./encode.h264.sh
> ~/Src/git.zoltan.ffmpeg/ffmpeg -loglevel debug -f rawvideo -pix_fmt nv12 -s:v 1280:720 -r 25 -i ~/Videos/raw/freeway.yuv -c:v h264_v4l2m2m out/out.h264.mp4
> -loglevel debug -f rawvideo -pix_fmt nv12 -s:v 1280:720 -r 25 -i ~/Videos/raw/freeway.yuv -c:v h264_v4l2m2m out/out.h264.mp4
> ffmpeg version N-87159-ga1e41bc747 Copyright (c) 2000-2017 the FFmpeg developers
> built with gcc 6.3.0 (Debian 6.3.0-21) 20170628
> configuration: --disable-optimizations --disable-stripping
> libavutil 55. 74.100 / 55. 74.100
> libavcodec 57.104.101 / 57.104.101
> libavformat 57. 80.100 / 57. 80.100
> libavdevice 57. 8.100 / 57. 8.100
> libavfilter 6.101.100 / 6.101.100
> libswscale 4. 7.103 / 4. 7.103
> libswresample 2. 8.100 / 2. 8.100
> Splitting the commandline.
> Reading option '-loglevel' ... matched as option 'loglevel' (set logging level) with argument 'debug'.
> Reading option '-f' ... matched as option 'f' (force format) with argument 'rawvideo'.
> Reading option '-pix_fmt' ... matched as option 'pix_fmt' (set pixel format) with argument 'nv12'.
> Reading option '-s:v' ... matched as option 's' (set frame size (WxH or abbreviation)) with argument '1280:720'.
> Reading option '-r' ... matched as option 'r' (set frame rate (Hz value, fraction or abbreviation)) with argument '25'.
> Reading option '-i' ... matched as input url with argument '/home/linaro/Videos/raw/freeway.yuv'.
> Reading option '-c:v' ... matched as option 'c' (codec name) with argument 'h264_v4l2m2m'.
> Reading option 'out/out.h264.mp4' ... matched as output url.
> Finished splitting the commandline.
> Parsing a group of options: global .
> Applying option loglevel (set logging level) with argument debug.
> Successfully parsed a group of options.
> Parsing a group of options: input url /home/linaro/Videos/raw/freeway.yuv.
> Applying option f (force format) with argument rawvideo.
> Applying option pix_fmt (set pixel format) with argument nv12.
> Applying option s:v (set frame size (WxH or abbreviation)) with argument 1280:720.
> Applying option r (set frame rate (Hz value, fraction or abbreviation)) with argument 25.
> Successfully parsed a group of options.
> Opening an input file: /home/linaro/Videos/raw/freeway.yuv.
> [rawvideo @ 0xaaaafee46490] Opening '/home/linaro/Videos/raw/freeway.yuv' for reading
> [file @ 0xaaaafee46d10] Setting default whitelist 'file,crypto'
> [rawvideo @ 0xaaaafee46490] Before avformat_find_stream_info() pos: 0 bytes read:32768 seeks:0 nb_streams:1
> [rawvideo @ 0xaaaafee46490] All info found
> [rawvideo @ 0xaaaafee46490] Estimating duration from bitrate, this may be inaccurate
> [rawvideo @ 0xaaaafee46490] After avformat_find_stream_info() pos: 1382400 bytes read:1382400 seeks:0 frames:1
> Input #0, rawvideo, from '/home/linaro/Videos/raw/freeway.yuv':
> Duration: 00:00:09.28, start: 0.000000, bitrate: 276480 kb/s
> Stream #0:0, 1, 1/25: Video: rawvideo, 1 reference frame (NV12 / 0x3231564E), nv12, 1280x720, 0/1, 276480 kb/s, 25 tbr, 25 tbn, 25 tbc
> Successfully opened the file.
> Parsing a group of options: output url out/out.h264.mp4.
> Applying option c:v (codec name) with argument h264_v4l2m2m.
> Successfully parsed a group of options.
> Opening an output file: out/out.h264.mp4.
> [file @ 0xaaaafee53930] Setting default whitelist 'file,crypto'
> Successfully opened the file.
> Stream mapping:
> Stream #0:0 -> #0:0 (rawvideo (native) -> h264 (h264_v4l2m2m))
> Press [q] to stop, [?] for help
> cur_dts is invalid (this is harmless if it occurs once at the start per stream)
> [rawvideo @ 0xaaaafee500a0] PACKET SIZE: 1382400, STRIDE: 1920
> detected 4 logical cores
> [graph 0 input from stream 0:0 @ 0xaaaafee55430] Setting 'video_size' to value '1280x720'
> [graph 0 input from stream 0:0 @ 0xaaaafee55430] Setting 'pix_fmt' to value '25'
> [graph 0 input from stream 0:0 @ 0xaaaafee55430] Setting 'time_base' to value '1/25'
> [graph 0 input from stream 0:0 @ 0xaaaafee55430] Setting 'pixel_aspect' to value '0/1'
> [graph 0 input from stream 0:0 @ 0xaaaafee55430] Setting 'sws_param' to value 'flags=2'
> [graph 0 input from stream 0:0 @ 0xaaaafee55430] Setting 'frame_rate' to value '25/1'
> [graph 0 input from stream 0:0 @ 0xaaaafee55430] w:1280 h:720 pixfmt:nv12 tb:1/25 fr:25/1 sar:0/1 sws_param:flags=2
> [AVFilterGraph @ 0xaaaafee47080] query_formats: 3 queried, 2 merged, 0 already done, 0 delayed
> [h264_v4l2m2m @ 0xaaaafee51b70] probing device /dev/video1
> [h264_v4l2m2m @ 0xaaaafee51b70] driver 'qcom-venus' on card 'Qualcomm Venus video decoder'
> [h264_v4l2m2m @ 0xaaaafee51b70] output format not supported
> [h264_v4l2m2m @ 0xaaaafee51b70] probing device /dev/video0
> [h264_v4l2m2m @ 0xaaaafee51b70] driver 'qcom-venus' on card 'Qualcomm Venus video encoder'
> [h264_v4l2m2m @ 0xaaaafee51b70] Using device /dev/video0
> [h264_v4l2m2m @ 0xaaaafee51b70] driver 'qcom-venus' on card 'Qualcomm Venus video encoder'
> [h264_v4l2m2m @ 0xaaaafee51b70] v4l2_out queuing 16 buffers
> [h264_v4l2m2m @ 0xaaaafee51b70] v4l2_cap queuing 4 buffers
> [h264_v4l2m2m @ 0xaaaafee51b70] Encoder: number of B-frames = 0
> [h264_v4l2m2m @ 0xaaaafee51b70] Encoder: header mode = 0
> [h264_v4l2m2m @ 0xaaaafee51b70] Encoder: bit rate = 200000
> [h264_v4l2m2m @ 0xaaaafee51b70] Encoder: gop size = 12
> [h264_v4l2m2m @ 0xaaaafee51b70] Encoder Context: id (28), profile (-99), frame rate(25/1), number b-frames (0), gop size (12), bit rate (200000), qmin (2), qmax (31)
> [h264_v4l2m2m @ 0xaaaafee51b70] h264 profile not found
> [h264_v4l2m2m @ 0xaaaafee51b70] Encoder adjusted: qmin (0), qmax (51)
> [h264_v4l2m2m @ 0xaaaafee51b70] Encoder: minimum video quantizer scale = 0
> [h264_v4l2m2m @ 0xaaaafee51b70] Encoder: maximum video quantizer scale = 51
> Output #0, mp4, to 'out/out.h264.mp4':
> Metadata:
> encoder : Lavf57.80.100
> Stream #0:0, 0, 1/12800: Video: h264 (h264_v4l2m2m), 1 reference frame (avc1 / 0x31637661), nv12, 1280x720, 0/1, q=2-31, 200 kb/s, 25 fps, 12800 tbn, 25 tbc
> Metadata:
> encoder : Lavc57.104.101 h264_v4l2m2m
> Clipping frame in rate conversion by 0.000008
> [h264_v4l2m2m @ 0xaaaafee51b70] enqued raw frame 0
> [rawvideo @ 0xaaaafee500a0] PACKET SIZE: 1382400, STRIDE: 1920
> [h264_v4l2m2m @ 0xaaaafee51b70] enqued raw frame 1
> [rawvideo @ 0xaaaafee500a0] PACKET SIZE: 1382400, STRIDE: 1920
> [h264_v4l2m2m @ 0xaaaafee51b70] enqued raw frame 2
> [rawvideo @ 0xaaaafee500a0] PACKET SIZE: 1382400, STRIDE: 1920
> [h264_v4l2m2m @ 0xaaaafee51b70] enqued raw frame 3
> [rawvideo @ 0xaaaafee500a0] PACKET SIZE: 1382400, STRIDE: 1920
> [h264_v4l2m2m @ 0xaaaafee51b70] enqued raw frame 4
> [rawvideo @ 0xaaaafee500a0] PACKET SIZE: 1382400, STRIDE: 1920
> [h264_v4l2m2m @ 0xaaaafee51b70] enqued raw frame 5
> [rawvideo @ 0xaaaafee500a0] PACKET SIZE: 1382400, STRIDE: 1920
> [h264_v4l2m2m @ 0xaaaafee51b70] enqued raw frame 6
> [rawvideo @ 0xaaaafee500a0] PACKET SIZE: 1382400, STRIDE: 1920
> [h264_v4l2m2m @ 0xaaaafee51b70] enqued raw frame 7
> [rawvideo @ 0xaaaafee500a0] PACKET SIZE: 1382400, STRIDE: 1920
> [h264_v4l2m2m @ 0xaaaafee51b70] enqued raw frame 8
> [rawvideo @ 0xaaaafee500a0] PACKET SIZE: 1382400, STRIDE: 1920
> [h264_v4l2m2m @ 0xaaaafee51b70] enqued raw frame 9
> [rawvideo @ 0xaaaafee500a0] PACKET SIZE: 1382400, STRIDE: 1920
> [h264_v4l2m2m @ 0xaaaafee51b70] enqued raw frame 10
> [rawvideo @ 0xaaaafee500a0] PACKET SIZE: 1382400, STRIDE: 1920
> [h264_v4l2m2m @ 0xaaaafee51b70] enqued raw frame 11
> [rawvideo @ 0xaaaafee500a0] PACKET SIZE: 1382400, STRIDE: 1920
> [h264_v4l2m2m @ 0xaaaafee51b70] enqued raw frame 12
> [rawvideo @ 0xaaaafee500a0] PACKET SIZE: 1382400, STRIDE: 1920
> [h264_v4l2m2m @ 0xaaaafee51b70] enqued raw frame 13
> [rawvideo @ 0xaaaafee500a0] PACKET SIZE: 1382400, STRIDE: 1920
> [h264_v4l2m2m @ 0xaaaafee51b70] enqued raw frame 14
> [rawvideo @ 0xaaaafee500a0] PACKET SIZE: 1382400, STRIDE: 1920
> [h264_v4l2m2m @ 0xaaaafee51b70] enqued raw frame 15
> [rawvideo @ 0xaaaafee500a0] PACKET SIZE: 1382400, STRIDE: 1920
> [h264_v4l2m2m @ 0xaaaafee51b70] enqued raw frame 16
> [h264_v4l2m2m @ 0xaaaafee51b70] dequeued encoded packet 0
> [h264_v4l2m2m @ 0xaaaafee51b70] dequeued encoded packet 1
> [h264_v4l2m2m @ 0xaaaafee51b70] dequeued encoded packet 2
> [h264_v4l2m2m @ 0xaaaafee51b70] dequeued encoded packet 3
> [h264_v4l2m2m @ 0xaaaafee51b70] dequeued encoded packet 4
> [h264_v4l2m2m @ 0xaaaafee51b70] dequeued encoded packet 5
> [h264_v4l2m2m @ 0xaaaafee51b70] dequeued encoded packet 6
> [h264_v4l2m2m @ 0xaaaafee51b70] dequeued encoded packet 7
> [h264_v4l2m2m @ 0xaaaafee51b70] dequeued encoded packet 8
> [h264_v4l2m2m @ 0xaaaafee51b70] dequeued encoded packet 9
> [h264_v4l2m2m @ 0xaaaafee51b70] dequeued encoded packet 10
> [h264_v4l2m2m @ 0xaaaafee51b70] dequeued encoded packet 11
> [h264_v4l2m2m @ 0xaaaafee51b70] dequeued encoded packet 12
> [h264_v4l2m2m @ 0xaaaafee51b70] dequeued encoded packet 13
> [h264_v4l2m2m @ 0xaaaafee51b70] dequeued encoded packet 14
> [h264_v4l2m2m @ 0xaaaafee51b70] dequeued encoded packet 15
> [h264_v4l2m2m @ 0xaaaafee51b70] dequeued encoded packet 16
>
> [at this point it blocks for ever since it has consumed all the input and we are blocking)
>
> The code in the encoder:
>
> static int v4l2m2m_send_frame(AVCodecContext *avctx, const AVFrame *frame)
> {
> V4L2m2mContext *s = avctx->priv_data;
> V4L2Context *const capture = &s->capture;
> V4L2Context *const output = &s->output;
> int ret;
> static int cnt = 0;
>
> if (!output->streamon) {
> ret = ff_v4l2_context_set_status(output, VIDIOC_STREAMON);
> if (ret) {
> av_log(avctx, AV_LOG_ERROR, "VIDIOC_STREAMOFF failed on output context\n");
> return ret;
> }
> }
>
> if (!capture->streamon) {
> ret = ff_v4l2_context_set_status(capture, VIDIOC_STREAMON);
> if (ret) {
> av_log(avctx, AV_LOG_ERROR, "VIDIOC_STREAMON failed on capture context\n");
> return ret;
> }
> }
>
> ret = ff_v4l2_enqueue_frame(output, frame);
> if (!ret) {
> av_log(avctx, AV_LOG_ERROR, "enqued raw frame %d\n", cnt++);
> }
>
> return ret;
> }
>
> /* Send and receive frame happen on the same thread, hence the need for a polling timeout */
> static int v4l2m2m_receive_packet(AVCodecContext *avctx, AVPacket *avpkt)
> {
> V4L2m2mContext *s = avctx->priv_data;
> V4L2Context *const capture = &s->capture;
> V4L2Context *const output = &s->output;
> unsigned int timeout = -1; /* block for ever until packets are ready to dequeue */
> static int cnt = 0;
> int ret;
>
> if (s->draining)
> goto dequeue;
>
> if (!s->data_queued) {
> if (output->ops.get_buffer(output))
> return AVERROR(EAGAIN);
> else s->data_queued = 1;
> }
>
> dequeue:
>
> ret = ff_v4l2_dequeue_packet(capture, avpkt, timeout);
> if (!ret)
> av_log(avctx, AV_LOG_ERROR, "dequeued encoded packet %d\n", cnt++);
>
> return ret;
> }
So the sequence of calls is:
send_frame(frame 0) -> success
receive_packet() -> EAGAIN
send_frame(frame 1) -> success
receive_packet() -> EAGAIN
...
send_frame(frame 15) -> success
receive_packet() -> EAGAIN
send_frame(frame 16) -> success
receive_packet() -> packet 0
receive_packet() -> packet 1
...
receive_packet() -> packet 15
receive_packet() -> packet 16
receive_packet() -> blocks
This appears correct to me - since EAGAIN has not been returned by a receive_packet() call, it can be called again repeatedly (as documented in avcodec.h). Do you disagree?
I think that the problem is that you are only polling on the frame buffer queue when blocking, so you don't see the packet buffers becoming free in the packet buffer queue - if you did see and dequeue them, you would then return EAGAIN to indicate that more input is needed. (See comments in <e4c6a8d7-798a-dfdb-b748-42936e944829 at jkqxz.net>.)
- Mark
More information about the ffmpeg-devel
mailing list