[FFmpeg-devel] [EXT] [PATCH v5 7/7] avcodec/v4l2_m2m_dec: setup capture queue before enqueue the first frame

[Ming Qian]

> From: Andriy Gelman <andriy.gelman at gmail.com>
> Sent: Monday, April 4, 2022 6:29 AM
> To: Ming Qian <ming.qian at nxp.com>
> Cc: FFmpeg development discussions and patches <ffmpeg-
> devel at ffmpeg.org>
> Subject: Re: [FFmpeg-devel] [EXT] [PATCH v5 7/7] avcodec/v4l2_m2m_dec:
> setup capture queue before enqueue the first frame
> 
> Caution: EXT Email
> 
> Hi Ming,
> 
> Sorry for the late reply.
> 
> On Tue, 29. Mar 06:51, Ming Qian wrote:
> > > From: Andriy Gelman [mailto:andriy.gelman at gmail.com]
> > > Sent: Saturday, March 26, 2022 11:43 PM
> > > To: FFmpeg development discussions and patches
> > > <ffmpeg-devel at ffmpeg.org>
> > > Cc: Ming Qian <ming.qian at nxp.com>
> > > Subject: Re: [FFmpeg-devel] [EXT] [PATCH v5 7/7]
> avcodec/v4l2_m2m_dec:
> > > setup capture queue before enqueue the first frame
> > >
> > > Caution: EXT Email
> > >
> > > Hi Ming,
> > >
> > > On Mon, 21. Mar 07:27, Ming Qian wrote:
> > > > Hi Andriy,
> > > >
> > > >     What do you think of this patch?
> > > >
> > > >     The Initialization flow defined in
> > > linux/Documentation/userspace-api/media/v4l/dev-decoder.rst
> > > > 1. Set the coded format on OUTPUT via VIDIOC_S_FMT().
> > > > 2. Allocate source (bytestream) buffers via VIDIOC_REQBUFS() on
> OUTPUT.
> > > > 3. Start streaming on the OUTPUT queue via VIDIOC_STREAMON().
> > > > 4. This step only applies to coded formats that contain resolution
> > > information in the stream. Continue queuing/dequeuing bytestream
> > > buffers to/from the OUTPUT queue via VIDIOC_QBUF() and
> > > VIDIOC_DQBUF(). The buffers will be processed and returned to the
> > > client in order, until required metadata to configure the CAPTURE
> > > queue are found. This is indicated by the decoder sending a
> > > V4L2_EVENT_SOURCE_CHANGE event with changes set to
> V4L2_EVENT_SRC_CH_RESOLUTION.
> > > >       Note: A client capable of acquiring stream parameters from
> > > > the
> > > bytestream on its own may attempt to set the width and height of the
> > > OUTPUT format to non-zero values matching the coded size of the
> > > stream, skip this step and continue with the Capture Setup sequence.
> > > >
> > > > 5. Continue with the Capture Setup sequence.
> > > >
> > > >     In ffmpeg's implementation, ffmpeg will set non-zero width and
> > > > height
> > > on output queue, so the step 4 should be skipped, and setup the
> > > capture queue directly.
> > > >     So the flow should be:
> > > > 1. Set the coded format and valid resolution on OUTPUT via
> VIDIOC_S_FMT().
> > > > 2. Allocate source (bytestream) buffers via VIDIOC_REQBUFS() on
> OUTPUT.
> > > > 3. Start streaming on the OUTPUT queue via VIDIOC_STREAMON().
> > > > 5. Continue with the Capture Setup sequence.
> > > >
> > > > And this patch is just following the above flow.
> > > >
> > > > Ming
> > >
> > > The v4 version didn't work for me on odroid xu4
> > >
> https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fffm
> > >
> peg.o%2F&data=04%7C01%7Cming.qian%40nxp.com%7Cfa858e4a13434
> ee727
> > >
> 3508da15c1695b%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C1%7C6378
> 4621
> > >
> 7684591813%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIj
> oiV2luM
> > >
> zIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=wl%2BZaEBNIPZ
> TOyv
> > > 7dgmnHhSQyAs37PjZCkvhwPL%2B6xg%3D&reserved=0
> > > rg%2Fpipermail%2Fffmpeg-devel%2F2022-
> January%2F290679.html&dat
> > >
> a=04%7C01%7Cming.qian%40nxp.com%7Cca988d592253473cbac408da0f3f4
> > > 801%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637839061704
> > >
> 093567%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2
> l
> > >
> uMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=I80bpE70S6JI
> > > 8A%2BaXQGq4JyxL0eBbtLN58p5SpqlA6s%3D&reserved=0
> > >
> > > I haven't had time retest but don't think anything has changed.
> > >
> > > --
> > > Andriy
> >
> > Hi Andriy,
> >     I check the S5P MFC Video Codec driver, and indeed it requires the client
> enqueue the output buffer which contains the sequence header. And in
> vidioc_g_fmt(), driver will block and wait until the header is parsed.
> >     And indeed my patch will not work on it.
> >     But it seems don't meet the specification.
> >     For S5P MFC, the step 4 can't be skipped, and the client should wait the
> V4L2_EVENT_SOURCE_CHANGE event before the step 5 setting up capture
> queue.
> >     The ffmpeg v4l2 decoder don't skip step 4, but also doesn't wait the
> V4L2_EVENT_SOURCE_CHANGE event.
> 
> >
> > So in current, we should make it work on more devices instead of more
> spec?
> 
> Personally I'd prefer not to break users. Odroid xu4 is a popular board afaik.
> But I'm also not the maintainer of v4l2m2m.
> 
> IMO it would be better the fix the problem in the driver first.
> 

Hi Andriy,
    Thanks for your reply. Maybe I can try to work on the driver first.

Ming.

> >
> >     The Initialization flow defined in linux/Documentation/userspace-
> api/media/v4l/dev-decoder.rst
> >       1. Set the coded format on OUTPUT via VIDIOC_S_FMT().
> >       2. Allocate source (bytestream) buffers via VIDIOC_REQBUFS() on
> OUTPUT.
> >       3. Start streaming on the OUTPUT queue via VIDIOC_STREAMON().
> >       4. This step only applies to coded formats that contain resolution
> information in the stream. Continue queuing/dequeuing bytestream buffers
> to/from the OUTPUT queue via VIDIOC_QBUF() and VIDIOC_DQBUF(). The
> buffers will be processed and returned to the client in order, until required
> metadata to configure the CAPTURE queue are found. This is indicated by the
> decoder sending a V4L2_EVENT_SOURCE_CHANGE event with changes set
> to V4L2_EVENT_SRC_CH_RESOLUTION.
> >       Note: A client capable of acquiring stream parameters from the
> bytestream on its own may attempt to set the width and height of the
> OUTPUT format to non-zero values matching the coded size of the stream,
> skip this step and continue with the Capture Setup sequence.
> >       5. Continue with the Capture Setup sequence.
> 
> 
> --
> Andriy