[FFmpeg-devel] [PATCH] avformat: add AV1 RTP depacketizer and packetizer

[Tristan Matthews]

Sent with Proton Mail secure email.

On Thursday, December 12th, 2024 at 4:18 AM, Chris Hodges <Chris.Hodges at axis.com> wrote:

> Hi Tristan,
>
> I have a draft reply ongoing the last couple of days, but I got
> sidetracked with other work, sorry. Will send it out without a new patch
> so we can discuss the changes and it gives me more time to look into the
> issue below.
>
> On 12/11/24 23:53, Tristan Matthews via ffmpeg-devel wrote:
>
> > Aside from my minor comments on the patch, one thing that came up in terms of testing is that while the packetization seems to be working well, I'm unable to depacketize a stream being sent by ffmpeg with ffmpeg as the receiver.
>
>
> There was a bug that triggered on fragmented OBUs >= 16384 bytes if it
>
> was not the last OBU in the TU:
>
> // move existing data for OBU back a bit
> memmove(lebptr + extra_bytes, lebptr,
> pkt->size - extra_bytes - data->frag_pkt_leb_pos);
>
> + // move pktpos further down for following OBUs in same
> packet.
> + pktpos += extra_bytes;
>
> This might have been one of the problems.
>
> > This is fairly easy to reproduce locally, start an RTSP server:
> > $ docker run --rm -it --network=host bluenviron/mediamtx:latest
> >
> > Push an RTP stream:
> > $ ./ffmpeg -re -i Chimera-AV1-8bit-1280x720-3363kbps.ivf -c copy -strict experimental -f rtsp rtsp://127.0.0.1:8554/foobar
> >
> > Pull the same stream:
> > $ ./ffmpeg -i rtsp://127.0.0.1:8554/foobar -copy -f matroska foobar.mkv
>
>
> Thanks for these examples, helps a lot!
>
> > This fails with a bunch of errors on trying decode e.g.:
> > [libdav1d @ 0x5c50bb8d1580] No sequence header available: unable to decode frame header.
> > [libdav1d @ 0x5c50bb8d1580] obu_forbidden_bit out of range: 1, but must be in [0,0].
> > etc.
>
>
> I don't seem to be able to replay the Chimera video directly with my
> local build of ffplay, get's all choppy and skips big deals of the
> video. Might be an outdated libdav1d?
>
> Anyway, the AV1 RTP decoder currently does not wait for a keyframe
> before sending down frames. Should it do that? Also, should it wait for
> another keyframe if any frame or packet had been dropped (the GOP is
> going to be useless for the decoder anyway)?

That's a good question, I don't believe any of the RTP depacketizers in libavformat do this currently so maybe best left as a future enhancement (behind an option/flag). 

On a related note, one feature that only the VP8 depacketizer supports now is signalling via RTCP that it needs a keyframe (in the event of loss or the start of a stream), to do this you just have to implement the .need_keyframe callback (see https://git.ffmpeg.org/gitweb/ffmpeg.git/blob/eb79c316c7b205f45e7003e91f77c253818e6685:/libavformat/rtpdec_vp8.c#l274)

>
> > The good news is that I can pull that same stream and save it to disk with e.g. gstreamer so it's clearly not that far off. It's not clear to me yet if this is an issue on the packetization side, the intermediate RTSP server side, the depacketization side or some combination therein.
>
>
> I haven't been working with such complex frame hierarchies, but I will
> try to get the Chimera working.

FWIW I was seeing similar behaviour with smaller/simpler files as well.