[FFmpeg-devel] [PATCH] avformat: add AV1 RTP depacketizer and packetizer

[Tristan Matthews]

Hi,

On Friday, December 6th, 2024 at 12:39 PM, Tristan Matthews via ffmpeg-devel <ffmpeg-devel at ffmpeg.org> wrote:

> Hi (also apologies if my client mangles the inline version of the patch, it's the first time I've tried to review an attached patch with it)...
> 
> On Monday, December 2nd, 2024 at 9:57 AM, Chris Hodges Chris.Hodges at axis.com wrote:
> 
> > Hi,
> > 
> > On 11/25/24 18:47, Tristan Matthews via ffmpeg-devel wrote:
> > 
> > > One nit I'd add is that since the RTP AV1 spec is still in draft (according to https://aomediacodec.github.io/av1-rtp-spec/) this feature should probably be marked experimental as is done for VP9 in RTP, see:
> > > https://git.ffmpeg.org/gitweb/ffmpeg.git/blob/f8e91ab05ff3d111626ab8a3b5d570865a934f07:/libavformat/rtpenc.c#l221
> > > 
> > > in which case CLI users will have to add `-strict experimental` to their options.
> > 
> > I've added this as suggested.
> > 
> > > For the keyframe detection issue I'm not sure if this is something missing in FFMPEG's RTP stack (e.g. I've noticed that both GStreamer and libwebrtc signal that a buffer contains a keyframe at a higher level), but if not could you set it if you're dealing with a FRAME OBU of type 0 (keyframe) or 2 (intra-only)? You'd need to parse the OBU to extract that however.
> > 
> > It turns out this was a lapsus on my side, when I used the flags field
> > in the RTPMuxContext structure instead of the flags in AVPacket. After
> > transferring the key frame information to the encoder function, I could
> > remove the workaround.
> > 
> > I also added the key frame info to the AVPacket during demuxing that was
> > missing before.
> > 
> > New patch attached.
> > 
> > --
> > Best regards, Chris_______________________________________________

<snip>

Aside from my minor comments on the patch, one thing that came up in terms of testing is that while the packetization seems to be working well, I'm unable to depacketize a stream being sent by ffmpeg *with* ffmpeg as the receiver.

This is fairly easy to reproduce locally, start an RTSP server:
$ docker run --rm -it --network=host bluenviron/mediamtx:latest

Push an RTP stream:
$ ./ffmpeg -re -i Chimera-AV1-8bit-1280x720-3363kbps.ivf -c copy -strict experimental -f rtsp rtsp://127.0.0.1:8554/foobar

Pull the same stream:
$ ./ffmpeg -i rtsp://127.0.0.1:8554/foobar -copy -f matroska foobar.mkv

This fails with a bunch of errors on trying decode e.g.:
[libdav1d @ 0x5c50bb8d1580] No sequence header available: unable to decode frame header.
[libdav1d @ 0x5c50bb8d1580] obu_forbidden_bit out of range: 1, but must be in [0,0].
etc.

and finally:
[matroska @ 0x5c50bb8f5f00] dimensions not set

The good news is that I *can* pull that same stream and save it to disk with e.g. gstreamer so it's clearly not that far off. It's not clear to me yet if this is an issue on the packetization side, the intermediate RTSP server side, the depacketization side or some combination therein.

Best,
Tristan