[FFmpeg-devel] [FEATURE PROPOSAL] Extracting codec-level data to binary files

[Ramiro Polla]

Hi,

On Thu, May 22, 2025 at 1:59 PM Ronald S. Bultje <rsbultje at gmail.com> wrote:
> On Wed, May 21, 2025 at 9:34 AM Timothée <
> timothee.informatique at regaud-chapuy.fr> wrote:
> > I am interested in expanding ffmpeg's capabilities to extract
> > low-level data from video codecs. Specifically, I'd like to implement
> > functionality that would allow exporting frame data, macroblock
> > information, quantization tables, and similar codec-specific elements
> > to binary files for further analysis.
> >
> > After searching through the documentation and existing features, I
> > haven't found similar functionality, though I may have missed
> > something. Has this been implemented before, or are there related
> > features I should examine?
>
> Some older codecs implement minor variants for this, e.g. grep
> for AV_FRAME_DATA_MOTION_VECTORS, which attaches a frame's motion vectors
> to the picture data. I believe there's an example app and possibly a filter
> to overlay MVs on top of the video frame based on this concept. You could
> extend this to cover other (macro)block info. There used to be a variant of
> this for quant-tables also but I can't find it, maybe it was removed.

If I recall correctly, we wanted to move away from exporting more of
this kind of codec-specific information as picture data.

Timothée, this kind of feature depends a lot on the codec that you
want to work with. It will also depend on which syntax elements you
need from the codec. In FFmpeg, there is the CBS code which describes
the bitstream for a few codecs, which might help you.

I have a separate project (called FFglitch), which lets you export
some elements (such as quantization tables, dct coefficients, and
motion vectors) from some codecs (jpeg, mpeg2, mpeg4) to JSON format,
and even modify them in the bitstream.

In the end, it will depend on what you want to do with the extracted
low-level data, and for which codecs.

Ramiro