[FFmpeg-devel] [PATCH 1/1] avcodec/mpegutils: add motion_vec debug mode

[Chema Gonzalez]

Hi Paul,

I tried to understand the per-frame side-data (and metadata)
mechanism. Adding my notes here in case they help a future reader.

Metadata and side-data seem like similar mechanisms to add auxiliary
information to each frame. Main difference seems to be that metadata
is simpler (a key/value dictionary), while side-data allows adding any
data struct.

Metadata operation is also simpler: There are filters that generate
metadata (e.g. "signalstats" generates key-values such as
`lavfi.signalstats.YMIN=3`, while "silencedetect" generates
audio-related key-values like `lavfi.silence_start=0`). There are also
2x filters that print metadata ("vf_metadata" and "af_ametadata"). So
e.g. to see what signalstats/silencedetect are generating, you can do:

```
$ ffmpeg -i in.264 -vf signalstats,metadata=mode=print -f null -
...
[Parsed_metadata_1 @ ...] frame:0    pts:0       pts_time:0
[Parsed_metadata_1 @ ...] lavfi.signalstats.YMIN=3
...
```

or:
```
$ ffmpeg -y -i in.wav -af "silencedetect=n=-10dB:d=1,ametadata=print"
/tmp/out.wav
...
[Parsed_ametadata_1 @ ...] frame:23   pts:47104   pts_time:0.981333
[Parsed_ametadata_1 @ ...] lavfi.silence_start=0
...
```

Side-data operation is more complicated. There is some side-data
information already generated (e.g. SEI_UNREGISTERED side-data). Some
information requires explicitly asking for it. For example, the
`MOTION_VECTORS` side-data, you need to enable
AV_CODEC_FLAG2_EXPORT_MVS (which means calling ffmpeg/ffplay/ffprobe
with "-flags2 +export_mvs"). The main filter to print side-data
information is showinfo (`vf_showinfo` and `af_ashowinfo`). Now, the
`vf_showinfo` filter only knows how to dump some of the side-data
structs. In particular, it does not know how to dump MOTION_VECTORS
side-data. So, if we add the motion vectors, and then ask showinfo to
print it, we see:

```
$ ffmpeg -hide_banner -flags2 +export_mvs -export_side_data +mvs
-export_side_data +prft -export_side_data +venc_params
-export_side_data +film_grain -i /tmp/in.264 -vf showinfo -f null
/dev/null
...
-- frame 0 is a key frame: We can see SEI_UNREGISTERED and VIDEO_ENC_PARAMS info
[Parsed_showinfo_0 @ 0x308fd40] config in time_base: 1/1200000, frame_rate: 25/1
[Parsed_showinfo_0 @ 0x308fd40] config out time_base: 0/0, frame_rate: 0/0
[Parsed_showinfo_0 @ 0x308fd40] n:   0 pts:      0 pts_time:0
duration:  48000 duration_time:0.04    pos:        0 fmt:yuv420p
sar:0/1 s:1920x1080 i:P iskey:1 type:I checksum:F6BBEA9F
plane_checksum:[AFB1432E 63F2F255 2887B50D] mean:[109 119 138]
stdev:[43.0 12.7 13.3]
[Parsed_showinfo_0 @ 0x308fd40]   side data - User Data Unregistered:
[Parsed_showinfo_0 @ 0x308fd40] UUID=47564adc-5c4c-433f-94ef-c5113cd143a8
[Parsed_showinfo_0 @ 0x308fd40] User Data=01ffccccff0200e4dd42
[Parsed_showinfo_0 @ 0x308fd40]
[Parsed_showinfo_0 @ 0x308fd40]   side data - video encoding
parameters: type 1; qp=26; 8160 blocks;
[Parsed_showinfo_0 @ 0x308fd40] color_range:tv color_space:bt709
color_primaries:bt709 color_trc:bt709
...
-- frame 1 is a P-frame: we can see VIDEO_ENC_PARAMS info, and a
complain about "side-data type 8" (MOTION_VECTORS)
[Parsed_showinfo_0 @ 0x308fd40] n:   1 pts:  48000 pts_time:0.04
duration:  48000 duration_time:0.04    pos:   259304 fmt:yuv420p
sar:0/1 s:1920x1080 i:P iskey:0 type:B checksum:BC4E5C12
plane_checksum:[AEA8857A 34697DA4 805E58E5] mean:[109 119 138]
stdev:[43.0 12.6 13.3]
[Parsed_showinfo_0 @ 0x308fd40]   side data - video encoding
parameters: type 1; qp=26; 8160 blocks;
-- showinfo does not dump MOTION_VECTORS side-data
[Parsed_showinfo_0 @ 0x308fd40]   side data - unknown side data type 8
(547280 bytes)
[Parsed_showinfo_0 @ 0x308fd40]
[Parsed_showinfo_0 @ 0x308fd40] color_range:tv color_space:bt709
color_primaries:bt709 color_trc:bt709
...
```

So the best way right now to see the MVs is to use
`doc/examples/extract_mvs`, which does exactly that:

```
$ make examples -j
...
$ doc/examples/extract_mvs in.264 | head -40 | \
    csvcut -C framenum,source,flags |csvlook
| blockw | blockh |  srcx | srcy |  dstx | dsty | motion_x | motion_y
| motion_scale |
| ------ | ------ | ----- | ---- | ----- | ---- | -------- | --------
| ------------ |
|     16 |     16 |    20 |   26 |     8 |    8 |       49 |       72
|            4 |
|     16 |     16 |   152 |   15 |   136 |    8 |       65 |       28
|            4 |
|     16 |      8 |   360 |    3 |   360 |    4 |        1 |       -6
|            4 |
|     16 |      8 |   360 |   13 |   360 |   12 |       -1 |        4
|            4 |
|     16 |     16 |   440 |   10 |   440 |    8 |        3 |       10
|            4 |
|      8 |     16 |   829 |    7 |   836 |    8 |      -31 |       -6
|            4 |
|      8 |     16 |   844 |    7 |   844 |    8 |       -1 |       -4
|            4 |
```

> Yes, it's called codecview. We can help understand how it works if you ask more specific questions, but something like "git grep EXPORT_DATA_MVS ../libavcodec/mpeg*.c" and checking the complement code in codecview should explain the basics.

codecview will get the motion vectors (assuming you asked for them),
and will overlay arrows showing the MV on top of the original video.

```
$ ffmpeg -flags2 +export_mvs -i input -vf codecview=mv=pf+bf+bb output.%04d.png
```

Adding one of the resulting frames in the next email.

-Chema


On Sat, Sep 3, 2022 at 5:08 AM Ronald S. Bultje <rsbultje at gmail.com> wrote:
>
> Hi Chema,
>
> On Fri, Sep 2, 2022 at 11:12 AM Chema Gonzalez <chema at berkeley.edu> wrote:
>>
>> So is there a filter that already dumps this information?
>
>
> Yes, it's called codecview. We can help understand how it works if you ask more specific questions, but something like "git grep EXPORT_DATA_MVS ../libavcodec/mpeg*.c" and checking the complement code in codecview should explain the basics.
>
> Ronald