[FFmpeg-devel] ffv1enc: question about "Cannot allocate worst case packet size, the encoding could fail"

[Jerome Martinez]

On 11/10/2018 14:08, Paul B Mahol wrote:
> On 10/11/18, Jerome Martinez <jerome at mediaarea.net> wrote:
>> Hi,
>>
>> Testing FFmpeg FFV1 encoder on big frames (more than 4K: 4300x3956), I
>> have a warning for each frame encoded (so a lot of warnings!):
>> [ffv1 @ 0000024a6bcfe880] Cannot allocate worst case packet size, the
>> encoding could fail
>>
>> Checking avcodec/ffv1enc.c, it is due to the following lines:
>>       int64_t maxsize =   AV_INPUT_BUFFER_MIN_SIZE
>>                         + avctx->width*avctx->height*37LL*4;
>> [...]
>>      if (maxsize > INT_MAX - AV_INPUT_BUFFER_PADDING_SIZE - 32) {
>>           av_log(avctx, AV_LOG_WARNING, "Cannot allocate worst case
>> packet size, the encoding could fail\n");
>>           maxsize = INT_MAX - AV_INPUT_BUFFER_PADDING_SIZE - 32;
>>       }
>>
>> The value of maxsize is 2517614784, more than INT_MAX, sure, but I don't
>> understand:
>> - why does FFmpeg need to allocate 148 times the size of a frame?
>> - why is having a number > INT_MAX an issue? modern machines are 64-bit
>> and have 8+ GB of RAM, and in practice I currently saw no encoding
>> failure on thousands of frames.
>>
>> Additionally I didn't get why maxsize is reduced ("only" need of 12
>> times the size of the frame) in case of FFV1 version > 3 (experimental
>> right now):
>>       if (f->version > 3)
>>           maxsize = AV_INPUT_BUFFER_MIN_SIZE +
>> avctx->width*avctx->height*3LL*4;
>>
>> maxsize is used for calling ff_alloc_packet2(), which accepts 64-bit
>> numbers.
>>      if ((ret = ff_alloc_packet2(avctx, pkt, maxsize, 0)) < 0)
>>           return ret;
>>
>> Is it possible to reduce this "37x4" multiplier without risk and/or
>> remove maxsize reduction in case of sizeof(size_t) == 8 and/or show the
>> warning only once?
>>
>> Issue can be reproduced with:
>> ./ffmpeg.exe -f lavfi -i testsrc2=size=4300x3956 -t 0.040 -pix_fmt
>> rgba64 rgba64.dpx
>> ./ffmpeg.exe -i rgba64.dpx -c:v ffv1 rgba64.mkv
> You are not using latest version of FFv1.


As far as I know, version 3 is the latest (stable) version of FFV1, and 
I use this one.
version > 3 are experimental.
I am not sure that advising to use an experimental version for "fixing" 
a warning is good.

Additionally, in theory a 16kx16k content would still produce this 
warning with FFV1 (experimental) v4 encoding, even if there is 16 
Terabytes of RAM (more than enough for having the MAX_INT = 2 GiB raw 
frame in RAM), so I understand that the warning would still be there (in 
theory) with FFV1 (experimental) v4.


> In latest version, encoder will not produce packets which are bigger than
> input ones -- uncompressed raw. And instead if that happens it will encode
> as raw video.
>
> Theoretically output packets for old versions of FFv1 could be several
> times bigger than raw input frame.


In this case, the ratio (148 times the size of input) is proportional to 
the input frame size, and the test is on a fixed value (MAX_INT).
I get now the reason to reserve more than the size of the input frame.
I don't get the relationship between your explanation and these hard 
coded values (148 times the size of input and MAX_INT limit).

Why 148x (37x4) and not 147 or 149?
Why a warning and maxsize forced to MAX_INT if maxsize > MAX_INT? Is it 
an internal constraint of FFmpeg which would be not able to store 
encoded frames bigger than MAX_INT?

Can any of that be changed e.g. warning only once for avoiding "spam" of 
such warning which hides other warnings during encoding?

Jérôme