[FFmpeg-devel] [PATCH] Fix quadratic memory use in ff_h2645_extract_rbsp() when multiple NALUs exist in packet.

[Hendrik Leppkes]

On Fri, Nov 3, 2017 at 12:38 AM, Derek Buitenhuis
<derek.buitenhuis at gmail.com> wrote:
> On 11/2/2017 10:48 PM, Kieran Kunhya wrote:
>> I have tried this using the following patch but it does not work:
>> https://www.irccloud.com/pastebin/qobTcW9d/
>>
>> Nothing obviously seems wrong so I suspect it's not possible to do this
>> whilst reusing the code between decoder and parser.
>> The old code use to write to a per-slice context, not a per packet one so
>> this might be related.
>
> As discussed on IRC, the issue was dangling pointers due to the realloc.
>
> It can be allocated based on the full packet size, outside the function,
> or on first call, I think.
>
> Aside:
>
> Do we have some preferred way to address the cumulative memory issue
> I mentioned in my previous mail, or is it a DNC situation? I haven't
> thought of a good way to solve that - only stuff like going back
> to in-place NAL parsing, or shrinking the buffer after some time based
> on some metric (ew...).
>

Personally I don't think the problem of a single buffer getting way
too large is a huge problem, at least it should not accumulate and
keep growing. Its also not a "new" problem, the old code would've
suffered from the same issue as well, since it also cached the rbsp
buffer - just only one, not multiple.

Another option would be to re-arrange the API to keep it shared but go
back to an iterator-style parsing of the NALs. Parse one, process it,
parse the next. That way you can also avoid the issue with a multitude
of NAL structs being allocated and sitting there forever.
Obviously H264 would be compatible with that, as its how it was used
before. I didn't check HEVC, however.

- Hendrik