[FFmpeg-devel] [PATCH] Fix quadratic memory use in ff_h2645_extract_rbsp() when multiple NALUs exist in packet.

[Michael Niedermayer]

Hi

On Mon, Oct 23, 2017 at 04:43:55PM -0700, Niki Bowe wrote:
> On Thu, Oct 19, 2017 at 3:39 PM, Carl Eugen Hoyos <ceffmpeg at gmail.com>
> wrote:
> 
> > 2017-10-19 20:46 GMT+02:00 Nikolas Bowe <nbowe-at-google.com at ffmpeg.org>:
> > > Found via fuzzing.
> > > /tmp/poc is a 1 MB mpegts file generated via fuzzing, where 1 packet has
> > many NALUs
> > > Before this change:
> > >   $ /usr/bin/time -f "\t%M Max Resident Set Size (Kb)"  ./ffprobe
> > /tmp/poc 2>&1 | tail -n 1
> > >         2158192 Max Resident Set Size (Kb)
> > > After this change:
> > >   $ /usr/bin/time -f "\t%M Max Resident Set Size (Kb)"  ./ffprobe
> > /tmp/poc 2>&1 | tail -n 1
> > >         1046812 Max Resident Set Size (Kb)
> >
> > This does not look like a fix for a "quadratic" memory consumption or
> > do I misunderstand?
> >
> 
> Before this patch, for each NALU in the packet, rbsp_buffer would be sized
> from the start of the NALU to the end of the packet, not the end of the
> NALU.
> This would occur for each NALU in the packet. Total memory allocated in all
> the rbsp_buffers for all the NALUs in the packet would be N + (N+x1) +
> (N+x2) + ...
> This is quadratic in the number of NALUs in the packet.

A better solution would be to allocate a buffer sized based on the
whole packet and then have all teh NALs point into that.
That way theres no need to know the size of the first NAL during
allocation.
This should have even lower memory overhead than your code
fewer alloc/free calls, and no speed loss

[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

I am the wisest man alive, for I know one thing, and that is that I know
nothing. -- Socrates
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: Digital signature
URL: <http://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20171031/51091d58/attachment.sig>