[FFmpeg-devel] [PATCH] Unbreak av_malloc_max(0) API/ABI

[Joakim Tjernlund]

On Tue, 2020-10-13 at 20:11 +0200, Andreas Rheinhardt wrote:
> 
> Joakim Tjernlund:
> > On Tue, 2020-10-13 at 15:24 +0100, Derek Buitenhuis wrote:
> > > 
> > > On 13/10/2020 15:19, Joakim Tjernlund wrote:
> > > > For now just fixing av_malloc_max(0) will do, av_max_malloc2() etc. is beyond this patch.
> > > 
> > > As Ronald mentioned, if you're going to fix the ABI/API, you should actually
> > > fix every use of that ABI/API, not just the one you care about (0).
> > > 
> > > So max values of 1-31 should also be handled.
> > 
> > OK, how far do you want to take this, will this do?
> >     if (max < 32)
> >         max = INT_MAX;
> > 
> 
> The old behaviour was to effectively limit allocations to max - 32 in
> this case (near to SIZE_MAX), not to INT_MAX. And it does not restore
> default behaviour (the effective default limit was INT_MAX - 32).
> 
> > Complete API fix would need to revert the patch that caused the break to begin with and I don't think you want that ?
> > 
> Actually the old behaviour was against the documented API (or where do
> you see the 32 mentioned in the docs?).

The old behaviour in code, what the docs say isn't relevant to a failing app.

> 
> - Andreas
> 
> PS: My rationale for this patch was to match actual behaviour and
> documented behaviour and to make the full range of INT available for
> allocations if the maximum has not been overridden by av_max_alloc().
> Otherwise allocations of e.g. packets with sizes in the range (INT_MAX -
> 31 - AV_INPUT_BUFFER_PADDING_SIZE)..(INT_MAX -
> AV_INPUT_BUFFER_PADDING_SIZE) will fail.

Please suggest a patch to your liking, I am just trying to repair the breakage, I am not a ffmpeg dev.

 Jocke