[FFmpeg-devel] [PATCH] rl.h: Use on-stack temporary VLC tables instead of having them static.

[Reimar Döffinger]

On Mon, Sep 01, 2014 at 10:19:36PM +0200, Michael Niedermayer wrote:
> On Mon, Sep 01, 2014 at 08:59:43PM +0200, Reimar Döffinger wrote:
> > Signed-off-by: Reimar Döffinger <Reimar.Doeffinger at gmx.de>
> > ---
> >  libavcodec/mpeg12.c    | 21 ++++++++++-----------
> >  libavcodec/mpegvideo.c | 12 ++++++++----
> >  libavcodec/rl.h        |  8 ++------
> >  3 files changed, 20 insertions(+), 21 deletions(-)
> 
> assuming the amount is ok on the stack the patch LGTM

I reduced the mpeg12 to the 680 required - since that's a
local function I think it's unnecessary to be conservative there.
For the other one...
Well, HP-UX seems to use 16 kB stack. I guess that we'd have
problems there (probably even without the patch). But I
don't think we'll have any volunteers to test and fix FFmpeg
on that (I doubt anyone would use HP-UX voluntarily at all..).
Next closest is OpenBSD with 64 kB (though maybe that is outdated
information?).
We'll be testing that, but of course that is small enough that
it can easily fail if the calling application already used up quite
a bit of it (and the ff_init_vlc_sparse uses up another 3 kB).
On the other hand, ffv1enc uses a lot more than that:
uint8_t best_state[256][256];
(I suspect that should probably be fixed > 64 kB I'd surely consider
unreasonable, and it will definitely crash on older OpenBSD at least).