[Ffmpeg-devel-irc] ffmpeg-devel.log.20181229

[burek]

[00:53:49 CET] <cone-695> ffmpeg 03Michael Niedermayer 07master:fccba32b4c4f: tools: add target_dec_fate.sh
[02:28:32 CET] <ShadowOfThePast> hello guys
[02:28:58 CET] <ShadowOfThePast> i'm losing my mind tracing an ENOMEM issue in libavformat
[02:29:53 CET] <ShadowOfThePast> is there any sorcery involved in the av_fast_realloc function?
[02:31:03 CET] <ShadowOfThePast> i added some logging and it returns 1114144 in its size argument at the call site i'm interested in
[02:32:37 CET] <ShadowOfThePast> it does so hundreds of times so that it should exceed the memory of my box pretty quickly, but it only finally errors out after ~1400 of those 1 MB allocations (my box has 384 MB of ram, runs Linux 2.6.34 and has no paging from what i know, because it runs off a 32 mb nand flash)
[02:33:26 CET] <ShadowOfThePast> the particular function that calls it in my case is mov_try_read_block in mov.c
[02:34:00 CET] <ShadowOfThePast> i'm reading a cenc-encrypted mp4 file and it uses this function while parsing saiz atoms
[02:50:45 CET] <iive> ShadowOfThePast, if i read the code correctly, it just allocates 6% bigger buffer (1/16 of the target size).
[02:51:24 CET] <ShadowOfThePast> yea alright, that explains why it returns that odd number instead of 1024*1024
[02:52:42 CET] <ShadowOfThePast> i'm really mind-blown though how it works, because those blocks aren't ever freed (until it runs out of memory and stops playback)
[02:53:05 CET] <ShadowOfThePast> they're only freed in mov_free_encryption_index, which isn't called unless the context is closed
[02:54:00 CET] <ShadowOfThePast> and like i said, my file has 1559 of those atoms so it should be taking a ton of memory (stupidly, because the data itself is only like 50 bytes every time)
[02:54:41 CET] <iive> how big is this file?
[02:54:53 CET] <ShadowOfThePast> 500 MB
[02:55:17 CET] <iive> so... the atoms are not really 1mb each?
[02:55:30 CET] <ShadowOfThePast> yea they're just a couple of bytes
[02:55:47 CET] <ShadowOfThePast> that read_block function just allocates at least a meg every time no matter what
[02:56:31 CET] <iive> oh, i see. mov probably allocates max size, then reallocs to the real size. but fast-realloc never shrinks
[02:58:02 CET] <iive> sorry, I'm not familiar with mov. does atoms have size parameter?
[02:59:37 CET] <ShadowOfThePast> yes they do, but in this case it reads a part of the atom only, where the size of that part is specified by an integer in the atom
[02:59:45 CET] <ShadowOfThePast> check out the mov_read_saiz function
[03:02:02 CET] <ShadowOfThePast> a realloc to shrink it would make sense, but i don't see that happening anywhere for the auxiliary_info_sizes buffer
[03:10:54 CET] <iive> so, the problem is in mov_try_read_block()
[03:12:37 CET] <iive> ShadowOfThePast, wb
[03:12:44 CET] <ShadowOfThePast> thanks, net died
[03:12:59 CET] <iive> just put an av_realloc() before the return.
[03:16:15 CET] <iive> if i read the code correctly, it should be "size" bytes long.
[03:17:00 CET] <ShadowOfThePast> yea i was under the same impression
[03:17:02 CET] <ShadowOfThePast> i'll try that now
[03:17:40 CET] <iive> i guess that this function starts with 1mb buffer, just in case...
[03:18:31 CET] <ShadowOfThePast> i'll know whether it helped in 46 minutes
[03:20:20 CET] <iive> and it stays at 1mb
[03:20:30 CET] <iive> well, tell me when i come back.
[03:20:32 CET] <iive> gtg
[03:20:40 CET] <ShadowOfThePast> aight
[03:21:26 CET] <iive> don't forget to open ticket and send patch :D
[03:27:33 CET] <iive> who wrote that function?...
[03:28:34 CET] <iive> anyway...
[03:28:39 CET] <iive> n8 ppl
[04:15:01 CET] <ShadowOfThePast> shrinking the buffer did indeed fix the issue, the stream plays through fine now
[04:41:17 CET] <ShadowOfThePast> there we go
[04:41:20 CET] <ShadowOfThePast> good night
[10:57:17 CET] <j-b> 'morning
[00:00:00 CET] --- Sun Dec 30 2018