[Ffmpeg-devel-irc] ffmpeg-devel.log.20180128

[burek]

[00:00:00 CET] <CoreX> :/
[00:30:25 CET] <kierank> durandal_1707: what os and compiler?
[02:02:12 CET] <kierank> is there a guide to building ffmpeg with tsan
[03:07:17 CET] <kierank> michaelni: are you aware mpeg4video is full of tsan errors
[03:14:39 CET] <michaelni> are they real errors (outside tsan) ? its a while since i looked at tsan and i had the feeling long ago that it was alot of not really bugs but maybe i was mistaken
[03:14:54 CET] <kierank> according to durandal_1707 with my patch yes
[03:15:00 CET] <kierank> but i can't reproduce to tsan is all i got
[03:15:02 CET] <kierank> carl also saw errors
[03:17:19 CET] <durandal_1707> kierank: ubuntu, gcc
[03:17:46 CET] <kierank> i can see lots of errors now but no crash
[03:17:50 CET] <kierank> on ubuntu 16.04
[03:17:54 CET] <kierank> lots of tsan errors
[03:21:30 CET] <kierank> some seem related to 10-bit dct
[03:21:43 CET] <kierank> but some happen with normal mpeg4
[03:23:20 CET] <durandal_1707> its trigered with error resilence for some reason
[03:24:15 CET] <kierank> probably ER doesn't understand 10-bit
[03:24:37 CET] <durandal_1707> doesnt happen always when starting ffplay, mpv
[03:25:42 CET] <durandal_1707> it should happen always or never...
[03:26:03 CET] <kierank> probably because s->block32 is an array
[03:26:09 CET] <kierank> whereas it perhaps expects a pointer
[03:26:48 CET] <durandal_1707> kierank: try corrupting bitstream with bsf to trigger er
[03:26:57 CET] <kierank> I can see the error in tsan
[03:27:02 CET] <wm4> (is there such a bsf)
[03:27:30 CET] <durandal_1707> yes, there is
[03:30:18 CET] <durandal_1707> noise bitstream filter
[03:30:52 CET] <wm4> (is that useful over a fuzzer?)
[08:33:16 CET] <vrd> Hello! Vishal here, I wanted to take up the super resolution task , if no one's already doing it.If somebody has already done it,please let me know! Thanks.
[11:29:30 CET] <kaushik_> hi can anyone help me with the "Add a filter which performs a convolution of two images." qualifying task.
[11:29:54 CET] <kaushik_> Where should I start to gain some background knowledge?
[12:35:10 CET] <BtbN> philipl, is it only me, or shouldn't the code that unmaps the input resource also unregister it? Unregister is currently only called when re-assigning an unmapped registered_frame
[13:14:22 CET] <cone-706> ffmpeg 03Timo Rothenpieler 07master:bbe1b21022e4: avcodec/nvenc: refcount input frame mappings
[13:14:22 CET] <cone-706> ffmpeg 03Timo Rothenpieler 07master:32bc4e77f61a: avcodec/nvenc: unregister input resource when unmapping
[13:14:22 CET] <cone-706> ffmpeg 03Timo Rothenpieler 07master:48e52e4edd12: avcodec/nvenc: add some more error case checks
[13:14:22 CET] <cone-706> ffmpeg 03Timo Rothenpieler 07master:932037c6bb6b: avcodec/nvenc: also clear data pointer after unregistering a resource
[13:38:54 CET] <cone-706> ffmpeg 03Timo Rothenpieler 07release/3.4:fbb27e291183: avcodec/nvenc: refcount input frame mappings
[13:38:55 CET] <cone-706> ffmpeg 03Timo Rothenpieler 07release/3.4:a7c60c5b7bc5: avcodec/nvenc: unregister input resource when unmapping
[13:38:56 CET] <cone-706> ffmpeg 03Timo Rothenpieler 07release/3.4:d36714f72702: avcodec/nvenc: add some more error case checks
[13:38:57 CET] <cone-706> ffmpeg 03Timo Rothenpieler 07release/3.4:93c8720b914e: avcodec/nvenc: also clear data pointer after unregistering a resource
[14:00:40 CET] <kierank> durandal_1707: ping
[14:01:36 CET] <durandal_1707> kierank: pong
[14:01:46 CET] <kierank> durandal_1707: can you test new patch please
[14:01:54 CET] <kierank> it's on ml
[14:02:07 CET] <kierank> dunno about segfault but it should fix thread problems
[14:07:18 CET] <durandal_1707> kierank: its worse, now also ffmpeg segv
[14:07:35 CET] <kierank> huh, works for me
[14:09:39 CET] <cone-706> ffmpeg 03Timo Rothenpieler 07release/3.3:802ebfae3b5b: avcodec/nvenc: refcount input frame mappings
[14:09:40 CET] <cone-706> ffmpeg 03Timo Rothenpieler 07release/3.3:4bb40c32ee09: avcodec/nvenc: unregister input resource when unmapping
[14:09:41 CET] <cone-706> ffmpeg 03Timo Rothenpieler 07release/3.3:d68d537f0a78: avcodec/nvenc: add some more error case checks
[14:09:42 CET] <cone-706> ffmpeg 03Timo Rothenpieler 07release/3.3:dfd2f4ee265e: avcodec/nvenc: also clear data pointer after unregistering a resource
[14:09:43 CET] <cone-706> ffmpeg 03Timo Rothenpieler 07release/3.3:09419de21620: configure: add support for libnpp* from cuda sdk 9
[14:10:10 CET] <durandal_1707> kierank: im on 64bit
[14:11:20 CET] <durandal_1707> perhaps you need to resend idct patch
[14:15:05 CET] <kierank> durandal_1707: so am I but in vm
[14:15:37 CET] <kierank> durandal_1707: can you send bt?
[14:17:56 CET] <durandal_1707> kierank: http://pastebin.com/gB2Gru3e
[14:18:29 CET] <kierank> durandal_1707: which file?
[14:19:20 CET] <durandal_1707> sr lite
[14:45:00 CET] <kierank> durandal_1707: weird, so recompiling without tsan shows the issue
[14:52:13 CET] <kierank> ah fixed
[15:11:41 CET] <kierank> durandal_1707: thanks
[15:13:38 CET] <cone-706> ffmpeg 03Michael Niedermayer 07master:3f621455d62e: avfilter/vf_transpose: Fix regression with packed pixel formats
[15:13:39 CET] <cone-706> ffmpeg 03Michael Niedermayer 07master:293f24b42c5d: fate: test the transpose filter more fully
[15:43:28 CET] <durandal_1707> kierank: where is patch?
[15:43:41 CET] <kierank> trying to fix cutting segfault first
[15:43:52 CET] <durandal_1707> ok
[15:44:55 CET] <kierank> it's all unchecked bitstream reads
[15:45:16 CET] <kierank> i can turn it off but ricers won't like that
[16:26:13 CET] <philipl> BtbN: So, I don't know about nvenc, but with nvdec interop, you keep the frame registered as long as you care about it, but you only map it to get a pointer and then immediately unmap it.
[16:26:54 CET] <philipl> Looks like you fixed whatever the problem was :-)
[18:00:31 CET] <kierank> michaelni: so can you explain all this padding heuristics stuff?
[18:01:21 CET] <kierank> why does decode_mb need to return < 0?
[18:02:03 CET] <kierank> the crash is actually in your av_log message
[18:08:26 CET] <durandal_1707> kierank: er still trigers random crash with ffplay/mpv
[18:10:08 CET] <kierank> durandal_1707: same backtrace?
[18:10:58 CET] <kierank> michaelni: imo the bug is in existing mpeg4video code
[18:12:03 CET] <kierank> you do show_bits(&s->gb, 24) without checking you can read 24 bits
[18:12:04 CET] <michaelni> decode_mb needs to return errors in case of errors / end otherwise it would continue and corrupt the next slice, waste cpu, potentially crash with overread
[18:12:22 CET] <michaelni> kierank, how can i reprodeuce with existing code ?
[18:13:12 CET] <durandal_1707> kierank: yes, same bt
[18:13:16 CET] <kierank> the problem is I return SLICE_OK
[18:13:25 CET] <kierank> but perhaps I need to return SLICE_END?
[18:13:33 CET] <kierank> who knows what these things mean?
[18:14:41 CET] <kierank> durandal_1707: but works in ffmpeg?
[18:14:47 CET] <kierank> imo this is all existing tsan bugs
[18:15:35 CET] <durandal_1707> yes, ffmpeg doesnt trigger er, it appears
[18:17:48 CET] <kierank> michaelni: you know at some point someone is going to send a field coded sample
[18:18:03 CET] <kierank> and then I will have to unwind the hacks upon hacks in the mpeg4video/h263 codebase
[18:20:30 CET] <kierank> the segfault happens because I return SLICE_OK and so i fall through all the s->padding_bug_score categories
[18:20:33 CET] <kierank> whatever those hacks do
[18:20:58 CET] <michaelni> mpeg2 already supports field coded pictures with the mpegvideo framework. 
[18:22:11 CET] <kierank> can you explain to me the difference between SLICE_OK and SLICE_END
[18:22:20 CET] <kierank> and why one falls through and one doesnt?
[18:25:28 CET] <durandal_1707> are there alternative mpeg4 decoders?
[18:26:23 CET] <michaelni> its long ago but IIRC SLICE_END means the slice end has been found, SLICE_OK means all fine and there is more data in this slice
[18:28:53 CET] <michaelni> looking at the code it looks like it reads the startcode of the next slice not just checks if its there. 
[18:29:54 CET] <kierank> michaelni: what does?
[18:31:22 CET] <michaelni> maybe i misunderstood next_start_code_studio()
[18:32:27 CET] <kierank> that's exactly how it says to do it in the spec (because studio allows all sorts of what I guess are predecessors to SEIs)
[18:36:22 CET] <kierank> durandal_1707: can you provide tsan for ffplay or mpv please
[18:36:38 CET] <michaelni> theres also a check_marker in mpeg4_decode_studio_block() that doesnt have its return code checked
[18:37:29 CET] <michaelni> also group is used as array index before being checked <ß
[18:37:37 CET] <michaelni> <0 
[18:37:59 CET] <kierank> michaelni: fixed that one locally
[18:38:21 CET] <kierank> actually fixed in latest patchset
[18:38:27 CET] <kierank> check_marker isn't checked anywhere in the code
[18:38:37 CET] <kierank> but I can't do dpcm (yet) so I bail out anyway
[18:39:32 CET] <michaelni> the markers check_marker checks are bits in long runs of potential zeros, not checking them can get the code probably deeper over the end
[18:41:06 CET] <kierank> i'd still bail out nonetheless
[18:41:18 CET] <kierank> the problem is as i said you read 24 bits without checking you have 24 bits available
[18:41:54 CET] <kierank> and that may have worked in the past because of all this padding heuristics
[18:42:58 CET] <michaelni> AV_INPUT_BUFFER_PADDING_SIZE requires 64 bytes over the end, thats 512 bits, for the 24 bits read to be too far and over that the code would have to be quite a bit over it before already
[18:44:46 CET] <michaelni> mpeg4 shouldnt need any tricks at all to not overread, mpeg4 doesnt allow start code emulation in the bitstream so a valid bitstream cannot contain long runs of 0. the padding is all 0 so it cannot be a valid bitstream and checkibg the stream for validity should catch this before the end
[18:47:13 CET] <kierank> https://0x0.st/sbZn.mxf
[18:47:33 CET] <kierank> that's the same, it's a cut version of one of the samples
[18:47:35 CET] <kierank> sample*
[18:49:05 CET] <kierank> michaelni: is there an assumption in the code that a slice is horizontal?
[18:49:17 CET] <kierank> one row of MBs perhaps
[18:49:26 CET] <michaelni> not for mpeg4
[18:52:02 CET] <kierank> this is something to do with ff_h263_resync perhaps
[18:52:36 CET] <kierank> seems i read some uninitialised stack memory
[18:56:31 CET] <Gusar_> Delf: there's a --hwdec=crystalhd in mpv, so theoretically it works... but can you actually buy the thing anywhere anymore?
[18:56:34 CET] <Gusar_> oops
[18:58:39 CET] <kierank> michaelni: what is the purpose of ff_h263_resync
[18:58:48 CET] <kierank> to get to the beginning of a slice?
[19:00:15 CET] <kierank> studio doesn't have resync markers like normal mpeg4
[19:18:08 CET] <kierank> durandal_1707: possibly fixed your issue, but the segfaults i dunno, mpeg4video/h263 code is impossible to understand
[19:25:56 CET] <cone-116> ffmpeg 03Marton Balint 07master:5bf774a4a448: avfilter/vf_framerate: unify luma and chroma blending
[19:25:56 CET] <cone-116> ffmpeg 03Marton Balint 07master:1b6ffe9aca1b: avfilter/vf_framerate: factorize blend functions and unify filter_slice
[19:25:56 CET] <cone-116> ffmpeg 03Marton Balint 07master:2cbe6bac0337: avfilter/vf_framerate: change blend factor precision
[19:25:56 CET] <cone-116> ffmpeg 03Marton Balint 07master:4d95c6d5d7d8: avfilter/vf_framerate: add SIMD functions for frame blending
[19:35:59 CET] <michaelni> ff_h263_resync() finds the next slice (video packet header / gob header)
[19:53:07 CET] <kierank> durandal_1707: can you test new patch that i just posted
[19:56:20 CET] <durandal_1707> will do
[19:57:30 CET] <kierank> durandal_1707: thanks, i am starting to give up being able to fixing the fuzz crashes
[19:59:19 CET] <durandal_1707> why?
[19:59:40 CET] <kierank> mpeg4video/ER code is impenetrable
[20:00:24 CET] <atomnuker> can't you just disble ER?
[20:01:43 CET] <kierank> dunno, it's quite tied into mpeg4video/mpegvideo
[20:08:20 CET] <durandal_1707> kierank: issue is fixed \Ë/
[20:08:54 CET] <kierank> !!!
[20:23:01 CET] <cone-116> ffmpeg 03Martin Vignali 07master:3a230ce5fa10: avfilter/x86/vf_blend : avfilter/x86/vf_blend : add AVX2 version for each func except divide
[20:25:00 CET] <cone-116> ffmpeg 03Martin Vignali 07master:8f9c38b19629: avcodec/utvideoenc : add SIMD (avx) for sub_left_prediction
[20:25:01 CET] <cone-116> ffmpeg 03Martin Vignali 07master:78b982d3b9f1: checkasm : add test for losslessvideoencdsp for diff bytes and sub_left_pred
[20:29:31 CET] <kierank> michaelni: so I understand correctly, should error_resilience report MV errors on an intra video?
[20:31:43 CET] <kierank> some kind of "special" MVs for error resilience?
[21:06:48 CET] <michaelni> possibly it does print a non zero mv error number. Shouldnt really matter.
[21:11:28 CET] <jamrial> jkqxz: can you reproduce the above ticket?
[21:11:51 CET] <jamrial> also, wonder if the amd guys even look at trac for amf issues
[21:12:25 CET] <jamrial> assuming it's amf and not d3d11 in general
[21:17:05 CET] <kierank> michaelni: it changes the idct codepaths which are not 10-bit
[21:17:24 CET] <kierank> it tries to do mc
[21:26:29 CET] <jkqxz> jamrial:  I don't have D3D11 input to AMF working at all.
[21:26:56 CET] <jamrial> it doesn't work for you? what system?
[21:27:26 CET] <jkqxz> Windows 7.  No Windows 10 system I can put a graphics card in.
[21:27:44 CET] <jamrial> ah, yeah, d3d11va is win8 minimum i think
[21:28:10 CET] <jkqxz> D3D11VA does work fine, it's just AMF says no when trying to give it the D3D11 frames.
[21:28:42 CET] <jamrial> can you try nvenc or some other encoder that accepts d3d11 frames?
[21:29:01 CET] <jamrial> to know if the issue is amf, or d3d11 hwaccel with d3d11 format output
[21:30:25 CET] <jkqxz> Can't try Nvidia.  libmfx does kindof work with D3D11 input, but the array texture problems are still there (though that should be fixed soon).
[21:30:48 CET] <jkqxz> OpenCL input to AMF did work for me, but apparently that's doing a secret download/upload so probably isn't representative.
[21:35:03 CET] <BtbN> I remember the AMD guy mentioning d3d11 only working on Win8.1+ due to new APIs
[21:38:02 CET] <jkqxz> Yeah, I didn't mind that in review once it was clear that was the problem.  It does mean I didn't test that part at all, though.
[21:47:59 CET] <jkqxz> jamrial:  Presumably it doesn't happen if you "... -hwaccel d3d11 -hwaccel_output_format d3d11 -i ... -vf 'hwdownload,format=nv12' ..."?
[21:48:21 CET] <jkqxz> I think you should probably just email the AMD guy.
[21:57:37 CET] <jamrial> jkqxz: doesn't happen. same good output as doing -hwaccel d3d11 -hwaccel_output_format nv12
[22:15:33 CET] <michaelni> kierank, "mc" (rather copyying from last frame) makes sense for ER even with intra only. The last frame may be the best source of picture content available
[22:56:53 CET] <BtbN> is -hwaccel cuda really a thing? oO
[22:56:56 CET] <BtbN> Never heard of that.
[23:04:17 CET] <kierank> michaelni: why does it involve calling dct outside out of the mb_intra branch
[23:07:18 CET] <jkqxz> BtbN:  It's a slightly silly consequence of the names coming from the hwcontext devices there.  (The "nvdec" name only works because of <http://git.videolan.org/?p=ffmpeg.git;a=blob;f=fftools/ffmpeg_opt.c;h=92199b3ac28f231738d3017186b3cb43d7386f8b;hb=HEAD#l802>.)
[23:07:34 CET] <michaelni> maybe block_last_index is set >= 0, that would make the code belive there are non zero coeffs that need idct
[23:18:49 CET] <wm4> jkqxz: haha I have the same thing in mpv
[23:19:26 CET] <wm4> that's the only name fudging left though, so not sure if anything needs to be done about it
[23:19:36 CET] <BtbN> wasn't the -hwaccel thing a hardcoded list in ffmpeg.c?
[23:26:40 CET] <cone-116> ffmpeg 03Marton Balint 07master:dc5d1515681b: avformat/hlsenc: use av_bprintf without buffer limit in replace_int_data_in_filename
[23:26:41 CET] <cone-116> ffmpeg 03Marton Balint 07master:ea3672b7d67c: avformat: add url field to AVFormatContext
[23:26:42 CET] <cone-116> ffmpeg 03Marton Balint 07master:25a2d269bdd9: fftools, tools, examples: migrate to AVFormatContext->url
[23:26:43 CET] <cone-116> ffmpeg 03Marton Balint 07master:4bb04098204a: avdevice: migrate to AVFormatContext->url
[23:26:44 CET] <cone-116> ffmpeg 03Marton Balint 07master:45ec2e44be91: avformat/hls: migrate to AVFormatContext->url
[23:26:45 CET] <cone-116> ffmpeg 03Marton Balint 07master:18ac64235939: avformat: migrate to AVFormatContext->url
[23:26:46 CET] <cone-116> ffmpeg 03Marton Balint 07master:fa8308d3d4f2: avformat: deprecate AVFormatContext filename field
[23:29:46 CET] <wm4> BtbN: I think that's completely removed now
[23:29:51 CET] <wm4> (and in mpv too)
[00:00:00 CET] --- Mon Jan 29 2018