[Ffmpeg-devel-irc] ffmpeg-devel.log.20170108
burek
burek021 at gmail.com
Mon Jan 9 03:05:02 EET 2017
[00:06:09 CET] <kierank> atomnuker: because it was ported from that foo86 decoder i ugess
[00:06:45 CET] <kierank> oh mdct
[00:06:46 CET] <kierank> never mind
[00:11:45 CET] <atomnuker> there's no way the elvis presley of coding foo86 can make a mistake like that
[00:12:16 CET] <atomnuker> or make a mistake at all
[00:13:08 CET] <kierank> mistake like what?
[00:15:39 CET] <atomnuker> dunno, big segfault-y bugs?
[00:16:11 CET] <atomnuker> (he had to go through only 1 revision IIRC for his big decoder replacement patches)
[00:16:51 CET] <kierank> oh crap one of the mpeg-4 sstp samples has dpcm
[00:24:53 CET] <durandal_1707> dpcm?
[00:26:48 CET] <kierank> yes some rice coded block that doesn't go through dct
[00:33:11 CET] <kierank> mpegvideo forces you to memcpy :(
[00:56:21 CET] <durandal_1707> just rip off own version
[00:57:51 CET] <Compn> thats what i suggested , just make a seperate decoder
[00:58:00 CET] <Compn> dont worry about sharing code because it will slow down 8bit anyhow
[01:01:59 CET] <Compn> no problem, dont mind me
[01:33:06 CET] <BBB> kierank: h264 has pcm also ;)
[01:33:19 CET] <BBB> they trying to emulate that?
[01:33:39 CET] <kierank> it was probably developed concurrently
[01:42:38 CET] <kierank> michaelni: "The format is choosen by the AVCodecContext.get_format() callback
[01:42:38 CET] <kierank> from the list of choices a decoder presents to it.
[01:42:38 CET] <kierank> if you present yuv420 as a choice it can pick that.
[01:42:38 CET] <kierank> the current code will present the full list from the AVCodec as is
[01:42:38 CET] <kierank> and the first in the list is supposed to be the best choice so it
[01:42:38 CET] <kierank> likely will be choosen"
[01:42:39 CET] <kierank> eh?
[01:42:47 CET] <kierank> why does it not choose the format I tell it to use
[01:45:28 CET] <kierank> and this -1U just crashes if I leave it there
[01:46:39 CET] <kierank> mpegvideo.c is pain
[02:39:27 CET] <Sesse> so, where should I send my shq decoder for review?
[02:39:32 CET] <Gramner> BBB: don't trash talk the pcm mode in H.264, it's great. I mean, it makes it really easy to encode YUV pixels into H.264 with just a few lines of code. who said writing an encoder needs to be difficult?
[02:44:27 CET] <kierank> Sesse: ffmpeg-devel
[02:44:33 CET] <kierank> git format-patch
[02:46:10 CET] <BBB> Gramner: thats not an encoder, thats a bitstream writer
[02:46:28 CET] <Sesse> kierank: okay, will write up some nice commit message tomorrow
[02:46:39 CET] <Sesse> kierank: and then I'll probably introduce like 20 new CVEs once the fuzzers arrive
[02:46:49 CET] <Gramner> it would also be useful if you'd ever need to create a single-frame H.264 bitstream in a hex editor by hand from a drawn picture
[02:46:50 CET] <BBB> Sesse: thats normal
[02:47:14 CET] <BBB> Gramner: again, not useful at all
[02:47:37 CET] <Gramner> please, that's totally a part of my workflow
[02:48:07 CET] <BBB> dont we all decode jpeg and mpeg in a hex editor, like the matrix?
[02:48:11 CET] <BBB> you dont need pcm for that
[02:52:57 CET] <Gramner> I also read my mail by staring into a 10GBASE-SR (850nm wavelength is fine, just need good eyesight) fiber
[03:04:44 CET] <BBB> \o/
[03:51:28 CET] <cone-272> ffmpeg 03Tobias Stoeckmann 07master:95d9a85ca3e6: ffserver: local OOB write with custom program name
[06:28:06 CET] <wm4> jkqxz: what's up with libva adding I010? which seems like P010, except the data is in the LSBs
[12:17:07 CET] <durandal_1707> Compn: ping
[12:38:51 CET] <durandal_1707> Sesse: how do one know which function is being called via that debug info?
[12:39:19 CET] <Sesse> durandal_1707: the debug log includes the function name, iirc
[12:40:22 CET] <durandal_1707> bit i need functions being called from inside dll, not what's exported
[12:47:13 CET] <Sesse> I don't understand
[12:47:21 CET] <Sesse> could you give an example?
[13:10:25 CET] <Sesse> speedhq patch sent to the mailing list
[13:11:28 CET] <atomnuker> Sesse: what's with the ++integer in loops, old habbit?
[13:11:42 CET] <Sesse> atomnuker: K&R
[13:12:13 CET] <atomnuker> the old testament
[13:12:22 CET] <Sesse> well, ffmpeg seemingly swears by C90
[13:12:26 CET] <Sesse> that's old testament indeed
[13:12:34 CET] <Sesse> C99 came out eighteen years ago :-)
[13:12:38 CET] <atomnuker> we prefer integer++ in loops
[13:12:44 CET] <Sesse> fair enough, easy to change
[13:20:33 CET] <wm4> ffmpeg is actually C99
[13:20:40 CET] <wm4> it won't compile with a C89 compiler
[13:20:48 CET] <Sesse> wm4: I get errors if I try to use C99 constructs
[13:20:51 CET] <wm4> and ++x is heresy and ugly
[13:20:55 CET] <Sesse> for (int i = 0; i < 100; i++) won't compile
[13:21:08 CET] <Chloe> huh?
[13:21:11 CET] <wm4> that's because ffmpeg enables a warning that breaks some valid C99 code
[13:21:24 CET] <wm4> because apparently we prefer putting all decls at the start of a block
[13:21:34 CET] <wm4> (not sure why this affects those for loops though?)
[13:21:38 CET] <wm4> or what is the error you get
[13:22:05 CET] <wm4> I mean, we definitely use some real C99 constructs like designated initializers (not just "easy" stuff like // comments)
[13:22:19 CET] <Sesse> wait, maybe for works, non-inline decls don't
[13:22:44 CET] <wm4> the reason for decls-at-start-of-block change whoever you ask
[13:23:08 CET] <Sesse> I don't really care personally, I'm not going to spend a lot of time within ffmpeg code
[13:23:14 CET] <wm4> one times it's for stylistic reasons, another time because it's "better", and often it's because supposedly to deal with shitty compilers
[13:23:19 CET] <Sesse> so as long as I can figure out what the rules are to get my patch applied..
[13:23:22 CET] <wm4> (the last argument is definitely the dumbest)
[13:23:57 CET] <Chloe> "better" is obviously the most convincing argument :D
[13:35:12 CET] <jkqxz> wm4: libva I010 is ffmpeg YUV420P10. It might be helpful for some mapping or copy cases (have mapped 10-bit surfaces without converting to P010), but the implementation is as-yet incomplete so I don't think it's currently useful.
[13:36:06 CET] <wm4> that's a really bad name then
[13:36:14 CET] <wm4> why does libva think it needs fourccs anyway
[13:36:22 CET] <wm4> they could just use proper names and enums
[13:42:37 CET] <jkqxz> Yes, they could.
[13:45:59 CET] <atomnuker> wm4: yeah, seems like its a bug in lavfi, loop happens outside the filter's i/o, told nicholas
[14:03:51 CET] <KGB> [13FFV1] 15michaelni pushed 2 new commits to 06master: 02 https://git.io/vMChI
[14:03:51 CET] <KGB> 13FFV1/06master 1409ab181 15Dave Rice: define abs log2 min functions...
[14:03:51 CET] <KGB> 13FFV1/06master 1469a377c 15Dave Rice: add missing a*=b function
[14:09:25 CET] <KGB> [13FFV1] 15michaelni pushed 2 new commits to 06master: 02 https://git.io/vMChE
[14:09:25 CET] <KGB> 13FFV1/06master 14c43a167 15Dave Rice: have 'make clean' be version independent
[14:09:25 CET] <KGB> 13FFV1/06master 14210958c 15Dave Rice: bump to draft version 01
[14:09:58 CET] <KGB> [13FFV1] 15michaelni closed pull request #45: Bump to draft version 01 (06master...06bump-to-draft-version-01) 02 https://git.io/vMC08
[15:42:48 CET] <cone-481> ffmpeg 03Michael Niedermayer 07master:bd83c295fc1b: avcodec/omx: Do not pass negative value into av_malloc()
[15:42:48 CET] <cone-481> ffmpeg 03Michael Niedermayer 07master:762bf6f4afa9: avcodec/bsf: Fix av_bsf_list_free()
[16:24:54 CET] <Sesse> wow, there are files in ffmpeg that require >30 sec to compile on my yasm version
[16:26:49 CET] <JEEB> yeah
[16:27:00 CET] <JEEB> they had a very low size for a hash map IIRC
[16:27:06 CET] <BtbN> update your yasm
[16:27:13 CET] <JEEB> 1.3 had an update which just bounced that way up
[16:27:18 CET] <JEEB> and suddenly it sped up. a lot
[16:27:23 CET] <BtbN> Or use nasm I guess
[16:30:50 CET] <atomnuker> I wonder how long that huge 16000 line asm file from x265 takes to compile on an old yasm
[16:31:29 CET] <JEEB> I think I did that once or so, and calculated it at a few minutes
[16:31:38 CET] <JEEB> although my memory is hazy
[16:45:13 CET] <Sesse> hm, how can I send the codec tag on the command line?
[16:45:20 CET] <Sesse> I tried something like: ./ffmpeg -f rawvideo -codec speedhq -codec_tag SHQ2 -video_size 640x360
[16:45:37 CET] <Sesse> but -codec_tag does not exist :-)
[16:49:17 CET] <durandal_1707> Sesse: vtag?
[16:51:46 CET] <Sesse> durandal_1707: no go
[16:52:05 CET] <Sesse> it exists, but i still get assert(0)
[16:52:11 CET] <Sesse> (due to unknown tag)
[16:55:26 CET] <durandal_1707> that command can not work anyway for nonfirst frame
[16:55:34 CET] <Sesse> that's fine
[16:55:46 CET] <Sesse> I just want to decode one
[17:03:01 CET] <durandal_1707> Sesse: where you put -tag:v ? at beginning?
[17:03:44 CET] <Sesse> yes
[17:03:44 CET] <Sesse> well, -codec speedhq -vtag SHQ2
[17:39:46 CET] <Sesse> okay, I hacked it a bit to always do SHQ2, so now I have a small target to fuzz
[17:40:11 CET] <Sesse> I suppose someone's already dealt with all the forkserver business to speed it up, though?
[17:42:10 CET] <Sesse> ah, kierank's stuff at https://github.com/OpenBroadcastSystems/fffuzz/blob/master/main.c
[17:42:13 CET] <Sesse> I suppose
[17:43:32 CET] <kierank> Yes
[17:44:14 CET] <Sesse> can I use it with rawvideo?
[17:44:14 CET] <Sesse> I just have a 16x32 frame
[17:44:14 CET] <Sesse> I don't really care about fuzzing the avi container
[17:47:06 CET] <Compn> durandal_1707 : yes?
[17:47:21 CET] <Compn> if you are still awake :D
[17:47:58 CET] <Sesse> (also, putting it in .avi takes it from 191 to 5902 bytes, not so nice)
[17:50:25 CET] <durandal_1707> Compn: add pixlet to MPlayer?
[17:50:48 CET] <Compn> yes i should do that
[17:51:28 CET] <Compn> vlc is bigger target though
[17:52:54 CET] <kierank> Sesse: I think, no because it's an FFmpeg restriction
[17:53:08 CET] <kierank> Unless you write an shq demuxer
[17:53:33 CET] <Sesse> Error: input video width/height/format changed:
[17:53:33 CET] <Sesse> old: width = 16, height = 32, format = yuv422p
[17:53:33 CET] <Sesse> new: width = 16, height = 32, format = yuv422p
[17:53:36 CET] <kierank> But you can't really because shq has no framing
[17:53:42 CET] <Sesse> kierank: I can decode it with the ffmpeg command-line tool
[17:53:44 CET] <Sesse> except the vtag
[17:53:53 CET] <kierank> Hmm dunno then
[17:54:01 CET] <Sesse> but why does it think the format changed
[17:54:04 CET] <Sesse> it doesn't really appear to change to me
[17:54:20 CET] <Compn> Sesse : how is the overhead in mkv ?
[17:54:22 CET] <Compn> vs avi?
[17:54:27 CET] <Sesse> no idea
[17:54:37 CET] <Compn> or nut :)
[17:54:48 CET] <kierank> Usual undocumented FFmpeg nonsense
[17:55:38 CET] <Sesse> heh, pix_fmt vs. format
[17:55:54 CET] <Sesse> and dec_ctx vs. frame
[17:56:42 CET] <Sesse> okay, I ran fffuzz and it segfaulted on my avi
[17:56:54 CET] <Sesse> the valid one :-)
[17:57:26 CET] <Sesse> bleh
[18:01:19 CET] <Compn> are you sad it crashed or sad to be fuzzing ?
[18:01:26 CET] <Sesse> sad it crashed when it shouldn't
[18:01:30 CET] <Sesse> I hacked it to sort-of work
[18:08:39 CET] <Sesse> how do I know whether persistent mode worked or not?
[18:11:00 CET] Action: Compn should have mentioned strncpy oh well
[18:11:09 CET] <Sesse> Compn: strncpy is not your friend
[18:11:24 CET] <Sesse> it can end up writing something that's not zero-terminated
[18:12:33 CET] <Sesse> ok, 340 execs/sec/thread, 40 threads
[18:13:54 CET] <Compn> yeah, i see what you are saying
[18:16:07 CET] Action: durandal_1707 tracing functions calls for fun and profit
[18:16:43 CET] <Sesse> durandal_1707: you never answered my question about what calls you were trying to trace
[18:17:55 CET] <durandal_1707> Sesse: that damn c++ pointers to pointers to pointers etc
[18:18:05 CET] <Sesse> vtables?
[18:18:22 CET] <Compn> Sesse : i think he wants to map out all of the function calls within the binary dll
[18:18:39 CET] <Sesse> Compn: my point was that this is underspecified
[18:18:46 CET] <Compn> theres a way to do that but i dont know... i've seen people make the maps before. kostya posted it i think, or maybe hyc
[18:18:52 CET] <Sesse> Compn: calls from within the DLL to other parts of the DLL can't be traced in a simple way
[18:19:06 CET] <Sesse> runtime, that is
[18:19:06 CET] <Sesse> this is something you want from your disassembler
[18:19:14 CET] <Sesse> but calls from the DLL to the emulated system calls is a different story
[18:19:46 CET] <Compn> yea disassembler is what i'm thinking of
[18:19:58 CET] <Compn> durandal_1707 : what do you want, functions to the wine code or function calls within the dll ?
[18:20:25 CET] <durandal_1707> Sesse: it's old codec im sInglestepping now, it will finish next milenium
[18:20:46 CET] <Sesse> durandal_1707: okay, if you don't want to answer the question, that's fine
[18:21:25 CET] <kierank> durandal_1707: the big trick Sesse told me is to use the osx binary
[18:21:53 CET] <durandal_1707> Sesse: it's vtables sure, with several layers
[18:22:14 CET] <Sesse> kierank: won't help if he's trying to run it, not disassemble
[18:22:35 CET] <Sesse> durandal_1707: but there are vtables in COM, too
[18:22:46 CET] <durandal_1707> now it's stuck in mmx pathe, probably doing colorspace shit
[18:22:53 CET] <Sesse> which is why I asked for an example call, which you for some reason completely ignore every time I'm saking
[18:22:56 CET] <Sesse> asking
[18:23:10 CET] <durandal_1707> I'm on phone
[18:23:12 CET] <Sesse> ok
[18:23:45 CET] <Compn> ah yes, this one is what i was talking about
[18:23:48 CET] <durandal_1707> call sword ptr [eat...
[18:23:49 CET] <Compn> kostya made it for g2m
[18:23:50 CET] <Compn> https://codecs.multimedia.cx/wp-content/uploads/2012/07/Screenshot-WinGraph32-Xrefs-from-compr2_decompress.png
[18:24:30 CET] <Sesse> Compn: that looks like a graph from some disassembler tool, and that won't be able to follow vtable pointers
[18:24:37 CET] <Sesse> or at least it will be hard
[18:26:20 CET] <durandal_1707> adding structures for those vtables is boring and daunting task
[18:27:17 CET] <kierank> durandal_1707: what codec?
[18:28:06 CET] <durandal_1707> ClearVideo
[18:28:40 CET] <durandal_1707> it's two dlls, to makeep things harder
[18:28:53 CET] <Compn> kostya wrote about clearvideo too, https://codecs.multimedia.cx/?p=449
[18:28:59 CET] <durandal_1707> wrappers
[18:29:03 CET] <Compn> https://codecs.multimedia.cx/?p=687
[18:29:31 CET] <Compn> ask kostya for some pointers , durandal_1707
[18:29:35 CET] <durandal_1707> he made intra , Im working on inter
[18:29:38 CET] <Compn> ah
[18:30:02 CET] <Compn> Sesse : i appreciate you helping durandal_1707 with this :)
[18:31:19 CET] <durandal_1707> if you disable functions tracing for little bit does olDer entries get lost?
[18:31:35 CET] <durandal_1707> in Ida*
[18:57:56 CET] <Compn> [07:34] <danieel> Stream #0:1: Audio: none (XiVs / 0x73566958), 48000 Hz, 2 channels, 112 kb/s (default)
[18:58:10 CET] <Compn> XiVs, one of the worst things in the world
[18:58:37 CET] <Compn> ogg vorbis audio wrapped in vorbis container, then stuffed in .mov :D
[19:05:00 CET] <Compn> Sesse : what license is the shq2 encoder ?
[19:05:09 CET] <Sesse> Compn: please do not distribute this license
[19:05:19 CET] <Sesse> which I don't intend to
[19:06:13 CET] <Compn> ah ok, just making sure they were not intending to offer us a encoder :P
[19:07:07 CET] <Compn> nice of them to help you out writing a decoder at least :)
[19:07:21 CET] <Compn> pretty rare in the codec world
[19:07:22 CET] <Sesse> yes
[19:08:01 CET] <Sesse> what really helped was the samples
[19:48:31 CET] <durandal_1707> Sesse: https://paste.ubuntu.com/23765903/
[20:01:11 CET] <jamrial_> nevcairiel: your windows fate clients haven't run in days
[20:08:20 CET] <Sesse> kierank: afl found an input for which fffuzz goes into an infinite loop printing 0 s :-P
[20:08:28 CET] <Sesse> [file @ 0x610000007f40] Setting default whitelist 'file,crypto'
[20:08:28 CET] <Sesse> [ivr @ 0x61b00001f180] Format ivr probed with size=2048 and score=100
[20:08:31 CET] <Sesse> [ivr @ 0x61b00001f180] O = '0xFF14146A0474590000000 [and so on]
[20:08:48 CET] <kierank> pfft probing
[20:08:53 CET] <JEEB> :D
[20:08:54 CET] <kierank> I don't personally test probing
[20:08:57 CET] <Sesse> obviously!
[20:09:00 CET] <kierank> it's all completely made up
[20:09:23 CET] <Sesse> I think maybe I've sent raw video to fffuzz
[20:09:23 CET] <kierank> Sesse: well if you don't force a codec/container all you do is waste time fuzzing probing functions
[20:09:29 CET] <Sesse> while I should have been giving it an .avi
[20:09:38 CET] <Sesse> except the avi writer has this JUNK tag
[20:10:25 CET] <kierank> Sesse: use perf top to make sure actual functions are being tested
[20:10:59 CET] <Sesse> something tells me it isn't
[20:11:18 CET] <kierank> usually you're just fuzzing mp3_probe because it's really easy for something to be an mp3
[20:11:23 CET] <Sesse> -rw-r----- 1 sesse sesse 458 Jan 8 20:11 micro.avi
[20:11:25 CET] <Sesse> perhaps better
[20:11:39 CET] <Sesse> than the 5k monster
[20:12:30 CET] <kierank> 5k isn't bad
[20:12:46 CET] <Sesse> the actual input data is 191 bytes
[20:14:56 CET] <kierank> are you building with afl-clang-fast?
[20:15:06 CET] <Sesse> yes
[20:15:18 CET] <Sesse> hm, no indications of the speedhq stuff in perf top
[20:15:47 CET] <Sesse> AFL_USE_ASAN=1 afl-fuzz -i testcases -o results -m none -M fuzzer01 -t 1000 -- ./fffuzz/fffuzz -f avi -c speedhq @@ /dev/null
[20:16:05 CET] <Sesse> that ought to have worked
[20:18:24 CET] <Sesse> perhaps the actual decoding just disappears in overhead
[20:19:47 CET] <kierank> ah asan I don't know about
[20:19:49 CET] <Sesse> if I break in gdb on speedhq_decode_frame, I do indeed get calls
[20:20:02 CET] <Sesse> so, it keeps finding hangs in the avi code :-P
[20:20:10 CET] <kierank> also you managed to get afl and asan which I find interesting
[20:20:29 CET] <Sesse> the documentation explains how
[20:20:29 CET] <Sesse> but it requires -m none
[20:20:46 CET] <Sesse> which I suppose is dangerous now with avi mux
[20:20:54 CET] <Sesse> wasn't dangerous when I was just using rawvideo and specifying the frame size
[20:23:58 CET] <kierank> Sesse: btw it's trivial to add frame threading
[20:24:14 CET] <Sesse> yeah, I suppose
[20:24:21 CET] <kierank> though the fuzz bugs you find are not so trivial sometimes
[20:27:43 CET] <Sesse> okay, I am only finding avi bugs
[20:27:43 CET] <Sesse> I need to go to rawvideo
[20:31:02 CET] <kierank> rawvideo is reading uncompressed data afaik
[20:31:12 CET] <Sesse> not rawvideo demuxer
[20:37:43 CET] <kierank> how are you doing that?
[20:38:44 CET] <Sesse> ./ffmpeg -f rawvideo -codec speedhq -vtag SHQ2 -video_size 16x32 -i micro.raw test.png
[20:38:49 CET] <Sesse> trying to replicate that from the C API now
[20:38:57 CET] <Sesse> without success so far
[20:39:51 CET] <kierank> ah yeah i have no idea how to do that
[20:39:53 CET] <kierank> patches welcome
[20:40:43 CET] <Sesse> evidently you can set AVFormatContext's video_codec_id, perhaps also av_format_set_video_codec
[20:41:15 CET] <kierank> you might have to use av_opt for that, not sure
[20:43:00 CET] <Sesse> didn't seem to help
[20:46:07 CET] <cone-343> ffmpeg 03Paul B Mahol 07master:0ddc24d23248: avfilter/af_dynaudnorm: fix hang with too short input
[20:46:18 CET] <Sesse> there we are
[20:46:33 CET] <Sesse> fmt_ctx->streams[i]->codecpar->codec_id = avcodec_find_decoder_by_name("speedhq")->id;
[20:46:37 CET] <Sesse> before avformat_find_stream_info
[20:47:06 CET] <Sesse> not that it ever invoked speedhq
[20:51:31 CET] <Sesse> oh, it's actually called
[20:51:34 CET] <Sesse> just gdb is confused
[20:51:55 CET] <KGB> [13FFV1] 15dericed opened pull request #48: add Michael's email (06master...06author's-email) 02 https://git.io/vMWc7
[20:53:36 CET] <Sesse> okay, trying new fuzz round... 850 execs/sec now
[20:53:41 CET] <Sesse> per thread
[20:54:06 CET] <Sesse> basically all of it is setup cost :-)
[20:54:22 CET] <Sesse> 0,06% fffuzz [.] speedhq_decode_init ’
[20:54:26 CET] <Sesse> 0,05% fffuzz [.] speedhq_decode_frame ’
[20:54:30 CET] <Sesse> 0,03% fffuzz [.] decode_speedhq_field ’
[20:54:34 CET] <Sesse> 0,01% fffuzz [.] speedhq_decode_end
[20:54:45 CET] <Sesse> 650:1 overhead
[21:10:21 CET] <jkqxz> durandal_1707: vf_scale_vaapi.c. (But I probably shouldn't be pointing to existing code I wrote to justify my own idiosyncrasies.)
[21:33:04 CET] <Sesse> [rawvideo @ 0x61b00001f180] id3v2 ver:4 flags:37 len:458763
[21:33:04 CET] <Sesse> [rawvideo @ 0x61b00001f180] Failed to alloc 0 bytes
[21:33:05 CET] <Sesse> wtf
[21:33:31 CET] <KGB> [13FFV1] 15michaelni pushed 1 new commit to 06master: 02 https://git.io/vMWlK
[21:33:31 CET] <KGB> 13FFV1/06master 14e69f975 15Dave Rice: extend colorspace descriptions...
[21:35:55 CET] <kierank> Sesse: lol
[21:37:43 CET] <Sesse> evidently id3v2 is not checked against the whitelist
[21:39:19 CET] <kierank> this whitelist business was added after I wrote fffuzz
[21:39:25 CET] <kierank> so ymmv
[21:39:30 CET] <kierank> I don't understand how it works tbh
[21:39:38 CET] <kierank> I consider libavformat a lost cause at this point
[21:40:16 CET] <Sesse> /* e.g. AVFMT_NOFILE formats will not have a AVIOContext */
[21:40:16 CET] <Sesse> if (s->pb)
[21:40:16 CET] <Sesse> ff_id3v2_read(s, ID3v2_DEFAULT_MAGIC, &id3v2_extra_meta, 0);
[21:40:21 CET] <Sesse> hardcoded
[21:42:52 CET] <kierank> ask wm4 about weird stuff like this
[21:43:01 CET] <kierank> 8:39 PM <"kierank> I consider libavformat a lost cause at this point
[21:44:05 CET] <Sesse> I commented out the id3 checks
[21:44:07 CET] <Sesse> speed doubled
[21:44:34 CET] <kierank> what does your profile look like?
[21:44:37 CET] <kierank> I still question whether it's working properly
[21:44:41 CET] <Sesse> I pasted it earlier
[21:44:50 CET] <kierank> i mean is speedhq still far down?
[21:44:52 CET] <Sesse> yes
[21:45:00 CET] <kierank> then i question whether it's working
[21:45:07 CET] <Sesse> why
[21:45:13 CET] <kierank> it should be at the top
[21:45:13 CET] <Sesse> it's decoding a 16x32 source
[21:45:23 CET] <Sesse> that's nothing compared to the huge amounts of setup
[21:45:35 CET] <kierank> aren't you using the mode where you don't need to setup
[21:45:42 CET] <KGB> [13FFV1] 15michaelni pushed 1 new commit to 06master: 02 https://git.io/vMW8z
[21:45:42 CET] <KGB> 13FFV1/06master 1435d5ae9 15Michael Niedermayer: Put table under ``` to avoid build xml build failure...
[21:45:49 CET] <Sesse> there's tons of setup still
[21:45:51 CET] <Sesse> just less
[21:46:04 CET] <kierank> ok
[21:46:09 CET] <Sesse> look at all the stuff going on within __AFL_LOOP
[21:46:39 CET] <Sesse> compared to the ~1000 cycles or whatever to do the actual decoding
[21:47:06 CET] <Sesse> I mean, just looking up "speedhq" in a hash table compared to that...
[21:54:31 CET] <KGB> [13FFV1] 15michaelni closed pull request #48: add Michael's email (06master...06author's-email) 02 https://git.io/vMWc7
[22:26:03 CET] <Compn> we dont have a quick zzuf tutorial for testing ffmpeg ?
[22:26:11 CET] <Compn> er testing new decoder/demuxer ?
[22:29:25 CET] <Sesse> none that I discovered, at least :-P
[22:43:36 CET] <j-b> hello
[22:43:40 CET] <Sesse> hi
[23:19:50 CET] <durandal_1707> Sesse: that memcpy really bothers me, what other codecs do that?
[23:20:14 CET] <Sesse> durandal_1707: memcpy or memset?
[23:20:34 CET] <durandal_1707> both
[23:20:44 CET] <Sesse> how would you ensure AV_INPUT_BUFFER_PADDING_SIZE without the memcpy?
[23:21:54 CET] <Sesse> seemingly most code goes through av_fast_padded_malloc
[23:22:05 CET] <durandal_1707> isn't there checked bitstream reader?
[23:22:10 CET] <Sesse> yes
[23:22:23 CET] <Sesse> it still requires AV_INPUT_BUFFER_PADDING_SIZE
[23:23:08 CET] <Sesse> * @param buffer bitstream buffer, must be AV_INPUT_BUFFER_PADDING_SIZE bytes
[23:23:12 CET] <Sesse> * larger than the actual read bits because some optimized bitstream
[23:23:14 CET] <Sesse> * readers read 32 or 64 bit at once and could read over the end
[23:23:16 CET] <Sesse> (init_get_bits8)
[23:25:49 CET] <durandal_1707> than why this padding is not in packet reading?
[23:26:16 CET] <durandal_1707> what other codecs uses this madness?
[23:27:28 CET] <Sesse> pannekake:~/nmu/ffmpeg> grep -l AV_INPUT_BUFFER_PADDING_SIZE libavcodec/*.c | wc -l
[23:27:31 CET] <Sesse> 84
[23:27:38 CET] <Sesse> pannekake:~/nmu/ffmpeg> grep -l padded_malloc libavcodec/*.c | wc -l
[23:27:41 CET] <Sesse> 32
[23:28:34 CET] <durandal_1707> that may not copy input at all
[23:29:09 CET] <Sesse> so, what do you suggest I do instead?
[23:31:11 CET] <durandal_1707> Sesse: how much it reads bits at once?
[23:31:25 CET] <Sesse> durandal_1707: I don't understand that sentence, sorry
[23:32:03 CET] <durandal_1707> get vlc stuff reads bits? right?
[23:32:52 CET] <durandal_1707> you said you get valgrind errors
[23:32:54 CET] <Sesse> the bitstream reader reads bits. you are typically allowed to consume up to MIN_CACHE_BITS at once, which is 25 or 32.
[23:33:11 CET] <Sesse> as what the bitstream reader consumes, that is in increments of 32 or 64 bits.
[23:33:33 CET] <Sesse> AV_INPUT_BUFFER_PADDING_SIZE is 32 bytes, which is probably overly conservative, but, yes.
[23:33:57 CET] <Sesse> the point is that the bitstream reader is allowed to read past the end of the buffer, for efficiency reasons
[23:34:15 CET] <Sesse> checked bitstream readers make sure the margin is bounded
[23:34:19 CET] <Sesse> but there's still a margin.
[23:34:41 CET] <durandal_1707> but not all deciders memcpy packet data
[23:34:54 CET] <durandal_1707> *decoders
[23:34:55 CET] <jamrial> Sesse: that amount is because of simd intructions that, at least on x86, can load up to 32 bytes from memory with a single instruction
[23:35:15 CET] <Sesse> durandal_1707: I still await your better solution
[23:35:18 CET] <jamrial> it will probably be changed to 64 in the coming years
[23:35:41 CET] <Sesse> durandal_1707: the documentation for the bitstream reader says I need to provide AV_INPUT_BUFFER_PADDING_SIZE margin, I'm complying with that
[23:35:48 CET] <Sesse> if you're saying it is unneccessary, please explain why
[23:35:57 CET] <Sesse> if you're saying there's a better way of doing it, please explain how
[23:36:03 CET] <durandal_1707> Sesse: do you get invalid reads without memcpy?
[23:36:24 CET] <Sesse> durandal_1707: no, I haven't bothered checking, but it will certainly depend on the specifics of the reader that's chosen
[23:36:39 CET] <Sesse> durandal_1707: are you saying I should willfully ignore a stated warning in the documentation on grounds of nah, I don't understand why this should be a problem ?
[23:37:11 CET] <durandal_1707> Sesse: i think that then bunch of decoders are wrong
[23:37:47 CET] <durandal_1707> but i think that packet reading function already do padding
[23:37:51 CET] <Sesse> durandal_1707: well, 100+ decoders evidently care enough to call the functions that pad
[23:38:10 CET] <Sesse> durandal_1707: if there's a guarantee that buffers coming in to the decoder functions are already padded, please point me to this guarantee
[23:38:16 CET] <Sesse> I'll be happy to remove the memcpy then
[23:39:07 CET] <durandal_1707> look in avformat
[23:39:24 CET] <Sesse> avformat doesn't matter, what matters is the stated API to decode_frame
[23:39:35 CET] <Sesse> you have to distinguish between guarantees and implementation accidents
[23:39:52 CET] <durandal_1707> i will try your decoder without memcpy and report back
[23:39:55 CET] <Sesse> ...
[23:40:30 CET] <durandal_1707> seriously having each decoder copy is simply nonsense
[23:40:44 CET] <Sesse> not all decoders need bitstream decoding
[23:41:03 CET] <Sesse> and not all decoders need bitstream decoding of all parts. if you know the last 32 bytes are safe, you can avoid copying.
[23:41:14 CET] <durandal_1707> yes, but im not talking about others
[23:41:52 CET] <durandal_1707> copy costs
[23:41:58 CET] <Sesse> according to the profile, it doesn't
[23:42:31 CET] <durandal_1707> its free?
[23:42:54 CET] <Sesse> 0.26% of decoding
[23:42:56 CET] <Sesse> I can live with that
[23:43:11 CET] <Sesse> at least if the alternative is a CVE down the road when something in avformat changes
[23:43:19 CET] <durandal_1707> even unchecked bitstream readers dont do copy
[23:43:25 CET] <Sesse> again, if there is a guarantee, please show it to me
[23:43:36 CET] <Sesse> comments, documentation, anything except I think there's some code here
[23:43:39 CET] <durandal_1707> tomorrow
[23:43:46 CET] <Compn> durandal_1707 : is it ok to commit this first patch, and then fix the problem with memcpy later ?
[23:43:55 CET] <durandal_1707> sure
[23:43:58 CET] <Compn> rephrased, is this memcpy a blocker?
[23:44:01 CET] <Compn> ok just checking :)
[23:44:13 CET] <durandal_1707> its just ugly in my eyes
[23:44:23 CET] <Compn> Sesse : please dont take questioning as insult, we are just discussing :)
[23:45:21 CET] <durandal_1707> but codec addition is very nice
[23:45:23 CET] Action: Compn tries to diffuse eastern european hostilities
[23:46:17 CET] <durandal_1707> im not from eastern europe
[23:47:22 CET] <Compn> where are you from again ?
[23:47:36 CET] <Compn> i was thinking cz but maybe i have mistaken :\
[23:48:05 CET] <kierank> iirc it's manadatory in the api to provide FF_INPUT_BUFFER_PADDING_SIZE
[23:48:16 CET] <jamrial> av_new_packet() seems to add padding. is that what this is about?
[23:48:40 CET] <kierank> but tbh there are also undocumented alignment requirements
[23:48:58 CET] <Sesse> kierank: the question for me is whether it's mandatory in the api down to my codec to provide FF_INPUT_BUFFER_PADDING_SIZE
[23:49:07 CET] <Sesse> kierank: so that I can pass these buffers on to the bitstream readers safely
[23:49:17 CET] <Sesse> kierank: if there is, there _really_ should be a comment about that
[23:50:04 CET] <kierank> like the output planes say 16-byte alignment or something but in practice it's 32
[23:50:08 CET] <Sesse> there seems to be such a guarantee for extradata
[23:50:35 CET] <Sesse> but I can't find it for actual packet data. maybe I'm just looking in the wrong place, though
[23:51:35 CET] <kierank> Sesse: is the comment in avcodec.h enough?
[23:52:26 CET] <Compn> Throughout Western Europe and in North America, the country is often incorrectly referred to as an Eastern European
[23:52:40 CET] <Compn> doh!
[23:53:40 CET] <Sesse> kierank: which one?
[23:53:56 CET] <kierank> https://git.videolan.org/?p=ffmpeg.git;a=blob;f=libavcodec/avcodec.h;h=ca8b786077efba0b32b8c47c4c40e808ab5f18cc;hb=HEAD#l4785
[23:54:28 CET] <Sesse> kierank: hm, avcodec_decode_video2() is just a wrapper around the decode function pointer?
[23:54:42 CET] <kierank> essentially yes i belivee
[23:54:44 CET] <kierank> believe
[23:54:55 CET] <Sesse> if it intends to just shove the buffer through, I think that's a strong enough guarantee, yes
[23:55:02 CET] <Sesse> if nothing else, it clearly shows the API's intent
[23:55:58 CET] <kierank> https://github.com/FFmpeg/FFmpeg/blob/314269118161ed7735a905332ccf847d1ec696a2/libavcodec/utils.c#L2214
[23:56:06 CET] <kierank> some stuff along the way but yes
[23:56:36 CET] <Sesse> there's some stuff about splitting the packet
[23:56:38 CET] <Sesse> for side data
[23:57:25 CET] <Sesse> sad that AVPacket doesn't just document this guarantee right in the size field
[23:57:28 CET] <jamrial> that doesn't split the packet. just the packet's side data
[23:57:36 CET] <Sesse> mm, ok
[00:00:00 CET] --- Mon Jan 9 2017
More information about the Ffmpeg-devel-irc
mailing list