[Ffmpeg-devel-irc] ffmpeg-devel.log.20170724

[burek]

[02:48:52 CEST] <cone-553> ffmpeg 03Michael Niedermayer 07master:69e7daf6ce2a: avcodec/dirac_vlc: Fix undefined shift
[02:48:52 CEST] <cone-553> ffmpeg 03Michael Niedermayer 07master:2dfb8c417891: avcodec/aacdec_fixed: fix: left shift of negative value -1
[02:48:52 CEST] <cone-553> ffmpeg 03Michael Niedermayer 07master:aff93e19297c: avcodec/mpegvideo_enc: Use intra/inter scantable matching mb type in quantization
[03:11:25 CEST] <KGB> [13FFV1] 15michaelni closed pull request #80: tiny typo (06master...06patch-1) 02https://git.io/v7ICy
[11:47:40 CEST] <ubitux> michaelni: can you upload http://b.pkh.me/fake-gp-media-with-real-gpmf.mp4 to fate-suite/mov?
[11:49:55 CEST] <ubitux> i'll push the gpmf remux with the following: http://sprunge.us/KeYX?diff
[12:06:00 CEST] <ubitux> i can push the test later though
[12:50:19 CEST] <J_Darnley> I've got a question regarding tests/ref/fate/sub-srt-badsyntax, does it have crlf line endings in master?  Is it supposed to?
[13:14:22 CEST] <J_Darnley> What does git mean with this message?  "warning: CRLF will be replaced by LF in tests/ref/fate/sub-srt-badsyntax.  The file will have its original line endings in your working directory."
[13:16:36 CEST] <J_Darnley> God dammit git!  Just commit the file as given!
[13:17:33 CEST] <TMM> Does git do automatic dos/unix line conversions?
[13:17:36 CEST] <TMM> that's weird
[13:19:08 CEST] <J_Darnley> All because Notepad is too stupid to understand lf endings.
[13:19:27 CEST] <J_Darnley> Although bash isn't any better and doesn't like crlf endings.
[13:28:11 CEST] <kierank> J_Darnley: ah yeah i have botched line endings in my repo
[13:30:47 CEST] <J_Darnley> damn you telenet
[13:30:48 CEST] <J_Darnley> Well, they may or may not be matching master at present.  I just commited regardless.
[13:30:57 CEST] Action: J_Darnley goes for lunch
[13:39:58 CEST] <michaelni> ubitux, uploaded
[13:53:57 CEST] <durandal_1707> what about fake color filter?
[13:54:31 CEST] <ubitux> michaelni: thanks
[13:56:24 CEST] <cone-362> ffmpeg 03Steven Liu 07master:f21457f8e0db: avformat/hlsenc: fix hls fmp4 extention name bug
[13:58:50 CEST] <J_Darnley> :( back to dirac
[14:44:09 CEST] <cone-362> ffmpeg 03Clément BSsch 07master:850a45aef10b: lavf/movenc: support GPMF track (gpmd) remuxing
[15:13:51 CEST] <ubitux> note to self: add an option to disable all the fucking autodetected libs
[15:14:08 CEST] <ubitux> (it makes builds completely unpredictible)
[15:15:33 CEST] <JEEB> aye
[15:17:43 CEST] <RiCON> --disable-all?
[15:21:30 CEST] <ubitux> only the autodetected stuff
[15:21:37 CEST] <ubitux> xcb, vdpau etc
[15:22:16 CEST] <BBB> ubitux: yes!!!!!
[15:22:27 CEST] <BBB> I really dislike how vda is automatically enabled in my osx builds
[15:27:52 CEST] <J_Darnley> kierank: I've got edge extension working but the other transforms produce just as poor quality as before.
[15:28:28 CEST] <kierank> J_Darnley: as before?
[15:28:38 CEST] <J_Darnley> as before edge extension
[15:28:56 CEST] <kierank> I guess the result is not bitexact?
[15:29:35 CEST] <J_Darnley> the output is the same with and without extension but the quality is rubbish compared to the whole plane transform
[15:29:47 CEST] <kierank> I would ask atomnuker 
[15:31:47 CEST] Action: J_Darnley needs to look at the spec again
[15:38:49 CEST] <ubitux> BBB: it's especially problematic in the context of automated builds
[16:06:13 CEST] <nevcairiel> typically things are enabled that are basically available everywhere, or require no runtime libraries
[17:54:42 CEST] <cone-362> ffmpeg 03Michael Niedermayer 07master:0764fe1d0983: avcodec/aacps: Fix multiple integer overflow in map_val_34_to_20()
[17:54:43 CEST] <cone-362> ffmpeg 03Michael Niedermayer 07master:03a9e6ff303a: avcodec/ylc: Fix shift overflow
[18:01:26 CEST] <BBB> does anyone feel like it would be more productive to amend the C standard and make the results of signed integer operation overflows implementation-defined instead of undefined instead of the madness currently going on in our codebase?
[18:02:49 CEST] <nevcairiel> Personally I would just treat it that way, and call it a day. But apparently so called "security researches" see these things as "serious issues" and if you don't listen to them they'll drown you in CVEs and ruin your day!
[18:03:27 CEST] <nevcairiel> All this shit probably doesnt even change the assembly one bit, or does it?
[18:05:16 CEST] <iive> BBB: it could be even better, they can create a new defined type, or repurpose one of the existing. .e.g int32_t could be defined to overflow and "int" could be left undefined.
[18:05:22 CEST] <BBB> Im worried about readability and wasting developer time
[18:05:30 CEST] <BBB> the output assembly is indeed the same
[18:05:53 CEST] <BBB> (or nearly the same, I believe for multiplies it can change a bit)
[18:27:41 CEST] <Blubberbub> Isn't the problem with overflows that some compilers say: "Hey, i know this can overflow, but overflow is undefined behaivior, so i can do whatever i want now" and "optimize" the applications in a way the developer never intended to?
[18:52:31 CEST] <kierank> atomnuker: is qp_idx=0 for dirac meant to be lossless?
[19:07:12 CEST] <kierank> atomnuker: also where do you handle the picture edges in vc2enc?
[19:07:15 CEST] <kierank> that's also confusing
[19:09:40 CEST] <nevcairiel> Blubberbub: its a cheap excuse to say that the compiler "notices" undefined behavior and then uses that to go off the rails, in reality I doubt any compiler even cares about that and just translates to the appropriate instructions, so it might as well be called implementation defined, since that what it ends up being, defined by the underlying hardware what happens
[19:11:35 CEST] <Blubberbub> nevcairiel, there is this old (>6 years) llvm blogpost "what every c programmer should know about undefined behaivior" that i once read and it has a different standpoint
[19:12:01 CEST] <nevcairiel> i'm sure there are other undefined parts that may be worse
[19:21:42 CEST] <Blubberbub> speaking about overflow: is there a method in ffmpeg that allows me to detect overflow when adding 2 audio samples?
[19:25:18 CEST] <iive> Blubberbub: compile with -ftrapv and it will call trap (int 2) when this happens. you could attach a debugger to it :D
[19:26:38 CEST] <iive> Blubberbub: or use mmx ops that saturate the result
[19:27:26 CEST] <atomnuker> kierank: yes, it should be lossless
[19:27:33 CEST] <Blubberbub> i'd rather prevent them from happening :D
[19:27:41 CEST] <Blubberbub> have not found mmx ops yet
[19:27:51 CEST] <atomnuker> the shift up of the transform and the shift down from quantization result in a shift of 0
[19:28:38 CEST] <nevcairiel> if you use floating sample formats, the "overflow" isnt so easy to automatically detect by the compiler, since overflow just means it goes above 1.0, not actually overflows any datatype. of course this method also allows you to actually manually check and recover from the overflow by attenuating
[19:28:58 CEST] <Blubberbub> i'm more worried about signed samples
[19:29:08 CEST] <Blubberbub> but mmx ops is probably the way to go
[19:29:18 CEST] <Blubberbub> now that i know they exist. Thanks iive :)
[19:29:19 CEST] <nevcairiel> any audio processing should really be done in float though
[19:29:33 CEST] <BBB> Blubberbub: please note that most blog posts about undefined behaviour vs. security issues are not at all related to integer overflows
[19:29:51 CEST] <BBB> Blubberbub: nobody denies that some forms of undefined behaviour (e.g. out of bounds array reads/writes) are extremely dangerous
[19:30:01 CEST] <nevcairiel> also, dont use mmx, use sse at the very least. it has the same saturating instructions
[19:30:12 CEST] <nevcairiel> mmx is just old and dead
[19:30:24 CEST] <Blubberbub> nevcairiel, i fear i will have to support old hardware :(
[19:30:25 CEST] <iive> all mmx ops are present in sse and avx
[19:30:26 CEST] <BBB> Blubberbub: but a signed integer multiplication overflow isnt quite the same undefined behaviour as an OOB write - yet theyre both undefined and thus some people will claim you can hack my computer with either
[19:31:09 CEST] <nevcairiel> Blubberbub: ... how old is old? sse1/2 are available for 15 years now
[19:32:14 CEST] <jdarnley> If you're on x86-64 then sse2 is always available
[19:33:04 CEST] <Blubberbub> have to target 32 bit :(
[19:35:37 CEST] <nevcairiel> thats fine, but what kind of level of old cpus?
[19:35:49 CEST] <nevcairiel> because mmx-only cpus are really so old that it seems very unrealistic
[19:36:42 CEST] <Blubberbub> BBB: Its not the same, true. An exploit is normally caused by something the programmer did not think of when writing the code. And overflows are easy to forget.
[19:37:55 CEST] <iive> one of the examples in the llvm article is a boundary check, that gets optimized out, because the compiler assumes overflow is not possible.
[19:39:06 CEST] <nevcairiel> there may be some cases where its relevant, but we're talking DSP functions here, the data is not used other then making pixels
[19:39:45 CEST] <Blubberbub> nevcairiel, the code is supposed to run on an Intel Pentium M... (Don't ask...)
[19:40:09 CEST] <iive> Blubberbub: Military?
[19:40:13 CEST] <nevcairiel> Pentium M is a P6, that has SSE and SSE2
[19:40:17 CEST] <nevcairiel> so no reason to bother with MMX
[19:40:28 CEST] <durandal_170> atomnuker: are you in good health?
[19:40:36 CEST] <Blubberbub> yea, just checked that as well. :D
[19:40:40 CEST] <Blubberbub> thanks :)
[19:40:52 CEST] <nevcairiel> like I said, MMX is unbeliavably old ;)
[19:41:11 CEST] <nevcairiel> even a 15 year old cpu from 2003 like the Pentium M supports SSE already. :)
[19:41:25 CEST] <nevcairiel> (and on that slow thing, you'll need it :p)
[19:41:30 CEST] <Blubberbub> iive, worse. Its family. :)
[19:41:42 CEST] <iive> :D
[19:44:05 CEST] <kierank> atomnuker: not lossless 
[19:44:37 CEST] <kierank> ./ffmpeg -loglevel info -lavfi testsrc2=hd1080,format=yuv422p10 -vframes 1 -vcodec vc2 -wavelet_type haar -strict unofficial -wavelet_depth 3 -threads 1 -qp 0 -y temp.nut
[19:45:36 CEST] <atomnuker> iive: you can't use 8 registers in 32 bit mode
[19:45:39 CEST] <atomnuker> you can use 7 grps
[19:45:56 CEST] <iive> atomnuker: 8 xmm registers
[19:46:45 CEST] <iive> i have ifdef-ery for the PIC base register
[19:46:50 CEST] <atomnuker> iive: all of our asm code has costants at the very top
[19:47:42 CEST] <iive> it has always been like this and it will always be like this. Amen.
[19:48:48 CEST] <atomnuker> well there's no reason to change the style randomly
[19:49:09 CEST] <iive> atomnuker: oh, forgot to ask you, benchmark also xmm avx2
[19:49:11 CEST] <atomnuker> I wouldn't mind if you just put the emulated instruction crap in x86utils
[19:49:19 CEST] <atomnuker> iive: I did, it made no difference
[19:49:34 CEST] <iive> now that's strange.
[19:49:45 CEST] <iive> can i see the numbers?
[19:50:43 CEST] <atomnuker> I posted them in the irc channel long ago when I did all the benchmarks on my machine
[19:50:57 CEST] <atomnuker> it was the same speed as avx, but I'll give it another go
[19:51:16 CEST] <iive> oh, that's ok
[19:51:48 CEST] <iive> i do expect them to be similar to avx1.
[19:53:27 CEST] <iive> you can see that there is small speed up on ryzen between v3 and v4, that's why I'd be happy if you run few more benchmarks.
[19:54:25 CEST] <iive> i'll make a patch that puts the emulated instructions in x86util ,
[19:54:55 CEST] <iive> there is a minor problem that i have improved version of a broadcast op that is already used in few places...
[19:57:47 CEST] <kierank> atomnuker: ah seems to be the way you calculate q_avg is false
[19:58:24 CEST] <kierank> s->q_avg = (s->q_avg + args->quant_idx)/2;
[19:58:30 CEST] <kierank> that's not equivalent to an average afaik
[19:59:31 CEST] <durandal_170> BBB: have you looked at ssim asm crash?
[19:59:44 CEST] <BBB> no, probably lack of padding, but not looked yet
[20:01:58 CEST] <atomnuker> kierank: it looks like an average to me, except I drop the rounding so in some cases it'll be off by one
[20:02:32 CEST] <kierank> I get qavg = 0 when clearly there are slices with not that
[20:03:36 CEST] <kierank> (a+b)/2 + c/2 != (a+b+c)/3
[20:04:15 CEST] <atomnuker> it works as indended then, make it a float and it'll give you more precision
[20:04:27 CEST] <kierank> stil 0.000000
[20:04:29 CEST] <kierank> ((a+b)/2 + c)/2 I mean
[20:04:32 CEST] <kierank> that's not an average
[20:04:37 CEST] <atomnuker> where do I do that?
[20:05:14 CEST] <kierank>     for (i = 0; i < s->num_x*s->num_y; i++) {
[20:05:14 CEST] <kierank>         SliceArgs *args = &enc_args[i];
[20:05:14 CEST] <kierank>         total_bytes_needed += args->bytes;
[20:05:14 CEST] <kierank>         s->q_avg = (s->q_avg + args->quant_idx)/2;
[20:05:14 CEST] <kierank>     }
[20:06:18 CEST] <atomnuker> it is an average, its just not accurate because of doing it on a per-slice basis with an integer
[20:06:33 CEST] <Blubberbub> i also don't think its an average
[20:06:33 CEST] <kierank> ((a+b)/2 + c)/2  != (a+b+c)/3
[20:06:34 CEST] <atomnuker> make it a sum and divide it by the number of slices then
[20:06:41 CEST] <kierank> QED it's not an average
[20:06:48 CEST] <Blubberbub> the last sample is only divided by 2, but the first by 2 ** a lot
[20:07:52 CEST] <atomnuker> it is an average, you're both tripping
[20:07:55 CEST] <kierank> I will send a patch
[20:08:05 CEST] <kierank> I have just proven it's not an average
[20:08:22 CEST] <atomnuker> your proof does not apply here
[20:08:30 CEST] <kierank> yes it does
[20:08:54 CEST] <kierank> the fact you are rounding doesn't make the maths suddenly correct
[20:09:26 CEST] <Blubberbub> summing and dividing has huge numeric instability, though.
[20:09:33 CEST] <iive> I reject your reality and substitute my own ;)
[20:09:47 CEST] <kierank> I just want to know the average slice quantiser
[20:09:48 CEST] <kierank> that's all
[20:09:54 CEST] <kierank> and it's printed incorrectly at the moment
[20:12:05 CEST] <atomnuker> kierank: IT IS AN AVERAGE, just with a sample size of 2
[20:12:14 CEST] <jdarnley> Fuck libavfilter and its rubbish syntax.  I hate you!  Everything crammed into one string with unknown levels of escaping.
[20:12:15 CEST] <kierank> it's not a sample size of two
[20:12:20 CEST] <Blubberbub> ah
[20:12:20 CEST] <kierank> you loop through all the slices
[20:12:26 CEST] <Blubberbub> its not += its =
[20:12:27 CEST] <atomnuker> I ignore that
[20:12:34 CEST] <Blubberbub> oh wait
[20:12:39 CEST] <atomnuker> its valid if the sample size is 2
[20:12:53 CEST] <kierank> then it's not an average slice quantiser?
[20:13:04 CEST] <atomnuker> no, it is not
[20:13:23 CEST] <kierank> https://github.com/FFmpeg/FFmpeg/blob/master/libavcodec/vc2enc.c#L1048
[20:13:26 CEST] <kierank> why print it then?
[20:13:38 CEST] <kierank> it's not used anywhere apart from printing
[20:14:09 CEST] <atomnuker> debugging, I wanted to know more or less a very coarse quantizer info
[20:14:34 CEST] <atomnuker> since quality goes to shit vert very quickly as you reduce bitrate
[20:14:37 CEST] <kierank> this is completely insane
[20:14:57 CEST] <atomnuker> x264 does it
[20:15:02 CEST] <kierank> x264 does it properly
[20:15:26 CEST] <atomnuker> I didn't want to spend time doing it properly
[20:15:40 CEST] <atomnuker> I wanted a rough indication and couldn't be bothered to make it correct
[20:15:55 CEST] <kierank> yes but it confuses people like me who are working on the code
[20:15:59 CEST] <kierank> who thought the output was lossless
[20:16:32 CEST] <kierank> h->stat.f_frame_qp[i_slice] / i_count
[20:16:35 CEST] <kierank> x264 QP is a proper average
[20:16:40 CEST] <atomnuker> no guarantees the output is lossless unless you fix the quantizer
[20:17:30 CEST] <kierank> an average of zero implies all quantisers are zero
[20:17:35 CEST] <atomnuker> well do send a patch then, not like I care
[20:17:36 CEST] <kierank> this is really misleading code, I will fix it
[20:58:51 CEST] <BBB> jdarnley: now now now, have you ever tried gst-launch?
[20:59:58 CEST] <BBB> jdarnley: more generally, yes its complex but it tries to put an incredibly complex structure in a single string, what do you expect :) use the API if you want it to be more logical (and wait for the cursing about how many lines of code it takes to do something thats only a single string in -lavfi syntex)
[21:00:00 CEST] <BBB> *syntax
[21:09:14 CEST] <atomnuker> kierank: suppose you have more than a single frame, what do you think happens
[21:09:34 CEST] <atomnuker> just make a tmp variable, add all the quantizers, divide it by the number of slices
[21:10:00 CEST] <atomnuker> then average that with the q_avg and you'll average it per frame
[21:11:44 CEST] <durandal_170> jdarnley: whats problem in your lavfi usage?
[21:11:55 CEST] <kierank> atomnuker: yes you are right thanks
[21:12:16 CEST] <BBB> so& that patch (https://lwn.net/Articles/697956/ and see ML)
[21:12:25 CEST] <BBB> is that Yet Another hwaccel-like interface?
[21:12:48 CEST] <kierank> yeah it's trying to shoehorn all these weird things into v4l2 imo
[21:13:23 CEST] <BBB> like, as if vda+dxva2+nvenc+qsv+vaapi+vdpau+xvmc wasnt bad enough
[21:13:36 CEST] <BBB> https://xkcd.com/927/ :D
[21:13:59 CEST] <BBB> so now we have v4l2 as Yet Another Hardwarewrapper (YAH)?
[21:14:26 CEST] <kierank> worse i think
[21:14:35 CEST] <kierank> because at least the ones above are somewhat constrained
[21:14:36 CEST] <BBB> but why v4l2
[21:14:44 CEST] <BBB> v4l2 was to capture video from bttv cards
[21:14:53 CEST] <kierank> because the hardware people need to have things as linux kernel interfaces
[21:14:54 CEST] <BBB> how is hw h264 decoding related to capturing video from bttv cards?
[21:14:56 CEST] <kierank> I agree
[21:15:05 CEST] <kierank> but imagine you have a webcam that sends h264
[21:15:07 CEST] <kierank> over usb
[21:15:14 CEST] <kierank> they want that in v4l2
[21:15:18 CEST] <BBB> sure
[21:15:27 CEST] <kierank> it's basically an extension to that
[21:15:35 CEST] <BBB> everything-to-everything
[21:15:39 CEST] <kierank> embedded people want stuff in the kernel
[21:15:42 CEST] <BBB> lets move gstreamer into the kernel next
[21:15:49 CEST] <BBB> KStreamer
[21:16:08 CEST] <kierank> when i went to kernel summit a few of them had somewhat justifiable reasons for doing this stuff in the kernel
[21:16:09 CEST] <BBB> ohwell, I guess its only on embedded devices so itll never build into my build
[21:16:34 CEST] <kierank> there was a guy doing QC on potatoes as hundreds went past a factory in realtime
[21:16:39 CEST] <kierank> and apparently userspace was too slow
[21:16:53 CEST] <kierank> bizzare use case
[21:17:14 CEST] <BBB> potatoes
[21:17:33 CEST] <BBB> okaayyyyyy
[21:19:48 CEST] <kierank> BBB: i can't say i fully understand what that libavcodec m2m thing does
[21:20:10 CEST] <kierank> doesn't seem to be codec related at all
[22:03:50 CEST] <iive> so, we can't have audio mixing in the kernel, but we can have video decoding in it...
[00:00:00 CEST] --- Tue Jul 25 2017