[Ffmpeg-devel-irc] ffmpeg.log.20170530

[burek]

[00:05:15 CEST] <DHE> so, no ideas?
[01:36:26 CEST] <bob_twinkles> is there a good way to get the ffmpeg CLI tool to print its status lines as seperate lines, instead of being cute and rewritting the same line over and over again?
[01:50:54 CEST] <bob_twinkles> DHE: are you getting any messages about dropped frames?
[01:51:38 CEST] <CoJaBo> you can always pipe the output to something
[01:52:17 CEST] <bob_twinkles> it has to go to stdout so it'll land in the CloudWatch logs =/
[01:53:21 CEST] <bob_twinkles> (I'm running a bunch of video transcoding jobs in The Cloud since I don't have a machine powerful enough to do the job locally)
[01:54:12 CEST] <CoJaBo> Something like tr should be able to change the \r to \n, then spit it out to stdout still
[02:12:23 CEST] <bob_twinkles> hrm
[02:12:39 CEST] <bob_twinkles> looks like it might have been me misunderstanding how cloudwatch batches events
[02:13:02 CEST] <bob_twinkles> seems they agregate similar lines together and I'm not doing the right incantation to get that to flush
[02:16:48 CEST] <DHE> bob_twinkles: pretty sure not..
[03:02:24 CEST] <frW1> i have 250,000.00 USdollar to invest , what should i buy
[03:12:17 CEST] <c3-Win> frW1: A life.
[03:12:52 CEST] <frW1> c3-Win huh
[03:16:40 CEST] <furq> it is extremely normal that you have a different nick every time you ask this
[03:17:16 CEST] <frW1> furq i don't remember the last nick i used
[03:17:28 CEST] <frW1> furq if you don't remember the last nick you used, then what would you do
[03:17:42 CEST] <dystopia_> doesn't #ffmpeg require a registered nick :O
[03:17:50 CEST] <furq> well for starters i wouldn't ask #ffmpeg for financial advice
[03:17:58 CEST] <furq> but then it's debatable whether that's what you're doing
[05:19:38 CEST] <m712> hello
[05:20:10 CEST] <m712> i am trying to copy part of a video and an audio stream from an mkv to another mkv
[05:20:38 CEST] <m712> however the location i set in -ss doesn't have a keyframe, so the video skips to 2-3 seconds in
[05:20:57 CEST] <m712> how can i make ffmpeg encode a first keyframe?
[05:23:11 CEST] <teratorn> m712: you can't do that while using vcodec copy
[05:23:56 CEST] <m712> i don't want to re-encode the entire thing though :/
[05:24:33 CEST] <m712> because when i reencode there are visible artifacts in the video
[05:25:05 CEST] <m712> is there really no way to encode the first frame as a keyframe and copy the rest?
[05:25:51 CEST] <teratorn> m712: unfortunately not
[05:26:34 CEST] <teratorn> m712: https://trac.ffmpeg.org/wiki/Seeking
[05:27:19 CEST] <teratorn> m712: maybe you can get rid of the artifacts if you play with the codec parameters
[05:27:27 CEST] <m712> maybe
[05:28:39 CEST] <m712> i was thinking this though: i can re-encode one frame with exactly the same codec and same parameters, and then concatenate it with the original video with v:copy, one frame off
[05:29:09 CEST] <teratorn> I was hoping I might be able to implement this feature (opportunistically copy GOPs while re-encoding beginning/ending keyframes to give frame-accurate seeking), but it's more involved than I hoped as the "copy" codec isn't a real codec at all, and has special handling in the front-end
[05:31:21 CEST] <teratorn> m712: hmm. how would you do the concatentaiton?
[05:31:31 CEST] <teratorn> *concatenation
[05:31:34 CEST] <m712> i think there was a filter like that
[05:31:48 CEST] <teratorn> concat filter, yes, but filters imply not using the copy codec
[05:31:49 CEST] <m712> i've done concatenation before, i forgot how to
[05:31:54 CEST] <teratorn> cuz filters deal in raw frames
[05:33:12 CEST] <m712> well, i'll give it a shot when i get home and share my findings (about 11 hours from now)
[05:33:33 CEST] <teratorn> hmm ok actually there is a "concat" format too, not just the concat filter
[05:33:43 CEST] <teratorn> https://trac.ffmpeg.org/wiki/Concatenate
[05:34:03 CEST] <m712> i've re-encoded it right now but as i said, there's ugly artifacts because of lossy encoding
[05:35:00 CEST] <m712> https://trac.ffmpeg.org/wiki/Concatenate#samecodec seems to be what i need, i'll give it a shot
[05:35:54 CEST] <teratorn> yeah I guess you can do a frame-accurate cut from your starting point till the next key-frame, but how do you know when that is?
[05:36:34 CEST] <m712> i can use mpv to find out the timecode
[05:37:00 CEST] <m712> mpv can show timecodes to the millisecond
[05:37:21 CEST] <m712> if i pause and try to seek to the beginning it will land on the first keyframe
[08:11:59 CEST] <m712> so how's life
[09:59:38 CEST] <Guest8925> Hello, I am facing a segfault in avcodec_video2 when using ffmpeg libs on Android (Unity app). all infos can be found here : https://pastebin.com/7BNPgcmT. Let me know if can help me, i'm really stuck here. If you need more infos, please ask me. Thanks!
[11:13:59 CEST] <livingbeef> I have something like 'ffmpeg -i a.png -i 'b.png' -i 'c.png' -lavfi [0]copy[x0],[x0][1]overlay=331:139[x1],[x1][2]overlay=109:139[x2] -map [x2] -y result.png
[11:14:41 CEST] <livingbeef> and the result is kinda blurry. Any idea what's the cause?
[11:18:05 CEST] <livingbeef> Oh, never mind, I'm an idiot. It uses yuv420 as the default format, png is obviously using rgb.
[13:26:14 CEST] <cousin_luigi> Greetings.
[14:14:19 CEST] <aaap> hello guys
[15:09:17 CEST] <dantedevil>  hi i have an error with motion, somebody can help me??? I have a Raspberry pi 3 and i use motion with 3 ipcamera. but ihave same error  in my log file,  as"[ERR] [ENC] [May 29 19:36:37] ffmpeg_avcodec_log: cabac decode of qscale diff failed at 21 21
[15:09:18 CEST] <dantedevil>    [ERR] [ENC] [May 29 19:36:37] ffmpeg_avcodec_log: error while decoding MB 21 21, bytestream 5013"
[15:32:19 CEST] <dantedevil>  hi i have an error with motion, somebody can help me??? I have a Raspberry pi 3 and i use motion with 3 ipcamera. but ihave same error  in my log file,  as"[ERR] [ENC] [May 29 19:36:37] ffmpeg_avcodec_log: cabac decode of qscale diff failed at 21 21
[15:32:19 CEST] <dantedevil>    [ERR] [ENC] [May 29 19:36:37] ffmpeg_avcodec_log: error while decoding MB 21 21, bytestream 5013"
[15:39:46 CEST] <aaap> hello guys
[15:39:53 CEST] <aaap> may i know what is the meaning of this?
[15:39:53 CEST] <aaap> [libx264 @ 0x2c415a0] height not divisible by 2 (678x777)
[15:39:54 CEST] <aaap> Error initializing output stream 0:0 -- Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height
[15:42:15 CEST] <DHE> aaap: exactly what it says. 777 (image width) is an odd number
[15:42:38 CEST] <DHE> err, height. not width
[15:46:34 CEST] <Zypho> I am noticing some extra noise and grain when encoding mp4 => hls using the source bitrate. Should I try to build HLS 2-pass support or lower the CPU preset (using medium) to resolve this?
[15:46:57 CEST] <Zypho> x264 codec btw
[15:49:46 CEST] <aaap> DHE, this is what command i use:
[15:49:47 CEST] <aaap> ffmpeg -f concat -safe 0 -i input.txt output.mp4
[15:49:57 CEST] <aaap> input.txt contain a list of images files
[15:58:20 CEST] <Zypho> 2-pass with HLS looks more complex than just lowering the preset, but time is a factor for me
[15:58:30 CEST] <Zypho> encode time is a factor, not dev time =)
[16:01:28 CEST] <DHE> aaap: then your images are 777 pixels tall which x264 can't take
[16:01:36 CEST] <BtbN> 2 pass will take quite exactly double the time to encode
[16:04:36 CEST] <DHE> BtbN: some encoders (including x264) support fast 1st pass mode
[16:05:05 CEST] <BtbN> I imagine 2 pass for the short HLS segments to not be worth it though
[16:05:38 CEST] <BtbN> it's an interesting idea though, to get perfectly constant bitrate, without adding too much padding
[16:05:47 CEST] <rsevero> Hi. There is an announcement from 2016-07-01 at ffmpeg's site mentioning that ffserver will be dropped (http://ffmpeg.org/index.html#ffserv). But I just checked the latest releases available at http://ffmpeg.org/download.html and ffserver seems to be there still. Will ffserver be dropped after all? Please say no ;)
[16:06:05 CEST] <BtbN> Once it gets in the way of anything, it will go.
[16:06:12 CEST] <BtbN> It's unmaintained
[16:06:23 CEST] <Mavrik> It probably doesn't work anyway.
[16:06:36 CEST] <BtbN> Some people are very actively defending it, but when it comes to fixing its issues, remain silent.
[16:06:53 CEST] <rsevero> Oh, the 3.0.8 version does work, I can assure you ;)
[16:07:11 CEST] <BtbN> Nobody has any knowledge on how to work with it anyway, as nobody uses it.
[16:07:18 CEST] <rsevero> BtbN: Will I be another one on this front?
[16:07:35 CEST] <BtbN> If you volunteer to fix and maintain it
[16:07:39 CEST] <BtbN> If not, it will go eventually
[16:07:46 CEST] <rsevero> Hey! I use it! But don't know ho to update it...
[16:07:54 CEST] <Mavrik> Why not just take a proper maintained streaming software?
[16:08:06 CEST] <Mavrik> Heck, even nginx has plugins for that these days.
[16:08:09 CEST] <rsevero> Mavrik: Suggestions?
[16:08:21 CEST] <BtbN> The common open source solution is nginx-rtmp
[16:09:14 CEST] <Zypho> ^^
[16:09:26 CEST] <rsevero> BtbN: Ok, I will take a look at it. Thanks.
[16:14:37 CEST] <Dex_> Hi, could some of you please have a look at my problem: https://superuser.com/questions/1214314/unable-to-set-ffmpeg-frei0r-facedetect-filter-parameter
[16:28:32 CEST] <ppw> Is Opus really better than AAC?
[16:28:43 CEST] <ppw> for music?
[16:29:25 CEST] <atomnuker> yes
[16:29:30 CEST] <atomnuker> for everything
[16:29:44 CEST] <nicholas_he> HI FFmpeg users. Should it be possible to extract separate audio channels from a 2 Dolby E Stream tracks in a .MOV file?
[16:29:58 CEST] <kerio> ppw: jury's still out
[16:30:08 CEST] <kerio> at least, at >128kbps
[16:30:37 CEST] <kerio> and even then, it's only because humans can't tell the difference anymore
[16:30:48 CEST] <atomnuker> if the jury says it can tell the difference between 256kbps opus and 256kbps aac I'll have them thrown out of the court
[16:31:33 CEST] <Dex_> is there a way to skip a  parameter for a filter ?
[16:32:22 CEST] <ppw> kerio / atomnuker: can you provide me with an audio example @ 256 kbps? or at least tell me what piece of music I should test to hear it? I have better-than-average hearing, I need to hear this for myself.
[16:32:35 CEST] <kerio> huh
[16:32:44 CEST] <kerio> just take a flac of your favourite song and convert it?
[16:32:52 CEST] <ppw> so -- any favorite song?
[16:33:01 CEST] <ppw> OK - shouldn't be a problem
[16:33:21 CEST] <bencoh> a proper flac encoded from some clean source, though
[16:33:27 CEST] <ppw> well obviously
[16:34:12 CEST] <bencoh> what do you meab by "better-than-average" though? more sensitive?
[16:34:24 CEST] <Dex_> he has 3 ears :P
[16:34:29 CEST] <bencoh> sweet :p
[16:35:35 CEST] <ppw> for exmaple, I always ace the hearing test at the doctor's. young kids to do. but a lot of adult people don't.
[16:35:41 CEST] <ppw> *do too
[16:36:14 CEST] <Dex_> so can I ... '-vf frei0r=facedetect:SKIP|0|0 '  ?
[16:36:30 CEST] <Dex_> or rather, how can i skip the first parameter
[16:36:36 CEST] <bencoh> ppw: meaning you're not affected by presbycusis? nice
[16:37:20 CEST] <bencoh> (I've always been a bit sad about presbycusis)
[16:37:48 CEST] <Dex_> at least you get rid of the moskito tone
[16:37:59 CEST] <bencoh> huhu
[16:43:04 CEST] <furq> atomnuker: what if it's faac
[16:43:46 CEST] <Mavrik> Don't use faac. :P
[16:44:26 CEST] <furq> ppw: http://listening-test.coresv.net/results.htm#list3
[16:44:32 CEST] <Dex_> Could some of you please have a look at my problem: https://superuser.com/questions/1214314/unable-to-set-ffmpeg-frei0r-facedetect-filter-parameter
[16:44:36 CEST] <furq> there's a test corpus you can use
[16:45:00 CEST] <furq> i guess you want to check the ones where opus scored low
[16:45:21 CEST] <ppw> furq thanks .. but that's qaac .. maybe fdk-aac is slightly better?
[16:45:31 CEST] <furq> i doubt it
[16:46:09 CEST] <Dex_> ah well, i give up for today
[16:46:16 CEST] <Dex_> see you guys tomorrow perhaps
[16:46:17 CEST] Action: Dex_ waves
[18:26:57 CEST] <PlanC> is it possible to add a parameter which makes the output streamable?
[18:27:58 CEST] <furq> is this for mp4
[18:27:59 CEST] <PlanC> my output is in MKV and although the video displays when I stream it, none of the meta data comes with it (e.g duration, bitrate)
[18:28:08 CEST] <PlanC> furq: mkv
[18:28:30 CEST] <furq> er
[18:28:33 CEST] <PlanC> I've noticed that FFMPEG adds the meta after it's done
[18:28:37 CEST] <furq> that sounds like it's already streamable
[18:28:49 CEST] <furq> you can't add the duration before it's done encoding
[18:28:58 CEST] <PlanC> furq: right, but how can I get the duration to show as well?
[18:29:11 CEST] <PlanC> right now it just says 00:00 when it's playing
[18:42:25 CEST] <PlanC> I guess it's streamable right now
[18:42:43 CEST] <PlanC> I just need to find a way to add the meta from the start
[18:47:34 CEST] <DHE> you mean stream directly from ffmpeg to a player before it's done processing?
[18:47:48 CEST] <PlanC> yeah
[18:52:06 CEST] <PlanC> or actually it's streamed to a browser
[18:52:31 CEST] <PlanC> if there was a way I could tell ffmpeg to add the metainfo to the end of the file instead of the start
[18:52:39 CEST] <PlanC> then I think it'd fix the problem
[18:52:59 CEST] <furq> but the end of the file is the end of the stream
[18:53:29 CEST] <PlanC> well it could be added to the start as well
[18:53:55 CEST] <furq> if you added it to the end then you wouldn't even be able to stream the video
[18:53:57 CEST] <furq> like with mp4
[18:54:06 CEST] <PlanC> I'm using mkv though
[18:54:59 CEST] <c_14> PlanC: -live 1 ?
[18:55:25 CEST] <Xys> Hello, when I ffprobe a video, it says : 29.85 fps, 29.97 tbr, 90k tbn, 59.94 tbc
[18:55:33 CEST] <Xys> I would like to change the fps to 30
[18:55:38 CEST] <c_14> the matroska muxer "recently" got some changes (I think the CRC ones) that make it not streamable without that parameter
[18:56:23 CEST] <Xys> And when I do ffmpeg -i video.mp4 -r 30 result.mp4, the FPS are still "29.85 fps"
[18:56:24 CEST] <furq> isn't that a webm dash private option
[18:57:08 CEST] <c_14> Nah, it's listed for standard matroska
[18:57:46 CEST] <c_14> And I've used it as such (and checked the code)
[18:57:46 CEST] <klaxa> in my matroska server i set the live option as well
[18:57:58 CEST] <furq> i don't see it in the docs
[18:58:05 CEST] <Xys> Does anyone has an idea why the FPS are still "29.85" instead of "30" please ? Thanks in advance
[18:58:29 CEST] <Cracki> hi. interested in the "scene" parameter (frame difference), but for all frames, output as text or something. haven't seen the filter to give me this output. wat do?
[18:59:00 CEST] <furq> nvm it's in -h encoder=matroska
[18:59:21 CEST] <c_14> The docs for the matroska muxer are (very) incomplete
[18:59:27 CEST] <furq> i can see that
[19:11:40 CEST] <PlanC> c_14: I just tried "-live 1"
[19:11:46 CEST] <PlanC> but the output is the same
[19:12:09 CEST] <PlanC> no duration is shown and I also can't jump to different parts of the video
[19:12:29 CEST] <PlanC> it plays perfectly if I just let it run on it's own though (from 00:00 to end of video)
[19:12:37 CEST] <c_14> probably missing the index
[19:12:44 CEST] <Cracki> showinfo could use a scene variable, no?
[19:13:17 CEST] <c_14> pretty sure you can't add that prior to finishing
[19:13:56 CEST] <PlanC> the thing is that the metainfo is added once ffmpeg finishes
[19:14:16 CEST] <PlanC> and it adds it to the start
[19:14:45 CEST] <PlanC> if there was a way to have it add the duration to the start right from the beginning then it would solve the problem
[19:15:05 CEST] <PlanC> instead of prepending it to the file in the end
[19:15:20 CEST] <c_14> ffmpeg doesn't know how long the file is before it finishes
[19:16:14 CEST] <PlanC> could I tell it with "-t"?
[19:16:29 CEST] <PlanC> or is that just used for cutting
[19:16:34 CEST] <c_14> No, you'd have to write a patch to add an option to specify
[19:17:31 CEST] <c_14> Even if you specify -t, the video could still be shorter
[19:17:37 CEST] <PlanC> oh I've got no idea how to do that so I don't think it's an option
[19:17:41 CEST] <PlanC> right
[19:18:16 CEST] <c_14> and the seeking is probably just your player being unable/refusing to seek without an index
[19:22:55 CEST] <PlanC> so there's no way I can add the index to the end of the file
[19:23:08 CEST] <PlanC> or even somewhere else in the file without having it prepend any data?
[19:23:31 CEST] <PlanC> because I can't take back or edit what's already sent out to the browser
[19:25:06 CEST] <c_14> Not really, you might want to rethink how you're serving content to the browser. Maybe webm-dash will work for you?
[19:25:54 CEST] <c_14> or regular dash or mpeg-ts or something
[19:26:45 CEST] <PlanC> but what I don't understand is why ffmpeg doesn't know the duration of the file
[19:26:56 CEST] <PlanC> because it knows the duration of the input
[19:27:29 CEST] <PlanC> I can even see the duration of the input in the ffmpeg logs before it starts processing
[19:27:37 CEST] <c_14> It only knows the duration the metadata of the input says it has, this doesn't have to be correct, and most probably won't be correct if ffmpeg seeks/truncates/filters
[19:45:24 CEST] <PlanC> I actually just tried to play the video in the MPC player and it shows the duration
[19:45:35 CEST] <PlanC> and even let's me jump to different parts of it
[19:45:44 CEST] <PlanC> VLC doesn't let me do anyone of that though
[19:46:02 CEST] <PlanC> is there anything I can add to the file that'll make it work in VLC as well?
[19:49:07 CEST] <PlanC> even windows media player shows the duration
[19:49:10 CEST] <PlanC> and I can even seek with that
[19:49:21 CEST] <PlanC> wonder why VLC (which should be more advanced) doesn't support it
[19:50:54 CEST] <furq> probably because vlc isn't very good
[19:52:40 CEST] <PlanC> lots of people use it though
[19:52:44 CEST] <PlanC> and it at least used to be good
[19:53:53 CEST] <jarainf> It's not bad
[20:09:53 CEST] <m712> teratorn: hey
[20:10:13 CEST] <m712> it came out kinda late because i forgot about this, but i got it working
[20:10:21 CEST] <m712> i think -ss is kinda buggy
[20:11:01 CEST] <m712> because there was a keyframe right at where i wanted to cut, at 00:01:30.000 but putting -ss 00:01:30.000 did not include the keyframe
[20:11:27 CEST] <m712> same thing happened when i tried to shorten the end of a video and i put -ss 00:00:00.000, first keyframe disappears
[20:11:39 CEST] <m712> removing that fixes it :S
[20:12:12 CEST] <m712> anyway, i managed to do it by using -f segment and then finding the videos that i needed
[20:12:33 CEST] <m712> which were 043.mkv to 067.mkv
[20:12:49 CEST] <m712> then i concatted them with the concat demuxer and it worked like magic
[20:14:45 CEST] <m712> no more lossy compression artifacts
[20:14:48 CEST] <m712> hooray!
[20:14:54 CEST] <m712> though -ss really needs to be fixed
[20:15:41 CEST] <Mista-D> is there a table of codecs and supported pix_fmt ?
[20:15:50 CEST] <m712> idk man i'm a noob
[20:17:51 CEST] <linkmauve1> Hi, when compiling latest master on ArchLinux on amd64, I get make: *** No rule to make target 'libavcodec/x86/simple_idct.c', needed by 'libavcodec/x86/simple_idct.o'.  Stop., it seems this file got replaced with a similarly named .asm file.
[20:19:12 CEST] <m712> i think avcodec is something else
[20:19:18 CEST] <PlanC> c_14: I actually found a way to change the duration with a "hack"
[20:19:23 CEST] <m712> ffmpeg and avcodec are not related
[20:19:33 CEST] <c_14> Mista-D: ffmpeg -encoders | grep ' V' | awk '{print $2}' | xargs -I'{}' sh -c "echo '{}'; ffmpeg -v quiet -h encoder='{}' | grep 'pixel formats'"
[20:19:56 CEST] <PlanC> c_14: I'm doing "-metadata duration=00:01:00.00" and now I can see the duration and even seek in the MKV
[20:20:15 CEST] <c_14> ah ye, that might work
[20:20:17 CEST] <Mista-D> c_14 - awesome, thank you
[20:20:31 CEST] <PlanC> c_14: it's really buggy though
[20:20:42 CEST] <PlanC> c_14: sometimes when I seek the entire VLC player crashes
[20:20:52 CEST] <furq> 19:19:23 ( m712) ffmpeg and avcodec are not related
[20:20:55 CEST] <furq> yeah this isn't true
[20:21:00 CEST] <m712> are they?
[20:21:12 CEST] <m712> i really don't know, i thought they were different
[20:21:32 CEST] <m712> they're separate packages in debian at least
[20:21:33 CEST] <furq> http://vpaste.net/KIl47
[20:21:46 CEST] <m712> i see
[20:22:17 CEST] <c_14> linkmauve1: might be a bug caused by https://git.videolan.org/?p=ffmpeg.git;a=commit;h=8e89f6fd37357361e0f4db5b6f3b422ce84175b0
[20:22:54 CEST] <furq> typical james darnley
[20:23:29 CEST] <linkmauve1> c_14, likely.
[20:24:30 CEST] <c_14> seems to work on a bunch of fate nodes though
[20:25:20 CEST] <linkmauve1> Hmm, maybe I should git clean and retry.
[20:25:51 CEST] <linkmauve1> But I ran configure again, there should be nothing left there.
[20:27:55 CEST] <linkmauve1> It built after I removed everything and recloned from the local copy.
[20:27:56 CEST] <linkmauve1> Weird.
[20:28:06 CEST] <linkmauve1> Sorry for thinking it was a bug. :)
[20:29:21 CEST] <c_14> probably some state file lying around somewhere
[20:29:50 CEST] <c_14> In unrelated news, when is fatebeta ever going to replace fate
[21:00:01 CEST] <Mavrik> hmm, is there a plugin for ffplay / ffmpeg to render graph of bitrate somewhere?
[21:00:59 CEST] <DHE> you could probably make one with ffprobe and show_packets (less muxing overhead)
[21:01:22 CEST] <m712>  /wi furq
[21:01:37 CEST] <m712> it's not awkward if you don't think it's awkward
[21:06:23 CEST] <Mista-D> is there a table of codecs and supported protocols and muxers please? `ffmpeg -v quiet -h muxer=mpegts` is not showing all supported codecs.
[23:12:01 CEST] <aib> ffmpeg seems to run about 50% slower whenever my CPU usage is above 100% . How can this be? (I have 4 CPU's, so 100% is actually 25%)
[23:25:03 CEST] <iive> aib: first check cooling, if cpu overheats it would start to throttle down.
[23:25:44 CEST] <iive> next, if you have virtual cores, then it is kind of normal... intel calls it HT or HyperThreading.
[23:26:10 CEST] <ppw> your usage can't be over 100%
[23:27:44 CEST] <DHE> multi-threading is a thing. usually percentage is measured in terms of 1 core
[23:27:59 CEST] <aib> here's how weird this is: I have two CPU eaters, a cryptocurrency miner and an increment loop I wrote (https://github.com/aib/bogomips) I execute two programs, let's call them A and B. A is at IDLE I/O priority, has SCHED_IDLE scheduling class and is in a cgroup with cpu.shares set to 16. B has B0 I/O prio, realtime priority (SCHED_RR/99) and has 10240 shares
[23:28:27 CEST] <DHE> SCHED_RR? seriously?
[23:28:36 CEST] <ppw> so you're running this beside ffmpeg and then wonder why it's 50% slower?
[23:29:07 CEST] <aib> now, in any given case, B completely destroys A as would be expected
[23:29:18 CEST] <furq> why are you running a bogomips calculator alongside ffmpeg
[23:29:36 CEST] <aib> the only thing is, if A is bogomips with 3 threads and B is ffmpeg, B finishes in 12 seconds. if A has no threads, B finishes in 6.
[23:29:49 CEST] <aib> furq: debugging. this is absolutely mind-boggling
[23:30:05 CEST] <furq> are you sure about that
[23:30:54 CEST] <furq> this is one of those cases where i can't tell if you're much smarter than me or much stupider
[23:30:58 CEST] <aib> furq: if you have an explanation I'm all ears
[23:31:15 CEST] <furq> i know shit about how cfs actually works but that makes intuitive sense to me
[23:32:05 CEST] <furq> given that you know what SCHED_RR is without googling it i'm going to assume there's a good reason it doesn't make sense to you
[23:33:42 CEST] <aib> that's just an example of how high priority B is, by the way. I've also tried nice -19 and other things. everything is as I would expect (high and low prio A/B share the cpu, they share according to their cpu.shares numbers and everything)
[23:33:46 CEST] <iive> isn't RR a realtime? basically blocking anything else?
[23:33:52 CEST] <aib> iive: yep
[23:33:53 CEST] <ppw> basically
[23:34:03 CEST] <iive> aib: what is your cpu?
[23:34:17 CEST] <aib> 4 cores on AWS
[23:34:30 CEST] <iive> what is aws?
[23:34:33 CEST] <aib> Intel(R) Xeon(R) CPU E5-2666 v3 @ 2.90GHz though there's a good chance it's virtualized
[23:34:35 CEST] <ppw> amazon web service?
[23:34:40 CEST] <aib> Amazon Web Services, yeah
[23:34:55 CEST] <iive> oh dear....
[23:35:04 CEST] <ppw> so the three threads saturate the three virtual cores and ffmpeg gets the one
[23:36:21 CEST] <aib> yes. in that case ffmpeg finishes in 12s.
[23:36:23 CEST] <iive> 10 cores, 20 threads
[23:36:33 CEST] <aib> if bogomips gets the core it scores at around 500.
[23:36:46 CEST] <aib> now, if I kill the 3-thread program, ffmpeg finishes in 6s
[23:36:53 CEST] <aib> but the bogomips score is still the same
[23:37:26 CEST] <ppw> how about running ffmpeg without ever starting the bogomips calculator
[23:37:41 CEST] <ppw> does it saturate all four cores? can you check with top?
[23:37:41 CEST] <aib> ffmpeg running alone is always 6s
[23:37:55 CEST] <aib> one sec
[23:38:39 CEST] <aib> https://paste.pound-python.org/show/oOCNx3cNrGCD0XcD2f5g/
[23:39:33 CEST] <aib> I'm beginning to think this is an artifact of virtualization
[23:39:41 CEST] <kepstin> aib: also if you're running on AWS, make sure to check your steal time stats
[23:39:45 CEST] <iive> aib: do you know how HT works?
[23:39:49 CEST] <aib> the hypervisor switches CPUs on me as soon as I start using SSE instructions or something
[23:40:00 CEST] <aib> iive: I think so
[23:40:26 CEST] <kepstin> iive: yeah, that's the other thing to note, on aws you get hyperthread vcpus, not cores.
[23:41:10 CEST] <aib> note that ffmpeg hardly ever runs out of quanta (or what was it called? its cpu timeslice)
[23:41:11 CEST] <ppw> it's entirely possible that SCHED_RR is interfering with the HT which would otherwise appear transparent.
[23:41:25 CEST] <aib> same case with other priorities
[23:42:55 CEST] <ppw> you could force ffmpeg to use 1 thread / 2 threads etc. see what happens
[23:43:29 CEST] <kepstin> aib: hyperthreading is weird, because if processes are scheduled on both hyperthread siblings on one core, they both appear to be using cpu to the OS, but the cpu is doing timesharing between them internally, so they're only running at about 1/2 the instruction rate.
[23:43:54 CEST] <aib> I mean, I don't have perfect measurement tools (and not much experience printing out raw process stats) but everything seems to be working as documented. shares are divided according to A/(A+B), higher prios are run more, higher scheduling classes completely destroy lower ones, etc.
[23:43:56 CEST] <kepstin> (of course, if there's a lot of waits for ram, then the throughput can appear higher than os-level timesharing)
[23:44:06 CEST] <furq> yeah HT goes a long way to explaining it
[23:44:27 CEST] <aib> ah, hmmm
[23:44:46 CEST] <aib> that would also explain the CPU apparently slowing down by 50% without anything visible changing
[23:44:58 CEST] <aib> such as the number of context switches
[23:45:40 CEST] <furq> 22:40:26 ( kepstin) iive: yeah, that's the other thing to note, on aws you get hyperthread vcpus, not cores.
[23:45:43 CEST] <furq> do they advertise this
[23:45:46 CEST] <furq> because if not then lol
[23:47:07 CEST] <furq> https://aws.amazon.com/ec2/instance-types/
[23:47:10 CEST] <furq> oh it is explicitly mentioned
[23:47:14 CEST] <aib> the reason I didn't pay much heed to HT is that bogomips is a very tight loop. should take one ALU and part of an MMU, not much else
[23:47:15 CEST] <furq> that's a bit less filthy than i thought then
[23:47:28 CEST] <aib> hmm, maybe it's taking half the MMU
[23:47:59 CEST] <aib> okay, okay, that should explain it. hold on, let me try some affinity stuff
[23:52:07 CEST] <kepstin> I don't know exactly how the topology sorts out with the amazon stuff - e.g. on a 2vcpu machine I don't know if you get 2 threads on the same core, 2 different cores, or if it varies
[23:52:26 CEST] <kepstin> I suspect they do 2 threads on the same core.
[23:54:40 CEST] <furq> that would be absolutely mental if they didn't
[23:55:16 CEST] <kepstin> it would mean really variable performance depending on other vms on the same host, yeah. which would be kinda crazy.
[23:55:26 CEST] <kbarry> I'm wanting to record my playback experience on my windows 10 box for my developers to hear "whatr i hear"
[23:55:43 CEST] <kbarry> like, piping vlc into ffmpeg
[23:56:03 CEST] <furq> actually i guess they have nodes with one vcpu
[23:56:09 CEST] <furq> so it is entirely likely ;_;
[23:56:44 CEST] <furq> a lot of aws users aren't exactly the brightest bulbs, so they could probably get away with it
[23:57:44 CEST] <aib> okay, so with a single thread bogomips running in CPU #1:
[23:58:09 CEST] <aib> ffmpeg running on #2 and #4 works fine. so does ffmpeg running on #1, with all the priority difference
[23:58:21 CEST] <aib> now ffmpeg on #3 runs 50% slower. so it *is* HT.
[23:59:21 CEST] <furq> it's slightly worrying that only one core is affected
[00:00:00 CEST] --- Wed May 31 2017