[FFmpeg-devel] [WIP] XComposite window capture demuxer (Linux)

[Lynne]

May 19, 2020, 11:25 by ema at fastwebnet.it:

> Hi Lynne,
>
> Thanks for the feedback.
> Some more discussion points below.
>
>> There already is a zero-overhead capture on linux - kmsgrab. It works on AMD and Intel.
>>
>
> Does it work on Nvidia too? Does it have smooth capture?
> Does it work for 3d applications?
>

Of course it does. Did you not bother to read the manual or wiki page? Why wouldn't it work for 3D?
It does not work on Nvidia, and you can only blame them.



>>> Wouldn't a hwaccel frame imply no choice for encoder (as in now we get a true uncompressed RGBA buffer which can be encoded as we see fit - if we were to get hwacell wouldn't we be forced to use the encoded data as is)?
>>>
>>
>> No.
>>
>
> My point was around if you use say NVenc to compress frames, you wouldn't want to decompress-recompress to avoid artefacts?
>

I have zero idea what you're talking about. Hardware frames are just like software frames, but
they live in VRAM.



>> I wouldn't be surprised if the xlib code has some PTS issues and schedules downloading a frame late.
>>
>
> I'm far from being an expert, but I would be surprised if that had issues considering it's the main Linux screen capture we have?
> The xcb calls to grab a frame are quite simple/straightforward (both shm and non shm versions).
>

The avdevice polls to get new frames. If it polls too late, it will miss a frame. I think that's where a possible issue is.



>> Maybe its worth fixing that instead of adding a yet another x11 capture?
>>
>
> This is a xcomposite capture, I would say it's different than a x11 raw capture.
>

It captures the same windowing system.
We do not want to end up with 2 capture systems that do the same thing. How would users know
which one to use?



>> I'm against any OpenGL code being ran anywhere in our libraries since it will disturb OpenGL's global
>> state, breaking any OpenGL users of our libraries (there are quite a lot). Yes, there are also people who
>> want to screencapture and use OpenGL. I'm one.
>> This is also why we don't have OpenGL filtering in libavfilter.
>> So that's a big NAK for me.
>>
>
> Fair enough - Understood why it would be a NAK from you.
> Would there be a way to mark this as experimental?
>

We don't have any way to mark devices as experimental or opengl.
All we have are subjective opinions that this is better than the current capture system,
which isn't very convincing its worth merging problematic code and adding new API to signal it.



> Out of curiosity if I was proposing/using the native NVidia proprietary library for screen capture (linked at runtime dynamically), would that be more acceptable in terms of conflicts (I wouldn't like it because it wouldn't support AMD/Intel based hardware)?
>

I'd be more okay with having the native Nvidia library for when kmsgrab doesn't work.



> I wrote this because I wanted to capture 3d apps at 60 FPS with high quality without having to drop frames.
> If there is a better way with ffmpeg/libav*, please feel free to suggest - otherwise currently folks would be forced to use OBS and not use ffmpeg/libav* based solutions.
>

kmsgrab is by far the fastest capture possible. Use it if performance isn't good enough.



> The video games segment under Linux is expanding, would ffmpeg/libav* have the right infra/codecs to cater for it?
> Can we capture true 60 FPS of a 3d app today? Do we have native Vulkan/OpenGL capturers?
>

There is no native Vulkan/OpenGL capture. No such thing exists. You can't just spy on other programs.
All captures on all platforms are managed by the windowing system.