[FFmpeg-devel] [WIP] XComposite window capture demuxer (Linux)

[Emanuele Oriani]

Hi Marton/all,

I've re-uploaded the firefox send link (expired) with some sample 
capture at 60 FPS:

https://send.firefox.com/download/51b45decae720c08/#r5o4J2SgCJZndRdMOxxRBg

Please note I have integrated usage of OpenGL's Pixel Buffer Objects - 
now the memory management of the captured framed can be managed by the 
drivers and hopefully should be DMA and less memory hops.
This new code works on both Nvidia and AMD.

CPU usage is still higher than x11grab (similar levels as below).

Let me know your thoughts.

Thanks,
Emanuele

Ps. quick link to demuxer: 
https://github.com/Emanem/replayer/blob/master/src/xcompgrab.c

On 08/05/2020 11:43, Emanuele Oriani wrote:
> Hi Marton,
> 
> Capturing at 60 FPS 1920x1080, the difference is even more noticeable: 
> x11grab is choppy, xcompgrab is smooth (this time I used H264 ultrafast 
> to encode).
> 
> https://send.firefox.com/download/17001ef60837a5ec/#FaQTa-dP4MB28YfPNMXxuw
> 
> CPU performance:
> 
>               real    user   sys
> - xcompgrab: 10.32s, 18.2s, 0.38s
> - x11grab:   10.20s, 11.6s, 0.22s
> 
> Definitely much more CPU usage, but the output quality can't be compared.
> Please note no frames were lost/skipped, everything was processed on 
> time for both tests.
> 
> Thanks,
> Emanuele
> 
> On 08/05/2020 11:05, Emanuele Oriani wrote:
>> Hi Marton,
>>
>> TL;DR
>> xcompgrab uses more CPU but produces much better streams than x11grab.
>>
>> I have the following CPU usage performance report (on my i7-8700k, 
>> Nvidia 2080 Ti RTX, governor set to 'performance', on Ubuntu 18.04 
>> using Gnome shell).
>> The capture has been a surface area of  1720x1376 for 10 seconds at 30 
>> FPS, only getting the frames, not trying to encode them into a file:
>>
>>               real    user   sys
>> - xcompgrab: 10.13s, 2.06s, 0.085s
>> - x11grab:   10.05s, 0.20s, 0.035s
>>
>> from a first outlook x11grab is much more CPU efficient, almost 90% 
>> less CPU usage, but upon starting to record the video, it feels like 
>> it's 'choppier' and loses frames.
>>
>> Below videos have been captured at 1920x1080, the encoder is exactly 
>> the same with same settings and the total CPU time is greater for 
>> xcompgrab (as expected).
>>
>> But if you look at both, the xcompgrab doesn't lose frames/is not 
>> choppier, instead x11grab seems to be much worse.
>>
>> I've uploaded both 10 secs segments at:
>> https://send.firefox.com/download/0cbf01f9126519c7/#DPKzIfjemAFm263ZSdMOBw 
>>
>>
>> The code used to grab and encode is https://github.com/Emanem/replayer 
>> again very simple, one thread gets the packets/frames, the other picks 
>> the frames and encodes them.
>>
>> Performance of this run has been:
>>
>>               real    user   sys
>> - xcompgrab: 10.42s, 12.5s, 0.22s
>> - x11grab:   10.21s, 8.90s, 0.31s
>>
>> As expected x11grab yields better CPU usage but the quality of output 
>> is somehow lacking.
>>
>> Let me know your thoughts.
>>
>> Thanks,
>> Emanuele
>>
>> On 07/05/2020 19:52, Marton Balint wrote:
>>>
>>>
>>> On Thu, 7 May 2020, Emanuele Oriani wrote:
>>>
>>>> Hi FFMPEG devel,
>>>>
>>>> I have been writing a simple XComposite window capture demuxer, 
>>>> heavily inspired from x11grab sources and OBS Window capture 
>>>> logic/code.
>>>
>>> Have you compared performance to x11grab for various resolutions and 
>>> frame rates? Do you have some numbers?
>>>
>>> Thanks,
>>> Marton
>>>
>>>>
>>>> I would like to give back to the community and share the sources; 
>>>> before I would formally submit for a review/patch, I would really 
>>>> appreciate if someone could have a high level pass at the code (it's 
>>>> relatively simple actually) and point out any major issues/problems/...
>>>>
>>>> The code is available as part of a simple test application on 
>>>> github, at:
>>>>
>>>> https://github.com/Emanem/replayer/blob/master/src/xcompgrab.c
>>>>
>>>> Being the author of the code I would be happy to re-license it to be 
>>>> used in ffmpeg/libav* mainline once I get the OK from you.
>>>> In order to compile/execute this code, one needs the following dev 
>>>> packages:
>>>>
>>>> libav*-dev, libx11-dev, libxcomposite-dev, OpenGL headers/libs.
>>>>
>>>> Libraries to be linked are:
>>>>
>>>> -lavcodec -lavformat -lavdevice -lavutil -lswscale -lX11 
>>>> -lXcomposite -lGL
>>>>
>>>> I've tested it and it's working fine on both AMD and Nvidia hardware.
>>>>
>>>> Let me know if this is of interest, otherwise no worries and thanks 
>>>> again for the fantastic library/software.
>>>>
>>>> Emanuele
>>>>
>>>> _______________________________________________
>>>> ffmpeg-devel mailing list
>>>> ffmpeg-devel at ffmpeg.org
>>>> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>>>>
>>>> To unsubscribe, visit link above, or email
>>>> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".
>>> _______________________________________________
>>> ffmpeg-devel mailing list
>>> ffmpeg-devel at ffmpeg.org
>>> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>>>
>>> To unsubscribe, visit link above, or email
>>> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".
>> _______________________________________________
>> ffmpeg-devel mailing list
>> ffmpeg-devel at ffmpeg.org
>> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>>
>> To unsubscribe, visit link above, or email
>> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".