[FFmpeg-devel] [WIP] XComposite window capture demuxer (Linux)

[Emanuele Oriani]

Hi Marton,

Thanks very much for the feedback; below answers to your points - let me 
know further feedback if any.

 > And sorry, I cannot say how useful this would be, maybe now is the time
 > for people to speak up if somebody is particularly against adding this
 > for any reason.

I haven't been able to capture video-games/3d apps running in full 
screen with x11grab, and when running in windowed mode the capture was 
sub optimal in terms of quality (lost frames/choppy/etc etc).
Unless we have better solutions with ffmpeg/libav* (which I'm not aware 
of :) this xcompgrab would target such audience (smooth capture alas 
more CPU usage).
But again, if there's already a better capture, then this has been an 
academic exercise :)

 > - Is there a way to keep the captured textures as an
 > OpenGL/VAAPI/NVenc/Vulkan object, so the frames can work as some kind of
 > hwaccel frames? Can this improve performance?

I think there would be. I would need to find more in terms of 
documentation for both hwaccel frames structures/management.
Please feel free to point me to guides/code/samples etc etc.

Wouldn't a hwaccel frame imply no choice for encoder (as in now we get a 
true uncompressed RGBA buffer which can be encoded as we see fit - if we 
were to get hwacell wouldn't we be forced to use the encoded data as is)?

 > - What can be the reason between the quality/smoothness difference of
 > x11grab and the Compositor capturer?

My suspicion is that OpenGL has access to the buffered data on the 
videocard itself, hence able to get smooth frames.

Related to both these questions, I've asked the same on Stackoverflow:

https://stackoverflow.com/questions/61613441/is-there-a-better-more-efficient-way-to-capture-composite-x-windows-in-linux

Unfortunately no one has been able to give me an answer (I did set a 
bounty for it) so I posted my own understanding.

 > As for the code, I only took a quick look, here are some comments
 >
 > - probably GCC atomics should be replaced by <stdatomic.h> atomic
 > functions.

Done.

 > - Unless there is a reason, I think av_malloc/av_free is preferred for
 > memory alloc functions.

Done.

 > - Maybe some openGL glue can be shared with libavdevice/opengl_enc.c?
 > Take a look, I am not sure, because that was implemented with SDL and
 > cross platform capabilities in mind, but since your capture device is
 > only for linux (or not?), then maybe it makes more sense to keep it
 > separete?

This is indeed a very specific implementation for Linux, I would agree 
that perhaps not sure this makes sense to use shared OpenGL functions?

 > - Can the Xlib functions replaced with xcb? I kind of think that would
 > be preferable if it is a sensible thing to do.

I understand where you're coming from and it makes sense to give it a 
shot - my only fear is that we may need to use specific Xlib functions 
anyway.
But let me give it a try and see where we go.

On 18/05/2020 20:11, Marton Balint wrote:
> 
> 
> On Mon, 18 May 2020, Emanuele Oriani wrote:
> 
>> Hi Marton/ffmpeg devs/all,
>>
>> Haven't seen any response to the proposed XComposite windows capture.
>>
>> As per below the reasoning for adding this code to libav*/ffmpeg is 
>> that seems to be better in quality than x11grab, although it uses more 
>> CPU in its current implementation (uses OpenGL and PBO - optional - to 
>> fetch XComposite windows).
>>
>> Please let me know if it's still of interest and/or you have further 
>> technical feedback.
>> I've posted some performance tests in the thread below.
>>
>> Here links of two captures (one with x11grab, the other with 
>> xcompgrab) and the sources:
>>
>> https://send.firefox.com/download/df859689f4cc6429/#EN6w9upyIMHqO7n_1hMTaQ 
>>
>> https://github.com/Emanem/replayer/blob/master/src/xcompgrab.c
>>
>> Thanks in advance for your time and feedback,
> 
> Thanks. I am not familiar with the details of opengl/compositing 
> capturing, but here are a few questions coming into my mind. Not only 
> for you, but for other developers as well:
> 
> - Is there a way to keep the captured textures as an 
> OpenGL/VAAPI/NVenc/Vulkan object, so the frames can work as some kind of 
> hwaccel frames? Can this improve performance?
> 
> - What can be the reason between the quality/smoothness difference of 
> x11grab and the Compositor capturer?
> 
> As for the code, I only took a quick look, here are some comments
> 
> - probably GCC atomics should be replaced by <stdatomic.h> atomic 
> functions.
> 
> - Unless there is a reason, I think av_malloc/av_free is preferred for 
> memory alloc functions.
> 
> - Maybe some openGL glue can be shared with libavdevice/opengl_enc.c? 
> Take a look, I am not sure, because that was implemented with SDL and 
> cross platform capabilities in mind, but since your capture device is 
> only for linux (or not?), then maybe it makes more sense to keep it 
> separete?
> 
> - Can the Xlib functions replaced with xcb? I kind of think that would 
> be preferable if it is a sensible thing to do.
> 
> And sorry, I cannot say how useful this would be, maybe now is the time 
> for people to speak up if somebody is particularly against adding this 
> for any reason.
> 
> Regards,
> Marton
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".