[FFmpeg-devel] [PATCH] Fix segfault in x11grab when drawing Cursor on Xservers that don't support the XFixes extension

[Michael Niedermayer]

On Fri, Jun 25, 2010 at 09:12:50AM +1000, Brenden Bain wrote:
> 2010/6/24 M?ns Rullg?rd <mans at mansr.com>:
> > Brenden Bain <brenden.bain at gmail.com> writes:
> >
> >> Index: libavdevice/x11grab.c
> >> ===================================================================
> >> --- libavdevice/x11grab.c ? ? (revision 23755)
> >> +++ libavdevice/x11grab.c ? ? (working copy)
> >> @@ -91,7 +91,7 @@
> >> ? ? ?XImage *image;
> >> ? ? ?int x_off = 0;
> >> ? ? ?int y_off = 0;
> >> - ? ?int use_shm;
> >> + ? ?int use_shm, ignore;
> >> ? ? ?char *param, *offset;
> >>
> >> ? ? ?param = av_strdup(s1->filename);
> >> @@ -115,6 +115,11 @@
> >> ? ? ? ? ?return AVERROR(EIO);
> >> ? ? ?}
> >>
> >> + ? ?if (!XFixesQueryExtension(dpy, &ignore, &ignore)) {
> >> + ? ? ? ?av_log(s1, AV_LOG_INFO, "Disabling cursor recording. Unable to query cursor shape.\n");
> >> + ? ? ? ?x11grab->nomouse = 1;
> >> + ? ?}
> >> +
> >> ? ? ?st = av_new_stream(s1, 0);
> >> ? ? ?if (!st) {
> >> ? ? ? ? ?return AVERROR(ENOMEM);
> >> @@ -245,8 +250,9 @@
> >> ? * ? ? ? ? ?coordinates
> >> ? * @param x Mouse pointer coordinate
> >> ? * @param y Mouse pointer coordinate
> >> + * @return 0 if an error occurred, 1 otherwise.
> >
> > Usual convention is 0 for success, negative for error.
> 
> Okay. Changed though this file seemed to use both in the past.
> 
> >> ? */
> >> -static void
> >> +static int
> >> ?paint_mouse_pointer(XImage *image, struct x11_grab *s)
> >> ?{
> >> ? ? ?int x_off = s->x_off;
> >> @@ -262,6 +268,10 @@
> >>
> >> ? ? ?xcim = XFixesGetCursorImage(dpy);
> >>
> >> + ? ?if (xcim == NULL) {
> >> + ? ? ? ?return 0;
> >> + ? ?}
> >> +
> >> ? ? ?x = xcim->x - xcim->xhot;
> >> ? ? ?y = xcim->y - xcim->yhot;
> >>
> >> @@ -284,6 +294,7 @@
> >>
> >> ? ? ?XFree(xcim);
> >> ? ? ?xcim = NULL;
> >> + ? ?return 1;
> >> ?}
> >>
> >>
> >> @@ -388,7 +399,11 @@
> >> ? ? ?}
> >>
> >> ? ? ?if(!s->nomouse){
> >> - ? ? ? ?paint_mouse_pointer(image, s);
> >> + ? ? ? ?if (!paint_mouse_pointer(image, s)) {
> >> + ? ? ? ? ? ?/* An error occured and will likely occur again. Just disable mouse capture from now on */
> >
> > That comment is superfluous. ?Anyone with half a brain can see pointer
> > capture is being turned off.
> 
> Done.
> 
> Brenden.

>  x11grab.c |   20 +++++++++++++++++---
>  1 file changed, 17 insertions(+), 3 deletions(-)
> 232ef00cd548c0a8ab48d48ec883d687b0e46eff  x11_segfault_fix.patch
> Index: libavdevice/x11grab.c
> ===================================================================
> --- libavdevice/x11grab.c	(revision 23763)
> +++ libavdevice/x11grab.c	(working copy)
> @@ -91,7 +91,7 @@
>      XImage *image;
>      int x_off = 0;
>      int y_off = 0;
> -    int use_shm;
> +    int use_shm, ignore;
>      char *param, *offset;
>  
>      param = av_strdup(s1->filename);
> @@ -115,6 +115,11 @@
>          return AVERROR(EIO);
>      }
>  
> +    if (!XFixesQueryExtension(dpy, &ignore, &ignore)) {
> +        av_log(s1, AV_LOG_INFO, "Disabling cursor recording. Unable to query cursor shape.\n");
> +        x11grab->nomouse = 1;
> +    }
> +
>      st = av_new_stream(s1, 0);
>      if (!st) {
>          return AVERROR(ENOMEM);
> @@ -245,8 +250,9 @@
>   *          coordinates
>   * @param x Mouse pointer coordinate
>   * @param y Mouse pointer coordinate
> + * @return !0 if error, 0 if successful

negative values generally means errors and in form of error
codes where a existing one make sense, otherwise -1 is a nice choice


[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

Awnsering whenever a program halts or runs forever is
On a turing machine, in general impossible (turings halting problem).
On any real computer, always possible as a real computer has a finite number
of states N, and will either halt in less than N cycles or never halt.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
URL: <http://lists.mplayerhq.hu/pipermail/ffmpeg-devel/attachments/20100702/8c4871ef/attachment.pgp>