pix format for vdpau hwaccel is AV_PIX_FMT_VDPAU. whithout this entry it is not used for mpeg1/2. I noticed this with version 1.2 which we have in use for XBMC. pull request: https://github.com/FFmpeg/FFmpeg/pull/17