[FFmpeg-devel] [PATCH] H264: Fix deblocking for first row in fields

[Jeff Downs]

H264 PAFF implementation is missing a change to filter_mb_fast to 
check for deblocking on the first row of a bottom field.  Without this 
check, filter_mb_fast might be called for the first row, which (from what 
I understand in a read of the existing code) is not appropriate.  In 
particular, a run through valgrind will show that the access to 
qscale_table[h->top_mb_xy] goes out of bounds (top_mb_xy is negative for 
first row).

This augments the existing check to look for row 1 in bottom fields.

Benchmarks here with non-interlaced content showed no real speed impact 
(overhead of the check), which is surprising to me; if anyone else wants 
to check that, please feel free.

Old:
User: avg: 6.222  stddev: 0.007  med: 6.222

New:
User: avg: 6.212  stddev: 0.007  med: 6.216

	-Jeff
-------------- next part --------------
Index: libavcodec/h264.c
===================================================================
--- libavcodec/h264.c	(revision 10937)
+++ libavcodec/h264.c	(working copy)
@@ -6371,12 +6371,13 @@
 
 static void filter_mb_fast( H264Context *h, int mb_x, int mb_y, uint8_t *img_y, uint8_t *img_cb, uint8_t *img_cr, unsigned int linesize, unsigned int uvlinesize) {
     MpegEncContext * const s = &h->s;
+    int mb_y_firstrow = s->picture_structure == PICT_BOTTOM_FIELD;
     int mb_xy, mb_type;
     int qp, qp0, qp1, qpc, qpc0, qpc1, qp_thresh;
 
     mb_xy = mb_x + mb_y*s->mb_stride;
 
-    if(mb_x==0 || mb_y==0 || !s->dsp.h264_loop_filter_strength || h->pps.chroma_qp_diff ||
+    if(mb_x==0 || mb_y==mb_y_firstrow || !s->dsp.h264_loop_filter_strength || h->pps.chroma_qp_diff ||
        (h->deblocking_filter == 2 && (h->slice_table[mb_xy] != h->slice_table[h->top_mb_xy] ||
                                       h->slice_table[mb_xy] != h->slice_table[mb_xy - 1]))) {
         filter_mb(h, mb_x, mb_y, img_y, img_cb, img_cr, linesize, uvlinesize);