[FFmpeg-devel] libx264.c:encode_nals can overwrite buffers

[Paul Flinders]

Despite being passed in the buffer size libx264.c:encode_nals (and 
libxavs:encode_nals) make no checks and assume that the buffer is large 
enough to hold the output data and will overflow silently if it isn't.

This shows up with current versions of kdenlive - trying to use their 
"lossless fast" preset causes a crash in the mlt command line renderer. 
Ultimately this is because mlt allocates too small a buffer and the 
lossless encoder can generate quite large frames but ffmpeg should 
really be able to catch this and generate an error, rather than just 
causing a crash.

The following patch adds the necessary checks.


--- libx264.c    2010-12-31 22:28:07.000000000 +0000
+++ libx264.c~    2010-12-26 16:32:46.000000000 +0000
@@ -60,13 +60,8 @@

      /* Write the SEI as part of the first frame. */
      if (x4->sei_size > 0 && nnal > 0) {
-    if (size < x4->sei_size){
-        av_log(ctx, AV_LOG_ERROR, "supplied buffer too small\n");
-        return 0;
-    }
          memcpy(p, x4->sei, x4->sei_size);
          p += x4->sei_size;
-    size -=  x4->sei_size;
          x4->sei_size = 0;
      }

@@ -78,13 +73,8 @@
              memcpy(x4->sei, nals[i].p_payload, nals[i].i_payload);
              continue;
          }
-    if (size < nals[i].i_payload){
-        av_log(ctx, AV_LOG_ERROR, "supplied buffer too small\n");
-        return p - buf;
-    }
          memcpy(p, nals[i].p_payload, nals[i].i_payload);
          p += nals[i].i_payload;
-    size -= nals[i].i_payload;
      }

      return p - buf;

[Previously noted Issue1586 BTW]