[FFmpeg-cvslog] flacenc: ensure the order is within the min/max range in LPC order search

[Justin Ruggles]

ffmpeg | branch: release/0.7 | Justin Ruggles <justin.ruggles at gmail.com> | Wed Nov  7 14:48:28 2012 -0500| [3d0c9c9af687fd2dd4fbaefdb4d7d85c59f7d19f] | committer: Reinhard Tartler

flacenc: ensure the order is within the min/max range in LPC order search

This fixes use of uninitialized values when the FLAC encoder uses the
2-level, 4-level, and 8-level search methods. Fixes failure of the
fate-flac-24-comp-8 test when run using valgrind.
(cherry picked from commit 3a2731cbd31d0c5681ddbc7c78edd5c53c4d0032)

Conflicts:

	libavcodec/flacenc.c

Signed-off-by: Reinhard Tartler <siretart at tauware.de>

> http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=3d0c9c9af687fd2dd4fbaefdb4d7d85c59f7d19f
---

 libavcodec/flacenc.c |    8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/libavcodec/flacenc.c b/libavcodec/flacenc.c
index be775ca..1c10d2b 100644
--- a/libavcodec/flacenc.c
+++ b/libavcodec/flacenc.c
@@ -915,14 +915,16 @@ static int encode_residual_ch(FlacEncodeContext *s, int ch)
         omethod == ORDER_METHOD_8LEVEL) {
         int levels = 1 << omethod;
         uint32_t bits[1 << ORDER_METHOD_8LEVEL];
-        int order;
+        int order       = -1;
         int opt_index   = levels-1;
         opt_order       = max_order-1;
         bits[opt_index] = UINT32_MAX;
         for (i = levels-1; i >= 0; i--) {
+            int last_order = order;
             order = min_order + (((max_order-min_order+1) * (i+1)) / levels)-1;
-            if (order < 0)
-                order = 0;
+            order = av_clip(order, min_order - 1, max_order - 1);
+            if (order == last_order)
+                continue;
             encode_residual_lpc(res, smp, n, order+1, coefs[order], shift[order]);
             bits[i] = find_subframe_rice_params(s, sub, order+1);
             if (bits[i] < bits[opt_index]) {