[Ffmpeg-devel] h264: fix removing NAL unit's trailing zero bytes

[Reinhard Nissl]

Hi,

the current code doesn't remove a trailing zero byte when dst_length is 1.

Consider the following byte sequence

    00 00 01 0a 00 00 00 01 09 ...
               ^  ^
               A  B

decode_nal() determines dst_length to be 1 (i. e. the byte between label
A and B above). Though, this byte is a trailing zero byte as the spec
says the the current NAL unit is terminated by a byte sequence 00 00 00.

Therefore the current code uses a loop to decrement dst_length
accordingly. But the loop doesn't start as the loop condition checks for
dst_length > 1, which should read dst_length > 0. The attached patch
fixes this.

As a result of the fix, dst_length can now get 0 and therefore the
calculation of bit_length may result in negative values, not to mention
the access to ptr[-1] as argument to decode_rbsp_trailing().

Regarding the access to ptr[-1]: it is save memory, i. e. the byte
before label A in the sequence above.

Regarding bit_length to be negative: the next use of this variable is
the call to init_get_bits() and there, a negative value is handled already.

Bye.
-- 
Dipl.-Inform. (FH) Reinhard Nissl
mailto:rnissl at gmx.de
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ffmpeg-h264-fix-trailing-zero-bytes.patch
Type: text/x-patch
Size: 465 bytes
Desc: not available
URL: <http://lists.mplayerhq.hu/pipermail/ffmpeg-devel/attachments/20070409/faa01247/attachment.bin>