[FFmpeg-devel] [PATCH 08/12] avformat/avc, hevc: Check the allocations implicit in dynamic buffers

[Andreas Rheinhardt]

Dynamic buffers involve implicit allocations that are currently usually
unchecked; ff_avc_parse_nal_units_buf() and ff_hevc_annexb2mp4buf() are
no exceptions to this. So add checks for them.

Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt at gmail.com>
---
One might argue that this should also check for whether the size returned
by ff_hevc_annexb2mp4() resp. ff_avc_parse_nal_units() equals the size
of the buffer (dynamic buffers have an implicit INT_MAX/2 allocation
limit and if it is hit, no further writes are performed, but the already
written data is not discarded); given that I prefer to drop this limit
(and replace it by INT_MAX) later I have not added such a check here.

(In case of an unchecked allocation failure in which the returned buffer
is NULL, the returned size is -AV_INPUT_BUFFER_PADDING_SIZE. This is
something that put_ebml_num() (the int is converted to uint64_t for it)
doesn't like at all (it should assert, yet it actually runs into an
infinite loop in ebml_num_size()).)

 libavformat/avc.c  | 3 +++
 libavformat/hevc.c | 2 ++
 2 files changed, 5 insertions(+)

diff --git a/libavformat/avc.c b/libavformat/avc.c
index cd15ac3cdb..aef5d3c35d 100644
--- a/libavformat/avc.c
+++ b/libavformat/avc.c
@@ -102,6 +102,9 @@ int ff_avc_parse_nal_units_buf(const uint8_t *buf_in, uint8_t **buf, int *size)
     ff_avc_parse_nal_units(pb, buf_in, *size);
 
     *size = avio_close_dyn_buf(pb, buf);
+    if (!*buf)
+        return AVERROR(ENOMEM);
+
     return 0;
 }
 
diff --git a/libavformat/hevc.c b/libavformat/hevc.c
index f621cb2f19..a4e53bc4ab 100644
--- a/libavformat/hevc.c
+++ b/libavformat/hevc.c
@@ -1061,6 +1061,8 @@ int ff_hevc_annexb2mp4_buf(const uint8_t *buf_in, uint8_t **buf_out,
     }
 
     *size = avio_close_dyn_buf(pb, buf_out);
+    if (!*buf_out)
+        return AVERROR(ENOMEM);
 
     return 0;
 }
-- 
2.20.1