[FFmpeg-devel] [PATCH] Never unmap unallocated space

[Tobias Stoeckmann]

The function av_file_map can use mmap to memory-map the content of a
file into address space. Just like every other alternative, mmap() could
fail due to various reasons, e.g. if not enough address space is
available.

Unfortunately, av_file_map writes the requested size even on error into
the supplied size pointer, which will eventually passed to munmap.

As the base pointer will be NULL, the first 'size' bytes of the address
space will be unmapped, which will most likely hit a lot of libraries.
If that happens, a segmentation fault is most likely to occur.

This example can trigger the issue on 32 bit systems. Adjust the seek
value if necessary (free memory < seek_size < physical RAM installed):

$ uname -m
i686
$ dd if=/dev/zero of=large bs=1 count=1 seek=3000000000
$ ffmpeg -f lavfi -i life=filename=large
Error occurred in mmap(): Cannot allocate memory
Error initializing filter 'life' with args 'filename=large'
Segmentation fault
$ _

---
I chose to set *size back to 0 in error-cases just because each error
case already handles close(fd), too. Adding a new variable and setting
*size at the end would have introduced too much noise in this diff.

Feel free to adjust. :)
---
 libavutil/file.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/libavutil/file.c b/libavutil/file.c
index 7bdf6cde84..8e12efe3ce 100644
--- a/libavutil/file.c
+++ b/libavutil/file.c
@@ -87,6 +87,7 @@ int av_file_map(const char *filename, uint8_t **bufptr, size_t *size,
         err = AVERROR(errno);
         av_strerror(err, errbuf, sizeof(errbuf));
         av_log(&file_log_ctx, AV_LOG_ERROR, "Error occurred in mmap(): %s\n", errbuf);
+        *size = 0;
         close(fd);
         return err;
     }
@@ -98,6 +99,7 @@ int av_file_map(const char *filename, uint8_t **bufptr, size_t *size,
         mh = CreateFileMapping(fh, NULL, PAGE_READONLY, 0, 0, NULL);
         if (!mh) {
             av_log(&file_log_ctx, AV_LOG_ERROR, "Error occurred in CreateFileMapping()\n");
+            *size = 0;
             close(fd);
             return -1;
         }
@@ -106,6 +108,7 @@ int av_file_map(const char *filename, uint8_t **bufptr, size_t *size,
         CloseHandle(mh);
         if (!ptr) {
             av_log(&file_log_ctx, AV_LOG_ERROR, "Error occurred in MapViewOfFile()\n");
+            *size = 0;
             close(fd);
             return -1;
         }
@@ -116,6 +119,7 @@ int av_file_map(const char *filename, uint8_t **bufptr, size_t *size,
     *bufptr = av_malloc(*size);
     if (!*bufptr) {
         av_log(&file_log_ctx, AV_LOG_ERROR, "Memory allocation error occurred\n");
+        *size = 0;
         close(fd);
         return AVERROR(ENOMEM);
     }
-- 
2.11.0