[FFmpeg-devel] av_fopen_utf8 and cross-DLL CRT object sharing issue on Windows

[Martin Storsjö]

Hi,

I just became aware of the av_fopen_utf8 function - which was introduced 
to fix path name translations on Windows - actually has a notable design 
flaw.


Background:

On Windows, a process can contain more than one C runtime (CRT); the 
system comes with two shared ones (UCRT and msvcrt.dll) and in MSVC 
builds, each DLL/EXE can have one statically linked in instead of linking 
against a shared library CRT (and that's actually the default 
configuration when building with MSVC).

This means that CRT objects (file descriptors from open(), FILE* opened 
with fopen/fdopen) mustn't be shared across DLLs; such an object must be 
opened, accessed and closed within the same DLL.


The issue:

This was fixed for the avpriv_open() function in 
e743e7ae6ee7e535c4394bec6fe6650d2b0dbf65 by duplicating the file_open.c 
source file in all libav* libraries (in MSVC builds, and renaming 
avpriv_open to ff_open), so that each library gets their own copy of 
ff_open (which isn't exported), so that all calls to open/read/write/close 
are done within the same DLL.

When av_fopen_utf8 was added afterwards, in 
85cabf1ca98fcc502fcf5b8d6bfb6d8061c2caef, this issue wasn't taken into 
account - although the issue is somewhat eased by lucky coincidence.

As av_fopen_utf8 is implemented in the same source file, 
libavutil/file_open.c, which gets duplicated in all libraries that use it, 
all uses of the function in other libraries (such as libavformat) actually 
end up using their own copy of it. (This also means that all the libav* 
DLLs export this function.) But for any users of the function outside of 
the ffmpeg libraries, this function (in libavutil, or whichever library it 
ends up imported from) returns a FILE* allocated by libavutil's CRT, which 
then can't be used with the fread/fwrite/fclose/whatever functions in the 
DLL/EXE that called it.

One concrete example of this is that the function is used for the twopass 
log file in fftools/ffmpeg_opt.c. To see the issue in action, build ffmpeg 
with MSVC with this config:

     ../ffmpeg/configure --enable-shared --toolchain=msvc --prefix=<dest>

Then try to do a twopass encoding with it:

     ffmpeg -i <input> -an -c:v ffv1 -pass 1 -f null -

     ffmpeg -i <input> -an -c:v ffv1 -pass 2 -y test-2pass.mkv

The same issue would appear anywhere this function is used from libavutil 
built as a DLL, from a caller built with a different CRT choice (e.g. 
often if mixing mingw/MSVC builds, which otherwise is supported just 
fine). (If built with a shared CRT, i.e. configured with 
--extra-cflags=-MD, it does work though.)


As this is a public function, we can't really do many tricks like we do 
within the libraries. (On the other hand, while it is a public function, 
it doesn't seem to be used much outside of ffmpeg, other than in ffmpeg 
API bindings for other languages.)

I guess the only really robust solution would be to turn av_fopen_utf8 
into a static inline function within the headers, essentially inlineing a 
copy of wchar_filename.h there, so that it expands to a call to the 
callers' _wfopen or similar. But that would end up polluting users' code 
by implicitly including windows.h everywhere, which really isn't nice to 
do either.

Or should we just document this issue, discourage further external use of 
the function, and fold this function into a ffmpeg-internal inline helper 
like libavutil/whcar_filename.h?


// Martin