[FFmpeg-devel] [PATCH] configure: speedup x2-x8

[avih]

Hi,

I noticed that configure can be a bit slow - few minutes on my macOS
system, 1:30m on linux/bash, 30s on linux/dash, and even if
pathological, still a too-long 10 minutes on Windows (MSYS2) (same
duration as a full build with make -j4 after configure).

I added some timing information printouts for various parts in
configure, and eventually identified few culprits.

The attachment "config-timing.patch" adds these printouts and sorted
summary. It's attached for reference and not intended to be merged.
It applies cleanly before or after the main patch.


The attached "main.patch" addresses three areas I identified as slow:

1. About 50-70% of configure runtime was being spent inside one
   function: flatten_extralibs() and callees resolve() and unique().
   It manipulates strings and invoked nearly 20K (20000) subshells.
   It was rewritten to avoid subshells, and ended up x50-x250 faster.

2. print_enabled_components() was invoking sed about 350 times on one
   file. This is never instant but takes many seconds where fork is
   slow (Windows). Invoke sed only once instead = x4-x10 speedup.

3. After the previous speedups, configure spent 20-60% of its runtime
   at check_deps(). It's particularly slow with bash. After some local
   optimizations - mainly avoid pushvar/popvar and abort early in one
   notable case (empty deps), it's now x4-x25 faster.


Some resulting speedups (more details at "before-after.txt"):

- macOS/bash:  total: 98s -> 22s    flatten_extralibs:  53s -> 0.7s
- Linux/bash:  total: 87s -> 11s    flatten_extralibs:  59s -> 0.9s
- Linux/dash:  total: 27s ->  8s    flatten_extralibs:  17s -> 0.2s
- FreeBSD/sh:  total: 34s ->  9s    flatten_extralibs:  23s -> 0.2s
- MSYS2/bash:  total: 10m -> 2:30m  flatten_extralibs: 400s -> 1.5s (!)


Notes:

- unique() is modified with flatten_extralibs. It now outputs different
  order: it was keeping the last instance of recurring items, now it
  keeps the first. It affects libs order at ffbuild/config.{mak,sh} -
  but I don't think it matters. If it does, "opt1-reorder-unique.patch"
  restores the original order. Let me know if/why it matters and I'll
  squash it and update the commit message accordingly if required.

- After the check_deps() patch, pushvar() and popvar() are not used but
  I was hesitant to remove them (nice to have). If you think it should
  be removed, the patch "opt2-remove-pushvar.patch" removes them.

- The patches assume POSIX shell and don't use anything "tricky".
  It was tested with dash, bash, busybox-ash, freebsd-sh, ksh93u, mksh.

- Thanks to tmm1 and atomnuker for their help with testing.


This is my first mail to this list, and I wasn't sure how to split the
patches. After asking a veteran, it was suggested to me that I should
put all of them in one email. Just let me know if you prefer something
else (dev mailing lists are not my forte...).

For convinience, the patches are also available at:
https://gist.github.com/avih/f51008225d4a20a0981daed1faca4bc2

Regards,
Avi
-------------- next part --------------
A non-text attachment was scrubbed...
Name: config-timing.patch
Type: text/x-patch
Size: 6993 bytes
Desc: not available
URL: <http://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20180825/a90cac28/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: main.patch
Type: text/x-patch
Size: 11344 bytes
Desc: not available
URL: <http://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20180825/a90cac28/attachment-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: opt1-reorder-unique.patch
Type: text/x-patch
Size: 1247 bytes
Desc: not available
URL: <http://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20180825/a90cac28/attachment-0002.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: opt2-remove-pushvar.patch
Type: text/x-patch
Size: 1012 bytes
Desc: not available
URL: <http://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20180825/a90cac28/attachment-0003.bin>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: before-after.txt
URL: <http://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20180825/a90cac28/attachment.txt>