[MPlayer-dev-eng] recursive make considered harmful (was: Re: The new Makefiles)

[Ivan Kalvachev]

On 6/2/08, Diego Biurrun <diego at biurrun.de> wrote:
> On Mon, Jun 02, 2008 at 07:07:22PM +0200, Nico Sabbi wrote:
>> Il Monday 02 June 2008 17:18:49 Diego Biurrun ha scritto:
>> > I did read your patch when you sent it to me.  However, I do not
>> > remember every single detail and it does not apply anymore.  By "I'll
>> > look at your patch" I just meant to say that I will investigate how it
>> > can be used to fix our linking problems, not that I wanted to look at
>> > it for the first time.
>> 
>> sorry, but I really don't understand the reason for a single and totally
>> flat
>> Makefile: IMO a Makefile per directory is a kind of modularization (that
>> is generally considered "A good thing" ) , clean for its own nature.
>> Can you explain what we gain now? or in what respect a single Makefile
>> is cleaner?
>
> The single Makefile is shorter than the single-directory Makefiles
> combined, even with common infrastructure factorized into mpcommon.mak.
> Our Makefile is just above 1000 lines.  Given that it consists largely
> of long lists, it is neither complicated nor particularly long.  In such
> a case I prefer having all the information in one central place.
>
> At work I once had the misfortune of having to debug a recursive make
> system where everything was split into tiny snippets.  I continuously
> lost track of which part got included from where and after staring at it
> for a few hours, I rewrote it non-recursively in a fraction of that
> time.  The end result was a very simple, single Makefile with about 30%
> of the previous total line count.  It was also faster and correct.
>
> So unless you go over a certain threshold I generally prefer 60 lines
> over 6 x 10 lines.
>
> But this is not the main reason.  The problem is that recursive Make is
> incorrect.  What the Make program does is build a directed acyclic graph
> of all dependencies/targets and then perform all the necessary steps the
> target you request requires, but no more than those necessary steps.
>
> Recursive uses of make cut this graph into pieces.  Unfortunately this
> process is not lossless because inter-directory dependencies are left
> out.  This results in Make systems that continuously recompile things
> unnecessarily, fail to recompile necessary parts and generally have
> horrible performance.
>
> For a more in-depth explanation, read Peter Miller's short but
> informative paper "Recursive Make Considered Harmful":
>
> http://miller.emu.id.au/pmiller/books/rmch/
>
> I hope this clears it up.  If you have more questions, ask, but it's
> really been explained many times in many places, the best I know being
> the above-mentioned paper.

There is Bulgarian saying "The frog saw that farrier is shoeing a horse and riced leg to get iron shoe".

Diego had found some article about good habits and implements it without fully understanding it. He was inspired by the FFmpeg change in same direction.

Most of the drawbacks of recursive compilation do not apply to MPlayer.
The main problem solved in the paper is when files that are needed for the compilation must be generated beforehand and they have/cause dependences from other branches. This in extreme cases requiring running make twice in the same directory/project.

In MPlayer such cases are 2, mp_msg.h and codecs-conf* . I can't remember any of these causing any major problems. (I'm not sure about vidix it may create some file)

The recursive makefile theoretically should be faster, because running new instance of make is slower operation. However it also requires more memory as it was exposed by make bug with memory handling that caused FFmpeg to require newer version of make.
The computational slowdown caused by the larger lists could be negated from not having to compute same stuff for every makefile instance. So it must be benchmarked.

The paper also suggest that there is nothing bad in including per-directory makefiles.inc files, if developers prefer to do so.

The biggest problem as it was already demonstrated, is that the MPlayer Makefile cannot be fully made non-recursive. We do have includes of different projects that are imported into MPlayer and we must call them recursively.