[FFmpeg-devel] Upgrade Trouble

[Reinhard Tartler]

Michael Niedermayer <michaelni at gmx.at> writes:

> On Fri, Dec 25, 2009 at 11:28:49PM +0100, Reinhard Tartler wrote:
>> Michael Niedermayer <michaelni at gmx.at> writes:
>> 
>> >> I've not addressed points 1 and 3 because I haven't managed (yet) to
>> >> reproduce the symptoms. With the testcase I've written earlier, I've
>> >> encountered not an assertion failure, but a failure to find a symbol. So
>> >> I think you're trying something more obscure than I'm doing.
>> >
>> 
>> Okay, I've succeeded now at reproducing your test case. So let's see:
>> 
>> >
>> > gcc -shared -fPIC -c libA0.c -o libA0.o
>> > ld -shared libA0.o -o libA0.so
>> >
>> > gcc -shared -fPIC -c libA1.c -o libA1.o
>> > ld -shared libA1.o -o libA1.so
>> >
>> > gcc -shared -fPIC -c libB0.c -o libB0.o
>> > ld -shared libB0.o -o libB0.so -L. -lA0
>> >
>> > gcc app.c -o app -L. -lB0 -lA0
>> 
>> So we're here leaving out details like using proper SONAMEs for the
>> libraries, which makes objdump output a bit unfamiliar, but for this
>> example it doesn't matter
>> 
>> >
>> > ./app
>> > #libA0
>> > #B libB0
>> > #libA0
>> > #libA0
>> > #        (correct)
>> >
>
>> > gcc -shared -fPIC -c libA1.c -o libA1.o
>> 
>> this is pretty  unnecessary, it was already compiled above
>
> yes, thats my mistake, i could have ommited a few lines
>
>
>> 
>> > ld -shared --version-script libA1.v libA1.o -o libA1.so
>> 
>> now libA1 is introduced *with* symbol versioning. This is IMO wrong,
>> what I propose would be to first introduce symbol versioning in
>> libA0.so, recompile libB.so and app, and then introduce symbol
>> versioning in libA1.so.
>
> You do realize that what you call wrong is
> "introducing a new lib with new soname & new ABI with versioning."
> It might not be what you plan to do but its surely something one would
> expect to work ...

We obviously have different expectations about this feature of ld.

>
>> 
>> But for the sake of this example, let's proceed.
>> 
>> > gcc -shared -fPIC -c libB0.c -o libB0.o
>> 
>> this makes libB0.c pickup the versioned symbol of libA1.
>> 
>> > ld -shared libB0.o -o libB0.so -L. -lA1
>> >
>> > ./app
>> > #libA0
>> > #B libB0
>> > #libA0
>> > #libA0
>> > #        (libB0 is linked by the linker to the wrong lib here)
>> 
>> This mixes a library with both versioned and unversioned symbols. The
>> "Base" symbol seems to be treated as a corner case by ld, that's why all
>> applications needs to be rebuilt to pickup the versioned symbols.
>> 
>> >
>> > gcc -shared -fPIC -c libA0.c -o libA0.o
>> 
>> again, superfluous
>> 
>> > ld -shared --version-script libA0.v libA0.o -o libA0.so
>> 
>> now we introduce symbol versioning to the 'old' lib, which removes the
>> @Base symbol from the application's view. This unsurprisingly results in:
>> 
>> > ./app
>> > #libA0
>> > #B libB0
>> > #libA1
>> > #libA0 
>> > #        (correct)
>> >
>> 
>> the expected behavior!
>> 
>> > gcc app.c -o app -L. -lB0 -lA0
>> 
>> now app picks up the versioned symbol from libA0. This should have been
>> done earlier.
>> 
>> > ./app
>> > #libA0
>> > #B libB0
>> > #libA1
>> > #libA0 
>> > #        (correct)
>> 
>> yupp
>> 
>> > gcc -shared -fPIC -c libA0.c -o libA0.o
>> > ld -shared libA0.o -o libA0.so
>> 
>> so now you remove the version tags. This essentially breaks the ABI and
>> results in:
>> 
>> >
>> > ./app
>> > #./app: ./libA0.so: no version information available (required by ./app)
>> > #Inconsistency detected by ld.so: do-lookup.h: 115: check_match: Assertion `version->filename == ((void *)0) || ! _dl_name_match_p (version->filename, map)' failed!
>> 
>> The error message above correctly indicates the problem. The assertion
>> failure could be probably seen as a very minor cosmetic issue.
>
> uhm you got that seriously backward.
> just out comment that assert and recompile ld.so
> (you need 10+ vomit bags and a bunch of "gnu is not posix tools" and patience)
> when you did this, it still prints the harmless warning but works otherwise

Well, I can imagine that this works in this case. Still, when we
consider symbols with and without version tags as totally different
symbols, the error message is correct (except that it does not fail
gracefully but by an confusing assert()). Of course this view is too
simplistic; for transition purposes, if an application references an
unversioned symbol, the runtime linker can satisfy this request with an
versioned symbol. However the reverse, (the application references an
versioned symbol but only an unversioned one is available) may or may not
succeed (as in "unspecified behavior").

> Sorry for the delay in my reply, i just had to retry a few times to get
> the gnu crap to compile far enough to have a executeable ld.so to proof to
> myself that the assert() is pure asshatry and theres no problem with any
> symbols not being found
>
>> 
>> > gcc -shared -fPIC -c libA0.c -o libA0.o
>> > ld -shared --version-script libA0.v libA0.o -o libA0.so
>> 
>> so symbol versioning in libA0.so is restored, the application should
>> work again.
>> 
>> > gcc -shared -fPIC -c libA1.c -o libA1.o
>> > ld -shared libA1.o -o libA1.so
>> 
>> now the version tags from libA1 is removed, which again results in
>> unsurprising behavior:
>> 
>> >
>> > ./app
>> > #./app: ./libA1.so: no version information available (required by ./libB0.so)
>> > #libA0
>> > #B libB0
>> > #Inconsistency detected by ld.so: do-lookup.h: 115: check_match: Assertion `version->filename == ((void *)0) || ! _dl_name_match_p (version->filename, map)' failed!
>> >
>> 
>> the runtime linker now fails to find the versioned symbols for libA1,
>> against which libB0 is linked.
>
> no, its all found, its just that some developers should have had their
> alzheimer pill dosages raised again before they attemped to add that assert
>
>
>> 
>> I fail to see any bug in ld here, this example demonstrates that broken
>> usage of ld can produce some confusing error messages.
>
> you know, even if that was the case ... A confusing error
> message in itself is a bug too.
>
>
>> 
>> I'd suggest to just use symbol versioning as it is intended:
>> 
>>  - first introduce symbol tags
>>  - then recompile applications
>>  - do not remove symbol versioning afterwards without at least a SONAME
>>    bump (preferably never)
>
> thats what we do

in ffmpeg yes, your testcase doesn't follow this procedure closely.

in any case, I think I've really done my homework now, checked this
approach with the debian release team and got the green light to
proceed. No I've proposed patches in the hope to have them accepted in
trunk. Once I get confirmation that this approach will get into ffmpeg
trunk in a forseeable timeframe, I'll start the transition in debian and
ubuntu.


>> >> I think that when introducing symbol versioning all applications need
>> >> to be rebuilt in order to pick up the new versioned symbols and the
>> >> problem doesn't arise in the first place.
>> >
>> > An application that is not rebuild links to the correct symbols
>> > A not rebuilt lib can link to the wrong symbols, you can rebuild all libs
>> > But does this ensure that the user doesnt upgrade just a subset?
>> 
>> By using versioned dependencies in application packages, it is
>> impossible to install a rebuilt application without also installing a
>> rebuilt library. We use the dpkg-shlibdeps mechanism to automate this,
>> so nothing to worry about that here.
>
> Its great that you can workaroud ld.so bugs by adding hundreads of otherwise
> unneeded dependancies. My oppinion stands though that the linker should be
> fixed. Part of that would be to comment that idiotic assert out and donate
> some more alzheimer pills to the guy who added it in the first place

I think we can agree to disagree: IMO, this is not a bug but undefined
behavior.  Can we move on or is this disagreement so critical to you that
you would block introducing symbol versioning in the ffmpeg svn?

> [...]
>
>> Okay, so here we go with the initial patches:
>> 
>> The first patch adds the versioning scripts. random thoughts on them:
>> 
>>  - I generated them automatically from configure at first, but dropped
>>    that idea because it doesn't make too much sense, IMO.
>
>>  - should they be moved to some subdirectory? if yes to which?
>
> id say that each file should be in the matching libs dir
>
> [...]

implemented in a new thread. let's keep this thread for general
discussion only and the other one for discussion on the implementation.

-- 
Gruesse/greetings,
Reinhard Tartler, KeyID 945348A4