[FFmpeg-devel] Upgrade Trouble

[Michael Niedermayer]

On Fri, Dec 25, 2009 at 11:28:49PM +0100, Reinhard Tartler wrote:
> Michael Niedermayer <michaelni at gmx.at> writes:
> 
> >> I've not addressed points 1 and 3 because I haven't managed (yet) to
> >> reproduce the symptoms. With the testcase I've written earlier, I've
> >> encountered not an assertion failure, but a failure to find a symbol. So
> >> I think you're trying something more obscure than I'm doing.
> >
> 
> Okay, I've succeeded now at reproducing your test case. So let's see:
> 
> >
> > gcc -shared -fPIC -c libA0.c -o libA0.o
> > ld -shared libA0.o -o libA0.so
> >
> > gcc -shared -fPIC -c libA1.c -o libA1.o
> > ld -shared libA1.o -o libA1.so
> >
> > gcc -shared -fPIC -c libB0.c -o libB0.o
> > ld -shared libB0.o -o libB0.so -L. -lA0
> >
> > gcc app.c -o app -L. -lB0 -lA0
> 
> So we're here leaving out details like using proper SONAMEs for the
> libraries, which makes objdump output a bit unfamiliar, but for this
> example it doesn't matter
> 
> >
> > ./app
> > #libA0
> > #B libB0
> > #libA0
> > #libA0
> > #        (correct)
> >

> > gcc -shared -fPIC -c libA1.c -o libA1.o
> 
> this is pretty  unnecessary, it was already compiled above

yes, thats my mistake, i could have ommited a few lines


> 
> > ld -shared --version-script libA1.v libA1.o -o libA1.so
> 
> now libA1 is introduced *with* symbol versioning. This is IMO wrong,
> what I propose would be to first introduce symbol versioning in
> libA0.so, recompile libB.so and app, and then introduce symbol
> versioning in libA1.so.

You do realize that what you call wrong is
"introducing a new lib with new soname & new ABI with versioning."
It might not be what you plan to do but its surely something one would
expect to work ...


> 
> But for the sake of this example, let's proceed.
> 
> > gcc -shared -fPIC -c libB0.c -o libB0.o
> 
> this makes libB0.c pickup the versioned symbol of libA1.
> 
> > ld -shared libB0.o -o libB0.so -L. -lA1
> >
> > ./app
> > #libA0
> > #B libB0
> > #libA0
> > #libA0
> > #        (libB0 is linked by the linker to the wrong lib here)
> 
> This mixes a library with both versioned and unversioned symbols. The
> "Base" symbol seems to be treated as a corner case by ld, that's why all
> applications needs to be rebuilt to pickup the versioned symbols.
> 
> >
> > gcc -shared -fPIC -c libA0.c -o libA0.o
> 
> again, superfluous
> 
> > ld -shared --version-script libA0.v libA0.o -o libA0.so
> 
> now we introduce symbol versioning to the 'old' lib, which removes the
> @Base symbol from the application's view. This unsurprisingly results in:
> 
> > ./app
> > #libA0
> > #B libB0
> > #libA1
> > #libA0 
> > #        (correct)
> >
> 
> the expected behavior!
> 
> > gcc app.c -o app -L. -lB0 -lA0
> 
> now app picks up the versioned symbol from libA0. This should have been
> done earlier.
> 
> > ./app
> > #libA0
> > #B libB0
> > #libA1
> > #libA0 
> > #        (correct)
> 
> yupp
> 
> > gcc -shared -fPIC -c libA0.c -o libA0.o
> > ld -shared libA0.o -o libA0.so
> 
> so now you remove the version tags. This essentially breaks the ABI and
> results in:
> 
> >
> > ./app
> > #./app: ./libA0.so: no version information available (required by ./app)
> > #Inconsistency detected by ld.so: do-lookup.h: 115: check_match: Assertion `version->filename == ((void *)0) || ! _dl_name_match_p (version->filename, map)' failed!
> 
> The error message above correctly indicates the problem. The assertion
> failure could be probably seen as a very minor cosmetic issue.

uhm you got that seriously backward.
just out comment that assert and recompile ld.so
(you need 10+ vomit bags and a bunch of "gnu is not posix tools" and patience)
when you did this, it still prints the harmless warning but works otherwise

Sorry for the delay in my reply, i just had to retry a few times to get
the gnu crap to compile far enough to have a executeable ld.so to proof to
myself that the assert() is pure asshatry and theres no problem with any
symbols not being found


> 
> > gcc -shared -fPIC -c libA0.c -o libA0.o
> > ld -shared --version-script libA0.v libA0.o -o libA0.so
> 
> so symbol versioning in libA0.so is restored, the application should
> work again.
> 
> > gcc -shared -fPIC -c libA1.c -o libA1.o
> > ld -shared libA1.o -o libA1.so
> 
> now the version tags from libA1 is removed, which again results in
> unsurprising behavior:
> 
> >
> > ./app
> > #./app: ./libA1.so: no version information available (required by ./libB0.so)
> > #libA0
> > #B libB0
> > #Inconsistency detected by ld.so: do-lookup.h: 115: check_match: Assertion `version->filename == ((void *)0) || ! _dl_name_match_p (version->filename, map)' failed!
> >
> 
> the runtime linker now fails to find the versioned symbols for libA1,
> against which libB0 is linked.

no, its all found, its just that some developers should have had their
alzheimer pill dosages raised again before they attemped to add that assert


> 
> I fail to see any bug in ld here, this example demonstrates that broken
> usage of ld can produce some confusing error messages.

you know, even if that was the case ... A confusing error
message in itself is a bug too.


> 
> I'd suggest to just use symbol versioning as it is intended:
> 
>  - first introduce symbol tags
>  - then recompile applications
>  - do not remove symbol versioning afterwards without at least a SONAME
>    bump (preferably never)

thats what we do


> 
> >> I think that when introducing symbol versioning all applications need
> >> to be rebuilt in order to pick up the new versioned symbols and the
> >> problem doesn't arise in the first place.
> >
> > An application that is not rebuild links to the correct symbols
> > A not rebuilt lib can link to the wrong symbols, you can rebuild all libs
> > But does this ensure that the user doesnt upgrade just a subset?
> 
> By using versioned dependencies in application packages, it is
> impossible to install a rebuilt application without also installing a
> rebuilt library. We use the dpkg-shlibdeps mechanism to automate this,
> so nothing to worry about that here.

Its great that you can workaroud ld.so bugs by adding hundreads of otherwise
unneeded dependancies. My oppinion stands though that the linker should be
fixed. Part of that would be to comment that idiotic assert out and donate
some more alzheimer pills to the guy who added it in the first place


[...]
> NB: I do not intend to update the FFmpeg package from 0.5 to current
> trunk yet, so the transition in Debian does not have to deal with the
> SONAME change yet. The FFmpeg "trunk" package will be provided as add-on
> package. 

> So in this specific situation, you could indeed force by
> partial upgrades a situation with that the runtime linker would pickup a
> wrong symbol.

interresting


> But this is a corner case I can live with at the moment.
> And even if I couldn't, I do have a plan how to tackle this. I'd rather
> avoid that at this point in order to make the transition not more even
> more complicated (I would need to rename binary packages and so on...)

do you think fixing the linker is more difficult than repeatly working around
it?


[...]
> 
> >> > Now if  1+3  where fixed without the "hierarchic symbol tables"
> >> > as your colleague called my point 2 that still would be a huge simplification
> >> > for distro packagers.
> >> > The reasons should be obvious
> >> >
> >> > * currently introducing versioning one needs to make sure that no application
> >> >   that has been build with the new versioned lib can end up being linked to
> >> >   the older lib without versioning but otherwise 100% identical ABI.
> >> >   Because ld.so goes bye bye with assert(0) in that case
> >> 
> >> That's why we have a documented and recommended migration strategy for
> >> that. [1]
> >
> > That guide also says all libs depending on one that changed the soname must
> > themself change the soname (thats just a ridiculously extreem way to work
> > around linker bugs).
> > You didnt plan to do that, but what matters more, is the rest of the guide
> > still correct if one doesnt do that?
> 
> yes.
> 
> >> I guess I'll just go ahead and start working on the plan with
> >> introducing symbol versioning.
> >
> > fine with me
> 
> Okay, so here we go with the initial patches:
> 
> The first patch adds the versioning scripts. random thoughts on them:
> 
>  - I generated them automatically from configure at first, but dropped
>    that idea because it doesn't make too much sense, IMO.

>  - should they be moved to some subdirectory? if yes to which?

id say that each file should be in the matching libs dir

[...]

-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

Dictatorship naturally arises out of democracy, and the most aggravated
form of tyranny and slavery out of the most extreme liberty. -- Plato
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
URL: <http://lists.mplayerhq.hu/pipermail/ffmpeg-devel/attachments/20091226/2a9440c3/attachment.pgp>