Bugzilla Bug 234710

antarus requested that we vote on whether to add it to the default LDFLAGS.

Preparation: Antarus will post a deployment plan to -dev for discussion. We can
vote on it on -council as soon as it solidifies.

Adding other interested folks to CC...I know I talked to someone (coldwind?)
about this on irc regarding the points we needed to cover; but I don't think I
logged the conversation; I'll have to see.

-Alec

If we decide to push through this, the right way would be, IMHO, changing the
GCC specs file. The problem here is that you'd have to make sure that all of
the tree builds with that too, and I can tell you, this is not yet the case.

I hope to be able to push out some patches to fix that where it's possible
during the fall.

Alec: ping. I know you were going to work up something on this.

I have nothing, feel free to act independent of me or wait on me ;)

What was the hold up on this item anyway? Why can't we just implement it.

(In reply to comment #5)
> What was the hold up on this item anyway? Why can't we just implement it.
> 

Implementing it is not trivial. Someone has to elaborate a migration plan for
ebuild developers, arch teams and users, properly document the new policy, test
heavily the whole stable tree and filter --as-needed where necessary... And
then, pull the trigger.

Some people on @-dev also asked for real numbers about as-needed benefits.

Doug's going to take the lead on getting this done.

Patrick, when you're building the whole tree, are you using --as-needed? If
not, could you?

(In reply to comment #6)
> (In reply to comment #5)
> Implementing it is not trivial. Someone has to elaborate a migration plan for
> ebuild developers, arch teams and users, properly document the new policy, test
> heavily the whole stable tree and filter --as-needed where necessary... And
> then, pull the trigger.

Filtering it is not the fully correct solution - passing --no-as-needed before
libraries that mustn't be filtered out due to questionable reliance of static
initialization, and restoring --as-needed afterwards if set would be more
correct at eupstream oatching and patch forwarding level. For the purpose of
being able to add --as-needed by default earlier filtering will do, but in 99%
of the cases it is a bug in the build system and in ~1% the reliance on static
initialization (in which case --no-as-needed where apporpriate) and so the
appropriate bug should remain open until the filtering is replaced with a
proper fix. So you can consider temporary filtering as a proposal for getting
it in earlier, when plans to fix them properly is not forgotten.

There is nothing to report at this time. I could not find users or developers
with the time or resources to currently investigate this.

If anyone has any detailed statistics they can provide on as-needed usage for
building the tree, please provide them. 

I've been experimenting with forced --as-needed by default on my system through
GCC specs files. I sincerely think that this would be the correct approach to
add --as-needed by default because it also sidesteps bugs like libtool bugs and
packages ignoring LDFLAGS until we can get those to actually work.

Up to now I think the results aren't that bad: in my system the only package
still bearing huge problems is xmlrpc-c. On the other hand, some other packages
needed special attention and I'm sure there are more that are not currently
causing failures with the LDFLAGS setting method that are instead problematic
with forced --as-needed.

Using forced --as-needed require stricter checks, for instance when libraries
are passed through the LDFLAGS variable in autotools (and not), and it affects
package that otherwise ignore LDFLAGS (which are an issue by themselves since
they breach QA policy), and thus haven't been looked up until now.

Filtering _certainly is not the right solution_ and should really really really
be avoided; appending -Wl,--no-as-needed is a bit more decent but is still a
very nasty workaround since I haven't really found any software in the wild
that cannot be made to work with --as-needed; upstream coordination _is_
necessary though, I was able just today for instance to get --as-needed fixes
in for alsa-tools on the upstream repositories, which means it's just a
question of time and of showing upstream the way.

If I may suggest a course of action here, I'd first ask toolchain to revbump
gcc for both the stable series and the ~arch series (so that people wanting to
experiment with it on stable can; I'd add gcc 4.2 to the list too so that one
can use it for packages like virtualbox that insist on not building with 4.3),
adding a further --as-needed variant to the specs files with install, that can
be chosen through gcc-config(1).

Then we can get more people (developers and power users) to test the forced
--as-needed and we can decided for a clean transition in the future.

I renew my offer to help as much as I can, time permitting, to port the
packages to work with --as-needed and forced --as-needed.

For those interested in the progress, I started a built of the whole ~arch tree
(as far as humanly possible) with the forced --as-needed as I explained in
http://blog.flameeyes.eu/2008/11/14/problems-and-mitigation-strategies-for-as-needed
.

It's going to take time to complete, but since this weekend I'm not working it
shouldn't take _too_ long to have some decent statistics. (Well it could be
better if people weren't forcing -j1 rather than fixing it but...).

As I already explained, a good percentage of failures I'm sure will end up
being caused by simple custom single makefiles, which is going probably to
become the most boring bit.

I'd also like for the current _filtering_ (which is wrong, --no-as-needed
should be appended instead) to be removed from the tree at once..

I think this is going to run into problems all over the board for anything that
has poorly written autotools.

Fameeyes: I spent 20 hours this weekend hacking around in libwww and I had to
cut corners to get the libwww-r8 ebuild to work without completely rewriting
their autotools files, and I wasn't even using --as-needed.

I was basically doing the same thing by implementing use_with and use_enable.

I am no expert on autotools myself, but I can spot many places where the
generated makefiles simply ignore autotools-based configuration settings.

In your blog I noticed you were getting the XML chain of errors this causes.

I'll keep up with this and work in parallel with you if you like.

(In reply to comment #10)
> I've been experimenting with forced --as-needed by default on my system through
> GCC specs files. I sincerely think that this would be the correct approach to
> add --as-needed by default because it also sidesteps bugs like libtool bugs and
> packages ignoring LDFLAGS until we can get those to actually work.
> 

I'm not convinced that forced --as-needed testing is the way to go here. We
glean no information as to weather the tree is ready or not for --as-needed in
LDFLAGS by default. I do respect the work however...I am just not sure what
this is telling us.

Let me give a very quick and realistic example of why we should go for
spec-forced --as-needed from the start rather than go through LDFLAGS first.

Package foo does not respect LDFLAGS, thus builds fine with --as-needed. The
package maintainer or upstream finally fix the package to build respecting
LDFLAGS, but it's not tested with --as-needed before the new ebuild enters
portage. Users using --as-needed try to build the package and it fails, it's a
regression, a nasty one. With spec-forced --as-needed, this is tested
beforehand, with or without respecting LDFLAGS.

Let's make it a bit more interesting: right now --as-needed is ignored for
libraries built with libtool because of the reordering bug. Let's say a new
libtool that fixes this problem is released, now all the packages that relied
on the reordering not to fail will be broken at once (if libtoolize is called)
or one by one as upstream migrates to the new version.

With forced --as-needed we trigger the bugs early in the process.

(In reply to comment #14)
> Let me give a very quick and realistic example of why we should go for
> spec-forced --as-needed from the start rather than go through LDFLAGS first.
<snip>

I don't get your point about future versions of packages. Shouldn't it be the
maintainer's resposibility to actually test what they are putting into the
tree?

To summarize my points: This *forced* as-needed testing is great from a
software development aspect and makes us all warm and fuzzy feeling if Gentoo
can "make software better" in general. Also, our stuff will go up then trickle
back down to other distros...great, I'm all for it. So, what do we do now? Wait
for all fixes to go upstream (unlikely), hit Gentoo, hit Gentoo's stable
tree...1 month (optimistic) - 6-12 months down the road - enable --as-needed by
default. Then run this tree wide forcing again and find that an entire new set
of packages fail? This is an endless cycle, I'm afraid.

However, many of us have --as-needed in LDFLAGS now and my emerge -e world
works fine. Of course, --as-needed isn't hitting *every* package that I have
installed but, I don't really care. It has saved me time recompiling packages
that obviously don't need to be recompiled. (No, I don't have any concrete
numbers, sorry)

I have little trust in each maintainer doublechecking his stuff. I'm not
pretending to have a perfect answer to everything here, I'm just trying to give
arguments for the solution.

What can we do is another matter. Most of the bugs I've open in the last week
are solvable in about ten minutes each, yes I know that ten minutes for all
those bugs is a lot of time, but the idea is that each maintainer takes care of
his packages and we'll be done.

Let's start with making this feasibly usable by making the most common packages
not fail nor break (xmlrpc-c for instance, which got so common because of
cmake), and then suggest developers to use forced --as-needed on their
development.

Make arch teams refuse to stable new packages that fail with --as-needed, make
it a QA problem to add packages that filter --as-needed in the wrong way. I
sincerely think step by step is a good solution.

To quote David Philippi in my blog's comments:

> I'd just like to thank you and the others involved in this effort. I think
> --as-needed is a very usefull flag and have it in my LDFLAGS for years now.
> In the beginning reporting bugs regarding this sometimes got a very
> unpleasant receiving, those days matters are much better. Things like the big
> expat breakage really show how much difference this can make...

When the --as-needed idea started to come to users around two and a half years
ago, we weren't expecting to have a whole tree to build properly with it in
LDFLAGS, yet now we came to this point more or less. Aim a little higher and we
can have something even better in probably much less time since other
distributions are also aiming at that.

Also if binutils decided to make --as-needed the default, which I don't find
too far fetched, we'd be having trouble if we're not to fix the spec-forced
--as-needed.

If you're concerned about the amount of bugs, I want to let you know that I
found a huge amount of bugs not related to --as-needed to with this run, just
like Patrick does find them when he adds more tests or packages to his run, a
lot of the stuff comes out of packages that almost nobody ever uses.

See if there is any feedback on $URL that I just added.

[No need to highlight and yes I'm monitoring the topic already, I asked for
that post myself ;)]

[Since Sébastine reported it in the first place, I'm adding him to CC ;)]

Upstream decided to change the way --as-needed works in the latest snapshot and
in the upcoming 2.20 version:

* --as-needed now links in a dynamic library if it satisfies undefined
  symbols in regular objects, or in other dynamic libraries.  In the
  latter case the library is not linked if it is found in a DT_NEEDED
  entry of one of the libraries already linked.

I'm sincerely a bit undecided on whether suggesting upstream to change this or
give at least a switch because IMHO this is something that could bite us quite
a bit. On the other hand, it might work around the fact that libtool does not
allow to pass -Wl,--no-as-needed -lsomething -Wl,--as-needed in LIBADD/LDADD to
force linkage when it's needed, and I still think that the use case that
Sébastien provided should be handled more like FreeBSD handles multiple
pthread libraries, but that's beside the point.

At any rate, while I still think that these situations should be corrected so
-lfoo works without further links, since otherwise we can find problems just as
bad as expat in the future, version 2.20 of binutils should be much less prone
to break things than 2.17-2.19 have been, and might be worth revisiting this
whole issue once that hits stable.

Although I should add that it doesn't fix the recursive linking dependencies in
PulseAudio I was hoping actually went to fix, which seems to me like it should
fix so it might actually still be broken.

Guys the current state is that 99% of the tree compiles with --as-needed by
default testing and stable. 90% of that compiles even with forced one.

Acutaly not using --as-needed causes bugs, for example the broken .la files and
whining all over libtool avare autotools using packages. See kde3 bugs after
kde3.5.10 bump or the fun we are going to have in x11 because of the dropping
.la file by upstream.

We should really set up some timeframe when we should start
enforcing/recomending it. (Btw amarok2 itself wont compile without --as-needed
so it enforces it, quite funny huh)

(In reply to comment #21)
> 
> We should really set up some timeframe when we should start
> enforcing/recomending it. (Btw amarok2 itself wont compile without --as-needed
> so it enforces it, quite funny huh)
> 

Actually upstreams forcing --as-needed is perfectly valid behavior as said by
ciaramn before and more upstream should do it. Just provide some statistics
from tinderbox runs and I don't see a problem bringing it to the council agenda
again.