| Summary: | as-needed by default | ||
|---|---|---|---|
| Product: | Gentoo Linux | Reporter: | Donnie Berkholz (RETIRED) <dberkholz> |
| Component: | [OLD] Unspecified | Assignee: | Doug Goldstein (RETIRED) <cardoe> |
| Status: | RESOLVED FIXED | ||
| Severity: | normal | CC: | ab4bd, antarus, arfrever, awickedshimmy, bicatali, billie, binki, coldwind, council, darkside, flameeyes, gef.kornflakes, gentoo.cart9, gentoo, jaak, jlec, johnparmitage, kanelxake, kfm, leio, levertond, n-roeser, nao.nakashima, nirbheek, pacho, patrick, pchrist, please.no.spam.here, polynomial-c, prefix, scarabeus, skyleach, spatz, tcunha, zorry |
| Priority: | High | ||
| Version: | unspecified | ||
| Hardware: | All | ||
| OS: | Linux | ||
| URL: | http://forums.gentoo.org/viewtopic-t-715157.html | ||
| Whiteboard: | |||
| Package list: | Runtime testing required: | --- | |
| Bug Depends on: | 129413, 297541 | ||
| Bug Blocks: | 327809 | ||
|
Description
Donnie Berkholz (RETIRED)
2008-08-14 08:51:31 UTC
Adding other interested folks to CC...I know I talked to someone (coldwind?) about this on irc regarding the points we needed to cover; but I don't think I logged the conversation; I'll have to see. -Alec If we decide to push through this, the right way would be, IMHO, changing the GCC specs file. The problem here is that you'd have to make sure that all of the tree builds with that too, and I can tell you, this is not yet the case. I hope to be able to push out some patches to fix that where it's possible during the fall. Alec: ping. I know you were going to work up something on this. I have nothing, feel free to act independent of me or wait on me ;) What was the hold up on this item anyway? Why can't we just implement it. (In reply to comment #5) > What was the hold up on this item anyway? Why can't we just implement it. > Implementing it is not trivial. Someone has to elaborate a migration plan for ebuild developers, arch teams and users, properly document the new policy, test heavily the whole stable tree and filter --as-needed where necessary... And then, pull the trigger. Some people on @-dev also asked for real numbers about as-needed benefits. Doug's going to take the lead on getting this done. Patrick, when you're building the whole tree, are you using --as-needed? If not, could you? (In reply to comment #6) > (In reply to comment #5) > Implementing it is not trivial. Someone has to elaborate a migration plan for > ebuild developers, arch teams and users, properly document the new policy, test > heavily the whole stable tree and filter --as-needed where necessary... And > then, pull the trigger. Filtering it is not the fully correct solution - passing --no-as-needed before libraries that mustn't be filtered out due to questionable reliance of static initialization, and restoring --as-needed afterwards if set would be more correct at eupstream oatching and patch forwarding level. For the purpose of being able to add --as-needed by default earlier filtering will do, but in 99% of the cases it is a bug in the build system and in ~1% the reliance on static initialization (in which case --no-as-needed where apporpriate) and so the appropriate bug should remain open until the filtering is replaced with a proper fix. So you can consider temporary filtering as a proposal for getting it in earlier, when plans to fix them properly is not forgotten. There is nothing to report at this time. I could not find users or developers with the time or resources to currently investigate this. If anyone has any detailed statistics they can provide on as-needed usage for building the tree, please provide them. I've been experimenting with forced --as-needed by default on my system through GCC specs files. I sincerely think that this would be the correct approach to add --as-needed by default because it also sidesteps bugs like libtool bugs and packages ignoring LDFLAGS until we can get those to actually work. Up to now I think the results aren't that bad: in my system the only package still bearing huge problems is xmlrpc-c. On the other hand, some other packages needed special attention and I'm sure there are more that are not currently causing failures with the LDFLAGS setting method that are instead problematic with forced --as-needed. Using forced --as-needed require stricter checks, for instance when libraries are passed through the LDFLAGS variable in autotools (and not), and it affects package that otherwise ignore LDFLAGS (which are an issue by themselves since they breach QA policy), and thus haven't been looked up until now. Filtering _certainly is not the right solution_ and should really really really be avoided; appending -Wl,--no-as-needed is a bit more decent but is still a very nasty workaround since I haven't really found any software in the wild that cannot be made to work with --as-needed; upstream coordination _is_ necessary though, I was able just today for instance to get --as-needed fixes in for alsa-tools on the upstream repositories, which means it's just a question of time and of showing upstream the way. If I may suggest a course of action here, I'd first ask toolchain to revbump gcc for both the stable series and the ~arch series (so that people wanting to experiment with it on stable can; I'd add gcc 4.2 to the list too so that one can use it for packages like virtualbox that insist on not building with 4.3), adding a further --as-needed variant to the specs files with install, that can be chosen through gcc-config(1). Then we can get more people (developers and power users) to test the forced --as-needed and we can decided for a clean transition in the future. I renew my offer to help as much as I can, time permitting, to port the packages to work with --as-needed and forced --as-needed. For those interested in the progress, I started a built of the whole ~arch tree (as far as humanly possible) with the forced --as-needed as I explained in http://blog.flameeyes.eu/2008/11/14/problems-and-mitigation-strategies-for-as-needed . It's going to take time to complete, but since this weekend I'm not working it shouldn't take _too_ long to have some decent statistics. (Well it could be better if people weren't forcing -j1 rather than fixing it but...). As I already explained, a good percentage of failures I'm sure will end up being caused by simple custom single makefiles, which is going probably to become the most boring bit. I'd also like for the current _filtering_ (which is wrong, --no-as-needed should be appended instead) to be removed from the tree at once.. I think this is going to run into problems all over the board for anything that has poorly written autotools. Fameeyes: I spent 20 hours this weekend hacking around in libwww and I had to cut corners to get the libwww-r8 ebuild to work without completely rewriting their autotools files, and I wasn't even using --as-needed. I was basically doing the same thing by implementing use_with and use_enable. I am no expert on autotools myself, but I can spot many places where the generated makefiles simply ignore autotools-based configuration settings. In your blog I noticed you were getting the XML chain of errors this causes. I'll keep up with this and work in parallel with you if you like. (In reply to comment #10) > I've been experimenting with forced --as-needed by default on my system through > GCC specs files. I sincerely think that this would be the correct approach to > add --as-needed by default because it also sidesteps bugs like libtool bugs and > packages ignoring LDFLAGS until we can get those to actually work. > I'm not convinced that forced --as-needed testing is the way to go here. We glean no information as to weather the tree is ready or not for --as-needed in LDFLAGS by default. I do respect the work however...I am just not sure what this is telling us. Let me give a very quick and realistic example of why we should go for spec-forced --as-needed from the start rather than go through LDFLAGS first. Package foo does not respect LDFLAGS, thus builds fine with --as-needed. The package maintainer or upstream finally fix the package to build respecting LDFLAGS, but it's not tested with --as-needed before the new ebuild enters portage. Users using --as-needed try to build the package and it fails, it's a regression, a nasty one. With spec-forced --as-needed, this is tested beforehand, with or without respecting LDFLAGS. Let's make it a bit more interesting: right now --as-needed is ignored for libraries built with libtool because of the reordering bug. Let's say a new libtool that fixes this problem is released, now all the packages that relied on the reordering not to fail will be broken at once (if libtoolize is called) or one by one as upstream migrates to the new version. With forced --as-needed we trigger the bugs early in the process. (In reply to comment #14) > Let me give a very quick and realistic example of why we should go for > spec-forced --as-needed from the start rather than go through LDFLAGS first. <snip> I don't get your point about future versions of packages. Shouldn't it be the maintainer's resposibility to actually test what they are putting into the tree? To summarize my points: This *forced* as-needed testing is great from a software development aspect and makes us all warm and fuzzy feeling if Gentoo can "make software better" in general. Also, our stuff will go up then trickle back down to other distros...great, I'm all for it. So, what do we do now? Wait for all fixes to go upstream (unlikely), hit Gentoo, hit Gentoo's stable tree...1 month (optimistic) - 6-12 months down the road - enable --as-needed by default. Then run this tree wide forcing again and find that an entire new set of packages fail? This is an endless cycle, I'm afraid. However, many of us have --as-needed in LDFLAGS now and my emerge -e world works fine. Of course, --as-needed isn't hitting *every* package that I have installed but, I don't really care. It has saved me time recompiling packages that obviously don't need to be recompiled. (No, I don't have any concrete numbers, sorry) I have little trust in each maintainer doublechecking his stuff. I'm not pretending to have a perfect answer to everything here, I'm just trying to give arguments for the solution.
What can we do is another matter. Most of the bugs I've open in the last week are solvable in about ten minutes each, yes I know that ten minutes for all those bugs is a lot of time, but the idea is that each maintainer takes care of his packages and we'll be done.
Let's start with making this feasibly usable by making the most common packages not fail nor break (xmlrpc-c for instance, which got so common because of cmake), and then suggest developers to use forced --as-needed on their development.
Make arch teams refuse to stable new packages that fail with --as-needed, make it a QA problem to add packages that filter --as-needed in the wrong way. I sincerely think step by step is a good solution.
To quote David Philippi in my blog's comments:
> I'd just like to thank you and the others involved in this effort. I think
> --as-needed is a very usefull flag and have it in my LDFLAGS for years now.
> In the beginning reporting bugs regarding this sometimes got a very
> unpleasant receiving, those days matters are much better. Things like the big
> expat breakage really show how much difference this can make...
When the --as-needed idea started to come to users around two and a half years ago, we weren't expecting to have a whole tree to build properly with it in LDFLAGS, yet now we came to this point more or less. Aim a little higher and we can have something even better in probably much less time since other distributions are also aiming at that.
Also if binutils decided to make --as-needed the default, which I don't find too far fetched, we'd be having trouble if we're not to fix the spec-forced --as-needed.
If you're concerned about the amount of bugs, I want to let you know that I found a huge amount of bugs not related to --as-needed to with this run, just like Patrick does find them when he adds more tests or packages to his run, a lot of the stuff comes out of packages that almost nobody ever uses.
See if there is any feedback on $URL that I just added. [No need to highlight and yes I'm monitoring the topic already, I asked for that post myself ;)] [Since Sébastine reported it in the first place, I'm adding him to CC ;)] Upstream decided to change the way --as-needed works in the latest snapshot and in the upcoming 2.20 version: * --as-needed now links in a dynamic library if it satisfies undefined symbols in regular objects, or in other dynamic libraries. In the latter case the library is not linked if it is found in a DT_NEEDED entry of one of the libraries already linked. I'm sincerely a bit undecided on whether suggesting upstream to change this or give at least a switch because IMHO this is something that could bite us quite a bit. On the other hand, it might work around the fact that libtool does not allow to pass -Wl,--no-as-needed -lsomething -Wl,--as-needed in LIBADD/LDADD to force linkage when it's needed, and I still think that the use case that Sébastien provided should be handled more like FreeBSD handles multiple pthread libraries, but that's beside the point. At any rate, while I still think that these situations should be corrected so -lfoo works without further links, since otherwise we can find problems just as bad as expat in the future, version 2.20 of binutils should be much less prone to break things than 2.17-2.19 have been, and might be worth revisiting this whole issue once that hits stable. Although I should add that it doesn't fix the recursive linking dependencies in PulseAudio I was hoping actually went to fix, which seems to me like it should fix so it might actually still be broken. Guys the current state is that 99% of the tree compiles with --as-needed by default testing and stable. 90% of that compiles even with forced one. Acutaly not using --as-needed causes bugs, for example the broken .la files and whining all over libtool avare autotools using packages. See kde3 bugs after kde3.5.10 bump or the fun we are going to have in x11 because of the dropping .la file by upstream. We should really set up some timeframe when we should start enforcing/recomending it. (Btw amarok2 itself wont compile without --as-needed so it enforces it, quite funny huh) (In reply to comment #21) > > We should really set up some timeframe when we should start > enforcing/recomending it. (Btw amarok2 itself wont compile without --as-needed > so it enforces it, quite funny huh) > Actually upstreams forcing --as-needed is perfectly valid behavior as said by ciaramn before and more upstream should do it. Just provide some statistics from tinderbox runs and I don't see a problem bringing it to the council agenda again. Flameeye's suggestion to inject --as-needed via GCC spec-files is fine. Could also do it with the binutils-config wrapper that Prefix uses. In any case, before doing this, keep in mind that not all linkers accept --as-needed, so it should be easy to disable for those combos (like on Darwin, or Solaris with Sun ld). Since this would affect a fair amount of the Prefix platforms, adding to CC. We could also suggest the use of a newer version of binutils and using LD_AS_NEEDED environment variable instead, it should be present in 2.20 iirc but I'm not sure, need to double-check. (In reply to comment #21) > Guys the current state is that 99% of the tree compiles with --as-needed by > default testing and stable. 90% of that compiles even with forced one. If most packages already build with "--as-needed" (without forcing it), maybe we could default to it just now and try to use forced one when more packages build with it :-/ Fixed by today's council decision to approve -Wl,--as-needed in LDFLAGS: http://sources.gentoo.org/viewvc.cgi/gentoo-x86/profiles/default/linux/make.defaults?r1=1.6&r2=1.7 |
