Bugzilla Bug 94453

Here are some errors that happen, the full output and emerge info are attached. 

x86_64-pc-linux-gnu-gcc: : No such file or directory 
make[2]: *** [valid.lo] Error 1 

Much thanks for any help!  I'm stuck on this one! 

Reproducible: Always
Steps to Reproduce:
1.  emerge libxml2 -v  
2.  
3.  

Actual Results:  
Error messages will be attached 

Expected Results:  
Finished compiling KDE

Created an attachment (id=60154) [details]
emerge output

This is what happens after doing "emerge libxml2"

Created an attachment (id=60155) [details]
emerge -info

This is the output of emerge -info. 

When I try to do a sandbox emerge, it says they are all masked (even with
~amd64)

I have also tried with no FEATURES to test without sandbox.   No go. 

Please try to emerge this again with more conservative CFLAGS. Maybe something
like just CFLAGS="-march=opteron -O2 -pipe".

Just did it with CFLAGS="-march=opteron -O2 -pipe" and I still get the error    

x86_64-pc-linux-gnu-gcc: : No such file or directory  

Except now it is nanoftp.lo causing it: 

make[2]: *** [nanoftp.lo] Error 1  

Oh my goodness.  I just tried with an earlier kernel
(gentoo-sources-2.6.11-gentoo-r9)   
and everything compiles fine!    

I think the reason it worked is because I am using a new dual core opteron and
it  
seems that the 2.6.11 kernel doesn't initialize/use the second processor in the
dual  
core.    

With the new kernel release candidate 2.6.12-rc5 (which does use both cores of
the 
processor), the errors are rampant, and with other packages too.    

The errors seem to be random, except that they always are giving errors about
not 
finding the x86_64-pc-linux-gnu-gcc file (sometimes it is just gcc).   

Very weird, so it seems that this could be a strange 2.6.12-rc5 kernel bug?  
What to 
do? 

reassigning

In comment 6, I mean that the errors are generated on random files, but are
always 
about a missing compiler executable... 

Yes, and libkpimexchange-3.4.0 is another package.   
 
gcc-3.4.3-r1 is another. 
 
Maybe it is something to do with the build process in the packages that do not merge.  
Some packages emerge just fine, like openoffice-bin! 
 
How about better build process reports?  No problem, I'll give better attachments soon 
as this is a widespread problem.  I haven't informed anyone else of this bug yet. 

Did it on sys-libs/libieee1284-0.2.8 too. 
 
Then, I try again, and it works fine.  So it is random.  I've also seen it seg fault before, 
I'll keep trying to catch it, but will emerge --debug be enough? 

tried running gcc-config -l and select a new gcc profile ? 

Created an attachment (id=60496) [details]
emerge gettext also gives the error, this is one with emerge --debug

Still having problems.	Here is one with emerge --debug.  I'm surprised this
isn't happening to someone else!?

Yes, just tried with x86_64-pc-linux-gnu-3.4.3-hardened but it still says:     
   
   
x86_64-pc-linux-gnu-gcc: : No such file or directory   
  
I wonder why it didn't change the compiler?  
  
I ran:   gcc-config  x86_64-pc-linux-gnu-gcc-hardened 
 
emerge gives: 
 
make[3]: *** [argmatch.lo] Error 1   
make[3]: *** Waiting for unfinished jobs....   
 x86_64-pc-linux-gnu-gcc -DEXEEXT=\"\" -DDEPENDS_ON_LIBINTL=1   
-DDEPENDS_ON_LIBICONV=1 -DHAVE_CONFIG_H -DLIBDIR=\"/usr/lib64\" -I. -I. -I..   
-I. -I. -I.. -I../intl -I../intl -march=opteron -O2 -pipe -msse -msse2 -funit-at-a-time   
-fomit-frame-pointer -frename-registers -fweb -c allocsa.c  -fPIC -DPIC -o .libs/allocsa.o   
 x86_64-pc-linux-gnu-gcc -DEXEEXT=\"\" -DDEPENDS_ON_LIBINTL=1   
-DDEPENDS_ON_LIBICONV=1 -DHAVE_CONFIG_H -DLIBDIR=\"/usr/lib64\" -I. -I. -I..   
-I. -I. -I.. -I../intl -I../intl -march=opteron -O2 -pipe -msse -msse2 -funit-at-a-time   
-fomit-frame-pointer -frename-registers -fweb -c allocsa.c -o allocsa.o >/dev/null 2>&1   
make[3]: Leaving directory   
`/var/tmp/portage/gettext-0.14.1-r1/work/gettext-0.14.1/gettext-tools/lib'   
make[2]: *** [all-recursive] Error 1   
make[2]: Leaving directory   
`/var/tmp/portage/gettext-0.14.1-r1/work/gettext-0.14.1/gettext-tools'   
make[1]: *** [all] Error 2   
make[1]: Leaving directory   
`/var/tmp/portage/gettext-0.14.1-r1/work/gettext-0.14.1/gettext-tools'   
make: *** [all-recursive] Error 1   
+ diefunc src_compile 49 2   
+ local funcname=src_compile lineno=49 exitcode=2   
+ shift 3   
+ echo   
   

Created an attachment (id=60689) [details]
another example with emerge -debug, dual core, 2.6.12-rc5  (with different
error messages)

Here's full output from compiling sys-devel/gettext-0.14.1-r1.	This is another
completely different error, showing that this is a random event.


Once again:

x86_64-pc-linux-gnu-gcc: : No such file or directory
make[3]: *** [l10nflist.lo] Error 1

Here is some evidence of segfaults when trying to compile arts: 
 
 
---------------------------- 
from emerge.log 
---------------------------- 
Compiling/Merging (kde-base/arts-3.4.1::/usr/portage/kde-base/arts/arts-3.4.1.ebuild) 
 
----------------------------- 
from /var/log/messages 
----------------------------- 
 
Jun  7 01:03:03 optimator grep[9741]: segfault at 0000000000014000 rip 
00002aaaaad37130 rsp 00007fffffb119b8 error 4 
Jun  7 01:03:07 optimator sed[11775] general protection rip:4087aa rsp:7fffffb17860 
error:0 
Jun  7 01:03:47 optimator sed[16029] general protection rip:404c1f rsp:7ffffff17460 
error:0 
Jun  7 01:03:56 optimator egrep[19653]: segfault at 0000000000014000 rip 
00002aaaaad36b60 rsp 00007ffffff108c8 error 4 
Jun  7 01:03:58 optimator sed[20073] general protection rip:404c1f rsp:7ffffff17ba0 
error:0 
Jun  7 01:04:01 optimator rm[20641] general protection rip:2aaaaad38a90 
rsp:7fffffb06b08 error:0 
Jun  7 01:07:58 optimator sed[22817] general protection rip:404c1f rsp:7ffffff18050 
error:0 
Jun  7 01:07:59 optimator grep[23120]: segfault at 0000000000014000 rip 
00002aaaaad37130 rsp 00007fffffd11228 error 4 
Jun  7 01:08:02 optimator sed[24828] general protection rip:404c1f rsp:7fffff9187a0 
error:0 
Jun  7 01:08:03 optimator rm[24859] general protection rip:2aaaaad38a90 
rsp:7fffffd07478 error:0 
Jun  7 01:08:03 optimator sed[25036] general protection rip:404c1f rsp:7ffffff1a800 
error:0 
Jun  7 01:08:08 optimator x86_64-pc-linux[26625] trap stack segment rip:2aaaaabc30fb 
rsp:7fffffb03a80 error:0 
Jun  7 01:08:14 optimator sed[27649] general protection rip:404c1f rsp:7fffffb19d70 
error:0 
Jun  7 01:08:15 optimator sed[27793] general protection rip:404c1f rsp:7ffffff18470 
error:0 
Jun  7 01:08:16 optimator grep[27931]: segfault at 0000000000014000 rip 
00002aaaaad37130 rsp 00007fffff9121c8 error 4 
 

I don't know why, but this bug disappeared.  Maybe gcc or glibc wasn't compiled 
correctly or something.  Anyway, it quit happening, so if you want to close the
bug it is 
up to you. 

sounds fixed

I wouldn't say that it was "fixed"   
 
We never found out why this happened. 

and it's even harder to find out why if it's no longer reproducible...

ok, I won't call it "fixed", but it's not longer an issue and is not
recreatable

*** Bug 100994 has been marked as a duplicate of this bug. ***

I have this same problem on Dual-Opteron servers. I have verified that this 
happens on two different (but identical hardware) systems. It doesn't seem to
happen while emerging while booted under the liveCD, but after I reboot and
begin to emerge packages, this is triggered in random user programs. Usually 
things like mkdir, sed, awk, grep. Often the ebuild completes even though 
faults are in the log. Sometimes the emerge hangs, other times it fails.

Here are some lines from dmesg:
egrep[5380]: segfault at 0000000000000010 rip 00002aaaaac2ed50 rsp
00007fffff914348 error 4
...
Floca1284.exe[1553] general protection rip:2aaaaabc519b rsp:7fffffd2ce78 error:0
...
sed[17065] general protection rip:2aaaaaaac274 rsp:7ffffffbe440 error:0
...
sed[10679] general protection rip:2aaaaaaac274 rsp:7fffffdbe8a0 error:0


The egrep appears to have happened during boot (still booted though).

I'm running gentoo-sources 2.6.12-r9. I will try this with 2.6.9-r9 and see
if it goes away.

I will attach emerge info and lspci.

Created an attachment (id=66552) [details]
emerge info + lspci + /proc/cpuinfo

Correction, I was running 2.6.12-r6. I will try 2.6.9-r9. Then I will try
2.6.12-r9 and report on if either have the same issues. If all three do then I
will try vanilla sources.

Please include 2.6.13-rc6 in your tests

OK, well I'm definitely getting these with:
gentoo-sources-2.6.12-r6
gentoo-sources-2.6.12-r9
vanilla-sources-2.6.13-rc6

I don't seem to get it with 2.6.9-gentoo-r9. I'm going to reboot and try it for
a longer run.

Yep, apparently 2.6.9-gentoo-r9 does not have the problem.

Under 2.6.13-rc6 please enable:
Kernel Hacking --> Kernel Debugging --> kobject Debugging
Kernel Hacking --> Kernel Debugging --> Debug preemptible kernel
Kernel Hacking --> Kernel Debugging --> Debug memory allocations
Kernel Hacking --> Kernel Debugging --> Page alloc debugging

Hopefully this won't cause too much of a slowdown. Then please attach a full
dmesg output after a crash has occurred.

Since it appears to be a rare timing window, hopefully this won't make the
problem 
go away because of the slower execution.

I'm not running a preemptible kernel so the "Debug preemptible kernel" option
was not available. I also didn't find the "Page alloc debugging" option.

So I'm running right now with "kobject Debugging" and "Debug memory
allocations"
on vanilla-source 2.6.13-rc6. If I hit the problem, I will attach the results.

Happened again, but I didn't get any additional information.

emerge gcc crashed. No additional output in dmesg after boot messages:
....
xgcc[2492] general protection rip:403485 rsp:7ffffff19100 error:0

The option "Page alloc debugging" is available on x86 but not x86_64.

I'm having this exact same problem. I have dual single-core Opterons. 
linux-2.6.11-gentoo-r11 worked fine. When I upgraded to linux-2.6.12-gentoo-r6, 
I first encountered the problem. KDE packages (kdegraphics and kdenetwork to be 
specific, off the top of my head) would fail to compile in the exact manner 
described here. When I tried to recompile GCC, the compilation also crashed 
this way. I too found that booting from the Gentoo CD allowed me to chroot into 
my partition and emerge GCC just fine. I ended up downgrading back to 
linux-2.6.11-gentoo-r11, and the problem disappeared. Yesterday I upgraded to 
linux-2.6.12-gentoo-r9, and the bug showed back up. The last few lines of an 
attempted emerge of kdegraphics are as follows: 
 
/bin/sh ../../libtool --silent --tag=CXX --mode=compile x86_64-pc-linux-gnu-g++ 
-DHAVE_CONFIG_H -I. -I. -I../.. -I./.. -I./../cursors -I./../interfaces 
-I./../pixmapfx -I./../tools -I./../views -I./../widgets -I/usr/kde/3.4/include 
-I/usr/qt/3/include -I.   -DQT_THREAD_SUPPORT  -D_REENTRANT  -Wnon-virtual-dtor 
-Wno-long-long -Wundef -ansi -D_XOPEN_SOURCE=500 -D_BSD_SOURCE -Wcast-align 
-Wconversion -Wchar-subscripts -Wall -W -Wpointer-arith -Wwrite-strings 
-DNDEBUG -DNO_DEBUG -O2 -march=opteron -O3 -pipe -Wformat-security 
-Wmissing-format-attribute -fno-exceptions -fno-check-new -fno-common 
-DQT_CLEAN_NAMESPACE -DQT_NO_ASCII_CAST -DQT_NO_STL -DQT_NO_COMPAT 
-DQT_NO_TRANSLATION  -c -o kptoolairspray.lo kptoolairspray.cpp 
/bin/sh ../../libtool --silent --tag=CXX --mode=compile x86_64-pc-linux-gnu-g++ 
-DHAVE_CONFIG_H -I. -I. -I../.. -I./.. -I./../cursors -I./../interfaces 
-I./../pixmapfx -I./../tools -I./../views -I./../widgets -I/usr/kde/3.4/include 
-I/usr/qt/3/include -I.   -DQT_THREAD_SUPPORT  -D_REENTRANT  -Wnon-virtual-dtor 
-Wno-long-long -Wundef -ansi -D_XOPEN_SOURCE=500 -D_BSD_SOURCE -Wcast-align 
-Wconversion -Wchar-subscripts -Wall -W -Wpointer-arith -Wwrite-strings 
-DNDEBUG -DNO_DEBUG -O2 -march=opteron -O3 -pipe -Wformat-security 
-Wmissing-format-attribute -fno-exceptions -fno-check-new -fno-common 
-DQT_CLEAN_NAMESPACE -DQT_NO_ASCII_CAST -DQT_NO_STL -DQT_NO_COMPAT 
-DQT_NO_TRANSLATION  -c -o kptoolbrush.lo kptoolbrush.cpp 
/bin/sh ../../libtool --silent --tag=CXX --mode=compile x86_64-pc-linux-gnu-g++ 
-DHAVE_CONFIG_H -I. -I. -I../.. -I./.. -I./../cursors -I./../interfaces 
-I./../pixmapfx -I./../tools -I./../views -I./../widgets -I/usr/kde/3.4/include 
-I/usr/qt/3/include -I.   -DQT_THREAD_SUPPORT  -D_REENTRANT  -Wnon-virtual-dtor 
-Wno-long-long -Wundef -ansi -D_XOPEN_SOURCE=500 -D_BSD_SOURCE -Wcast-align 
-Wconversion -Wchar-subscripts -Wall -W -Wpointer-arith -Wwrite-strings 
-DNDEBUG -DNO_DEBUG -O2 -march=opteron -O3 -pipe -Wformat-security 
-Wmissing-format-attribute -fno-exceptions -fno-check-new -fno-common 
-DQT_CLEAN_NAMESPACE -DQT_NO_ASCII_CAST -DQT_NO_STL -DQT_NO_COMPAT 
-DQT_NO_TRANSLATION  -c -o kptoolcolorpicker.lo kptoolcolorpicker.cpp 
/bin/sh ../../libtool --silent --tag=CXX --mode=compile x86_64-pc-linux-gnu-g++ 
-DHAVE_CONFIG_H -I. -I. -I../.. -I./.. -I./../cursors -I./../interfaces 
-I./../pixmapfx -I./../tools -I./../views -I./../widgets -I/usr/kde/3.4/include 
-I/usr/qt/3/include -I.   -DQT_THREAD_SUPPORT  -D_REENTRANT  -Wnon-virtual-dtor 
-Wno-long-long -Wundef -ansi -D_XOPEN_SOURCE=500 -D_BSD_SOURCE -Wcast-align 
-Wconversion -Wchar-subscripts -Wall -W -Wpointer-arith -Wwrite-strings 
-DNDEBUG -DNO_DEBUG -O2 -march=opteron -O3 -pipe -Wformat-security 
-Wmissing-format-attribute -fno-exceptions -fno-check-new -fno-common 
-DQT_CLEAN_NAMESPACE -DQT_NO_ASCII_CAST -DQT_NO_STL -DQT_NO_COMPAT 
-DQT_NO_TRANSLATION  -c -o kptoolcolorwasher.lo kptoolcolorwasher.cpp 
/bin/sh ../../libtool --silent --tag=CXX --mode=compile x86_64-pc-linux-gnu-g++ 
-DHAVE_CONFIG_H -I. -I. -I../.. -I./.. -I./../cursors -I./../interfaces 
-I./../pixmapfx -I./../tools -I./../views -I./../widgets -I/usr/kde/3.4/include 
-I/usr/qt/3/include -I.   -DQT_THREAD_SUPPORT  -D_REENTRANT  -Wnon-virtual-dtor 
-Wno-long-long -Wundef -ansi -D_XOPEN_SOURCE=500 -D_BSD_SOURCE -Wcast-align 
-Wconversion -Wchar-subscripts -Wall -W -Wpointer-arith -Wwrite-strings 
-DNDEBUG -DNO_DEBUG -O2 -march=opteron -O3 -pipe -Wformat-security 
-Wmissing-format-attribute -fno-exceptions -fno-check-new -fno-common 
-DQT_CLEAN_NAMESPACE -DQT_NO_ASCII_CAST -DQT_NO_STL -DQT_NO_COMPAT 
-DQT_NO_TRANSLATION  -c -o kptoolcurve.lo kptoolcurve.cpp 
/bin/sh ../../libtool --silent --tag=CXX --mode=compile x86_64-pc-linux-gnu-g++ 
-DHAVE_CONFIG_H -I. -I. -I../.. -I./.. -I./../cursors -I./../interfaces 
-I./../pixmapfx -I./../tools -I./../views -I./../widgets -I/usr/kde/3.4/include 
-I/usr/qt/3/include -I.   -DQT_THREAD_SUPPORT  -D_REENTRANT  -Wnon-virtual-dtor 
-Wno-long-long -Wundef -ansi -D_XOPEN_SOURCE=500 -D_BSD_SOURCE -Wcast-align 
-Wconversion -Wchar-subscripts -Wall -W -Wpointer-arith -Wwrite-strings 
-DNDEBUG -DNO_DEBUG -O2 -march=opteron -O3 -pipe -Wformat-security 
-Wmissing-format-attribute -fno-exceptions -fno-check-new -fno-common 
-DQT_CLEAN_NAMESPACE -DQT_NO_ASCII_CAST -DQT_NO_STL -DQT_NO_COMPAT 
-DQT_NO_TRANSLATION  -c -o kptoolellipse.lo kptoolellipse.cpp 
/bin/sh ../../libtool --silent --tag=CXX --mode=compile x86_64-pc-linux-gnu-g++ 
-DHAVE_CONFIG_H -I. -I. -I../.. -I./.. -I./../cursors -I./../interfaces 
-I./../pixmapfx -I./../tools -I./../views -I./../widgets -I/usr/kde/3.4/include 
-I/usr/qt/3/include -I.   -DQT_THREAD_SUPPORT  -D_REENTRANT  -Wnon-virtual-dtor 
-Wno-long-long -Wundef -ansi -D_XOPEN_SOURCE=500 -D_BSD_SOURCE -Wcast-align 
-Wconversion -Wchar-subscripts -Wall -W -Wpointer-arith -Wwrite-strings 
-DNDEBUG -DNO_DEBUG -O2 -march=opteron -O3 -pipe -Wformat-security 
-Wmissing-format-attribute -fno-exceptions -fno-check-new -fno-common 
-DQT_CLEAN_NAMESPACE -DQT_NO_ASCII_CAST -DQT_NO_STL -DQT_NO_COMPAT 
-DQT_NO_TRANSLATION  -c -o kptooleraser.lo kptooleraser.cpp 
/bin/sh ../../libtool --silent --tag=CXX --mode=compile x86_64-pc-linux-gnu-g++ 
-DHAVE_CONFIG_H -I. -I. -I../.. -I./.. -I./../cursors -I./../interfaces 
-I./../pixmapfx -I./../tools -I./../views -I./../widgets -I/usr/kde/3.4/include 
-I/usr/qt/3/include -I.   -DQT_THREAD_SUPPORT  -D_REENTRANT  -Wnon-virtual-dtor 
-Wno-long-long -Wundef -ansi -D_XOPEN_SOURCE=500 -D_BSD_SOURCE -Wcast-align 
-Wconversion -Wchar-subscripts -Wall -W -Wpointer-arith -Wwrite-strings 
-DNDEBUG -DNO_DEBUG -O2 -march=opteron -O3 -pipe -Wformat-security 
-Wmissing-format-attribute -fno-exceptions -fno-check-new -fno-common 
-DQT_CLEAN_NAMESPACE -DQT_NO_ASCII_CAST -DQT_NO_STL -DQT_NO_COMPAT 
-DQT_NO_TRANSLATION  -c -o kptoolflip.lo kptoolflip.cpp 
x86_64-pc-linux-gnu-g++: : No such file or directory 
/bin/sh ../../libtool --silent --tag=CXX --mode=compile x86_64-pc-linux-gnu-g++ 
-DHAVE_CONFIG_H -I. -I. -I../.. -I./.. -I./../cursors -I./../interfaces 
-I./../pixmapfx -I./../tools -I./../views -I./../widgets -I/usr/kde/3.4/include 
-I/usr/qt/3/include -I.   -DQT_THREAD_SUPPORT  -D_REENTRANT  -Wnon-virtual-dtor 
-Wno-long-long -Wundef -ansi -D_XOPEN_SOURCE=500 -D_BSD_SOURCE -Wcast-align 
-Wconversion -Wchar-subscripts -Wall -W -Wpointer-arith -Wwrite-strings 
-DNDEBUG -DNO_DEBUG -O2 -march=opteron -O3 -pipe -Wformat-security 
-Wmissing-format-attribute -fno-exceptions -fno-check-new -fno-common 
-DQT_CLEAN_NAMESPACE -DQT_NO_ASCII_CAST -DQT_NO_STL -DQT_NO_COMPAT 
-DQT_NO_TRANSLATION  -c -o kptoolfloodfill.lo kptoolfloodfill.cpp 
make[3]: *** [kptooleraser.lo] Error 1 
make[3]: *** Waiting for unfinished jobs.... 
make[3]: Leaving directory 
`/var/tmp/portage/kdegraphics-3.4.1-r1/work/kdegraphics-3.4.1/kolourpaint/tools' 
make[2]: *** [all-recursive] Error 1 
make[2]: Leaving directory 
`/var/tmp/portage/kdegraphics-3.4.1-r1/work/kdegraphics-3.4.1/kolourpaint' 
make[1]: *** [all-recursive] Error 1 
make[1]: Leaving directory 
`/var/tmp/portage/kdegraphics-3.4.1-r1/work/kdegraphics-3.4.1' 
make: *** [all] Error 2 
 
!!! ERROR: kde-base/kdegraphics-3.4.1-r1 failed. 
!!! Function kde_src_compile, Line 170, Exitcode 2 
!!! died running emake, kde_src_compile:make 
!!! If you need support, post the topmost build error, NOT this status message. 
 
The place where the crash occurs is random. I'm fixing to have to downgrade to 
2.6.11 again (for normal use at least), but if there's something I can do to 
help out the cause, let me know. This annoying little bastard of a bug had me 
all but convinced that I already had serious hardware problems in my new 
computer. 

Something else that it occurs to me may or may not be of help is that the 
crashes seem to occur at roughly the same time for each package. I always run 
emerges through the time command, so I can see how long they take. kdegraphics 
always seems to go down between 4-5 minutes in, although last night I think it 
lasted 6 minutes and some seconds (then again, last night was the only time 
I've tried with -r9 instead of -r6). GCC always lasts like 30-40 minutes before 
going down (at least on -r6). I don't know what to make of that, but it's a 
fact that's there, and sure as heck doesn't seem like random chance. 

Is this reproducible on gentoo-sources-2.6.13?

is this happening with a 64bit kernel only?

I don't know about the 64-bit only thing, but I can confirm that it does happen 
in 2.6.13 also. I had to try to emerge kdegraphics from a virtual console, 
since X wouldn't start, but indeed, the emerge failed a hair under four minutes 
in. 

================================ 
To those wanting to fix this bug: 
================================ 
It seems to me that there are now a significant amount of reports and information to 
proceed here.  How can we figure out where the seg faults are coming from.  Are cores 
dumped in any of the machines where we are currently getting this bug?  Obviously the 
seg faults will contain the source of the bug. 
 
================================ 
To those wanting to fix their machines: 
================================ 
As I indicated earlier in Comment 16, after recompiling the whole system a few times, 
(emerge --empty-tree system), things worked fine.  So, maybe it has to do with a 
misinstallation of one of the packages or there is something wrong with one of the 
2005.0 amd64 system packages? 
 
I'd try re-emerging glibc and gcc first.  I think that is what I did to get the whole system 
to be able to be compiled.   
 
Also, I remember that I had to remove the cached code compilations because they 
would be sort of "tainted" after being compiled with the older compiler/libraries.  In other 
words, it seemed to help to remove everything before after I had upgraded glibc  and 
gcc. 
 
I removed the cached files by "rm -rf /var/tmp/portage/*" and disabled ccache.  Maybe 
that wasn't a good command to use on my /var/tmp/portage directory, so make sure 
you know what you are doing before you do that. 
 
Hope this helps someone as this is a hard bug to reproduce and figure out where it is 
coming from.  

Well, I don't exactly want to hose or otherwise seriously screw up my system 
hunting this down. If there's any information left after a failed emerge that I 
could provide, just tell me where it's at. Doing an emerge system is tempting, 
but that wouldn't find the problem nor fix it for anyone else, it would just be 
a workaround (maybe the problem has been fixed since I installed my system, but 
that was only earlier in the summer, and there's no way to know until we find 
what the problem is). 

is anybody still experiencing this with the latest stable (2.6.13-r5) or the
latest testing (2.6.14-r2)?

(In reply to comment #38) 
> is anybody still experiencing this with the latest stable (2.6.13-r5) or the 
> latest testing (2.6.14-r2)? 
 
I upgraded from 2.6.11-r11 to 2.6.14-r2 the other day, and haven't experienced 
it. I did have the problem with 2.6.13-rsomething though. I don't know if 
2.6.14-r2 has actually fixed the problem, or if it was another package, since I 
wasn't updating anything as long as it insisted on emerging another 2.6.13 
kernel. If the problem really lies with another package, it could still be 
busted, and maybe it'll reappear in a later kernel version (or maybe even in 
the future with this one). Who knows, I just know that at the moment I can 
actually emerge things, although I haven't emerged anything too big lately, 
which is when the problem used to occur (KDE packages and GCC). 

thanks for the fast response. closing, as it seems fixed (dual-core got much
more popular in the past few weeks, that might be another reason for it beeing
fixed). if anybody still has these strange errors, feel free to reopen