Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 63734 - Threading issues on an NPTL enabled system
Summary: Threading issues on an NPTL enabled system
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: [OLD] Core system (show other bugs)
Hardware: All All
: High normal (vote)
Assignee: dotnet project
: 54603 63713 (view as bug list)
Depends on:
Reported: 2004-09-11 20:11 UTC by Brandon Hale (RETIRED)
Modified: 2005-02-25 22:16 UTC (History)
12 users (show)

See Also:
Package list:
Runtime testing required: ---


Note You need to log in before you can comment on or make changes to this bug.
Description Brandon Hale (RETIRED) gentoo-dev 2004-09-11 20:11:41 UTC
There appears to be an issue with threading on nptl enabled systems.
So far this is mainly demonstrated by Mono applications, but could potentially
affect other threaded apps as well. Im posting this as a new bug, as the
original bug focused (incorrectly) on Mono. Here's a fresh start.

Here is a sample program to better demonstrate the issue submitted by Canal Vorfeed on bug #54603. Here's Canal:


Ok, I've spent few hours on this issue and found where REAL problem lies.

Good news: it's NOT Boehm's GC and it's NOT mono.

Bad news: it's problem with glibc itself :-(

I've started with "GC is broken with nptl" sample by "Peter Johanson" and played with it for few hours. In the end I've just removed Boehm's GC completely (just plain old malloc) and... it still deadlocks somewhere.

So we should stop playing with mono and try to address REAL issue: deadlocks somewhere in nptl library itself :-( Unfortunatelly I'm not glibc guru.


#include <pthread.h>
#include <stdio.h>

void *thread_function (void *args)
        int j = 0;
        char *str;
        printf("starting thread!\n");
        for (j; j < 24; j++)
                str = (char *)malloc(240);
                printf("malloc in thread !\n");

int main (void)
        int i;
        pthread_t thread;
        char *str;
        for (i=0;i<1000;i++)
                pthread_create( &thread, NULL, thread_function, (void *)i);
                str = (char *)malloc(240);
                printf("%d threads\n", i);

$ ./testpgm | grep 'malloc in thread' | wc -l

Without ntpl I've got expected 24000 ...


Nice work tracking this down, let's have the toolchain team take a look
at this or try and push it upstream.
Comment 1 Brandon Hale (RETIRED) gentoo-dev 2004-09-11 20:12:38 UTC
*** Bug 54603 has been marked as a duplicate of this bug. ***
Comment 2 Canal Vorfeed 2004-09-11 21:41:35 UTC
*** Bug 63713 has been marked as a duplicate of this bug. ***
Comment 3 Canal Vorfeed 2004-09-12 08:01:06 UTC
Just a side comment: it does not look like a deadlock problem after all: it's just glibc/nptl can not create more the 384 threads (even if THREADS_MAX in glibc is 100000 and /proc/sys/kernel/threads-max is 14336). Since LinuxThreads CAN create requested 1000 threads on the same system (in fact it's not even two systems: I have Gentoo with glibc/LinuxThreads installed in chroot jail) it's not kernel issue.

Not sure about mono: can it actually hit this limit in real world applications ? If the answer is no then mono bug should be reopened...
Comment 4 Canal Vorfeed 2004-09-12 09:52:00 UTC
Oops. Of course even if it's not deadlock condition it still CAN affect mono (and a lot of other programs): it's not 384 simultaneous threads but rather 384 threads for lifetime of program!

#include <pthread.h>
#include <stdio.h>
#include <time.h>

void *thread_function (void *args)

int main (void)
        int i;
        pthread_t thread;
        char *str;
        for (i=0;i<1000;i++)
                if (pthread_create( &thread, NULL, thread_function, (void *)i)) {
                } else {
//                      int status;
//                      pthread_join(thread, (void **)&status);
                        printf("%d threads\n", i);
$ ./testpgm
1 threads
2 threads
380 threads
381 threads
testpgm: Cannot allocate memory
testpgm: Cannot allocate memory
testpgm: Cannot allocate memory
testpgm: Cannot allocate memory

If you'll uncomment pthread_join program will work Ok but it's not Ok not not wait for thread stop with pthreads, right ?

P.S. sleep is there to make sure no race conditions exist at all: thread is created and executed and then other will be started. You can remove it from sample - it does not change anything...
Comment 5 Scott Mohekey 2004-09-12 14:45:59 UTC
I've tried both these test programs on my system (which is NPTL enabled, and exhibits the referenced mono 'bug'). Surprisingly, they both work fine, easily reaching the end of the for loop. I attempted to replace the for loop with a while( 1 ), and they both sat there quite happliy chugging into the tens of thousands of threads created. Also, it should be mentioned that in all cases that I've seen this bug exhibited (in the C# thread test posted elsewhere, and in mono's xsp/mod-mono-server) the threads numbered in the 10s, and each of them was in one of the various wait functions (pthread_cond_timedwait, wait_sem, etc..).

In mono's case at least, the bug can be 'resolved' by disabling the garbage collector (libgc), obviously this isn't a viable solution. On the mono bugzilla (, I've mentioned that the freeze occurs when the garbage collector initiates a world stop (stops all threads except the current) and begins a full garbage collection. However, for some reason the thread performing the garbage collection then also waits.

When running the various test programs in gdb, I found that if I put a breakpoint in the libgc code that performed the garbage collection, the breakpoint would be reached repeatedly, however there was a period of one or two seconds in between each occurance. This occured when it would normally freeze completely. I'm not one hundred percent up on my unix signal handling etc., but it seems as if the breakpoint occuring may have altered some of the processes signals, perhaps waking one of the other threads, and therefore alleviating the freeze.
Comment 6 Canal Vorfeed 2004-09-12 16:39:15 UTC
Perhaps some details are different enough to see difference in behaviour.

I've tried to recompile system with gcc 3.3.4-r1 but it made no difference...

Anyway: looks like there are something goes on with thread synchronisation: in my case with pthread_join I can avoid problems in mono case GC trying to freeze all threads to safely do garbage collectiong. I think real problem is in nptl resources allocation: from outside it looks like there are something wrong with mutexes or something and sometimes nptl just can not do the right thing. Since "normal" programs (like MySQL or Apache) behave quite predictable in regard to threads communications (threads are created and work on separate task - sometimes with something like "manager" thread but never in "normal" programs some random thread will try to communicate with some other equally random thread!) it does not trigger bug in nptl. But GC can stop all threads to do it's work literally at ANY TIME and thus it triggers bug from time to time...

For reference:
Portage 2.0.50-r11 (gcc34-x86-2004.2, gcc-3.4.1, glibc-, 2.6.9-rc1-mm4)
System uname: 2.6.9-rc1-mm4 i686 Intel(R) Pentium(R) 4 CPU 3.00GHz
Gentoo Base System version 1.5.3
Autoconf: sys-devel/autoconf-2.59-r4
Automake: sys-devel/automake-1.8.5-r1
CFLAGS="-O2 -pipe -march=pentium4 -funroll-loops -ffast-math -fomit-frame-pointer -ffloat-store -fforce-addr -ftracer -mmmx -msse -msse2 -mfpmath=sse"
CONFIG_PROTECT="/etc /usr/X11R6/lib/X11/xkb /usr/kde/2/share/config /usr/kde/3.3/env /usr/kde/3.3/share/config /usr/kde/3.3/shutdown /usr/kde/3/share/config /usr/share/config /usr/share/texmf/dvipdfm/config/ /usr/share/texmf/dvips/config/ /usr/share/texmf/tex/generic/config/ /usr/share/texmf/tex/platex/config/ /usr/share/texmf/xdvi/ /var/qmail/control"
CONFIG_PROTECT_MASK="/etc/afs/C /etc/afs/afsws /etc/gconf /etc/terminfo /etc/env.d"
CXXFLAGS="-O2 -pipe -march=pentium4 -funroll-loops -ffast-math -fomit-frame-pointer -ffloat-store -fforce-addr -ftracer -mmmx -msse -msse2 -mfpmath=sse"
FEATURES="autoaddcvs ccache sandbox"
USE="X X509 Xaw3d aalib acl afs alsa apm arts avi berkdb bitmap-fonts bzlib calendar caps cjk crypt cups curl dga directfb djbfft doc emacs encode esd f77 fam fbcon fdftk firebird foomaticdb ftp gcj gd gd-external gdbm ggi gif gmp gnome gpm gtk gtk2 guile iconv imap imlib immqt immqt-bc ipv6 jack java javamail javascript jikes jpeg junit kde kerberos krb4 ldap libcaca libg++ libwww mad makecheck memlimit mhash mikmod mime ming mmx motif mozilla mpeg mysql nas ncurses nls nptl objc odbc oggvorbis opengl oss pam pcre pdflib perl php pic png postgres pwdb python qdbm qt quicktime readline ruby samba sasl sdl session skey slang slp snmp socks5 spell sqlite sse sse2 ssl svga tcltk tcpd tetex threads tiff truetype unicode x86 xinerama xml xml2 xmlrpc xmms xpm xprint xsl xv zlib"
Comment 7 solar (RETIRED) gentoo-dev 2004-09-13 08:59:53 UTC
This bug is probably a dup/dep of bug #45115
Comment 8 Peter Johanson (RETIRED) gentoo-dev 2004-09-13 09:14:32 UTC
@solar: I fail to see how a compile time bug about g++ not liking one of the pthread.h macros is related to this runtime threading problem. Care to elaborate?
Comment 9 solar (RETIRED) gentoo-dev 2004-09-13 09:37:05 UTC
"Care to elaborate?" Sorry not really. 

I'm not a nptl fan. If you/others want to try to solve this bug then I'm
simply suggesting to take a peek at that bug which may or may not be
related to the problems your having here. Either way if your using ntpl
then that bug concerns you.
Comment 10 Travis Tilley (RETIRED) gentoo-dev 2004-09-16 21:49:19 UTC
can someone here please test glibc
Comment 11 The Master 2004-09-18 20:41:34 UTC
Muine (a mono app) appears to still lock up under glibc
Comment 12 Disenchanted (RETIRED) gentoo-dev 2004-09-19 00:13:45 UTC
reaching 102 threads with latest glibc in portage 0918

adding as much info as i can think off
mono apps seem happier, since it is 1020 i will try to see if they lock up how they used to in past, compiled mono -r2 with nptl after remerging 0918 with nptl

ok, here all info

first, while i wrtoe the above, muine must have reached 1020 threads as it locked up now in the end

nullzone ~ # /lib/
GNU C Library 20040918 release version 2.3.4, by Roland McGrath et al.
Copyright (C) 2004 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
Compiled by GNU CC version 3.4.2  (Gentoo Linux 3.4.2-r1, ssp-3.4.1-1, pie-
Compiled on a Linux 2.6.7 system on 2004-09-19.
Available extensions:
        GNU libio by Per Bothner
        crypt add-on version 2.1 by Michael Glad and others
        Native POSIX Threads Library by Ulrich Drepper et al
        NIS(YP)/NIS+ NSS modules 0.19 by Thorsten Kukuk
Thread-local storage support included.
For bug reporting instructions, please see:
nullzone ~ #

nullzone ~ # emerge info
Portage 2.0.51_rc1 (default-linux/x86/2004.2/gcc34/2.6, gcc-3.4.2, glibc-, 2.6.9-rc1 i686)
System uname: 2.6.9-rc1 i686 AMD Athlon(tm) XP 3200+
Gentoo Base System version 1.5.3
ccache version 2.3 [enabled]
Autoconf: sys-devel/autoconf-2.59-r4
Automake: sys-devel/automake-1.8.5-r1
Binutils: sys-devel/binutils-
Headers:  sys-kernel/linux26-headers-2.6.7-r4
Libtools: sys-devel/libtool-1.5.2-r5
CFLAGS="-O2 -march=athlon-xp -pipe -fomit-frame-pointer"
CONFIG_PROTECT="/etc /usr/X11R6/lib/X11/xkb /usr/kde/2/share/config /usr/kde/3.2/share/config /usr/kde/3.3/share/config:/usr/kde/3.3/env:/usr/kde/3.3/shutdown /usr/kde/3/share/config /usr/lib/mozilla/defaults/pref /usr/share/config /usr/share/texmf/dvipdfm/config/ /usr/share/texmf/dvips/config/ /usr/share/texmf/tex/generic/config/ /usr/share/texmf/tex/platex/config/ /usr/share/texmf/xdvi/ /var/qmail/control"
CONFIG_PROTECT_MASK="/etc/gconf /etc/terminfo /etc/env.d"
CXXFLAGS="-O2 -march=athlon-xp -pipe -fomit-frame-pointer"
FEATURES="autoaddcvs ccache cvs sandbox sfperms"

i know longer till freezing of mono apps != fix but goes in right direction

just a fyi for all
1020 threads
starting thread!
GC_ALLOC in thread !
GC_ALLOC in thread !
GC_ALLOC in thread !
GC_ALLOC in thread !
GC_ALLOC in thread !
GC_ALLOC in thread !
GC_ALLOC in thread !
GC_ALLOC in thread !
GC_ALLOC in thread !
GC_ALLOC in thread !
GC_ALLOC in thread !
GC_ALLOC in thread !
GC_ALLOC in thread !
GC_ALLOC in thread !
GC_ALLOC in thread !
GC_ALLOC in thread !
GC_ALLOC in thread !
GC_ALLOC in thread !
GC_ALLOC in thread !
GC_ALLOC in thread !
GC_ALLOC in thread !
GC_ALLOC in thread !
GC_ALLOC in thread !
GC_ALLOC in thread !
1021 threads
1022 threads
1023 threads
1024 threads
then the usual endless list
Comment 13 Disenchanted (RETIRED) gentoo-dev 2004-09-19 00:17:03 UTC
s/102/1020/ sorry
Comment 14 Scott Mohekey 2004-09-19 15:19:37 UTC
The more I've looked into this, the more apparant it is that the issue is not that there is a bug in NPTL, but instead there are some applications (mono and libgc in particular) that are developed expecting certain behaviour in LinuxThreads that is non standard, or erroneous at best.
Comment 15 Defresne Sylvain (keiichi) 2004-09-20 14:04:23 UTC
I've straced program given in comment #4 on my nptl system and got :
$ strace ./testpgm2
... ... ...
30236 write(1, "253 threads\n", 12)     = 12
30236 mmap2(NULL, 8392704, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xbf25b000
30236 mprotect(0xbf25b000, 4096, PROT_NONE) = 0
30236 clone(child_stack=0xbfa5bb28, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID|CLONE_DETACHED, parent_tidptr=0xbfa5bbf8, {entry_number:6, base_addr:0xbfa5bbb0, limit:1048575, seg_32bit:1, contents:0, read_exec_only:0, limit_in_pages:1, seg_not_present:0, useable:1}, child_tidptr=0xbfa5bbf8) = 30491
30236 sched_yield( <unfinished ...>
30491 _exit(0)                          = ?
30236 <... sched_yield resumed> )       = 0
30236 write(1, "254 threads\n", 12)     = 12
30236 mmap2(NULL, 8392704, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory)
30236 dup(2)                            = 3
30236 fcntl64(3, F_GETFL)               = 0x2 (flags O_RDWR)
30236 fstat64(3, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 5), ...}) = 0
30236 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xbfa5c000
30236 _llseek(3, 0, 0xbffff358, SEEK_CUR) = -1 ESPIPE (Illegal seek)
30236 write(3, "nptl: Cannot allocate memory\n", 29) = 29
30236 close(3)                          = 0
... ... ...

So the 255-th pthread_create seems to fails because the mmap of 8M for
the thread stack fails. I thus tried to lower the maximum stack size to
1M (ulimit -s 1024), and the program terminate correctly as the kernel
is able to allocate all requested memory for kernel stacks (well it
pretend to because I don't have 1G of memory :).

Doing so little math, it appears the problem comes from the very large
default maximum stack size used. With a 8M stack size, and a 2G/2G split
(which seems to be how I compiled my kernel), a single process can not
map more than 2G of memory, that is 256 * 8M ... The original poster
seems to have a kernel compiled with a 3G/1G split which allow a single
process to map 3G (~ 380 * 8M) of memory ...

This is the same problem that affect the original problem. By reducing
the stack size, i get the awaited 24000 answer:
$ ulimit -s 8192
$ ./testpgm | grep 'malloc in thread' | wc -l
$ ulimit -s 512
$ testpgm | grep 'malloc in thread' | wc -l

So the failure of those sample programs is simply caused by memory exhaustion,
and this is no bug in glibc (maybe there is a problem for the very large
default stack size limit but that is another issue) and it is something
else that is causing the problem with glibc (after reading the different
links, it seems to come from difference in semantic from LinuxThreads and
NPTL, but its just a guess) ...
Comment 16 Canal Vorfeed 2004-09-21 03:40:33 UTC
Yes, problem is with "memory exhaustion", true. But why the hell memory is exhausted in the first place ?

You are missing fundamental fact: program from comment #4 WILL NEVER CREATE MORE THEN TWO THREADS! sleep is there for a reason: it's GUARANTEE that this program will never create more then two threads on any sane system.

If glibc is 100% sure every stack for every dead thread should be kept around forever then you are 100% correct. But it's just stupid: why all this dead memory is kept around forever ? I can see ABSOLUTELY NO REASON to keep stack for every thread which ever existed in program allocated forever. More: if you'll add pthread_join all stacks will be correctly deallocated. To me it looks like some strange error with nptl locking logic and NOT as normal behaviour.

If you'll think about this:
you'll see it's EXACTLY the same problem: thread is gone (and there are NO associated kernel process in memory - check with /proc !) but half-dead zombie is still in memory so no cleanup...
Comment 17 Defresne Sylvain (keiichi) 2004-09-21 05:44:39 UTC
From pthread_join(3) man page:

    When a joinable thread terminates, its memory resources (thread
    descriptor and stack) are not deallocated until another thread
    performs pthread_join on it. Therefore, pthread_join must be
    called once for each joinable thread created to avoid memory

And since pthread_create create thread in joinable state by default
(that is the case since the 'pthread_attr_t *' parameter is NULL)
and they are not detached by call to pthread_detach(3), the behaviour
is correct. The bug is in the program posted in comment #4, not the
glibc !

If you want the thread memory (stack, ...) to be automatically freeed
by the libc when the thread exit, you must create it in detached
state or call pthred_detach(3). Otherwise, you must call pthread_join(3).

I think the rationale for keeping thread memory is to be able to get
the return value from the thread when calling pthread_join. But in all
case, there is no bug since it is the behaviour POSIX seems to specify.
Comment 18 Ed Catmur 2004-09-27 17:59:33 UTC
As Defresne Sylvain pointed out this is a red herring. See bug 54603 and upstream for the fix.
Comment 19 Paul de Vrieze (RETIRED) gentoo-dev 2004-09-28 00:57:47 UTC
That fix is actually a workaround. It seems to be a gcc/glibc issue:
Basically there is a threading/exception problem caused by either of them.
Comment 20 Ed Catmur 2004-09-28 11:40:22 UTC
True, I guess, the underlying problem is that gcc -fexceptions is incompatible with glibc+NPTL. However mono doesn't need -fexceptions (it's pure C, -fexceptions is for mixed C/C++ code) so removing it will fix the mono build. 

If you want to try to fix the gcc/-fexceptions/glibc/nptl incompatibility then I think that would best be done on a separate bug, or upstream.
Comment 21 Peter Johanson (RETIRED) gentoo-dev 2004-10-29 10:10:54 UTC
Okay, I've just commited a change to the package.masked mono-1.0.2-r1 which adds the -fexpections fix as posted on the ximian bug. Still requires gcc-3.4 unfortunately.

I'd like to get some wide testing on that, and then un package.mask that version, so it is only ~x86. At that point I want to get the mono-1.0.2 and friends (gtk-sharp, etc). So that people with NPTL systems will only have to deal with marking a few select ebuilds ~x86 to have it functioning, and every one else can have a stable 1.0 mono.

TEST! (/me gets on hands and knees and begs)
Comment 22 pasi harkone 2004-11-02 10:52:37 UTC
Seems to be ok for me with mono 1.0.2-r1+muine 0.6.3
Comment 23 Gábor Farkas 2004-11-02 11:30:46 UTC
works fine with muine-from-cvs,
Comment 24 Peter Johanson (RETIRED) gentoo-dev 2005-02-18 11:19:02 UTC
Ok, one last *MAJOR* test request:

I've just commited mono-1.0.6 to the tree, but i've added it to package.mask as well.

I've done some testing with both muine, and BLAM! on this, and I'm 90% sure we're ok with NPTL *without* using gcc-3.4! The new version deps on gcc-3.3.5, which i believe may have been the piece that contained the fix. "may" being the operative word.

So I need some testing from people using gcc-3.3.5, as if this is the case, one of the major things holding up a mono-1.0.x marked stable will disappear. Thanks all for persevering.
Comment 25 Peter Johanson (RETIRED) gentoo-dev 2005-02-21 12:01:10 UTC
1.0.6 actually seems to have some threading issues unrelated to the NPTL issue. I've commited mono-1.0.5-r4, which includes these changes for the 1.0.5 series, and is working here and for one other user wonderfully. *PLEASE* test this, as it's the most realistic target for stabalization, given the flakiness of 1.0.6. Marking this TEST-REQUEST.
Comment 26 Jung Tae-young 2005-02-25 21:37:02 UTC
I build mono-1.0.6 with gcc-3.3.5 & glibc- (nptl only glibc)
it works fine for me..

muine doesn't hang any more :)

cafri ~ # emerge -pv glibc mono =sys-devel/gcc-3.3.5-r1

These are the packages that I would merge, in order:

Calculating dependencies ...done!
[ebuild   R   ] sys-libs/glibc-  -build -debug -erandom -hardened 
(-multilib) +nls -nomalloccheck +nptl +nptlonly +pic -userlocales 0 kB 
[ebuild   R   ] dev-dotnet/mono-1.0.6  -debug +nptl 0 kB 
[ebuild   R   ] sys-devel/gcc-3.3.5-r1  -bootstrap -boundschecking -build -debug
 -fortran -gcj -gtk -hardened (-ip28) (-multilib) -multislot (-n32) (-n64) +nls 
-nocxx +objc* -static (-uclibc) 0 kB 

Total size of downloads: 0 kB
cafri ~ # gcc -v
Reading specs from /usr/lib/gcc-lib/i686-pc-linux-gnu/3.3.5/specs
Configured with: /var/tmp/portage/gcc-3.3.5-r1/work/gcc-3.3.5/configure --enable
-version-specific-runtime-libs --prefix=/usr --bindir=/usr/i686-pc-linux-gnu/gcc
-bin/3.3.5 --includedir=/usr/lib/gcc-lib/i686-pc-linux-gnu/3.3.5/include --datad
ir=/usr/share/gcc-data/i686-pc-linux-gnu/3.3.5 --mandir=/usr/share/gcc-data/i686
-pc-linux-gnu/3.3.5/man --infodir=/usr/share/gcc-data/i686-pc-linux-gnu/3.3.5/in
fo --with-gxx-include-dir=/usr/lib/gcc-lib/i686-pc-linux-gnu/3.3.5/include/g++-v
3 --host=i686-pc-linux-gnu --disable-altivec --enable-nls --without-included-get
text --enable-__cxa_atexit --enable-clocale=gnu --with-system-zlib --disable-che
cking --disable-werror --disable-libunwind-exceptions --enable-shared --enable-t
hreads=posix --disable-multilib --disable-libgcj --enable-languages=c,c++
Thread model: posix
gcc version 3.3.5  (Gentoo Linux 3.3.5-r1, ssp-3.3.2-3, pie-

cafri ~ # gcc-config -l
[1] i686-pc-linux-gnu-3.3.5 *
[2] i686-pc-linux-gnu-3.3.5-hardened
[3] i686-pc-linux-gnu-3.3.5-hardenednopie
[4] i686-pc-linux-gnu-3.3.5-hardenednossp
[5] i686-pc-linux-gnu-3.4.3
[6] i686-pc-linux-gnu-3.4.3-hardened
[7] i686-pc-linux-gnu-3.4.3-hardenednopie
[8] i686-pc-linux-gnu-3.4.3-hardenednossp