Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 272388 - net-libs/gnutls-2.8.0: gnutls_global_init crashes after earlier init/deinit cycle
Summary: net-libs/gnutls-2.8.0: gnutls_global_init crashes after earlier init/deinit c...
Status: RESOLVED FIXED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: New packages (show other bugs)
Hardware: All Linux
: High normal (vote)
Assignee: Crypto team [DISABLED]
URL:
Whiteboard:
Keywords:
: 260630 (view as bug list)
Depends on:
Blocks:
 
Reported: 2009-06-03 08:59 UTC by Stephan Friedrichs
Modified: 2009-06-10 17:20 UTC (History)
21 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
paludis --info mozilla-firefox (p,14.13 KB, text/plain)
2009-06-03 09:01 UTC, Stephan Friedrichs
Details
valgrind output (valgrind.log,9.12 KB, text/plain)
2009-06-07 15:34 UTC, Philip L
Details
gnutls-2.8.1_pre20090608.ebuild (gnutls-2.8.1_pre20090608.ebuild,2.37 KB, text/plain)
2009-06-08 17:45 UTC, Arfrever Frehtes Taifersar Arahesis (RETIRED)
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Stephan Friedrichs 2009-06-03 08:59:53 UTC
mozilla-firefox-3.0.10 occasionally crashes on my ~amd64 machine (sometimes after clicking on a link sometimes after just moving the mouse over something, I can't tell that precisely, yet) and gives the following output:

*** glibc detected *** /usr/lib64/mozilla-firefox/firefox: corrupted double-linked list: 0x0000000003515ce0 ***
======= Backtrace: =========
/lib/libc.so.6[0x7fb34f35a258]
/lib/libc.so.6[0x7fb34f35a560]
/lib/libc.so.6[0x7fb34f35cd97]
/lib/libc.so.6(__libc_malloc+0x6a)[0x7fb34f35edea]
/usr/lib64/xulrunner-1.9/libxul.so[0x7fb34dfdfd04]
/usr/lib64/xulrunner-1.9/libxul.so[0x7fb34d8f0f9d]
/usr/lib64/xulrunner-1.9/libxul.so[0x7fb34d8ff148]
/usr/lib64/xulrunner-1.9/libmozjs.so[0x7fb34ee15f6b]
/usr/lib64/xulrunner-1.9/libmozjs.so[0x7fb34edffbe5]
/usr/lib64/xulrunner-1.9/libxul.so[0x7fb34d8e02a0]
/usr/lib64/xulrunner-1.9/libxul.so[0x7fb34e014236]
/usr/lib64/xulrunner-1.9/libxul.so[0x7fb34dc64c71]
/usr/lib64/xulrunner-1.9/libxul.so[0x7fb34dc6501c]
/usr/lib64/xulrunner-1.9/libxul.so[0x7fb34dfe92ec]
/usr/lib64/xulrunner-1.9/libxul.so[0x7fb34dfe95d2]
/usr/lib64/xulrunner-1.9/libxul.so[0x7fb34db9ed5a]
/usr/lib64/xulrunner-1.9/libxul.so[0x7fb34e00ca78]
/usr/lib64/xulrunner-1.9/libxul.so[0x7fb34e00cad6]
/usr/lib64/xulrunner-1.9/libxul.so[0x7fb34e00abbc]
/usr/lib64/xulrunner-1.9/libxul.so[0x7fb34dfe0d6e]
/usr/lib64/xulrunner-1.9/libxul.so[0x7fb34df65ad9]
/usr/lib64/xulrunner-1.9/libxul.so[0x7fb34de490e9]
/usr/lib64/xulrunner-1.9/libxul.so(XRE_main+0x1d1f)[0x7fb34d8d5a72]
/usr/lib64/mozilla-firefox/firefox[0x401708]
/lib/libc.so.6(__libc_start_main+0xe6)[0x7fb34f304a26]
/usr/lib64/mozilla-firefox/firefox[0x401249]

I've glibc-2.10.0 installed.

Reproducible: Sometimes

Steps to Reproduce:
1. start firefox
2. use it for a couple of minutes
Comment 1 Stephan Friedrichs 2009-06-03 09:01:40 UTC
Created attachment 193383 [details]
paludis --info mozilla-firefox
Comment 2 Fadi Adlouni 2009-06-03 15:24:46 UTC
i have the same on my ~x86. and although i did emerge -uDN world recently, but the only major component that got upgraded was glibc from 2.9.something to 2.10.1 .
in fact i had several crashes. sometimes nothing shows up on terminal. sometimes i get a double free crash, once i got a double-lined list crash and once i got an assert. i'm not sure if i should open seperate bugs for the other crashes or they have the same root cause as this one. below i past each one:
-double free crash
*** glibc detected *** <link:url>/usr/lib/mozilla-firefox/firefox</link:url>: double free or corruption (!prev): 0x09010490 ***
======= Backtrace: =========
<link:url>/lib/libc.so.6[0xb7d5d7c4</link:url>]
<link:url>/lib/libc.so.6[0xb7d5eea3</link:url>]
<link:url>/lib/libc.so.6[0xb7d62b1b</link:url>]
<link:url>/lib/libc.so.6(realloc+0xdd)[0xb7d62fbd</link:url>]
<link:url>/usr/lib/libgnutls.so.26(gnutls_ext_register+0x34)[0xb5aacc64</link:url>]
<link:url>/opt/netscape/plugins/libflashplayer.so[0xaf93db4d</link:url>]
<link:url>/opt/netscape/plugins/libflashplayer.so[0xafb06ae0</link:url>]
<link:url>/opt/netscape/plugins/libflashplayer.so[0xaf7c49b6</link:url>]
<link:url>/opt/netscape/plugins/libflashplayer.so[0xaf7b736c</link:url>]
<link:url>/opt/netscape/plugins/libflashplayer.so[0xaf7a8649</link:url>]
<link:url>/opt/netscape/plugins/libflashplayer.so[0xaf7acd49</link:url>]
<link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb76f2f0b</link:url>]
<link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb7702406</link:url>]
<link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb76f7bed</link:url>]
<link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb7701df6</link:url>]
<link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb715d80b</link:url>]
<link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb7161597</link:url>]
<link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb72b6b2e</link:url>]
<link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb72b6c49</link:url>]
<link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb72b77d6</link:url>]
<link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb72b8086</link:url>]
<link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb733c486</link:url>]
<link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb7302e60</link:url>]
<link:url>/usr/lib/xulrunner-1.9/libxul.so(NS_InvokeByIndex_P+0x29)[0xb78eb329</link:url>]
<link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb6f6b461</link:url>]
<link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb6f743ca</link:url>]
<link:url>/usr/lib/xulrunner-1.9/libmozjs.so(js_Invoke+0x464)[0xb7c1bbc4</link:url>]
<link:url>/usr/lib/xulrunner-1.9/libmozjs.so[0xb7c0ff97</link:url>]
<link:url>/usr/lib/xulrunner-1.9/libmozjs.so(js_Invoke+0x741)[0xb7c1bea1</link:url>]
<link:url>/usr/lib/xulrunner-1.9/libmozjs.so[0xb7c1c332</link:url>]
<link:url>/usr/lib/xulrunner-1.9/libmozjs.so(JS_CallFunctionValue+0x42)[0xb7bdd1f2</link:url>]
<link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb73d86af</link:url>]
<link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb73cbad7</link:url>]
<link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb73e5637</link:url>]
<link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb73e56e5</link:url>]
<link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb7293902</link:url>]
<link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb736fbd9</link:url>]
<link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb7306a53</link:url>]
<link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb7308bce</link:url>]
<link:url>/usr/lib/xulrunner-1.9/libxul.so(NS_InvokeByIndex_P+0x29)[0xb78eb329</link:url>]
<link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb6f6b461</link:url>]
<link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb6f7429c</link:url>]
<link:url>/usr/lib/xulrunner-1.9/libmozjs.so(js_Invoke+0x464)[0xb7c1bbc4</link:url>]
<link:url>/usr/lib/xulrunner-1.9/libmozjs.so[0xb7c1c332</link:url>]
<link:url>/usr/lib/xulrunner-1.9/libmozjs.so[0xb7c1c422</link:url>]
<link:url>/usr/lib/xulrunner-1.9/libmozjs.so[0xb7c26262</link:url>]
<link:url>/usr/lib/xulrunner-1.9/libmozjs.so[0xb7c0f613</link:url>]
<link:url>/usr/lib/xulrunner-1.9/libmozjs.so(js_Invoke+0x741)[0xb7c1bea1</link:url>]
<link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb6f6979f</link:url>]
<link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb6f63103</link:url>]
<link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb78ebed9</link:url>]
<link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb72df9dd</link:url>]
<link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb72e0052</link:url>]
<link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb72fc79a</link:url>]
<link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb72fc8e3</link:url>]
<link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb72fcec8</link:url>]
<link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb72fd170</link:url>]
<link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb728850a</link:url>]
<link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb7272c47</link:url>]
<link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb72899b7</link:url>]
<link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb7294e3c</link:url>]
<link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb78dbfc7</link:url>]
<link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb78a0363</link:url>]

assertion:
firefox: malloc.c:3552: mremap_chunk: Assertion `((size + offset) &amp; (mp_.pagesize-1)) == 0' failed.
Comment 3 Chad A. Simmons 2009-06-03 19:36:53 UTC
I had similar issues yesterday a reemerge of firefox and adobe-flash seem to have resolved here. You may want to try the same.
Comment 4 Chad A. Simmons 2009-06-03 22:20:29 UTC
Disregard comment #3 I;m still getting a similar backtrace again.
Comment 5 Fabio Coatti 2009-06-04 12:59:07 UTC
I suspect that latest gnutls has some responsability.
with 2.6.6 all works fine
with 2.8.0 I see firefox crashing now and then, even when apparently idle.

This happens both in 32 and 64 bit machines (amd64)
both systems are using glibc-2.10.1 and 4.3.3-r2

Check your gnutls and possibly try to revert to 2.6.6 
Comment 6 Chad A. Simmons 2009-06-04 15:07:35 UTC
(In reply to comment #5)
Reverting to gnutls 2.6.6 seems to solve the problem here for me. Both gnutls 2.8.0 and 2.9.0 cause these crashes.

Comment 7 Philip L 2009-06-05 22:40:53 UTC
I also had this problem and can confirm that reverting to gnutls 2.6.6 fixes it.


*** glibc detected *** /usr/lib64/mozilla-firefox/firefox: double free or corruption (!prev): 0x00000000076f5230 ***
======= Backtrace: =========
/lib/libc.so.6[0x7fb198509258]
/lib/libc.so.6[0x7fb19850ea2b]
/lib/libc.so.6(realloc+0xf1)[0x7fb19850ee91]
/usr/lib/libgnutls.so.26(gnutls_ext_register+0x51)[0x7fb18b72f331]
/usr/lib/libgnutls.so.26[0x7fb18b72f3f8]
/usr/lib/libgnutls.so.26(gnutls_global_init+0x165)[0x7fb18b735b75]
/usr/lib/libcurl.so.4(Curl_gtls_init+0x17)[0x7fb186e14cc7]
/usr/lib/libcurl.so.4(curl_global_init+0x8d)[0x7fb186e0c5fd]
/opt/netscape/plugins/libflashplayer.so[0x7fb16ee6c59a]
/opt/netscape/plugins/libflashplayer.so[0x7fb16efb4a19]
/opt/netscape/plugins/libflashplayer.so[0x7fb16f18404c]
/opt/netscape/plugins/libflashplayer.so[0x7fb16ee6e3be]
/opt/netscape/plugins/libflashplayer.so[0x7fb16ee653d9]
/opt/netscape/plugins/libflashplayer.so[0x7fb16ee5a9c8]
/opt/netscape/plugins/libflashplayer.so[0x7fb16ee5c398]
/usr/lib64/xulrunner-1.9/libxul.so[0x7fb196fba96e]
/usr/lib64/xulrunner-1.9/libxul.so[0x7fb196fc1328]
/usr/lib64/xulrunner-1.9/libxul.so[0x7fb196fc0111]
/usr/lib64/xulrunner-1.9/libxul.so[0x7fb196fc90ad]
/usr/lib64/xulrunner-1.9/libxul.so[0x7fb196b9094b]
/usr/lib64/xulrunner-1.9/libxul.so[0x7fb196b936ac]
/usr/lib64/xulrunner-1.9/libxul.so[0x7fb196c8db61]
/usr/lib64/xulrunner-1.9/libxul.so[0x7fb196c904c6]
/usr/lib64/xulrunner-1.9/libxul.so[0x7fb197121f76]
/usr/lib64/xulrunner-1.9/libxul.so[0x7fb1970f8132]
/usr/lib64/xulrunner-1.9/libxul.so[0x7fb19707ce9d]
/usr/lib64/xulrunner-1.9/libxul.so[0x7fb196f605a9]
/usr/lib64/xulrunner-1.9/libxul.so(XRE_main+0x1d1f)[0x7fb1969ecf22]
/usr/lib64/mozilla-firefox/firefox[0x40166c]
/lib/libc.so.6(__libc_start_main+0xe6)[0x7fb1984b3a26]
/usr/lib64/mozilla-firefox/firefox[0x401189]
Comment 8 hirakendu 2009-06-06 06:12:44 UTC
Many many thanks for pointing it out. Indeed reverting to gnutls-2.6.6 fixes these crashes. I have been pulling my hair all the last week over these crashes. This comes especially after I did a massive system update to gcc-4.4 and glibc-2.10 and so these were the usual suspects. But when I reverted to my backup image of gcc-4.3 and glibc-2.9 and these problems were still there, I was totally mad.

And it also led me to badmouthing adobe-flash over their incompatibility with glibc-2.10, which is not exactly true (unfortunately, when I ran the gdb traces, I didn't look carefully enough to notice the gnutls links). However, I imagine it is only flash, because disabling flash makes these crashes disappear (afaik).

(( In the meantime, I have been using Opera 10 beta and have been impressed. It also seems to encounter this problem with flash, but to a lesser extent - it hangs for several seconds and then recovers, unlike firefox which crashes/hangs out completely. And also offers a very snappy browsing experience.))

Also see bug 260630 which seems to be related.
Comment 9 Stephan Friedrichs 2009-06-06 13:31:14 UTC
I can confirm that the downgrade to gnutls-2.6.6 seems to solve the problem (firefox didn't crash for a couple of hours now, which was impossible before the downgrade).
Comment 10 Jory A. Pratt gentoo-dev 2009-06-06 21:53:48 UTC
This bug needs to be reassigned to proper maintainer of gnutls.
Comment 11 Arfrever Frehtes Taifersar Arahesis (RETIRED) gentoo-dev 2009-06-06 23:55:59 UTC
(In reply to comment #10)
> This bug needs to be reassigned to proper maintainer of gnutls.

They are already CC-ed.

There is a chance that Mozilla Firefox, XULRunner or something else incorrectly uses GnuTLS API...
Comment 12 hirakendu 2009-06-07 00:07:32 UTC
@ comment 11:
That might very well be the case, given that opera doesn't crash?
Comment 13 Nirbheek Chauhan (RETIRED) gentoo-dev 2009-06-07 00:13:21 UTC
Does this also occur with the -bin version? Is this with USE=32bit on adobe-flash? Could someone test if this also occurs with firefox 3.5_beta4 from the mozilla overlay as well?
Comment 14 hirakendu 2009-06-07 00:17:01 UTC
I actually use the daily firefox builds minefield trunk (64-bit), currently 3.6a1pre. That crashes too. I use the 64-bit flash.

PS: Aside, can someone verify that Yahoo! mail crashes with current build 3.6a1pre?
Comment 15 Nirbheek Chauhan (RETIRED) gentoo-dev 2009-06-07 01:35:28 UTC
(In reply to comment #14)
> I actually use the daily firefox builds minefield trunk (64-bit), currently
> 3.6a1pre. That crashes too. I use the 64-bit flash.
> 

Try with the 32-bit flash (multilib+32-bit). I'm inclined to think this is a bug in the beta 64-bit flash since it behaves the same way in Opera too.
Comment 16 Nirbheek Chauhan (RETIRED) gentoo-dev 2009-06-07 01:47:07 UTC
(In reply to comment #15)
> Try with the 32-bit flash (multilib+32-bit). I'm inclined to think this is a
> bug in the beta 64-bit flash since it behaves the same way in Opera too.
> 

Also, xulrunner doesn't use gnutls at all, it obviously uses nss instead. And the 32-bit flash plugin doesn't use gnutls either.

This is totally a 64-bit flash bug.
Comment 17 Fadi Adlouni 2009-06-07 09:28:55 UTC
I don't think this is due to 64 bit flash. i'm reporter of comment #2, and this is on ~x86 on intel x86 cpu with gnutls 2.8.0, after downgrade to 2.6.6 it's stable.

my home pc is amd64 running ~amd64 all 64 bit including flash plugin and it's stable, my home pc is still on gnutls 2.6.6 .

so we can definitely rule out 64 bit flash plugin causing the crash.
Comment 18 Stephan Friedrichs 2009-06-07 09:51:11 UTC
can someone reproduce the crash with gnutls-2.8 and disabled flash?
Comment 19 Nirbheek Chauhan (RETIRED) gentoo-dev 2009-06-07 11:59:09 UTC
(In reply to comment #17)
> I don't think this is due to 64 bit flash. i'm reporter of comment #2, and this
> is on ~x86 on intel x86 cpu with gnutls 2.8.0, after downgrade to 2.6.6 it's
> stable.
> 

Output of lddtree.sh on /usr/lib/xulrunner-1.9/libxul.so and /opt/netscape/plugins/libflashplayer.so please. I don't see anything in the stack that could be using GnuTLS (especially the xulrunner side)
Comment 20 Diego Elio Pettenò (RETIRED) gentoo-dev 2009-06-07 12:18:00 UTC
Neither have gnutls loaded by needed, but it is loaded in the address space (smem shows that); it's dlopen()'d directly or indirectly...
Comment 21 Diego Elio Pettenò (RETIRED) gentoo-dev 2009-06-07 12:21:39 UTC
Quick search on which libraries links to gnutls:

libgnutls.so.26  /usr/lib64/libcups.so.2
libgnutls.so.26  /usr/lib64/libneon.so.27.1.4
libgnutls.so.26  /usr/lib64/libgnutls-extra.so.26.14.7
libgnutls.so.26  /usr/lib64/libggz.so.2.3.0
libgnutls.so.26  /usr/lib64/libwireshark.so.0.0.1
libgnutls.so.26  /usr/lib64/libsoup-2.4.so.1.2.0
libgnutls.so.26  /usr/lib64/libgnutls-openssl.so.26.14.7
libgnutls.so.26  /usr/lib64/libgtk-vnc-1.0.so.0.0.1
libgnutls.so.26  /usr/lib64/libsoup-2.2.so.8.5.0
libgnutls.so.26  /usr/lib64/libcurl.so.4.1.1
libgnutls.so.26  /usr/lib64/libgnomevfs-2.so.0.2400.1
libgnutls.so.26  /usr/lib64/libgnutlsxx.so.26.14.7

guess which one of these _is_ dlopen'd by Firefox?

3....
2...
1...

The right answer is libgnomevfs-2!

And again I can't understand why they just don't link at build time these things in so we can make sure they are there.

So what's the problem? I'm almost ready to bet on symbol collisions...
Comment 22 Märt Bakhoff 2009-06-07 12:31:46 UTC
I had similar problems on my amd64. 
Downgrading adobe-flash (using with nspluginwrapper) seems to help. 

Working with: 
www-plugins/adobe-flash-10.0.15.3
www-client/mozilla-firefox-3.0.10
Comment 23 Martin von Gagern 2009-06-07 14:18:23 UTC
As I already wrote in bug 260630 comment 27, there is an indirect dependency of adobe-flash on gnutls via curl. Comment 7 has a backtrace showing this, and I had a very similar one myself, which I posted to the gnutls developers list: http://thread.gmane.org/gmane.comp.encryption.gpg.gnutls.devel/3621

A simple "strings /usr/lib/nsbrowser/plugins/libflashplayer.so" indicates references to libcurl for dynloading, but none for gnutls.

To account for the backtrace from comment 2, I have several ideas:
1. Optimized tail invocation. If a C function ends with
   return gnutls_ext_register(...);
   then the compiler might translate this into a jump instead of a call/ret
   combination. As a result, the stack frame with the curl function could get
   lost, even if flash invoced curl and not gnutls directly.
   I find no such reference in the curl source code, though.
2. Omitted stack frame. If the curl function called does little work of its own,
   it might get compiled in a way that avoids allocation of a stack frame. I
   thought this only happened for functions not calling other functions, but
   I'm not absolutely sure.
3. Memory corruption. If FF crashes due to memory problems, and glibc reports
   one memory corruption, there might be other corruptions that go undetected
   but affect the backtrace. As most frames in that region look pretty sane, I
   believe this to be unlikely, though.
4. Function pointer. Curl might pass a pointer to gnutls_ext_register back to
   the flash plugin, either intentionally or erroneously, which in turn might
   call the function via this pointer, and might violate some contract if the
   pointer was passed deliberately.
5. glibc might have gotten things wrong when printing the trace. Maybe skipped
   frames for some reason which shouldn't have been skipped.
Comment 24 Diego Elio Pettenò (RETIRED) gentoo-dev 2009-06-07 14:44:32 UTC
The double free trace from glibc tends to be mostly useless, you should check with valgrind instead to see what gets freed twice.

Good luck with that if you wish to try ;)

I would have done so myself but I have problems with SSE4 on my system.
Comment 25 Philip L 2009-06-07 15:34:25 UTC
Created attachment 193810 [details]
valgrind output

(In reply to comment #24)
> The double free trace from glibc tends to be mostly useless, you should check
> with valgrind instead to see what gets freed twice.

I've attached the log of valgrind --error-limit=no --undef-value-errors=no. It doesn't seem to be very helpful, though.
Comment 26 Diego Elio Pettenò (RETIRED) gentoo-dev 2009-06-07 15:48:50 UTC
You should probably rebuild gnutls with debug info enabled (-ggdb), this way we could know what this is:

==2845==    by 0x12A3CF9F: (within /usr/lib64/libgnutls.so.26.14.7)

which is where the invalid free starts, to move then inside libGL no idea how.
Comment 27 Philip L 2009-06-07 17:05:27 UTC
I remerged gnutls with CFLAGS+=-ggdb but I still got this:

==23088== Thread 1:
==23088== Invalid free() / delete / delete[]
==23088==    at 0x4C24AC1: realloc (vg_replace_malloc.c:429)
==23088==    by 0xC20B0E6: (within /usr/lib64/opengl/nvidia/lib/libGL.so.180.60)
==23088==    by 0x100000000: ???
==23088==    by 0x12A3CF9F: (within /usr/lib64/libgnutls.so.26.14.7)
==23088==  Address 0x1ed40ea8 is not stack'd, malloc'd or (recently) free'd

I didn't run it all the way to a crash, so that's the only relevant error.
Comment 28 Olaf Freyer 2009-06-07 17:15:21 UTC
*** glibc detected *** /usr/lib64/mozilla-firefox/firefox: double free or corruption (!prev): 0x0000000007bd26d0 ***                                                        
======= Backtrace: =========                                                                                                                                                
/lib/libc.so.6[0x7f8849442258]                                                                                                                                              
/lib/libc.so.6[0x7f8849447a2b]                                                                                                                                              
/lib/libc.so.6(realloc+0xf1)[0x7f8849447e91]                                                                                                                                
/usr/lib/libgnutls.so.26(gnutls_ext_register+0x51)[0x7f883ba8aa01]                                                                                                          
/usr/lib/libgnutls.so.26[0x7f883ba8aac8]                                                                                                                                    
/usr/lib/libgnutls.so.26(gnutls_global_init+0x165)[0x7f883ba91135]                                                                                                          
/usr/lib/libcurl.so.4(Curl_gtls_init+0x17)[0x7f8826324ae7]                                                                                                                  
/usr/lib/libcurl.so.4(curl_global_init+0x8d)[0x7f882631c52d]                                                                                                                
/opt/netscape/plugins/libflashplayer.so[0x7f88285c759a]    
Comment 29 Olaf Freyer 2009-06-07 17:20:41 UTC
*** glibc detected *** /usr/lib64/mozilla-firefox/firefox: realloc(): invalid pointer: 0x000000000791a330 ***
======= Backtrace: =========                                                                                 
/lib/libc.so.6[0x7f898b110258]                                                                               
/lib/libc.so.6(realloc+0x2f8)[0x7f898b116098]                                                                
/usr/lib/libgnutls.so.26(gnutls_ext_register+0x51)[0x7f897d758a01]                                           
/usr/lib/libgnutls.so.26[0x7f897d758ac8]                                                                     
/usr/lib/libgnutls.so.26(gnutls_global_init+0x165)[0x7f897d75f135]                                           
/usr/lib/libcurl.so.4(Curl_gtls_init+0x17)[0x7f8967ae1ae7]                                                   
/usr/lib/libcurl.so.4(curl_global_init+0x8d)[0x7f8967ad952d]                                                 
/opt/netscape/plugins/libflashplayer.so[0x7f8968d8259a]   


(gdb) bt
#0  0x00007f898b0ce645 in raise () from /lib/libc.so.6
#1  0x00007f898b0cfb63 in abort () from /lib/libc.so.6
#2  0x00007f898b10aac8 in __libc_message () from /lib/libc.so.6
#3  0x00007f898b110258 in malloc_printerr () from /lib/libc.so.6
#4  0x00007f898b116098 in realloc () from /lib/libc.so.6
#5  0x00007f897d758a01 in gnutls_ext_register (type=1, name=0x7f897d7b9da6 "MAX_RECORD_SIZE", parse_type=6, recv_func=0xffffffffffffffff, send_func=0)
    at gnutls_extensions.c:359
#6  0x00007f897d758ac8 in _gnutls_ext_init () at gnutls_extensions.c:283
#7  0x00007f897d75f135 in gnutls_global_init () at gnutls_global.c:240
#8  0x00007f8967ae1ae7 in Curl_gtls_init () from /usr/lib/libcurl.so.4
#9  0x00007f8967ad952d in curl_global_init () from /usr/lib/libcurl.so.4
#10 0x00007f8968d8259a in ?? () from /opt/netscape/plugins/libflashplayer.so
Comment 30 Nirbheek Chauhan (RETIRED) gentoo-dev 2009-06-07 17:23:48 UTC
(In reply to comment #27)
> I remerged gnutls with CFLAGS+=-ggdb but I still got this:
> 

You likely are still doing FEATURES=strip -- see http://www.gentoo.org/proj/en/qa/backtraces.xml
Comment 31 Diego Elio Pettenò (RETIRED) gentoo-dev 2009-06-07 17:24:44 UTC
Sounds like there can be two different _init calls... maybe gnutls stopped supporting that?
Comment 32 Philip L 2009-06-07 17:27:26 UTC
(In reply to comment #30)
> You likely are still doing FEATURES=strip

FEATURES="ccache distlocks fixpackages parallel-fetch protect-owned sandbox sfperms splitdebug strict unmerge-orphans userfetch"
Comment 33 Martin von Gagern 2009-06-07 18:00:11 UTC
(In reply to comment #31)
> Sounds like there can be two different _init calls... maybe gnutls stopped
> supporting that?

Could this be a race condition, two threads initializing gnutls concurrently? FF is heavily multithreaded, and I could imagine multiple flash plugins on a single page causing concurrent initialization.

Comment 34 Olaf Freyer 2009-06-07 19:32:50 UTC
> Could this be a race condition, two threads initializing gnutls concurrently?
> FF is heavily multithreaded, and I could imagine multiple flash plugins on a
> single page causing concurrent initialization.
> 

Sounds reasonable. A quite sure way for me to reproduce the issue (or at least similar issues) is to switch back and forth between two pages loading quite large flash movies.

I now also recompiled mozilla-firefox, xulrunner, libtasn1 and this is as much info as I were able to get so far (not the exact same issue as before, but most propably related to it):

Core was generated by `/usr/lib/mozilla-firefox/firefox'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007fa16a886d8b in raise () from /lib/libpthread.so.0
(gdb) bt
#0  0x00007fa16a886d8b in raise () from /lib/libpthread.so.0
#1  0x00007fa168160dab in nsProfileLock::FatalSignalHandler () from /usr/lib64/xulrunner-1.9/libxul.so
#2  <signal handler called>
#3  0x00007fa15c003a33 in asn1_delete_structure (structure=0x7fa15c4abfe8) at structure.c:305
#4  0x00007fa15c236fb5 in gnutls_global_deinit () at gnutls_global.c:269
#5  0x00007fa146a91ab2 in Curl_gtls_cleanup () from /usr/lib/libcurl.so.4
#6  0x00007fa146a93162 in Curl_ssl_cleanup () from /usr/lib/libcurl.so.4
#7  0x00007fa146a8948d in curl_global_cleanup () from /usr/lib/libcurl.so.4
#8  0x00007fa148d34328 in ?? () from /opt/netscape/plugins/libflashplayer.so
Comment 35 Olaf Freyer 2009-06-07 19:36:28 UTC
I'm able to cause a similar crash in konqueror...

Thread 1 (Thread 0x7f1984f83750 (LWP 21851)):
[KCrash Handler]
#5  0x00007f1980c4ccb9 in free () from /lib/libc.so.6
#6  0x00007f196d4eead6 in _asn1_remove_node (node=0xe33a60) at parser_aux.c:515
#7  0x00007f196d4f0a63 in asn1_delete_structure (structure=0x7f196d998fe8) at structure.c:316
#8  0x00007f196d723fb5 in gnutls_global_deinit () at gnutls_global.c:269
#9  0x00007f196e839ab2 in Curl_gtls_cleanup () from /usr/lib/libcurl.so.4
#10 0x00007f196e83b162 in Curl_ssl_cleanup () from /usr/lib/libcurl.so.4
#11 0x00007f196e83148d in curl_global_cleanup () from /usr/lib/libcurl.so.4
#12 0x00007f197631e328 in ?? () from /opt/netscape/plugins/libflashplayer.so
#13 0x00007f197646b2dd in ?? () from /opt/netscape/plugins/libflashplayer.so
#14 0x00007f1976634c0f in ?? () from /opt/netscape/plugins/libflashplayer.so
#15 0x00007f1976315bd8 in ?? () from /opt/netscape/plugins/libflashplayer.so
#16 0x00007f19763110fa in ?? () from /opt/netscape/plugins/libflashplayer.so
#17 0x00007f1976309e8e in ?? () from /opt/netscape/plugins/libflashplayer.so
#18 0x00007f197630e379 in ?? () from /opt/netscape/plugins/libflashplayer.so
#19 0x000000000040dc0c in NSPluginInstance::destroy (this=0xeb23e0) at /var/tmp/portage/kde-base/nsplugins-4.2.4/work/nsplugins-4.2.4/nsplugins/viewer/nsplugin.cpp:718
#20 0x000000000040dce9 in ~NSPluginInstance (this=0xeb23e0) at /var/tmp/portage/kde-base/nsplugins-4.2.4/work/nsplugins-4.2.4/nsplugins/viewer/nsplugin.cpp:682
#21 0x000000000040c45e in NSPluginClass::timer (this=<value optimized out>) at /var/tmp/portage/kde-base/nsplugins-4.2.4/work/nsplugins-4.2.4/nsplugins/viewer/nsplugin.cpp:1336
#22 0x000000000040c51f in NSPluginClass::destroyInstance (this=0xdb9210, inst=0xeb23e0) at /var/tmp/portage/kde-base/nsplugins-4.2.4/work/nsplugins-4.2.4/nsplugins/viewer/nsplugin.cpp:1488
#23 0x0000000000416e98 in InstanceAdaptor::qt_metacall (this=0xdeeb10, _c=QMetaObject::InvokeMetaMethod, _id=0, _a=0x7fff8cfc5730)
    at /var/tmp/portage/kde-base/nsplugins-4.2.4/work/nsplugins-4.2.4_build/nsplugins/viewer/instanceadaptor.moc:99
#24 0x00007f19843bd70b in QDBusConnectionPrivate::deliverCall () from /usr/lib64/qt4/libQtDBus.so.4
#25 0x00007f19843be351 in QDBusConnectionPrivate::activateCall () from /usr/lib64/qt4/libQtDBus.so.4
#26 0x00007f19843be8e5 in QDBusConnectionPrivate::activateObject () from /usr/lib64/qt4/libQtDBus.so.4
#27 0x00007f19843bebb8 in QDBusActivateObjectEvent::placeMetaCall () from /usr/lib64/qt4/libQtDBus.so.4
#28 0x00007f198252e851 in QObject::event () from /usr/lib64/qt4/libQtCore.so.4
#29 0x00007f198187b8ad in QApplicationPrivate::notify_helper () from /usr/lib64/qt4/libQtGui.so.4
#30 0x00007f198188417a in QApplication::notify () from /usr/lib64/qt4/libQtGui.so.4
#31 0x00007f19830c9b6b in KApplication::notify () from /usr/lib64/libkdeui.so.5
#32 0x00007f198251fc43 in QCoreApplication::notifyInternal () from /usr/lib64/qt4/libQtCore.so.4
#33 0x00007f1982520546 in QCoreApplicationPrivate::sendPostedEvents () from /usr/lib64/qt4/libQtCore.so.4
#34 0x00007f1982544ec3 in postEventSourceDispatch () from /usr/lib64/qt4/libQtCore.so.4
#35 0x00007f1983eef841 in g_main_context_dispatch () from /usr/lib64/libglib-2.0.so.0
#36 0x00007f1983ef2e60 in g_main_context_iterate () from /usr/lib64/libglib-2.0.so.0
#37 0x00007f1983ef2ffc in g_main_context_iteration () from /usr/lib64/libglib-2.0.so.0
#38 0x00007f1975dd9271 in gtk_main_iteration () from /usr/lib/libgtk-x11-2.0.so.0
#39 0x00007f1976310bab in ?? () from /opt/netscape/plugins/libflashplayer.so
#40 0x00007f1983eeff8b in g_timeout_dispatch () from /usr/lib64/libglib-2.0.so.0
#41 0x00007f1983eef841 in g_main_context_dispatch () from /usr/lib64/libglib-2.0.so.0
#42 0x00007f1983ef2e60 in g_main_context_iterate () from /usr/lib64/libglib-2.0.so.0
#43 0x00007f1983ef2ffc in g_main_context_iteration () from /usr/lib64/libglib-2.0.so.0
#44 0x00007f1982544b6f in QEventDispatcherGlib::processEvents () from /usr/lib64/qt4/libQtCore.so.4
#45 0x00007f1981900aef in QGuiEventDispatcherGlib::processEvents () from /usr/lib64/qt4/libQtGui.so.4
#46 0x00007f198251e642 in QEventLoop::processEvents () from /usr/lib64/qt4/libQtCore.so.4
#47 0x00007f198251e7d5 in QEventLoop::exec () from /usr/lib64/qt4/libQtCore.so.4
#48 0x00007f19825207d4 in QCoreApplication::exec () from /usr/lib64/qt4/libQtCore.so.4
#49 0x000000000041528f in main (argc=<value optimized out>, argv=<value optimized out>) at /var/tmp/portage/kde-base/nsplugins-4.2.4/work/nsplugins-4.2.4/nsplugins/viewer/viewer.cpp:144
Comment 36 Fabio Coatti 2009-06-07 20:42:42 UTC
Not sure if this can help (probably not) but FF is acting in a weird way with gnutls 2.8.0.
Besides the behaviour reported by others, after some segfaults without other messages, now I just got this:

Inconsistency detected by ld.so: dl-close.c: 344: _dl_close_worker: Assertion `tmap->l_ns == nsid' failed!
Comment 37 Philip L 2009-06-08 02:35:18 UTC
(In reply to comment #33)
> Could this be a race condition, two threads initializing gnutls concurrently?
> FF is heavily multithreaded, and I could imagine multiple flash plugins on a
> single page causing concurrent initialization.

From the docs for gnutls_global_init():
"Note! This function is not thread safe. If two threads call this function simultaneously, they can cause a race between checking the global counter and incrementing it, causing both threads to execute the library initialization code. That would lead to a memory leak. To handle this, your application could invoke this function after aquiring a thread mutex. To ignore the potential memory leak is also an option."

A race condition involving gnutls_global_deinit() would explain the double free.

From the docs for curl_global_cleanup():
"This function is not thread safe. You must not call it when any other thread in the program (i.e. a thread sharing the same memory) is running. This doesn't just mean no other thread that is using libcurl. Because curl_global_cleanup(3) calls functions of other libraries that are similarly thread unsafe, it could conflict with any other thread that uses these other libraries."

Since curl_global_cleanup() is being called at various times, it seems to me that the caller (flash) may be using curl improperly. 
Comment 38 Fabio Coatti 2009-06-08 06:07:08 UTC
(In reply to comment #37)
I understand, but something is obviously changed since gnutls 2.6.6. Maybe the earlier version (prior 2.8.0) was allowing leaks or was more forgiving? anyway, the net result is that ff was usable, now it isn't. yuck.

Comment 39 Raouf Bencheraiet 2009-06-08 07:46:18 UTC
have the same problem here, firefox crashs after a while for whatever reason. sometimes it just segfaults or give a bur error no more and  some other time it gives this (see bellow).

the strange thing is that it happens only with firefox. seemonkey and  opera work no crashs even after extensive abuse of youtube and flash player.
any how emerging curl without gnutls use flag has fixed it for now  

*** glibc detected *** /usr/lib64/mozilla-firefox/firefox: double free or corruption (!prev): 0x00000000067e7140 ***
======= Backtrace: =========
/lib/libc.so.6[0x7f1385637258]
/lib/libc.so.6[0x7f138563ca2b]
/lib/libc.so.6(realloc+0xf1)[0x7f138563ce91]
/usr/lib/libgnutls.so.26(gnutls_ext_register+0x51)[0x7f13790794d1]
/usr/lib/libgnutls.so.26[0x7f1379079598]
/usr/lib/libgnutls.so.26(gnutls_global_init+0x165)[0x7f1379080355]
/usr/lib/libcurl.so.4(Curl_gtls_init+0x17)[0x7f13641eb977]
/usr/lib/libcurl.so.4(curl_global_init+0x8d)[0x7f13641e1c1d]
/opt/netscape/plugins/libflashplayer.so[0x7f136548c59a]
/opt/netscape/plugins/libflashplayer.so[0x7f13655d4a19]
/opt/netscape/plugins/libflashplayer.so[0x7f13657a404c]
/opt/netscape/plugins/libflashplayer.so[0x7f136548e3be]
/opt/netscape/plugins/libflashplayer.so[0x7f13654853d9]
/opt/netscape/plugins/libflashplayer.so[0x7f136547a9c8]
/opt/netscape/plugins/libflashplayer.so[0x7f136547c398]
/usr/lib64/xulrunner-1.9/libxul.so[0x7f13841725d2]
/usr/lib64/xulrunner-1.9/libxul.so[0x7f1384178f8c]
/usr/lib64/xulrunner-1.9/libxul.so[0x7f1384177d75]
/usr/lib64/xulrunner-1.9/libxul.so[0x7f1384180d11]
/usr/lib64/xulrunner-1.9/libxul.so[0x7f1383d4856f]
/usr/lib64/xulrunner-1.9/libxul.so[0x7f1383d4b2d0]
/usr/lib64/xulrunner-1.9/libxul.so[0x7f1383e45605]
/usr/lib64/xulrunner-1.9/libxul.so[0x7f1383e456d7]
/usr/lib64/xulrunner-1.9/libxul.so[0x7f1383e464a6]
/usr/lib64/xulrunner-1.9/libxul.so[0x7f1383e46c65]
/usr/lib64/xulrunner-1.9/libxul.so[0x7f1383ea19c9]
/usr/lib64/xulrunner-1.9/libxul.so[0x7f1383e83450]
/usr/lib64/xulrunner-1.9/libxul.so(NS_InvokeByIndex_P+0x26e)[0x7f13842e3f82]
/usr/lib64/xulrunner-1.9/libxul.so[0x7f1383bcc236]
Comment 40 Diego Elio Pettenò (RETIRED) gentoo-dev 2009-06-08 10:05:36 UTC
If the GNOME integration is not enabled it should not be using gnutls by itself, and Flash would probably be standing alone.
Comment 41 Fabio Coatti 2009-06-08 10:16:23 UTC
(In reply to comment #40)
> If the GNOME integration is not enabled it should not be using gnutls by
> itself, and Flash would probably be standing alone.
> 

My installation has no gnome use flag enabled for firefox, but gnutls bug shows up.
Comment 42 Martin von Gagern 2009-06-08 14:38:30 UTC
OK, I built myself a LD_PRELOAD library to trace dlopen calls, as well as modified versions of curl and gnutls to trace invocations of the init and cleanup functions.

Obervations:

Sometimes libcurl gets dlopened several times, especially when multiple flash objects are loaded simultaneously. But I just got a hanging firefox without this, so this is not the cause of the issue, or at least not the only possible cause.

I almost always got curl initialized, deinitialized, and initialized again. Sounds proper to me, but in cases where the initialization code relies on some pre-initialized state, which the cleanup code doesn't restore, this will cause problems. Haven't looked for such errors in the code yet.

I dumped the thread IDs along with the PIDs. Looks like it's always the initial mozilla thread from which stuff gets loaded or initialized. There goes my idea of an initialization race causing all this mess.

Here is some output I just got from a FF run with a single flash object, causing FF to hang in the end. The lines up front are my own debug additions, where ... indicates a function entry, to be concluded by a later line ending in a semicolon. The numbers up front are PID.TID, and you'll notice they are always the same, indicating the single main thread invoking all this. No gnutls at all involved in the backtrace, but a prior memory corruption might cause all kinds of delayed strangeness.

9143.9143: dlopen("libcurl.so.4", 0x1) = 0x0xaee1c28;
9143.9143: curl_global_init(1) ...  // initialized=0
9143.9143: gnutls_global_init() ...  // _gnutls_init=0
9143.9143: gnutls_global_init() = 0;
9143.9143: curl_global_init(1) = CURLE_OK;
9143.9143: curl_global_cleanup() ...  // initialized=1
9143.9143: gnutls_global_deinit() ...  // _gnutls_init=1
9143.9143: curl_global_cleanup();
9143.9143: curl_global_init(1) ...  // initialized=0
9143.9143: gnutls_global_init() ...  // _gnutls_init=0
9143.9143: gnutls_global_init() = 0;
9143.9143: curl_global_init(1) = CURLE_OK;
*** glibc detected *** /usr/lib/mozilla-firefox/firefox: corrupted double-linked list: 0x0aee6438 ***
======= Backtrace: =========
/lib/libc.so.6[0x4c59d7c4]
/lib/libc.so.6[0x4c59da94]
/lib/libc.so.6[0x4c59fc95]
/lib/libc.so.6(__libc_malloc+0x5a)[0x4c5a1f8a]
/usr/lib/xulrunner-1.9/libxul.so[0x422ce334]
/usr/lib/xulrunner-1.9/libxul.so[0x422cdd6d]
/usr/lib/xulrunner-1.9/libxul.so[0x422ce14e]
/usr/lib/xulrunner-1.9/libxul.so[0x4230c292]
/usr/lib/xulrunner-1.9/libxul.so(NS_CycleCollectorSuspect_P+0x2a)[0x4230c2dd]
...
Comment 43 Philip L 2009-06-08 15:08:19 UTC
Since all the init/deinit calls are ordered properly and are coming from the main thread, and since the problem only occurs with certain versions of gnutls, it must be a bug in gnutls, right?
Comment 44 hirakendu 2009-06-08 16:01:53 UTC
Well, it was clear right from the beginning that the problem arises only after upgrading gnutls (and is fixed by downgrading). While it is natural to say that it is a bug in gnutls, but may be other packages are touching gnutls in an inappropriate manner ;). In any case, thanks for the efforts to actually identify the bug.
Comment 45 Martin von Gagern 2009-06-08 16:57:14 UTC
It's gnutls, and upstream has a patch for us:
http://git.savannah.gnu.org/cgit/gnutls.git/patch/?id=86ca9d6ce227a877
Comment 46 Jim Ramsay (lack) (RETIRED) gentoo-dev 2009-06-08 17:27:11 UTC
*** Bug 260630 has been marked as a duplicate of this bug. ***
Comment 47 Arfrever Frehtes Taifersar Arahesis (RETIRED) gentoo-dev 2009-06-08 17:45:35 UTC
Created attachment 193921 [details]
gnutls-2.8.1_pre20090608.ebuild

Please test this ebuild.
Comment 48 Olaf Freyer 2009-06-08 18:04:02 UTC
Works like a charm for me! Thanks alot everyone...
Comment 49 Nirbheek Chauhan (RETIRED) gentoo-dev 2009-06-09 00:56:54 UTC
(In reply to comment #47)
> Created an attachment (id=193921) [edit]
> gnutls-2.8.1_pre20090608.ebuild
> 
> Please test this ebuild.
> 

The patch applies to the tarball by itself, perhaps just the patch should be applied?
Comment 50 Arfrever Frehtes Taifersar Arahesis (RETIRED) gentoo-dev 2009-06-09 09:14:00 UTC
(In reply to comment #49)

net-libs/gnutls-2.8.1 will be released soon.
Comment 51 Fabio Coatti 2009-06-09 12:58:12 UTC
Confirmed here: gnutls-2.8.1_pre20090608 installed, no crashes so far.
Comment 52 Olaf Freyer 2009-06-09 15:25:46 UTC
Wouldn't it be a sane idea to mask broken gnutls-2.8.0 to prevent more people from running into this issue?
Comment 53 Arfrever Frehtes Taifersar Arahesis (RETIRED) gentoo-dev 2009-06-10 17:20:54 UTC
net-libs/gnutls-2.8.1 is now in the tree.