mozilla-firefox-3.0.10 occasionally crashes on my ~amd64 machine (sometimes after clicking on a link sometimes after just moving the mouse over something, I can't tell that precisely, yet) and gives the following output: *** glibc detected *** /usr/lib64/mozilla-firefox/firefox: corrupted double-linked list: 0x0000000003515ce0 *** ======= Backtrace: ========= /lib/libc.so.6[0x7fb34f35a258] /lib/libc.so.6[0x7fb34f35a560] /lib/libc.so.6[0x7fb34f35cd97] /lib/libc.so.6(__libc_malloc+0x6a)[0x7fb34f35edea] /usr/lib64/xulrunner-1.9/libxul.so[0x7fb34dfdfd04] /usr/lib64/xulrunner-1.9/libxul.so[0x7fb34d8f0f9d] /usr/lib64/xulrunner-1.9/libxul.so[0x7fb34d8ff148] /usr/lib64/xulrunner-1.9/libmozjs.so[0x7fb34ee15f6b] /usr/lib64/xulrunner-1.9/libmozjs.so[0x7fb34edffbe5] /usr/lib64/xulrunner-1.9/libxul.so[0x7fb34d8e02a0] /usr/lib64/xulrunner-1.9/libxul.so[0x7fb34e014236] /usr/lib64/xulrunner-1.9/libxul.so[0x7fb34dc64c71] /usr/lib64/xulrunner-1.9/libxul.so[0x7fb34dc6501c] /usr/lib64/xulrunner-1.9/libxul.so[0x7fb34dfe92ec] /usr/lib64/xulrunner-1.9/libxul.so[0x7fb34dfe95d2] /usr/lib64/xulrunner-1.9/libxul.so[0x7fb34db9ed5a] /usr/lib64/xulrunner-1.9/libxul.so[0x7fb34e00ca78] /usr/lib64/xulrunner-1.9/libxul.so[0x7fb34e00cad6] /usr/lib64/xulrunner-1.9/libxul.so[0x7fb34e00abbc] /usr/lib64/xulrunner-1.9/libxul.so[0x7fb34dfe0d6e] /usr/lib64/xulrunner-1.9/libxul.so[0x7fb34df65ad9] /usr/lib64/xulrunner-1.9/libxul.so[0x7fb34de490e9] /usr/lib64/xulrunner-1.9/libxul.so(XRE_main+0x1d1f)[0x7fb34d8d5a72] /usr/lib64/mozilla-firefox/firefox[0x401708] /lib/libc.so.6(__libc_start_main+0xe6)[0x7fb34f304a26] /usr/lib64/mozilla-firefox/firefox[0x401249] I've glibc-2.10.0 installed. Reproducible: Sometimes Steps to Reproduce: 1. start firefox 2. use it for a couple of minutes
Created attachment 193383 [details] paludis --info mozilla-firefox
i have the same on my ~x86. and although i did emerge -uDN world recently, but the only major component that got upgraded was glibc from 2.9.something to 2.10.1 . in fact i had several crashes. sometimes nothing shows up on terminal. sometimes i get a double free crash, once i got a double-lined list crash and once i got an assert. i'm not sure if i should open seperate bugs for the other crashes or they have the same root cause as this one. below i past each one: -double free crash *** glibc detected *** <link:url>/usr/lib/mozilla-firefox/firefox</link:url>: double free or corruption (!prev): 0x09010490 *** ======= Backtrace: ========= <link:url>/lib/libc.so.6[0xb7d5d7c4</link:url>] <link:url>/lib/libc.so.6[0xb7d5eea3</link:url>] <link:url>/lib/libc.so.6[0xb7d62b1b</link:url>] <link:url>/lib/libc.so.6(realloc+0xdd)[0xb7d62fbd</link:url>] <link:url>/usr/lib/libgnutls.so.26(gnutls_ext_register+0x34)[0xb5aacc64</link:url>] <link:url>/opt/netscape/plugins/libflashplayer.so[0xaf93db4d</link:url>] <link:url>/opt/netscape/plugins/libflashplayer.so[0xafb06ae0</link:url>] <link:url>/opt/netscape/plugins/libflashplayer.so[0xaf7c49b6</link:url>] <link:url>/opt/netscape/plugins/libflashplayer.so[0xaf7b736c</link:url>] <link:url>/opt/netscape/plugins/libflashplayer.so[0xaf7a8649</link:url>] <link:url>/opt/netscape/plugins/libflashplayer.so[0xaf7acd49</link:url>] <link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb76f2f0b</link:url>] <link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb7702406</link:url>] <link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb76f7bed</link:url>] <link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb7701df6</link:url>] <link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb715d80b</link:url>] <link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb7161597</link:url>] <link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb72b6b2e</link:url>] <link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb72b6c49</link:url>] <link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb72b77d6</link:url>] <link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb72b8086</link:url>] <link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb733c486</link:url>] <link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb7302e60</link:url>] <link:url>/usr/lib/xulrunner-1.9/libxul.so(NS_InvokeByIndex_P+0x29)[0xb78eb329</link:url>] <link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb6f6b461</link:url>] <link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb6f743ca</link:url>] <link:url>/usr/lib/xulrunner-1.9/libmozjs.so(js_Invoke+0x464)[0xb7c1bbc4</link:url>] <link:url>/usr/lib/xulrunner-1.9/libmozjs.so[0xb7c0ff97</link:url>] <link:url>/usr/lib/xulrunner-1.9/libmozjs.so(js_Invoke+0x741)[0xb7c1bea1</link:url>] <link:url>/usr/lib/xulrunner-1.9/libmozjs.so[0xb7c1c332</link:url>] <link:url>/usr/lib/xulrunner-1.9/libmozjs.so(JS_CallFunctionValue+0x42)[0xb7bdd1f2</link:url>] <link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb73d86af</link:url>] <link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb73cbad7</link:url>] <link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb73e5637</link:url>] <link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb73e56e5</link:url>] <link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb7293902</link:url>] <link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb736fbd9</link:url>] <link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb7306a53</link:url>] <link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb7308bce</link:url>] <link:url>/usr/lib/xulrunner-1.9/libxul.so(NS_InvokeByIndex_P+0x29)[0xb78eb329</link:url>] <link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb6f6b461</link:url>] <link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb6f7429c</link:url>] <link:url>/usr/lib/xulrunner-1.9/libmozjs.so(js_Invoke+0x464)[0xb7c1bbc4</link:url>] <link:url>/usr/lib/xulrunner-1.9/libmozjs.so[0xb7c1c332</link:url>] <link:url>/usr/lib/xulrunner-1.9/libmozjs.so[0xb7c1c422</link:url>] <link:url>/usr/lib/xulrunner-1.9/libmozjs.so[0xb7c26262</link:url>] <link:url>/usr/lib/xulrunner-1.9/libmozjs.so[0xb7c0f613</link:url>] <link:url>/usr/lib/xulrunner-1.9/libmozjs.so(js_Invoke+0x741)[0xb7c1bea1</link:url>] <link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb6f6979f</link:url>] <link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb6f63103</link:url>] <link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb78ebed9</link:url>] <link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb72df9dd</link:url>] <link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb72e0052</link:url>] <link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb72fc79a</link:url>] <link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb72fc8e3</link:url>] <link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb72fcec8</link:url>] <link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb72fd170</link:url>] <link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb728850a</link:url>] <link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb7272c47</link:url>] <link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb72899b7</link:url>] <link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb7294e3c</link:url>] <link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb78dbfc7</link:url>] <link:url>/usr/lib/xulrunner-1.9/libxul.so[0xb78a0363</link:url>] assertion: firefox: malloc.c:3552: mremap_chunk: Assertion `((size + offset) & (mp_.pagesize-1)) == 0' failed.
I had similar issues yesterday a reemerge of firefox and adobe-flash seem to have resolved here. You may want to try the same.
Disregard comment #3 I;m still getting a similar backtrace again.
I suspect that latest gnutls has some responsability. with 2.6.6 all works fine with 2.8.0 I see firefox crashing now and then, even when apparently idle. This happens both in 32 and 64 bit machines (amd64) both systems are using glibc-2.10.1 and 4.3.3-r2 Check your gnutls and possibly try to revert to 2.6.6
(In reply to comment #5) Reverting to gnutls 2.6.6 seems to solve the problem here for me. Both gnutls 2.8.0 and 2.9.0 cause these crashes.
I also had this problem and can confirm that reverting to gnutls 2.6.6 fixes it. *** glibc detected *** /usr/lib64/mozilla-firefox/firefox: double free or corruption (!prev): 0x00000000076f5230 *** ======= Backtrace: ========= /lib/libc.so.6[0x7fb198509258] /lib/libc.so.6[0x7fb19850ea2b] /lib/libc.so.6(realloc+0xf1)[0x7fb19850ee91] /usr/lib/libgnutls.so.26(gnutls_ext_register+0x51)[0x7fb18b72f331] /usr/lib/libgnutls.so.26[0x7fb18b72f3f8] /usr/lib/libgnutls.so.26(gnutls_global_init+0x165)[0x7fb18b735b75] /usr/lib/libcurl.so.4(Curl_gtls_init+0x17)[0x7fb186e14cc7] /usr/lib/libcurl.so.4(curl_global_init+0x8d)[0x7fb186e0c5fd] /opt/netscape/plugins/libflashplayer.so[0x7fb16ee6c59a] /opt/netscape/plugins/libflashplayer.so[0x7fb16efb4a19] /opt/netscape/plugins/libflashplayer.so[0x7fb16f18404c] /opt/netscape/plugins/libflashplayer.so[0x7fb16ee6e3be] /opt/netscape/plugins/libflashplayer.so[0x7fb16ee653d9] /opt/netscape/plugins/libflashplayer.so[0x7fb16ee5a9c8] /opt/netscape/plugins/libflashplayer.so[0x7fb16ee5c398] /usr/lib64/xulrunner-1.9/libxul.so[0x7fb196fba96e] /usr/lib64/xulrunner-1.9/libxul.so[0x7fb196fc1328] /usr/lib64/xulrunner-1.9/libxul.so[0x7fb196fc0111] /usr/lib64/xulrunner-1.9/libxul.so[0x7fb196fc90ad] /usr/lib64/xulrunner-1.9/libxul.so[0x7fb196b9094b] /usr/lib64/xulrunner-1.9/libxul.so[0x7fb196b936ac] /usr/lib64/xulrunner-1.9/libxul.so[0x7fb196c8db61] /usr/lib64/xulrunner-1.9/libxul.so[0x7fb196c904c6] /usr/lib64/xulrunner-1.9/libxul.so[0x7fb197121f76] /usr/lib64/xulrunner-1.9/libxul.so[0x7fb1970f8132] /usr/lib64/xulrunner-1.9/libxul.so[0x7fb19707ce9d] /usr/lib64/xulrunner-1.9/libxul.so[0x7fb196f605a9] /usr/lib64/xulrunner-1.9/libxul.so(XRE_main+0x1d1f)[0x7fb1969ecf22] /usr/lib64/mozilla-firefox/firefox[0x40166c] /lib/libc.so.6(__libc_start_main+0xe6)[0x7fb1984b3a26] /usr/lib64/mozilla-firefox/firefox[0x401189]
Many many thanks for pointing it out. Indeed reverting to gnutls-2.6.6 fixes these crashes. I have been pulling my hair all the last week over these crashes. This comes especially after I did a massive system update to gcc-4.4 and glibc-2.10 and so these were the usual suspects. But when I reverted to my backup image of gcc-4.3 and glibc-2.9 and these problems were still there, I was totally mad. And it also led me to badmouthing adobe-flash over their incompatibility with glibc-2.10, which is not exactly true (unfortunately, when I ran the gdb traces, I didn't look carefully enough to notice the gnutls links). However, I imagine it is only flash, because disabling flash makes these crashes disappear (afaik). (( In the meantime, I have been using Opera 10 beta and have been impressed. It also seems to encounter this problem with flash, but to a lesser extent - it hangs for several seconds and then recovers, unlike firefox which crashes/hangs out completely. And also offers a very snappy browsing experience.)) Also see bug 260630 which seems to be related.
I can confirm that the downgrade to gnutls-2.6.6 seems to solve the problem (firefox didn't crash for a couple of hours now, which was impossible before the downgrade).
This bug needs to be reassigned to proper maintainer of gnutls.
(In reply to comment #10) > This bug needs to be reassigned to proper maintainer of gnutls. They are already CC-ed. There is a chance that Mozilla Firefox, XULRunner or something else incorrectly uses GnuTLS API...
@ comment 11: That might very well be the case, given that opera doesn't crash?
Does this also occur with the -bin version? Is this with USE=32bit on adobe-flash? Could someone test if this also occurs with firefox 3.5_beta4 from the mozilla overlay as well?
I actually use the daily firefox builds minefield trunk (64-bit), currently 3.6a1pre. That crashes too. I use the 64-bit flash. PS: Aside, can someone verify that Yahoo! mail crashes with current build 3.6a1pre?
(In reply to comment #14) > I actually use the daily firefox builds minefield trunk (64-bit), currently > 3.6a1pre. That crashes too. I use the 64-bit flash. > Try with the 32-bit flash (multilib+32-bit). I'm inclined to think this is a bug in the beta 64-bit flash since it behaves the same way in Opera too.
(In reply to comment #15) > Try with the 32-bit flash (multilib+32-bit). I'm inclined to think this is a > bug in the beta 64-bit flash since it behaves the same way in Opera too. > Also, xulrunner doesn't use gnutls at all, it obviously uses nss instead. And the 32-bit flash plugin doesn't use gnutls either. This is totally a 64-bit flash bug.
I don't think this is due to 64 bit flash. i'm reporter of comment #2, and this is on ~x86 on intel x86 cpu with gnutls 2.8.0, after downgrade to 2.6.6 it's stable. my home pc is amd64 running ~amd64 all 64 bit including flash plugin and it's stable, my home pc is still on gnutls 2.6.6 . so we can definitely rule out 64 bit flash plugin causing the crash.
can someone reproduce the crash with gnutls-2.8 and disabled flash?
(In reply to comment #17) > I don't think this is due to 64 bit flash. i'm reporter of comment #2, and this > is on ~x86 on intel x86 cpu with gnutls 2.8.0, after downgrade to 2.6.6 it's > stable. > Output of lddtree.sh on /usr/lib/xulrunner-1.9/libxul.so and /opt/netscape/plugins/libflashplayer.so please. I don't see anything in the stack that could be using GnuTLS (especially the xulrunner side)
Neither have gnutls loaded by needed, but it is loaded in the address space (smem shows that); it's dlopen()'d directly or indirectly...
Quick search on which libraries links to gnutls: libgnutls.so.26 /usr/lib64/libcups.so.2 libgnutls.so.26 /usr/lib64/libneon.so.27.1.4 libgnutls.so.26 /usr/lib64/libgnutls-extra.so.26.14.7 libgnutls.so.26 /usr/lib64/libggz.so.2.3.0 libgnutls.so.26 /usr/lib64/libwireshark.so.0.0.1 libgnutls.so.26 /usr/lib64/libsoup-2.4.so.1.2.0 libgnutls.so.26 /usr/lib64/libgnutls-openssl.so.26.14.7 libgnutls.so.26 /usr/lib64/libgtk-vnc-1.0.so.0.0.1 libgnutls.so.26 /usr/lib64/libsoup-2.2.so.8.5.0 libgnutls.so.26 /usr/lib64/libcurl.so.4.1.1 libgnutls.so.26 /usr/lib64/libgnomevfs-2.so.0.2400.1 libgnutls.so.26 /usr/lib64/libgnutlsxx.so.26.14.7 guess which one of these _is_ dlopen'd by Firefox? 3.... 2... 1... The right answer is libgnomevfs-2! And again I can't understand why they just don't link at build time these things in so we can make sure they are there. So what's the problem? I'm almost ready to bet on symbol collisions...
I had similar problems on my amd64. Downgrading adobe-flash (using with nspluginwrapper) seems to help. Working with: www-plugins/adobe-flash-10.0.15.3 www-client/mozilla-firefox-3.0.10
As I already wrote in bug 260630 comment 27, there is an indirect dependency of adobe-flash on gnutls via curl. Comment 7 has a backtrace showing this, and I had a very similar one myself, which I posted to the gnutls developers list: http://thread.gmane.org/gmane.comp.encryption.gpg.gnutls.devel/3621 A simple "strings /usr/lib/nsbrowser/plugins/libflashplayer.so" indicates references to libcurl for dynloading, but none for gnutls. To account for the backtrace from comment 2, I have several ideas: 1. Optimized tail invocation. If a C function ends with return gnutls_ext_register(...); then the compiler might translate this into a jump instead of a call/ret combination. As a result, the stack frame with the curl function could get lost, even if flash invoced curl and not gnutls directly. I find no such reference in the curl source code, though. 2. Omitted stack frame. If the curl function called does little work of its own, it might get compiled in a way that avoids allocation of a stack frame. I thought this only happened for functions not calling other functions, but I'm not absolutely sure. 3. Memory corruption. If FF crashes due to memory problems, and glibc reports one memory corruption, there might be other corruptions that go undetected but affect the backtrace. As most frames in that region look pretty sane, I believe this to be unlikely, though. 4. Function pointer. Curl might pass a pointer to gnutls_ext_register back to the flash plugin, either intentionally or erroneously, which in turn might call the function via this pointer, and might violate some contract if the pointer was passed deliberately. 5. glibc might have gotten things wrong when printing the trace. Maybe skipped frames for some reason which shouldn't have been skipped.
The double free trace from glibc tends to be mostly useless, you should check with valgrind instead to see what gets freed twice. Good luck with that if you wish to try ;) I would have done so myself but I have problems with SSE4 on my system.
Created attachment 193810 [details] valgrind output (In reply to comment #24) > The double free trace from glibc tends to be mostly useless, you should check > with valgrind instead to see what gets freed twice. I've attached the log of valgrind --error-limit=no --undef-value-errors=no. It doesn't seem to be very helpful, though.
You should probably rebuild gnutls with debug info enabled (-ggdb), this way we could know what this is: ==2845== by 0x12A3CF9F: (within /usr/lib64/libgnutls.so.26.14.7) which is where the invalid free starts, to move then inside libGL no idea how.
I remerged gnutls with CFLAGS+=-ggdb but I still got this: ==23088== Thread 1: ==23088== Invalid free() / delete / delete[] ==23088== at 0x4C24AC1: realloc (vg_replace_malloc.c:429) ==23088== by 0xC20B0E6: (within /usr/lib64/opengl/nvidia/lib/libGL.so.180.60) ==23088== by 0x100000000: ??? ==23088== by 0x12A3CF9F: (within /usr/lib64/libgnutls.so.26.14.7) ==23088== Address 0x1ed40ea8 is not stack'd, malloc'd or (recently) free'd I didn't run it all the way to a crash, so that's the only relevant error.
*** glibc detected *** /usr/lib64/mozilla-firefox/firefox: double free or corruption (!prev): 0x0000000007bd26d0 *** ======= Backtrace: ========= /lib/libc.so.6[0x7f8849442258] /lib/libc.so.6[0x7f8849447a2b] /lib/libc.so.6(realloc+0xf1)[0x7f8849447e91] /usr/lib/libgnutls.so.26(gnutls_ext_register+0x51)[0x7f883ba8aa01] /usr/lib/libgnutls.so.26[0x7f883ba8aac8] /usr/lib/libgnutls.so.26(gnutls_global_init+0x165)[0x7f883ba91135] /usr/lib/libcurl.so.4(Curl_gtls_init+0x17)[0x7f8826324ae7] /usr/lib/libcurl.so.4(curl_global_init+0x8d)[0x7f882631c52d] /opt/netscape/plugins/libflashplayer.so[0x7f88285c759a]
*** glibc detected *** /usr/lib64/mozilla-firefox/firefox: realloc(): invalid pointer: 0x000000000791a330 *** ======= Backtrace: ========= /lib/libc.so.6[0x7f898b110258] /lib/libc.so.6(realloc+0x2f8)[0x7f898b116098] /usr/lib/libgnutls.so.26(gnutls_ext_register+0x51)[0x7f897d758a01] /usr/lib/libgnutls.so.26[0x7f897d758ac8] /usr/lib/libgnutls.so.26(gnutls_global_init+0x165)[0x7f897d75f135] /usr/lib/libcurl.so.4(Curl_gtls_init+0x17)[0x7f8967ae1ae7] /usr/lib/libcurl.so.4(curl_global_init+0x8d)[0x7f8967ad952d] /opt/netscape/plugins/libflashplayer.so[0x7f8968d8259a] (gdb) bt #0 0x00007f898b0ce645 in raise () from /lib/libc.so.6 #1 0x00007f898b0cfb63 in abort () from /lib/libc.so.6 #2 0x00007f898b10aac8 in __libc_message () from /lib/libc.so.6 #3 0x00007f898b110258 in malloc_printerr () from /lib/libc.so.6 #4 0x00007f898b116098 in realloc () from /lib/libc.so.6 #5 0x00007f897d758a01 in gnutls_ext_register (type=1, name=0x7f897d7b9da6 "MAX_RECORD_SIZE", parse_type=6, recv_func=0xffffffffffffffff, send_func=0) at gnutls_extensions.c:359 #6 0x00007f897d758ac8 in _gnutls_ext_init () at gnutls_extensions.c:283 #7 0x00007f897d75f135 in gnutls_global_init () at gnutls_global.c:240 #8 0x00007f8967ae1ae7 in Curl_gtls_init () from /usr/lib/libcurl.so.4 #9 0x00007f8967ad952d in curl_global_init () from /usr/lib/libcurl.so.4 #10 0x00007f8968d8259a in ?? () from /opt/netscape/plugins/libflashplayer.so
(In reply to comment #27) > I remerged gnutls with CFLAGS+=-ggdb but I still got this: > You likely are still doing FEATURES=strip -- see http://www.gentoo.org/proj/en/qa/backtraces.xml
Sounds like there can be two different _init calls... maybe gnutls stopped supporting that?
(In reply to comment #30) > You likely are still doing FEATURES=strip FEATURES="ccache distlocks fixpackages parallel-fetch protect-owned sandbox sfperms splitdebug strict unmerge-orphans userfetch"
(In reply to comment #31) > Sounds like there can be two different _init calls... maybe gnutls stopped > supporting that? Could this be a race condition, two threads initializing gnutls concurrently? FF is heavily multithreaded, and I could imagine multiple flash plugins on a single page causing concurrent initialization.
> Could this be a race condition, two threads initializing gnutls concurrently? > FF is heavily multithreaded, and I could imagine multiple flash plugins on a > single page causing concurrent initialization. > Sounds reasonable. A quite sure way for me to reproduce the issue (or at least similar issues) is to switch back and forth between two pages loading quite large flash movies. I now also recompiled mozilla-firefox, xulrunner, libtasn1 and this is as much info as I were able to get so far (not the exact same issue as before, but most propably related to it): Core was generated by `/usr/lib/mozilla-firefox/firefox'. Program terminated with signal 11, Segmentation fault. #0 0x00007fa16a886d8b in raise () from /lib/libpthread.so.0 (gdb) bt #0 0x00007fa16a886d8b in raise () from /lib/libpthread.so.0 #1 0x00007fa168160dab in nsProfileLock::FatalSignalHandler () from /usr/lib64/xulrunner-1.9/libxul.so #2 <signal handler called> #3 0x00007fa15c003a33 in asn1_delete_structure (structure=0x7fa15c4abfe8) at structure.c:305 #4 0x00007fa15c236fb5 in gnutls_global_deinit () at gnutls_global.c:269 #5 0x00007fa146a91ab2 in Curl_gtls_cleanup () from /usr/lib/libcurl.so.4 #6 0x00007fa146a93162 in Curl_ssl_cleanup () from /usr/lib/libcurl.so.4 #7 0x00007fa146a8948d in curl_global_cleanup () from /usr/lib/libcurl.so.4 #8 0x00007fa148d34328 in ?? () from /opt/netscape/plugins/libflashplayer.so
I'm able to cause a similar crash in konqueror... Thread 1 (Thread 0x7f1984f83750 (LWP 21851)): [KCrash Handler] #5 0x00007f1980c4ccb9 in free () from /lib/libc.so.6 #6 0x00007f196d4eead6 in _asn1_remove_node (node=0xe33a60) at parser_aux.c:515 #7 0x00007f196d4f0a63 in asn1_delete_structure (structure=0x7f196d998fe8) at structure.c:316 #8 0x00007f196d723fb5 in gnutls_global_deinit () at gnutls_global.c:269 #9 0x00007f196e839ab2 in Curl_gtls_cleanup () from /usr/lib/libcurl.so.4 #10 0x00007f196e83b162 in Curl_ssl_cleanup () from /usr/lib/libcurl.so.4 #11 0x00007f196e83148d in curl_global_cleanup () from /usr/lib/libcurl.so.4 #12 0x00007f197631e328 in ?? () from /opt/netscape/plugins/libflashplayer.so #13 0x00007f197646b2dd in ?? () from /opt/netscape/plugins/libflashplayer.so #14 0x00007f1976634c0f in ?? () from /opt/netscape/plugins/libflashplayer.so #15 0x00007f1976315bd8 in ?? () from /opt/netscape/plugins/libflashplayer.so #16 0x00007f19763110fa in ?? () from /opt/netscape/plugins/libflashplayer.so #17 0x00007f1976309e8e in ?? () from /opt/netscape/plugins/libflashplayer.so #18 0x00007f197630e379 in ?? () from /opt/netscape/plugins/libflashplayer.so #19 0x000000000040dc0c in NSPluginInstance::destroy (this=0xeb23e0) at /var/tmp/portage/kde-base/nsplugins-4.2.4/work/nsplugins-4.2.4/nsplugins/viewer/nsplugin.cpp:718 #20 0x000000000040dce9 in ~NSPluginInstance (this=0xeb23e0) at /var/tmp/portage/kde-base/nsplugins-4.2.4/work/nsplugins-4.2.4/nsplugins/viewer/nsplugin.cpp:682 #21 0x000000000040c45e in NSPluginClass::timer (this=<value optimized out>) at /var/tmp/portage/kde-base/nsplugins-4.2.4/work/nsplugins-4.2.4/nsplugins/viewer/nsplugin.cpp:1336 #22 0x000000000040c51f in NSPluginClass::destroyInstance (this=0xdb9210, inst=0xeb23e0) at /var/tmp/portage/kde-base/nsplugins-4.2.4/work/nsplugins-4.2.4/nsplugins/viewer/nsplugin.cpp:1488 #23 0x0000000000416e98 in InstanceAdaptor::qt_metacall (this=0xdeeb10, _c=QMetaObject::InvokeMetaMethod, _id=0, _a=0x7fff8cfc5730) at /var/tmp/portage/kde-base/nsplugins-4.2.4/work/nsplugins-4.2.4_build/nsplugins/viewer/instanceadaptor.moc:99 #24 0x00007f19843bd70b in QDBusConnectionPrivate::deliverCall () from /usr/lib64/qt4/libQtDBus.so.4 #25 0x00007f19843be351 in QDBusConnectionPrivate::activateCall () from /usr/lib64/qt4/libQtDBus.so.4 #26 0x00007f19843be8e5 in QDBusConnectionPrivate::activateObject () from /usr/lib64/qt4/libQtDBus.so.4 #27 0x00007f19843bebb8 in QDBusActivateObjectEvent::placeMetaCall () from /usr/lib64/qt4/libQtDBus.so.4 #28 0x00007f198252e851 in QObject::event () from /usr/lib64/qt4/libQtCore.so.4 #29 0x00007f198187b8ad in QApplicationPrivate::notify_helper () from /usr/lib64/qt4/libQtGui.so.4 #30 0x00007f198188417a in QApplication::notify () from /usr/lib64/qt4/libQtGui.so.4 #31 0x00007f19830c9b6b in KApplication::notify () from /usr/lib64/libkdeui.so.5 #32 0x00007f198251fc43 in QCoreApplication::notifyInternal () from /usr/lib64/qt4/libQtCore.so.4 #33 0x00007f1982520546 in QCoreApplicationPrivate::sendPostedEvents () from /usr/lib64/qt4/libQtCore.so.4 #34 0x00007f1982544ec3 in postEventSourceDispatch () from /usr/lib64/qt4/libQtCore.so.4 #35 0x00007f1983eef841 in g_main_context_dispatch () from /usr/lib64/libglib-2.0.so.0 #36 0x00007f1983ef2e60 in g_main_context_iterate () from /usr/lib64/libglib-2.0.so.0 #37 0x00007f1983ef2ffc in g_main_context_iteration () from /usr/lib64/libglib-2.0.so.0 #38 0x00007f1975dd9271 in gtk_main_iteration () from /usr/lib/libgtk-x11-2.0.so.0 #39 0x00007f1976310bab in ?? () from /opt/netscape/plugins/libflashplayer.so #40 0x00007f1983eeff8b in g_timeout_dispatch () from /usr/lib64/libglib-2.0.so.0 #41 0x00007f1983eef841 in g_main_context_dispatch () from /usr/lib64/libglib-2.0.so.0 #42 0x00007f1983ef2e60 in g_main_context_iterate () from /usr/lib64/libglib-2.0.so.0 #43 0x00007f1983ef2ffc in g_main_context_iteration () from /usr/lib64/libglib-2.0.so.0 #44 0x00007f1982544b6f in QEventDispatcherGlib::processEvents () from /usr/lib64/qt4/libQtCore.so.4 #45 0x00007f1981900aef in QGuiEventDispatcherGlib::processEvents () from /usr/lib64/qt4/libQtGui.so.4 #46 0x00007f198251e642 in QEventLoop::processEvents () from /usr/lib64/qt4/libQtCore.so.4 #47 0x00007f198251e7d5 in QEventLoop::exec () from /usr/lib64/qt4/libQtCore.so.4 #48 0x00007f19825207d4 in QCoreApplication::exec () from /usr/lib64/qt4/libQtCore.so.4 #49 0x000000000041528f in main (argc=<value optimized out>, argv=<value optimized out>) at /var/tmp/portage/kde-base/nsplugins-4.2.4/work/nsplugins-4.2.4/nsplugins/viewer/viewer.cpp:144
Not sure if this can help (probably not) but FF is acting in a weird way with gnutls 2.8.0. Besides the behaviour reported by others, after some segfaults without other messages, now I just got this: Inconsistency detected by ld.so: dl-close.c: 344: _dl_close_worker: Assertion `tmap->l_ns == nsid' failed!
(In reply to comment #33) > Could this be a race condition, two threads initializing gnutls concurrently? > FF is heavily multithreaded, and I could imagine multiple flash plugins on a > single page causing concurrent initialization. From the docs for gnutls_global_init(): "Note! This function is not thread safe. If two threads call this function simultaneously, they can cause a race between checking the global counter and incrementing it, causing both threads to execute the library initialization code. That would lead to a memory leak. To handle this, your application could invoke this function after aquiring a thread mutex. To ignore the potential memory leak is also an option." A race condition involving gnutls_global_deinit() would explain the double free. From the docs for curl_global_cleanup(): "This function is not thread safe. You must not call it when any other thread in the program (i.e. a thread sharing the same memory) is running. This doesn't just mean no other thread that is using libcurl. Because curl_global_cleanup(3) calls functions of other libraries that are similarly thread unsafe, it could conflict with any other thread that uses these other libraries." Since curl_global_cleanup() is being called at various times, it seems to me that the caller (flash) may be using curl improperly.
(In reply to comment #37) I understand, but something is obviously changed since gnutls 2.6.6. Maybe the earlier version (prior 2.8.0) was allowing leaks or was more forgiving? anyway, the net result is that ff was usable, now it isn't. yuck.
have the same problem here, firefox crashs after a while for whatever reason. sometimes it just segfaults or give a bur error no more and some other time it gives this (see bellow). the strange thing is that it happens only with firefox. seemonkey and opera work no crashs even after extensive abuse of youtube and flash player. any how emerging curl without gnutls use flag has fixed it for now *** glibc detected *** /usr/lib64/mozilla-firefox/firefox: double free or corruption (!prev): 0x00000000067e7140 *** ======= Backtrace: ========= /lib/libc.so.6[0x7f1385637258] /lib/libc.so.6[0x7f138563ca2b] /lib/libc.so.6(realloc+0xf1)[0x7f138563ce91] /usr/lib/libgnutls.so.26(gnutls_ext_register+0x51)[0x7f13790794d1] /usr/lib/libgnutls.so.26[0x7f1379079598] /usr/lib/libgnutls.so.26(gnutls_global_init+0x165)[0x7f1379080355] /usr/lib/libcurl.so.4(Curl_gtls_init+0x17)[0x7f13641eb977] /usr/lib/libcurl.so.4(curl_global_init+0x8d)[0x7f13641e1c1d] /opt/netscape/plugins/libflashplayer.so[0x7f136548c59a] /opt/netscape/plugins/libflashplayer.so[0x7f13655d4a19] /opt/netscape/plugins/libflashplayer.so[0x7f13657a404c] /opt/netscape/plugins/libflashplayer.so[0x7f136548e3be] /opt/netscape/plugins/libflashplayer.so[0x7f13654853d9] /opt/netscape/plugins/libflashplayer.so[0x7f136547a9c8] /opt/netscape/plugins/libflashplayer.so[0x7f136547c398] /usr/lib64/xulrunner-1.9/libxul.so[0x7f13841725d2] /usr/lib64/xulrunner-1.9/libxul.so[0x7f1384178f8c] /usr/lib64/xulrunner-1.9/libxul.so[0x7f1384177d75] /usr/lib64/xulrunner-1.9/libxul.so[0x7f1384180d11] /usr/lib64/xulrunner-1.9/libxul.so[0x7f1383d4856f] /usr/lib64/xulrunner-1.9/libxul.so[0x7f1383d4b2d0] /usr/lib64/xulrunner-1.9/libxul.so[0x7f1383e45605] /usr/lib64/xulrunner-1.9/libxul.so[0x7f1383e456d7] /usr/lib64/xulrunner-1.9/libxul.so[0x7f1383e464a6] /usr/lib64/xulrunner-1.9/libxul.so[0x7f1383e46c65] /usr/lib64/xulrunner-1.9/libxul.so[0x7f1383ea19c9] /usr/lib64/xulrunner-1.9/libxul.so[0x7f1383e83450] /usr/lib64/xulrunner-1.9/libxul.so(NS_InvokeByIndex_P+0x26e)[0x7f13842e3f82] /usr/lib64/xulrunner-1.9/libxul.so[0x7f1383bcc236]
If the GNOME integration is not enabled it should not be using gnutls by itself, and Flash would probably be standing alone.
(In reply to comment #40) > If the GNOME integration is not enabled it should not be using gnutls by > itself, and Flash would probably be standing alone. > My installation has no gnome use flag enabled for firefox, but gnutls bug shows up.
OK, I built myself a LD_PRELOAD library to trace dlopen calls, as well as modified versions of curl and gnutls to trace invocations of the init and cleanup functions. Obervations: Sometimes libcurl gets dlopened several times, especially when multiple flash objects are loaded simultaneously. But I just got a hanging firefox without this, so this is not the cause of the issue, or at least not the only possible cause. I almost always got curl initialized, deinitialized, and initialized again. Sounds proper to me, but in cases where the initialization code relies on some pre-initialized state, which the cleanup code doesn't restore, this will cause problems. Haven't looked for such errors in the code yet. I dumped the thread IDs along with the PIDs. Looks like it's always the initial mozilla thread from which stuff gets loaded or initialized. There goes my idea of an initialization race causing all this mess. Here is some output I just got from a FF run with a single flash object, causing FF to hang in the end. The lines up front are my own debug additions, where ... indicates a function entry, to be concluded by a later line ending in a semicolon. The numbers up front are PID.TID, and you'll notice they are always the same, indicating the single main thread invoking all this. No gnutls at all involved in the backtrace, but a prior memory corruption might cause all kinds of delayed strangeness. 9143.9143: dlopen("libcurl.so.4", 0x1) = 0x0xaee1c28; 9143.9143: curl_global_init(1) ... // initialized=0 9143.9143: gnutls_global_init() ... // _gnutls_init=0 9143.9143: gnutls_global_init() = 0; 9143.9143: curl_global_init(1) = CURLE_OK; 9143.9143: curl_global_cleanup() ... // initialized=1 9143.9143: gnutls_global_deinit() ... // _gnutls_init=1 9143.9143: curl_global_cleanup(); 9143.9143: curl_global_init(1) ... // initialized=0 9143.9143: gnutls_global_init() ... // _gnutls_init=0 9143.9143: gnutls_global_init() = 0; 9143.9143: curl_global_init(1) = CURLE_OK; *** glibc detected *** /usr/lib/mozilla-firefox/firefox: corrupted double-linked list: 0x0aee6438 *** ======= Backtrace: ========= /lib/libc.so.6[0x4c59d7c4] /lib/libc.so.6[0x4c59da94] /lib/libc.so.6[0x4c59fc95] /lib/libc.so.6(__libc_malloc+0x5a)[0x4c5a1f8a] /usr/lib/xulrunner-1.9/libxul.so[0x422ce334] /usr/lib/xulrunner-1.9/libxul.so[0x422cdd6d] /usr/lib/xulrunner-1.9/libxul.so[0x422ce14e] /usr/lib/xulrunner-1.9/libxul.so[0x4230c292] /usr/lib/xulrunner-1.9/libxul.so(NS_CycleCollectorSuspect_P+0x2a)[0x4230c2dd] ...
Since all the init/deinit calls are ordered properly and are coming from the main thread, and since the problem only occurs with certain versions of gnutls, it must be a bug in gnutls, right?
Well, it was clear right from the beginning that the problem arises only after upgrading gnutls (and is fixed by downgrading). While it is natural to say that it is a bug in gnutls, but may be other packages are touching gnutls in an inappropriate manner ;). In any case, thanks for the efforts to actually identify the bug.
It's gnutls, and upstream has a patch for us: http://git.savannah.gnu.org/cgit/gnutls.git/patch/?id=86ca9d6ce227a877
*** Bug 260630 has been marked as a duplicate of this bug. ***
Created attachment 193921 [details] gnutls-2.8.1_pre20090608.ebuild Please test this ebuild.
Works like a charm for me! Thanks alot everyone...
(In reply to comment #47) > Created an attachment (id=193921) [edit] > gnutls-2.8.1_pre20090608.ebuild > > Please test this ebuild. > The patch applies to the tarball by itself, perhaps just the patch should be applied?
(In reply to comment #49) net-libs/gnutls-2.8.1 will be released soon.
Confirmed here: gnutls-2.8.1_pre20090608 installed, no crashes so far.
Wouldn't it be a sane idea to mask broken gnutls-2.8.0 to prevent more people from running into this issue?
net-libs/gnutls-2.8.1 is now in the tree.
