After getting a Segmentation fault with no further details when running the Linux native 32 bit Adobe Acrobat Reader, I prefixed the executable within /opt/Adobe/Reader9/bin/acroread script file with gdb. Line #686 -- LaunchBinary "$ACRO_EXEC_CMD" ${1+"$@"} ++ LaunchBinary gdb "$ACRO_EXEC_CMD" ${1+"$@"} I'm now getting a good backtrace when running "/opt/Adobe/Reader9/bin/acroread" directly. (Executing the symbolic link file /opt/bin/acroread will only result in another silent error.) Below is the backtrace. Reproducible: Always Steps to Reproduce: 1. Prefix the executable within line #686 with "gdb" 2. Execute "/opt/Adobe/Reader9/bin/acroread" and type run within the gdb shell 3. Type bt (backtrace) Actual Results: Hey, I have problems with this binary package. Expected Results: Binaries should just run like magic without any problems! Program received signal SIGSEGV, Segmentation fault. _______________________________________________________________________________ eax:00000000 ebx:43F58000 ecx:09BD1948 edx:09BD1954 eflags:00010246 esi:0981CB28 edi:413612D4 esp:FFFFB508 ebp:09BD2180 eip:43F4F7A7 cs:0023 ds:002B es:002B fs:0000 gs:0063 ss:002B o d I t s Z a P c [002B:FFFFB508]---------------------------------------------------------[stack] FFFFB538 : 00 00 00 00 00 00 00 00 - CB D2 F3 43 00 B0 37 41 ...........C..7A FFFFB528 : 00 00 F0 3F 00 00 00 00 - 00 00 00 00 00 00 00 00 ...?............ FFFFB518 : 7C CB 81 09 54 19 BD 09 - 20 89 BF 09 00 00 00 00 |...T... ....... FFFFB508 : 28 CB 81 09 F8 06 BF 09 - 0A 02 00 00 00 28 00 00 (............(.. [002B:0981CB28]---------------------------------------------------------[ data] 0981CB28 : 10 09 BF 09 02 00 00 00 - 00 00 00 00 E0 FD BE 09 ................ 0981CB38 : 60 63 BA 09 18 CB 81 09 - 00 00 00 00 00 00 F0 3F `c.............? [0023:43F4F7A7]---------------------------------------------------------[ code] => 0x43f4f7a7 <_pango_cairo_font_get_metrics+151>: movapd %xmm0,0x40(%esp) 0x43f4f7ad <_pango_cairo_font_get_metrics+157>: movapd -0x32d0(%ebx),%xmm0 0x43f4f7b5 <_pango_cairo_font_get_metrics+165>: movapd %xmm0,0x50(%esp) 0x43f4f7bb <_pango_cairo_font_get_metrics+171>: xorpd %xmm0,%xmm0 0x43f4f7bf <_pango_cairo_font_get_metrics+175>: movapd %xmm0,0x60(%esp) 0x43f4f7c5 <_pango_cairo_font_get_metrics+181>: call 0x43f4e0a0 <pango_font_get_font_map@plt> ------------------------------------------------------------------------------ 0x43f4f7a7 in _pango_cairo_font_get_metrics () from /usr/lib32/libpangocairo-1.0.so.0 gdb> quit
I recompiled pango with the "debug" USE flag and now get the following for a backtrace: Program received signal SIGSEGV, Segmentation fault. _______________________________________________________________________________ eax:09C4ACF8 ebx:43AE1000 ecx:00000094 edx:00000001 eflags:00010292 esi:09C4AD7C edi:FFFF88F0 esp:FFFF8634 ebp:09C4ACF8 eip:439F890D cs:0023 ds:002B es:002B fs:0000 gs:0063 ss:002B o d I t S z A p c [002B:FFFF8634]---------------------------------------------------------[stack] FFFF8664 : 7C 00 00 00 58 8F C3 09 - 77 00 00 00 B9 47 F4 4F |...X...w....G.O FFFF8654 : 6E 00 00 00 00 00 00 00 - 77 00 00 00 00 00 00 00 n.......w....... FFFF8644 : 38 00 00 00 00 00 00 00 - 5B 00 00 00 00 00 00 00 8.......[....... FFFF8634 : 7C AD C4 09 28 00 00 00 - 03 00 00 00 00 00 00 00 |...(........... [002B:09C4AD7C]---------------------------------------------------------[ data] 09C4AD7C : 02 00 00 00 00 00 00 00 - 00 00 00 00 04 00 00 00 ................ 09C4AD8C : 02 00 00 00 01 00 00 00 - F0 56 A2 04 00 00 00 00 .........V...... [0023:439F890D]---------------------------------------------------------[ code] => 0x439f890d <_cairo_scaled_font_glyph_device_extents+45>: movdqa %xmm0,0x70(%esp) 0x439f8913 <_cairo_scaled_font_glyph_device_extents+51>: call 0x4399ba80 <_cairo_font_options_get_round_glyph_positions> 0x439f8918 <_cairo_scaled_font_glyph_device_extents+56>: mov 0x4(%ebp),%edx 0x439f891b <_cairo_scaled_font_glyph_device_extents+59>: mov %eax,0x40(%esp) 0x439f891f <_cairo_scaled_font_glyph_device_extents+63>: test %edx,%edx 0x439f8921 <_cairo_scaled_font_glyph_device_extents+65>: jne 0x439f9110 <_cairo_scaled_font_glyph_device_extents+2096> ------------------------------------------------------------------------------ 0x439f890d in _cairo_scaled_font_glyph_device_extents () from /usr/lib32/libcairo.so.2 gdb>
I have further backtraced this bug to likely x11-libs/cairo-1.14.2 breaks when compiling with"-Ofast" CFLAG. Noting open Bug #428032 "x11-libs/cairo: ignores cflags"
/proc/cpuinfo: Intel(R) Core(TM) i7-3770K CPU @ 3.50GHz The following packages appear to break =app-text/acroread-9.5.5-r3 when compiled with -Ofast: =x11-libs/cairo-1.14.2 =x11-libs/pango-1.36.8 WORKAROUND: File: /etc/portage/env/cflags-safe-nofast.conf CFLAGS="-march=corei7 -pipe" CXXFLAGS="${CFLAGS}" File: /etc/portage/package.env/env # Acroread breaks when cairo/pango are compiled with -Ofast x11-libs/cairo cflags-safe-nofast.conf x11-libs/pango cflags-safe-nofast.conf
On the one hand, this report reads like 100% user error. Enabling -Ofast globally is crazy (read the flag description: "disregard strict standards compliance", why would you do that when building code that you are not familiar with!), many libraries on your system should be broken, somehow you haven't noticed... On the other hand, cairo and pango are performance-critical libraries and some users might try compile them with aggressive flags in a misguided attempt to e.g. improve browser rendering performance on a slow machine. So perhaps we should explicitly filter out -Ofast in the ebuild.
Rostovtsev: Coding errors or poor coding techniques become more critical or more apparent when using more aggressive CFLAGS. (Or stuff just happens when coding. ;-) The easy method of avoiding coding problems, is to just avoid certain CFLAGS. Sometimes if I have time, I'll provide backtraces which allow the coders to fix their code. Most times, getting good backtraces can be extremely difficult, especially with binary packages. In this case, it looks like I lucked-out and successfully provided backtraces to two packages. The backtraces can be reported upstream to the pango/cairo package maintainers. Now that I recall, I was reading an Intel Developer article, which detailed some of their recommended CFLAGS. Some interesting information here. http://software.intel.com/en-us/blogs/2012/09/26/gcc-x86-performance-hints I agree with users adhering to safe CFLAGS if they do not want to spend time debugging or performing backtraces. But if people want to spend time debugging and performing backtraces, then go ahead and experiment as it will inevitably create more stable code. Cheers. BOTTOM LINE: Ditto, filter the CFLAG. If people here have more time available, push the backtraces upstream to the pango/cairo package maintainers.
The description of the -Ofast flag is "disregard strict standards compliance". When you use -Ofast, you are not compiling C. You are compiling an undocumented gcc-specific dialect which is 98% similar to C. If it works - great. If it doesn't work - you can't really blame upstream developers for expecting your compiler to follow the language standard.
So we blame GCC? ;-)
GCC is not to blame, -Ofast documentation specifically states that this option operates outside the C standard. If you want standards compliance, do not use -Ofast.
Can you reproduce the crash with -O3 instead of -Ofast? Also, which version of gcc?
-O3 appears safe here. These days, most if not all packages appear safe with this optimization level except for sometimes lower level packages such as sometimes Grub. CFLAGS="-Ofast -march=corei7 -pipe" CXXFLAGS="${CFLAGS}" Instead of looking for blame, probably a lot easier to look for what needs to be fixed in order for the problem to be resolved. If one spends enough time thinking, there's always a fix! ;-) ... just usually never enough time for coding the fix. Kind of ridiculous looking for whom to blame, as everybody is just going to say, "Not me!" Looks like I can further possibly easily narrow the specific flag, as -Ofast implies all three "-ffast-math and the Fortran-specific -fno-protect-parens and -fstack-arrays." flags. I've seen breaks usually occur with "-ffast-math", but have no apparent experience with "-fno-protect-parens and -fstack-arrays" flags. If the problem is with "-ffast-math", I think I've seen those resolved over time.
(In reply to Roger from comment #10) Yes, please check with -ffast-math. I would much rather filter just 2 known bad flags than force users to stick with plain -O2.
Oops. Seems I assumed I was already compiling with "-O3" level safely. Technically I was, but you'll notice I likely confused the terminology above. Further debugging shows x11-libs/cairo x11-libs/pango cannot handle "-O3" optimization level either. I'll see if I can find the specific optimization flag, but in terms of this bug, "-O3" and "-Ofast" breaks acroread. (Tried "-Os", and "-Os" is safe.)
"-ftree-loop-vectorize" breaks x11-libs/cairo x11-libs/pango compile. (configure: error: C compiler cannot create executables) (Syntax error? Odd as -O3 and -Ofast compile just fine, even odder that this is related with vertorize per the next and source of the problem.) "-ftree-slp-vectorize" is very likely the source cause of x11-libs/cairo & x11-libs/pango runtime segmentation fault for acroread. Upstream should have plenty of information, and likely just filtering "-O3" and "-Ofast" may resolve the problem temporarily. Might want to either: Place a note within the either pango, cair, or acroread ebuilds to test future versions against such CFLAGS or just advise users within stdout of the ebuilds noting the breaks with such CFLAG usage. Funny a binary package detected such a subtle problem not readily seen with other open source packages! Thanks for your time on this one.
... and now I've reversed the fix to further prove vectorizing the cause. Runtime x11-libs/cairo and x11-libs/pango will perform fine with acroread using the following: CFLAGS="-march=corei7 -O3 -pipe -fno-tree-slp-vectorize" CXXFLAGS="${CFLAGS}" MAKEOPTS="-j1" NOTE: "-j1" is required, as building x11-libs/cairo and x11-libs/pango seems to break during the configure test C compile scenarios when using "-j8". Probably again related to vectorizing, but haven't further narrowed. I'll be using the above for my compiling options for cairo & pango.
This afternoon I updated to pango-1.36.8-r1 (from -r0) and what do you know, acroread started dying, with exactly the same symptom/backtrace as described here. Only difference in my case: - I'm not using -O3, -Ofast or any other insane flags - it was now built with gcc 5.2 (previously 4.9.x) Since I was using the quite pedestrian default flags: -pipe -O2 -march=native -mfpmath=sse (the -mfpmath flag was a leftover from when the entire system was 32bit) ..I started wondering why this would fail since -r0 worked fine, and reduced the CFLAGS until I found the culprit: in my case it was, surprisingly, -mfpmath=sse! Since it's the default fp mode in the 64bit world and acroread is hardly performance-sensitive, I removed it from the default flags, rebuilt pango and acroread was immediately happy again. This makes me believe that the problem is not *only* -O3 (aka the tree-vectorizer). Hope this helps someone who has -mfpmath=sse enabled in their 32bit world.
Would have been nice to see a GDB backtrace to see if the break was similar to mine, but involves a little more tinkering around to employ gdb on the acroread script file. (ie. At the end of the file, add gdb just before the call to the main binary, and then execute the script.)
See also Bug #562612 - =app-text/acroread-9.5.5-r3 fails with "Cannot access memory ..." Looks like acroread cannot make use of media-libs/fontconfig when also compiled using vectorization. (Workaround, compile media-libs/fontconfig also using CFLAGS="-march=corei7 -Ofast -pipe -fno-tree-slp-vectorize") I'm wondering if the problem I'm seeing (also mentioned within the previous mentioned newer bug), if this problem is specifically localized to acroread and it's later versions, or if this is a problem localized to 32 bit compiled pango/cairo/fontconfig on a 64 bit platform using the USE ABI flag. In my many years using these CFLAGS and other CFLAGS, I've never seen this with any other source compiled binary until just recently. Just converted to the USE ABI flag this year for using 32 bit applications on a 64 bit platform, and just with these later versions of acroread. All other open source compiled binaries appear to function normally without any anomalies.
(In reply to Roger from comment #17) You will see this problem with acroread also with zlib when built with -O3 (i.e. vectorization). This is because acoread embeds parts of zlib in the package, but then _also load the shared library on top_. The mind boggles. All this really means is that the current multilib way is, in the long term, doomed.
Thanks for noting this possibility. Then it sounds like the current problems with multilib might be limited to pre-compiled 32 bit binaries, for example acroread? And these bugs could possibly be fixed upstream, for example acroread, if companies such as Adobe wish to do so?
(In reply to Roger from comment #19) > Thanks for noting this possibility. Then it sounds like the current > problems with multilib might be limited to pre-compiled 32 bit binaries, for > example acroread? And these bugs could possibly be fixed upstream, for > example acroread, if companies such as Adobe wish to do so? In theory they could, yes. In practice acroread for Linux is abandonware, i.e. dead.
*** Bug 552508 has been marked as a duplicate of this bug. ***
*** Bug 562612 has been marked as a duplicate of this bug. ***
There is a huge amount of crazy in this bug. Not something I want to encourage, must less support.
