Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 39828 - emerging avidemux-2.0.20 fails using hardened-gcc
Summary: emerging avidemux-2.0.20 fails using hardened-gcc
Status: RESOLVED WORKSFORME
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Current packages (show other bugs)
Hardware: x86 Linux
: High normal (vote)
Assignee: The Gentoo Linux Hardened Team
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2004-01-29 16:06 UTC by Antony Suter
Modified: 2004-09-08 18:26 UTC (History)
0 users

See Also:
Package list:
Runtime testing required: ---


Attachments
output of my failing emerge avidemux (avidemux.log,35.43 KB, text/plain)
2004-01-29 16:08 UTC, Antony Suter
Details
source file mpegvideo_mmx_template.c as requested (mpegvideo_mmx_template.c,15.61 KB, text/plain)
2004-01-29 17:44 UTC, Antony Suter
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Antony Suter 2004-01-29 16:06:04 UTC
I have hardened-gcc installed. Emerging avidemux-2.0.20 is successful after a "hcc -R", but not after a "hcc -A".

Reproducible: Always
Steps to Reproduce:
1. hcc -A
2. emerge =avidemux-2.0.20

Actual Results:  
The compile fails with the following error snippet:-
gcc  -O3 -g -Wall -O2 -DHAVE_AV_CONFIG_H -I.. -D_FILE_OFFSET_BITS=64
-D_LARGEFILE_SOURCE -D_GNU_SOURCE -I/usr/include/malloc -c -o
i386/mpegvideo_mmx.o i386/mpegvideo_mmx.c
In file included from i386/mpegvideo_mmx.c:494:
i386/mpegvideo_mmx_template.c: In function `dct_quantize_MMX':
i386/mpegvideo_mmx_template.c:92: error: can't find a register in class
`GENERAL_REGS' while reloading `asm'

Expected Results:  
A successful compile and emerge.

Output of emerge -v avidemux will be attached.

** date_time_of_last_emerge, name_version_of_dev_tool **
20040121040357 sys-devel/binutils-2.14.90.0.8
20040121133933 sys-libs/glibc-2.3.3_pre20040117
20040121143515 sys-devel/gcc-3.3.2-r5
20040121143554 sys-devel/hardened-gcc-3.3.2.1-r2
20040121165603 sys-kernel/linux-headers-2.4.21-r1
20040121165617 sys-devel/gcc-config-1.3.4

** emerge info **
Gentoo Base System version 1.4.3.10
Portage 2.0.49-r21 (default-x86-1.4, gcc-3.3.2,
glibc-2.3.2-r9,2.3.3_pre20040117-r0, 2.6.2-rc1-mm2-as024)
=================================================================
System uname: 2.6.2-rc1-mm2-as024 i686 AMD Athlon(tm) XP
distcc 2.11.1 i686-pc-linux-gnu (protocols 1 and 2) (default port 3632) [disabled]
ACCEPT_KEYWORDS="x86"
AUTOCLEAN="yes"
CFLAGS="-march=athlon-xp -O2 -fomit-frame-pointer -ffast-math -pipe
-fstack-protector"
CHOST="i686-pc-linux-gnu"
COMPILER="gcc3"
CONFIG_PROTECT="/etc /usr/X11R6/lib/X11/xkb /usr/kde/2/share/config
/usr/kde/3.1/share/config /usr/kde/3/share/config /usr/share/config
/var/qmail/control"
CONFIG_PROTECT_MASK="/etc/gconf /etc/env.d"
CXXFLAGS="-march=athlon-xp -O2 -fomit-frame-pointer -ffast-math -pipe
-fstack-protector"
DISTDIR="/usr/portage/distfiles"
FEATURES="autoaddcvs ccache sandbox userpriv usersandbox"
GENTOO_MIRRORS="ftp://mirror.internode.on.net/pub/gentoo
http://gentoo.oregonstate.edu/
http://www.ibiblio.org/pub/Linux/distributions/gentoo"
MAKEOPTS="-j2"
PKGDIR="/usr/portage/packages"
PORTAGE_TMPDIR="/usr/local/portage-extra/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY="/usr/local/portage"
SYNC="rsync://rsync6.us.gentoo.org/gentoo-portage"
USE="X alsa apm avi berkdb bonobo composite crypt cups dvd encode esd foomaticdb
gd gdbm gif gnome gphoto2 gpm gtk gtk2 gtkhtml guile imlib ipv6 jack java
joystick jpeg ladcca libg++ libwww mad matroska mikmod motif mozilla moznoirc
moznomail mpeg ncurses nls nptl oav odbc oggvorbis openal opengl oss pam pdflib
perl png postgres python qt quicktime readline samba sdl slang spell sse ssl
tcltk tcpd tiff truetype usb wxwindows x86 xfs xinerama xml2 xv zlib"
Comment 1 Antony Suter 2004-01-29 16:08:50 UTC
Created attachment 24606 [details]
output of my failing emerge avidemux
Comment 2 solar (RETIRED) gentoo-dev 2004-01-29 16:24:42 UTC
Sounds like this package needs to filter-flags -fPIC (I've seen this happen with mmx asm before)

Bonta can you try appending this to the ebuild.

inherit flag-o-matic
filter-flags -fPIC 
Comment 3 solar (RETIRED) gentoo-dev 2004-01-29 16:28:02 UTC
And I've only seen it happen on athlon-xp with mmx.
Does the althon even support the mmx instruction? 
Seems to me it does not even support it. I think it's supposed to use 3dnow.

Please do
grep flags /proc/cpuinfo 
Comment 4 Antony Suter 2004-01-29 16:35:10 UTC
Athlons certainly do support MMX, and some of their own MMX extensions as well. However I choose not to have "mmx" in my USE flags.

hikaru root # grep flags /proc/cpuinfo
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 mmx fxsr sse syscall mmxext 3dnowext 3dnow
Comment 5 Antony Suter 2004-01-29 16:47:48 UTC
The ebuild already contained an inherit line with flag-o-matic. Adding a "filter-flags -fPIC" after the inherit line had no effect.

** diff of logs: **
--- avidemux.log	2004-01-30 10:33:17.790708853 +1100
+++ avidemux2.log	2004-01-30 11:40:53.378986687 +1100
@@ -1,6 +1,6 @@
-]1;]2;Started emerge on: Jan 30, 2004 10:30:54]1;]2; *** emerge --verbose avidemux]1;]2; >>> emerge (1 of 1) media-video/avidemux-2.0.20 to /]1;]2; === (1 of 1) Cleaning (media-video/avidemux-2.0.20::/usr/portage/media-video/avidemux/avidemux-2.0.20.ebuild)eutils
+]1;]2;Started emerge on: Jan 30, 2004 11:38:22]1;]2; *** emerge --verbose avidemux]1;]2; >>> emerge (1 of 1) media-video/avidemux-2.0.20 to /]1;]2; === (1 of 1) Cleaning (media-video/avidemux-2.0.20::/usr/local/portage/media-video/avidemux/avidemux-2.0.20.ebuild)eutils
 flag-o-matic
-]1;]2; === (1 of 1) Compiling/Merging (media-video/avidemux-2.0.20::/usr/portage/media-video/avidemux/avidemux-2.0.20.ebuild)eutils
+]1;]2; === (1 of 1) Compiling/Merging (media-video/avidemux-2.0.20::/usr/local/portage/media-video/avidemux/avidemux-2.0.20.ebuild)eutils
 flag-o-matic
 eutils
 flag-o-matic
@@ -255,7 +255,7 @@
 aclocal.m4:10805: _AM_DIRNAME is expanded from...
 configure.in:761: the top level
 /bin/sh ./config.status --recheck
-running /bin/sh ./configure  --prefix=/usr --host=i686-pc-linux-gnu --mandir=/usr/share/man --infodir=/usr/share/info --datadir=/usr/share --sysconfdir=/etc --localstatedir=/var/lib --with-gnu-ld --disable-warnings CC=gcc CFLAGS=-march=athlon-xp -O2 -fomit-frame-pointer -ffast-math -pipe -fstack-protector CXXFLAGS=-march=athlon-xp -O2 -fomit-frame-pointer -ffast-math -pipe -fstack-protector CXX=g++ host_alias=i686-pc-linux-gnu  --no-create --no-recursion
+running /bin/sh ./configure  --prefix=/usr --host=i686-pc-linux-gnu --mandir=/usr/share/man --infodir=/usr/share/info --datadir=/usr/share --sysconfdir=/etc --localstatedir=/var/lib --with-gnu-ld --disable-warnings CC=gcc CFLAGS=-march=athlon-xp -O2 -fomit-frame-pointer -ffast-math -pipe -fstack-protector -yet_exec CXXFLAGS=-march=athlon-xp -O2 -fomit-frame-pointer -ffast-math -pipe -fstack-protector -yet_exec CXX=g++ LDFLAGS= -yet_exec host_alias=i686-pc-linux-gnu  --no-create --no-recursion
 configure: WARNING: If you wanted to set the --build type, don't use --host.
     If a cross compiler is detected then cross compile mode will be used.
 checking build system type... i686-pc-linux-gnu
@@ -592,7 +592,7 @@
 make: *** [all] Error 2
 
 !!! ERROR: media-video/avidemux-2.0.20 failed.
-!!! Function src_compile, Line 66, Exitcode 2
+!!! Function src_compile, Line 67, Exitcode 2
 !!! make failed
 
 Calculating dependencies    ...done!
Comment 6 solar (RETIRED) gentoo-dev 2004-01-29 17:00:49 UTC
Can you please attach me a copy of the i386/mpegvideo_mmx_template.c
Comment 7 Antony Suter 2004-01-29 17:44:01 UTC
Created attachment 24608 [details]
source file mpegvideo_mmx_template.c as requested

From my undeleted portage build directory...
Comment 8 solar (RETIRED) gentoo-dev 2004-01-29 17:49:08 UTC
Could you also try filter-flags -fstack-protector while I read this attachment.
Comment 9 solar (RETIRED) gentoo-dev 2004-01-29 18:05:24 UTC
It's somewhere in this chunk of code I just dont know where yet :)

        asm volatile(
            "movd %%eax, %%mm3                  \n\t" // last_non_zero_p1
            SPREADW(%%mm3)
            "pxor %%mm7, %%mm7                  \n\t" // 0
            "pxor %%mm4, %%mm4                  \n\t" // 0
            "movq (%2), %%mm5                   \n\t" // qmat[0]
            "pxor %%mm6, %%mm6                  \n\t"
            "psubw (%3), %%mm6                  \n\t" // -bias[0]
            "movl $-128, %%eax                  \n\t"
            ".balign 16                         \n\t" 
            "1:                                 \n\t"
            "pxor %%mm1, %%mm1                  \n\t" // 0
            "movq (%1, %%eax), %%mm0            \n\t" // block[i]
            "pcmpgtw %%mm0, %%mm1               \n\t" // block[i] <= 0 ? 0xFF : 0x00
            "pxor %%mm1, %%mm0                  \n\t"
            "psubw %%mm1, %%mm0                 \n\t" // ABS(block[i])   
            "psubusw %%mm6, %%mm0               \n\t" // ABS(block[i]) + bias[0]
            "pmulhw %%mm5, %%mm0                \n\t" // (ABS(block[i])*qmat[0] - bias[0]*qmat[0])>>16
            "por %%mm0, %%mm4                   \n\t"
            "pxor %%mm1, %%mm0                  \n\t"
            "psubw %%mm1, %%mm0                 \n\t" // out=((ABS(block[i])*qmat[0] - bias[0]*qmat[0])>>16)*sign(block[i])
            "movq %%mm0, (%5, %%eax)            \n\t"
            "pcmpeqw %%mm7, %%mm0               \n\t" // out==0 ? 0xFF : 0x00
            "movq (%4, %%eax), %%mm1            \n\t"
            "movq %%mm7, (%1, %%eax)            \n\t" // 0
            "pandn %%mm1, %%mm0                 \n\t"
            PMAXW(%%mm0, %%mm3)
            "addl $8, %%eax                     \n\t"
            " js 1b                             \n\t"
            "movq %%mm3, %%mm0                  \n\t"
            "psrlq $32, %%mm3                   \n\t"
            PMAXW(%%mm0, %%mm3)
            "movq %%mm3, %%mm0                  \n\t"
            "psrlq $16, %%mm3                   \n\t"
            PMAXW(%%mm0, %%mm3)
            "movd %%mm3, %%eax                  \n\t"
            "movzbl %%al, %%eax                 \n\t" // last_non_zero_p1
            : "+a" (last_non_zero_p1)
            : "r" (block+64), "r" (qmat), "r" (bias),
              "r" (inv_zigzag_direct16+64), "r" (temp_block+64)
        );


What I need next is to see the output of this command.

gcc  -O3 -g -Wall -O2 -DHAVE_AV_CONFIG_H -I.. -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_GNU_SOURCE -I/usr/include/malloc -E i386/mpegvideo_mmx.c

This will make a single pass with cpp which should resolve those macro's for us.

Also in the code I see this comment.
-----------
// note the asm is split cuz gcc doesnt like that many operands ...
-----------
ssp will add more operands so I'm wondering if we need to split that code up even more. I'm not sure if that makes sense to you or not.. If it does please try that as well.
Comment 10 solar (RETIRED) gentoo-dev 2004-01-29 18:07:23 UTC
Could you also try without -O3
Comment 11 Alexander Gabert (RETIRED) gentoo-dev 2004-03-01 06:50:11 UTC
yes, solar, SSP is prone to limit available registers because of the nature it fetches the guard and stores it in the canary on the stack.

this may have impact on compiling inline assembler into object code.

running with -yet_exec and -fno-stack-protector is suggested.

also look out for some kind of -DEFINE in the Makefile which can prohibit the compilation of specific MMX code.
Comment 12 Alexander Gabert (RETIRED) gentoo-dev 2004-03-03 04:55:01 UTC
what is the status here now?
shall we insert sed logic to make the file build without PIE and without SSP at all?
Comment 13 solar (RETIRED) gentoo-dev 2004-09-08 18:26:17 UTC
No feedback from bug reporter and hgcc is obsolete now. Please file a new bug is problem persists.