Changing the --enable-assembler to --disable-assembler in the ebuild does get rid of the textrel Blue root # check-textrel /usr/sbin/mysqld TEXTREL in exe /usr/sbin/mysqld 03:33 < pipacs> 0044cf60 00034f01 R_386_32 0051b860 _dig_vec 03:33 < pipacs> 0044cf75 00034f01 R_386_32 0051b860 _dig_vec 03:33 < pipacs> didn't we run into these they other day? 03:34 < swtaylor> yes minus debug symbols 03:34 < pipacs> ok, let's see where they are now 03:35 < pipacs> longlong2str 03:47 < pipacs> aha, we have in fact strings/longlong2str.c and strings/longlong2str-x86.s 03:47 < pipacs> let me guess, you used the latter and it's not PIC 03:48 < pipacs> yeah that's it 03:56 < pipacs> well, i think we can just use the C version for now 03:57 < pipacs> problem is that it's highly optimized asm as far as free registers are concerned (none ;P) 04:00 < pipacs> pass --disable-assembler to configure or at least don't pass --enable-assembler 04:01 < pipacs> probably the ebuild should be modified to do this depending on USE=pic or something 04:02 < pipacs> for now just take out the --enable-assembler from the ebuild and you'll be fine 04:02 < pipacs> and open a bug and suggest that it should be made dependent on pic
just turning off the assembler option is not an acceptable solution. it provides a large peformance boost. _dig_vec is char NEAR _dig_vec[] = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ"; in strings/int2str.c how does one get this to be PIC instead... ?
Its assembler code in the longlong2str-x86 that's the issue, and I'm not exactly sure what it'd take to get it to be that way. Perhaps an issue for upstream. It appears to be the only chunk of their code that isn't pic-happy though.
longlong2str-x86 isn't the only asm code in mysql, but it is the only asm that has an external reference to a variable (_dig_vec). I'm thinking variables are handled differently in asm when PIC is being used. any PIC gurus on this? I do program asm, but i've never delved into PIC.
Easiest place to work out how to write PIC asm code if you've not done so before is to look at the nasm help files. Try 'info nasm' and see section 8.2. When you need to do it with gcc, the simplest way to figure out the syntax is to run gcc over some example code with -S -fPIC and -S -fno-pic, compare and contrast. I just had a scan through the version of mysql installed here (4.0.24) and I think the references to _dig_vec in longlong2str-x86.s are the only problem. However as pipacs said, all registers are in use in the inner loop - so good luck :)
Hmm; a thought. _dig_vec is simply used to convert from 0-35 to '0'-'9''A'-'Z'. It might be efficient enough to just remove the lookup from inside the loop, and append a loop to do the conversion to ascii afterwards (perhaps change the loop at L160 to do that instead of a simple rep movsb). If you do the conversion with _dig_vec, you'll only need to acquire the GOT reference once; alternatively you could actually eliminate even that and hard-code the conversion (i.e. loop each char, add 48, if the result>57 add 7 more) - that may actually be faster depending on whether the contents of _dig_vec are likely to be in the cache.
Created attachment 59072 [details, diff] fix non-PIC code, 2nd try first stab at converting the x86 asm code to PIC while more or less preserving the original's speed, comments/fixes are welcome. on another note, bugs 92776 and 92778 can probably be marked as duplicates.
*** Bug 92776 has been marked as a duplicate of this bug. ***
*** Bug 63885 has been marked as a duplicate of this bug. ***
*** Bug 92778 has been marked as a duplicate of this bug. ***
*** Bug 86869 has been marked as a duplicate of this bug. ***
thanks for the patch, I'm hoping to roll out 4.0.24-r2 this evening (as part of the mysql-4.1 stuff), and I'll put the patch in that. It'll be hard-masked for a day or two, for further testing before it hits ~arch.
after testing, I find that the patch breaks mysql. TEST RESULT ------------------------------------------ alias [ pass ] alter_table [ fail ] Errors are (from /var/tmp/portage/mysql-4.0.24-r2/work/mysql/mysql- test/var/log/mysqltest-time) : /var/tmp/portage/mysql-4.0.24-r2/work/mysql/client/.libs/mysqltest: At line 15: query 'insert into t1 values (2,4,3,5,"PENDING",1,7)' failed: 2013: Lost connection to MySQL server during query ./mysql-test-run: line 615: 9686 Killed $@ >>$CUR_MYERR 2>&1 (wd: /var/tmp/portage/mysql-4.0.24-r2/work/mysql) (the last line(s) may be the ones that caused the die() in mysqltest) My port of the patch to 4.1.12 fares a little better, but not much func_math [ pass ] func_misc [ fail ] Errors are (from /dev/shm/portage/mysql-4.1.12/work/mysql/mysql- test/var/log/mysqltest-time) : /dev/shm/portage/mysql-4.1.12/work/mysql/client/.libs/mysqltest: At line 11: query 'select hex(inet_aton('127'))' failed: 2013: Lost connection to MySQL server during query (the last lines may be the most important ones) Aborting: func_misc failed in default mode. To continue, re-run with '--force'. I've put the new ebuilds into the tree without the patch for now.
(In reply to comment #12) > ./mysql-test-run: line 615: 9686 Killed $@ >>$CUR_MYERR 2>&1 > (wd: /var/tmp/portage/mysql-4.0.24-r2/work/mysql) > (the last line(s) may be the ones that caused the die() in mysqltest) under what kernel did this happen? in particular, was PaX in it and if so, did you get any messages logged by PaX? if you could get a coredump + bactkrace, i'd appreciate it too, i never set up/ran mysql myself, so reproducing this here will take a while.
sorry, I should have specified that when mentioned it to you. I haven't tested it with PaX kernels at all, only the existing kernels on my test machines (2.6.11-rc3-mm1 and 2.6.11-gentoo-r6).
Comment on attachment 59072 [details, diff] fix non-PIC code, 2nd try >--- mysql-4.0.24/strings/longlong2str-x86.s 2005-03-05 00:38:14.000000000 +0000 >+++ mysql-4.0.24-fixed/strings/longlong2str-x86.s 2005-05-17 01:37:52.000000000 +0100 >@@ -19,6 +19,13 @@ > .file "longlong2str.s" > .version "1.01" > >+ .section .rodata >+ .align 32 >+ .type _dig_vec, @object >+ .size _dig_vec, 37 >+_dig_vec: >+ .string "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ" >+ > .text > .align 4 > >@@ -31,11 +38,14 @@ longlong2str: > pushl %esi > pushl %edi > pushl %ebx >+ >+ call __i686.get_pc_thunk.bx >+ addl $_GLOBAL_OFFSET_TABLE_,%ebx >+ > movl 100(%esp),%esi # Lower part of val > movl 104(%esp),%ebp # Higher part of val > movl 108(%esp),%edi # get dst >- movl 112(%esp),%ebx # Radix >- movl %ebx,%eax >+ movl 112(%esp),%eax # Radix > testl %eax,%eax > jge .L144 > >@@ -50,7 +60,7 @@ longlong2str: > adcl $0,%ebp > negl %ebp > .L146: >- negl %ebx # Change radix to positive >+ negl 112(%esp) # Change radix to positive > jmp .L148 > .align 4 > .L144: >@@ -77,12 +87,12 @@ longlong2str: > > movl %ebp,%eax # High part of value > xorl %edx,%edx >- divl %ebx >+ divl 112(%esp) > movl %eax,%ebp > movl %esi,%eax >- divl %ebx >+ divl 112(%esp) > movl %eax,%esi # quotent in ebp:esi >- movb _dig_vec(%edx),%al # al is faster than dl >+ movb _dig_vec@GOTOFF(%ebx,%edx),%al # al is faster than dl > decl %ecx > movb %al,(%ecx) # store value in buff > .align 4 >@@ -93,14 +103,13 @@ longlong2str: > jl .L153 > je .L160 # Ready > movl %esi,%eax >- movl $_dig_vec,%ebp > .align 4 > > .L154: # Do rest with integer precision > cltd >- divl %ebx >+ divl 112(%esp) > decl %ecx >- movb (%edx,%ebp),%dl # bh is always zero as ebx=radix < 36 >+ movb _dig_vec@GOTOFF(%ebx,%edx),%dl > testl %eax,%eax > movb %dl,(%ecx) > jne .L154 >@@ -138,3 +147,13 @@ longlong10_to_str: > > .L10end: > .size longlong10_to_str,.L10end-longlong10_to_str >+ >+ .section .gnu.linkonce.t.__i686.get_pc_thunk.bx,"ax",@progbits >+.globl __i686.get_pc_thunk.bx >+ .hidden __i686.get_pc_thunk.bx >+ .type __i686.get_pc_thunk.bx, @function >+__i686.get_pc_thunk.bx: >+ movl (%esp), %ebx >+ ret >+ >+ .section .note.GNU-stack,"",@progbits >diff -Nurp mysql-4.0.24/strings/strings-x86.s mysql-4.0.24-fixed/strings/strings-x86.s >--- mysql-4.0.24/strings/strings-x86.s 2005-03-05 00:38:15.000000000 +0000 >+++ mysql-4.0.24-fixed/strings/strings-x86.s 2005-05-17 01:37:47.000000000 +0100 >@@ -403,3 +403,5 @@ next_str: > ret > .strxmov_end: > .size strxmov,.strxmov_end-strxmov >+ >+ .section .note.GNU-stack,"",@progbits
doh, silly typo in the patch, please test it again (this time i tried it on a sample app and it should work, compared to the previous one that did crash indeed, due to bad relocs, i still don't get why gcc/as/ld mistreated the wrong section name...).
just for we poor humans the diff from the last two patches of PaX Team is this one: 7c7 < + .section rodata --- > + .section .rodata
Created attachment 59631 [details, diff] mysql-4.1.12-x86_asm-pic-fixes-r1.patch The patch modified by "PaX Team" work for me on mysql-4.0.24 and with a vanilla kernel. The attached one is the same patch modified to work with 4.1.12 relase. Tests are still running.
> The attached one is the same patch modified to work with 4.1.12 relase. Tests > are still running. if 4.1.x has strings/strings-x86.s (and/or other .s files) then it/they will also need the one-liner marking for proper GNU-stack handling (that is, we don't want the resulting libraries to get marked with an executable PT_GNU_STACK).
Created attachment 59657 [details] #find . -perm +u+x -and -not -type d -print -exec eu-findtextrel {} \; Admitting that I'm totally a newbie on these arguments, can u have a look the attached output from this "eu-findtextrel" executed on the "image" of mysql-4.1.12 ready to be installed ? Is this all we need ? Do you need a "readelf -a" of the same? infobit n.2 longlong2str-x86.s for mysql-4.1 has a little additional section. For that I've understund untill now it's clean. infobit n.3 there is only one new asm file "my_strtoll10-x86.s" and also this seem to be clean. infobit n.4 mysql-4.1 compile successfully here. with standard kernel and gcc, compile it with an hardened one is needed ? Pax I've not understud comment #19 can you point me on some place where (briefly) study something ?
forgive me I've en that the original patch modify string.s too, pointer to understud that is apreciated anyway ;)
Created attachment 59662 [details, diff] mysql-4.1.12-x86_asm-pic-fixes-r1.patch looks better ?
(In reply to comment #20) > Created an attachment (id=59657) [edit] > #find . -perm +u+x -and -not -type d -print -exec eu-findtextrel {} \; > > Admitting that I'm totally a newbie on these arguments, can u have a look the > attached output from this "eu-findtextrel" executed on the "image" of > mysql-4.1.12 ready to be installed ? > Is this all we need ? Do you need a "readelf -a" of the same? well, eu-findtextrel should have printed at least something like "no text relocations reported in [...]", so i'm not sure what you did above ;-). in any case, the goal would be no textrels in any executable files, so once you can verify that, we're good to go. > infobit n.2 longlong2str-x86.s for mysql-4.1 has a little additional section. > For that I've understund untill now it's clean. > > infobit n.3 there is only one new asm file "my_strtoll10-x86.s" and also this > seem to be clean. i took a look at these files and they'll need additional patches for PIC, i'll work on them as time permits. > infobit n.4 mysql-4.1 compile successfully here. with standard kernel and gcc, > compile it with an hardened one is needed ? yes, it would be nice because the hardened gcc profile will create PIEs (position independent executables) that can have textrels. > Pax I've not understud comment #19 can you point me on some place where > (briefly) study something ? search the grsecurity forums, it has been discussed extensively there.
Created attachment 59750 [details, diff] PIC fix for mysql-4.1.12 this is for 4.1.x, let's hope it works as well ;-).
Created attachment 59794 [details] eu-findtextrel.txt change of strategy, the attachment eu-findtextrel.txt is the output of the following commands: mkdir tmp && cd tmp && tar -jxvf ../mysql-4.1.12-hardened.tbz2 find . -perm +u+x -and -not -type d -exec file {} \; \ | grep "ELF 32-bit LSB shared object" \ | sed -e "s|:.*$||" \ > list.txt while read i ; do eu-findtextrel "$i" ; done < list.txt 2> eu-findtextrel.txt Additionally at http://www.francesco-riosa.com/gentoo/misc/PaX_Team/ there are the following files: File name | Description ---------------------------+---------------- CFLAGS-hardened.txt | C*FLAGS (and emerge command) emerge--info-hardened.txt | emerge --info output gcc-hardened-output.txt.gz | compile output mysql-4.1.12-hardened.tbz2 | image after building | CFLAGS.txt | emerge--info.txt | gcc-output.txt.gz | mysql-4.1.12.tbz2 | | mysql_overlay_nightly-20 ->| overlay used (few diff from gentoo ones) -> 050525-1607.tar.gz | this ones will stay there untill this bug will be fixed (or the web server crashed) @PaX Team Check we have spoken via mail are still to do, I've scheduled them in next few days, mainly because of lack of knowledge this side (wanna learn) .
Using the patch attachment #59750 [details, diff] from comment #24 mysql compiles/starts up runs and is free of every text relocation. I did not run with FEATURES=test enabled.
I, however, DID try the corrected patch here on a hardened box (that is running bugzilla happily now) with FEATURES="maketest" and none of the tests failed, although a few were skipped. IF it interests you, Robin, solar, please let me know and I can paste the full output of the tests.
(In reply to comment #25) > @PaX Team > Check we have spoken via mail are still to do, I've scheduled them in next few > days, mainly because of lack of knowledge this side (wanna learn) . i think we have enough evidence now that 1. there're no more textrels, 2. stuff still works as expected. what i'd like to see is that someone post a textrel report without my PIC patches (for both hardened (PIE) and non-hardened systems). anyone with a fast box out there ;-)?
<ms> # eu-findtextrel ./libmysqld/examples/mysql <ms> the file containing the function 'longlong10_to_str' is not compiled with -fpic/-fPIC <ms> the file containing the function 'my_strtoll10' is not compiled with -fpic/-fPIC <ms> the file containing the function 'longlong2str' is not compiled with -fpic/-fPIC thanks ms, that confirmes the theory, it's only those files that i patched and now everything should be fine for hardened/PIE users as well.
Reassuming: - builds of mysql using "minimal" use flag are not affected. The libraries don't use those asm functions. - only "x86" arch is affected. This exclude "x86_64" too. For the record assembler for "sparc" exist too. - The behaviour of patched and unpatched source is exactly the same. Both on hardened and non hardened inastallations. Performances are the same in both conditions.
Thanks vivo, others have also reported good results. Please apply to our internal mysql ebuild. Also any of our mysql guys have a good upstream contact?
MySQL Bugs: #11642 contain http://bugs.mysql.com/file.php?id=1569 longlong2str-x86.s It's a first step in close the text relocation issue, with a little speed enhancement too. As is it reached the tree and is included into mysql-4.0.25, mysql-4.1.12-r1, mysql-5.0.9 ebuilds.
Maybe it was not clear in the previous comment, so expliciting that: That patch is the result of the efforts from people on this bug. In one way or the other.
the linked patch/file is still missing the .note.GNU-stack marking, it'd be important to add that as well, else all code linking in these functions will be marked with an executable GNU_STACK program header and will in turn cause the process to run with executable stacks for no good reason.
Created attachment 63496 [details, diff] 036_x86_asm-pic-fixes-r0.patch The attached patch complete the upstream one for >=mysql-4.1 until works are finished.
(In reply to comment #35) > Created an attachment (id=63496) [edit] > 036_x86_asm-pic-fixes-r0.patch > > The attached patch complete the upstream one for >=mysql-4.1 until works are > finished. in longlong2str-x86.s you don't anything but the GNU-stack section stuff, the rest is the PIC changes that upstream solved in a different way.
to comment #36 2 steps back. I've found things that need to be better analized in the upstream patch. At the moment we reverted back to your original patch.
mysql-4.0.25-r2.ebuild has upstream patches mysql-4.1.13-r1.ebuild has "PaX Team" patches mysql-5.0.10_beta.ebuild has upstream patches All three versions are free of text relocations and proper GNU-stack handling. All are avaiable to our user base in portage from a few days, no bugs reconducible to these patches have been submitted. Closing this bug without thanks "PAX team" only to not be boring.
*** Bug 101590 has been marked as a duplicate of this bug. ***
Can we leave this open till something reaches stable.
MySQL 4.0.25-r2 is marked stable on x86 .
