dev-libs/icu-4.4.1 won't compile when using a large CFLAGS string failing with an U_BUFFER_OVERFLOW_ERROR Reproducible: Always Steps to Reproduce: 1.CFLAGS="-O2 -pipe -fforce-addr -fomit-frame-pointer -funswitch-loops -fbranch-target-load-optimize --param max-gcse-passes=3 -fweb -frename-registers -freorder-blocks-and-partition -fpredictive-commoning -fira-region=all -fira-coalesce -fgcse-sm -fgcse-las -fgcse-after-reload -freschedule-modulo-scheduled-loops -fipa-struct-reorg -fipa-matrix-reorg -fno-sched-stalled-insns -fsee -ftree-loop-im -ftree-loop-linear -ftree-loop-distribution -ftree-vectorize -fvect-cost-model -fivopts -fmodulo-sched -fmodulo-sched-allow-regmoves -fmerge-all-constants -march=core2" emerge -v1 dev-libs/icu 2.fails Actual Results: The package won't compile Expected Results: The package compiles Below is a small portion of the output of the failure. LD_LIBRARY_PATH=../lib:../stubdata:../tools/ctestfw:$LD_LIBRARY_PATH ../bin/gensprep -d ./out/build/icudt44l -i ./out/build/icudt44l -s ./sprep -b rfc4013 -m ./unidata -u 3.2.0 rfc4013.txt LD_LIBRARY_PATH=../lib:../stubdata:../tools/ctestfw:$LD_LIBRARY_PATH ../bin/gensprep -d ./out/build/icudt44l -i ./out/build/icudt44l -s ./sprep -b rfc4505 -m ./unidata -u 3.2.0 rfc4505.txt LD_LIBRARY_PATH=../lib:../stubdata:../tools/ctestfw:$LD_LIBRARY_PATH ../bin/gensprep -d ./out/build/icudt44l -i ./out/build/icudt44l -s ./sprep -b rfc4518 -m ./unidata -u 3.2.0 rfc4518.txt LD_LIBRARY_PATH=../lib:../stubdata:../tools/ctestfw:$LD_LIBRARY_PATH ../bin/gensprep -d ./out/build/icudt44l -i ./out/build/icudt44l -s ./sprep -b rfc4518ci -m ./unidata -u 3.2.0 rfc4518ci.txt LD_LIBRARY_PATH=../lib:../stubdata:../tools/ctestfw:$LD_LIBRARY_PATH echo ALL_CFU_SOURCE: ./unidata/confusables.txt ./unidata/confusablesWholeScript.txt ALL_CFU_SOURCE: ./unidata/confusables.txt ./unidata/confusablesWholeScript.txt LD_LIBRARY_PATH=../lib:../stubdata:../tools/ctestfw:$LD_LIBRARY_PATH echo CFU_FILES: ./out/build/icudt44l/confusables.cfu CFU_FILES: ./out/build/icudt44l/confusables.cfu LD_LIBRARY_PATH=../lib:../stubdata:../tools/ctestfw:$LD_LIBRARY_PATH echo CFU_FILES_SHORT: confusables.cfu CFU_FILES_SHORT: confusables.cfu LD_LIBRARY_PATH=../lib:../stubdata:../tools/ctestfw:$LD_LIBRARY_PATH ../bin/gencfu -c -i ./out/build/icudt44l -r ./unidata/confusables.txt -w ./unidata/confusablesWholeScript.txt -o out/build/icudt44l/confusables.cfu gencfu: tool completed successfully. generating out/tmp/icudata.lst (list of data files) LD_LIBRARY_PATH=../stubdata:../tools/ctestfw:../lib:$LD_LIBRARY_PATH MAKEFLAGS= ../bin/pkgdata -O ../data/icupkg.inc -q -c -s /var/tmp/portage/dev-libs/icu-4.4.1/work/icu/source/data/out/build/icudt44l -d ../lib -e icudt44 -T ./out/tmp -p icudt44l -m dll -r 44.1 -L icudata ./out/tmp/icudata.lst Unable to open or read "../data/icupkg.inc" option file. status = U_BUFFER_OVERFLOW_ERROR make[1]: *** [packagedata] Error 255 make[1]: se sale del directorio `/var/tmp/portage/dev-libs/icu-4.4.1/work/icu/source/data' make: *** [all-recursive] Error 2 * ERROR: dev-libs/icu-4.4.1 failed: * emake failed * * Call stack: * ebuild.sh, line 54: Called src_compile * environment, line 2944: Called _eapi2_src_compile * ebuild.sh, line 646: Called die * The specific snippet of code: * emake || die "emake failed" * * If you need support, post the output of 'emerge --info =dev-libs/icu-4.4.1', * the complete build log and the output of 'emerge -pqv =dev-libs/icu-4.4.1'. * The complete build log is located at '/var/tmp/portage/dev-libs/icu-4.4.1/temp/build.log'. * The ebuild environment file is located at '/var/tmp/portage/dev-libs/icu-4.4.1/temp/environment'. * S: '/var/tmp/portage/dev-libs/icu-4.4.1/work/icu/source' Also, using a smaller CFLAGs solves the problem.
Attach full build log.
Created attachment 237199 [details] Full build log Attached Full build log
Created attachment 237201 [details] ebuild environment
Created attachment 237203 [details] emerge --info =dev-libs/icu-4.4.1
Created attachment 237205 [details] emerge -pqv =dev-libs/icu-4.4.1
Confirmed with GCC 4.5.1 and a quite different set of flags...
Created attachment 244483 [details, diff] Patch to fix "pkgdata" tool hard-wired buffer sizes I have written a patch that fixes this issue; the buffer sizes in the "pkgdata" tool were hard-wired constants. My patch uses the old "double-and-retry" method to ensure that the flags will fit into the buffer. It's hardly the most elegant code I've ever written, but sometimes brute force is the right answer...
Oh, forgot to mention... there's also an updated e-build at my overlay at git://www.valleyhold.org/gentoo-portage.git or http://www.valleyhold.org/repo/dists/gentoo/portage.git (it's a bit of a mess yet, which is why I've not asked to add it to the list...)
(In reply to comment #7) Please report the patch in: https://bugs.icu-project.org/trac
I've reported this upstream as https://bugs.icu-project.org/trac/ticket/7956
(In reply to comment #10) > I've reported this upstream as https://bugs.icu-project.org/trac/ticket/7956 Thanks for filing this. Please change URL to https://ssl.icu-project.org/trac/ticket/7956
(In reply to comment #10) > I've reported this upstream as https://bugs.icu-project.org/trac/ticket/7956 It's been fixed in the upstream. Diff/patch here http://bugs.icu-project.org/trac/changeset/28643
I don't think the patch in upstream will definitely solve the problem. The only thing done is increasing the size of the buffer but that's only a temporary patch, because somebody with even a larger number of flags would end hitting this again.
(In reply to comment #13) > I don't think the patch in upstream will definitely solve the problem. The only > thing done is increasing the size of the buffer but that's only a temporary > patch, because somebody with even a larger number of flags would end hitting > this again. If nobody objects, I'd like to offer them my patch; that *should* be a real fix. (Yes, it's more "expensive" as it's doing heap allocation, but it's just options parsing, so who cares...)
I give a +1 I don't think the patch is so ugly and the bigger buffers can be applied already.
(In reply to comment #14) > (In reply to comment #13) > > I don't think the patch in upstream will definitely solve the problem. The only > > thing done is increasing the size of the buffer but that's only a temporary > > patch, because somebody with even a larger number of flags would end hitting > > this again. > > If nobody objects, I'd like to offer them my patch; that *should* be a real > fix. (Yes, it's more "expensive" as it's doing heap allocation, but it's just > options parsing, so who cares...) Thanks for mentioning it there- the patch ought to have been posted to the upstream bug in the first place. BTW there's no registry on our trac, just use your e-mail address.
I hit the same problem, but when compiling app-text/texlive-core-2010 with dev-libs/icu-4.6 and gcc-4.5.1-r1. CFLAGS="-march=atom -m32 --param l1-cache-line-size=64 --param l1-cache-size=32 --param l2-cache-size=512 -O2 -funswitch-loops -fpredictive-commoning -fgcse-after-reload -ftree-vectorize -mstackrealign -ftree-loop-linear -ftree-loop-im -fweb -frename-registers -fomit-frame-pointer -mfpmath=sse -fexcess-precision=fast -pipe" Build fails at: make[7]: Entering directory `/var/tmp/portage/app-text/texlive-core-2010/work/texlive-20100722-source/libs/icu/icu-build/data' rm -rf icupkg.inc mkdir -p -- ./out ./out/build ./out/build/icudt44l ./out/build/icudt44l/curr ./out/build/icudt44l/lang ./out/build/icudt44l/region ./out/build/icudt44l/zone ./out/build/icudt44l/brkitr ./out/build/icudt44l/coll ./out/build/icudt44l/rbnf ./out/build/icudt44l/translit ./out/tmp ./out/tmp/curr ./out/tmp/lang ./out/tmp/region ./out/tmp/zone ./out/tmp/coll ./out/tmp/rbnf ./out/tmp/translit ./out/tmp/brkitr mv build-dir.tmp build-dir Unpacking /var/tmp/portage/app-text/texlive-core-2010/work/texlive-20100722-source/libs/icu/icu-4.4/data/in/icudt44l.dat and generating out/tmp/icudata.lst (list of data files) LD_LIBRARY_PATH=../lib:../stubdata:../tools/ctestfw:$LD_LIBRARY_PATH ../bin/icupkg -d ./out/build/icudt44l --list -x \* /var/tmp/portage/app-text/texlive-core-2010/work/texlive-20100722-source/libs/icu/icu-4.4/data/in/icudt44l.dat -o out/tmp/icudata.lst make[7]: Leaving directory `/var/tmp/portage/app-text/texlive-core-2010/work/texlive-20100722-source/libs/icu/icu-build/data' echo timestamp > build-local LD_LIBRARY_PATH=../stubdata:../tools/ctestfw:../lib:$LD_LIBRARY_PATH MAKEFLAGS= ../bin/pkgdata -O ../data/icupkg.inc -q -c -s /var/tmp/portage/app-text/texlive-core-2010/work/texlive-20100722-source/libs/icu/icu-build/data/out/build/icudt44l -d ../lib -e icudt44 -T ./out/tmp -p icudt44l -L icudata -m static -r 44.0 ./out/tmp/icudata.lst Unable to open or read "../data/icupkg.inc" option file. status = U_BUFFER_OVERFLOW_ERROR make[6]: *** [packagedata] Error 255
Created attachment 257539 [details] build.log
Created attachment 257541 [details] environment
Created attachment 257543 [details] emerge --info
Hmm, it seems Gordon's patch is already present in a slightly different way for the icu-4.6.
I found that it fails due to too long LDFLAGS: with reduced length of LDFLAGS and untouched CFLAGS/CXXFLAGS it builds. It looks like connected but different issue, so I'll open a new bug.
Well 4.6 solves the issue for me but with very large CFLAGS now it will crash due to a double free made by ../bin/pkgdata Here is the bt: ======= Backtrace: ========= /lib/libc.so.6(+0x77df6)[0x20a2252edf6] /lib/libc.so.6(cfree+0x73)[0x20a22535613] ../bin/pkgdata(+0x243d)[0x335548e43d] ../bin/pkgdata(+0x36e7)[0x335548f6e7] ../bin/pkgdata(main+0x5ba)[0x335548fdea] /lib/libc.so.6(__libc_start_main+0xfd)[0x20a224d5bfd] ../bin/pkgdata(+0x1ae9)[0x335548dae9] ======= Memory map: ======== 335548c000-3355494000 r-xp 00000000 fd:01 540695 /var/tmp/portage/dev-libs/icu-4.6/work/icu/source/bin/pkgdata 3355693000-3355694000 r--p 00007000 fd:01 540695 /var/tmp/portage/dev-libs/icu-4.6/work/icu/source/bin/pkgdata 3355694000-3355695000 rw-p 00008000 fd:01 540695 /var/tmp/portage/dev-libs/icu-4.6/work/icu/source/bin/pkgdata 3355695000-3355736000 rw-p 00000000 00:00 0 [heap] 20a21575000-20a21595000 r-xp 00000000 fd:01 247290 /lib64/ld-2.11.2.so 20a21595000-20a21596000 r-xp 00000000 00:00 0 [vdso] 20a21596000-20a2160f000 rw-p 00000000 00:00 0 20a21794000-20a21795000 r--p 0001f000 fd:01 247290 /lib64/ld-2.11.2.so 20a21795000-20a21796000 rw-p 00020000 fd:01 247290 /lib64/ld-2.11.2.so 20a21796000-20a21797000 rw-p 00000000 00:00 0 20a21797000-20a217a8000 r-xp 00000000 fd:01 221013 /usr/lib64/libsandbox.so 20a217a8000-20a219a8000 ---p 00011000 fd:01 221013 /usr/lib64/libsandbox.so 20a219a8000-20a219a9000 r--p 00011000 fd:01 221013 /usr/lib64/libsandbox.so 20a219a9000-20a219aa000 rw-p 00012000 fd:01 221013 /usr/lib64/libsandbox.so 20a219aa000-20a219ad000 rw-p 00000000 00:00 0 20a219ad000-20a219d3000 r-xp 00000000 fd:01 539725 /var/tmp/portage/dev-libs/icu-4.6/work/icu/source/lib/libicutu.so.46.0 20a219d3000-20a21bd2000 ---p 00026000 fd:01 539725 /var/tmp/portage/dev-libs/icu-4.6/work/icu/source/lib/libicutu.so.46.0 20a21bd2000-20a21bd4000 r--p 00025000 fd:01 539725 /var/tmp/portage/dev-libs/icu-4.6/work/icu/source/lib/libicutu.so.46.0 20a21bd4000-20a21bd5000 rw-p 00027000 fd:01 539725 /var/tmp/portage/dev-libs/icu-4.6/work/icu/source/lib/libicutu.so.46.0 20a21bd5000-20a21bee000 rw-p 00000000 00:00 0 20a21bee000-20a21d54000 r-xp 00000000 fd:01 539626 /var/tmp/portage/dev-libs/icu-4.6/work/icu/source/lib/libicuuc.so.46.0 20a21d54000-20a21f54000 ---p 00166000 fd:01 539626 /var/tmp/portage/dev-libs/icu-4.6/work/icu/source/lib/libicuuc.so.46.0 20a21f54000-20a21f63000 r--p 00166000 fd:01 539626 /var/tmp/portage/dev-libs/icu-4.6/work/icu/source/lib/libicuuc.so.46.0 20a21f63000-20a21f65000 rw-p 00175000 fd:01 539626 /var/tmp/portage/dev-libs/icu-4.6/work/icu/source/lib/libicuuc.so.46.0 20a21f65000-20a21f68000 rw-p 00000000 00:00 0 20a21f68000-20a21f80000 r-xp 00000000 fd:01 247227 /lib64/libpthread-2.11.2.so 20a21f80000-20a2217f000 ---p 00018000 fd:01 247227 /lib64/libpthread-2.11.2.so 20a2217f000-20a22180000 r--p 00017000 fd:01 247227 /lib64/libpthread-2.11.2.so 20a22180000-20a22181000 rw-p 00018000 fd:01 247227 /lib64/libpthread-2.11.2.so 20a22181000-20a22186000 rw-p 00000000 00:00 0 20a22186000-20a22299000 r-xp 00000000 fd:01 238531 /usr/lib64/gcc/x86_64-pc-linux-gnu/4.4.4/libstdc++.so.6.0.13 20a22299000-20a22499000 ---p 00113000 fd:01 238531 /usr/lib64/gcc/x86_64-pc-linux-gnu/4.4.4/libstdc++.so.6.0.13 20a22499000-20a224a2000 r--p 00113000 fd:01 238531 /usr/lib64/gcc/x86_64-pc-linux-gnu/4.4.4/libstdc++.so.6.0.13 20a224a2000-20a224a3000 rw-p 0011c000 fd:01 238531 /usr/lib64/gcc/x86_64-pc-linux-gnu/4.4.4/libstdc++.so.6.0.13 20a224a3000-20a224b7000 rw-p 00000000 00:00 0 20a224b7000-20a22625000 r-xp 00000000 fd:01 247292 /lib64/libc-2.11.2.so 20a22625000-20a22825000 ---p 0016e000 fd:01 247292 /lib64/libc-2.11.2.so 20a22825000-20a22829000 r--p 0016e000 fd:01 247292 /lib64/libc-2.11.2.so 20a22829000-20a2282a000 rw-p 00172000 fd:01 247292 /lib64/libc-2.11.2.so 20a2282a000-20a2282f000 rw-p 00000000 00:00 0 20a2282f000-20a22831000 r-xp 00000000 fd:01 247185 /lib64/libdl-2.11.2.so 20a22831000-20a22a31000 ---p 00002000 fd:01 247185 /lib64/libdl-2.11.2.so 20a22a31000-20a22a32000 r--p 00002000 fd:01 247185 /lib64/libdl-2.11.2.so 20a22a32000-20a22a33000 rw-p 00003000 fd:01 247185 /lib64/libdl-2.11.2.so 20a22a33000-20a22a34000 rw-p 00000000 00:00 0 20a22a34000-20a22c36000 r-xp 00000000 fd:01 539704 /var/tmp/portage/dev-libs/icu-4.6/work/icu/source/lib/libicui18n.so.46.0 20a22c36000-20a22e36000 ---p 00202000 fd:01 539704 /var/tmp/portage/dev-libs/icu-4.6/work/icu/source/lib/libicui18n.so.46.0 20a22e36000-20a22e42000 r--p 00202000 fd:01 539704 /var/tmp/portage/dev-libs/icu-4.6/work/icu/source/lib/libicui18n.so.46.0make[1]: *** [packagedata] Abortado And the CFLAGS used: -O2 -pipe -fforce-addr -fomit-frame-pointer -fbranch-target-load-optimize --param max-gcse-passes=3 -fweb -frename-registers -freorder-blocks-and-partition -fpredictive-commoning -fira-region=all -fira-coalesce -fgcse-sm -fgcse-las -fgcse-after-reload -freschedule-modulo-scheduled-loops -fipa-struct-reorg -fipa-matrix-reorg -fno-sched-stalled-insns -ftree-loop-im -ftree-loop-linear -ftree-loop-distribution -ftree-vectorize -fvect-cost-model -fivopts -fmodulo-sched -fmodulo-sched-allow-regmoves -fmerge-all-constants -march=native -Dveryyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyylongstring
Please file a new bug upstream and mention this bug, and the older ICU bug. (In reply to comment #23) > Well 4.6 solves the issue for me but with very large CFLAGS now it will crash > due to a double free made by ../bin/pkgdata > > Here is the bt
upstream is now marked as fixed. http://bugs.icu-project.org/trac/ticket/8281#comment:4 (In reply to comment #24) > Please file a new bug upstream and mention this bug, and the older ICU bug. > > (In reply to comment #23) > > Well 4.6 solves the issue for me but with very large CFLAGS now it will crash > > due to a double free made by ../bin/pkgdata > > > > Here is the bt > (In reply to comment #23) > Well 4.6 solves the issue for me but with very large CFLAGS now it will crash > due to a double free made by ../bin/pkgdata > > Here is the bt:
Not yet fixed in Gentoo.
Fixed in dev-libs/icu-4.6. Steven R. Loomis: Please backport the fix to maint/maint-4-6 branch.
