ncbi-tools++ (-2009.05.15) needs significant work to include the various keyword options in the Gentoo builds. For a start it looks like the default is to build the entire package with static libraries (good for long-term maintenance, bad from a system disk space perspective as the package is *large*). Reproducible: Always Steps to Reproduce: 1. Add sci-biology/ncbi-tools++ ~x86 to /etc/portage/package.keywords. 2. emerge ncbi-tools++ > ncbi-tools++.log 2>&1 Look at the log file and build directories during the lengthy build process. Actual Results: Build takes a long time and is very large. Programs accumulating in the bin subdirectory appear to all be loaded using static libraries (only the C libraries appear to shared). Produces 50+ binaries from 1-7 MB in size (average 3-4 MB) and 90+ test programs approximately the same size. Expected Results: There should be keyword options for many of the more "standard" options, e.g. static/-static, various databases (mysql/sybase/sqlite2|3/bdb), +/-boost, +/-python, +/- ncbi-c/SSS libraries, SSS UTIL library, SSS DB library, the Genome Workbench (gbench), etc. There should be USE flag documentation on these. There should also be doc and test USE options for people who want to conserve disk space and/or reduce compile time. It looks like the CFLAGS options may be overridden by the configuration to a generic "-O", so system-wide -O2 or -O3 may have no effect. If I can compile my entire Linux system with -O2, I should be able to compile ncbi-tools++. I suspect many of the USE flags that ncbi-tools++ needs added should be added to ncbi-ttols (Bug #266569) as well. Though I haven't completed an install yet (I'm not sure I have sufficient disk space for it) I also suspect that there are no man pages or "normal" Linux documentation on the various programs.
Created attachment 199567 [details] Output of configure --help for ncbi-tools++ Shows the variety of options which can be set for ncbi-tools++ configuration. Esp. important would be, static/-static, static-exe/-static-exe, test (check-tools, check), and doc (???) [there are two or three html directories present but there doesn't appear to be a lot of documentation for a package with 40-50 executables].
Patches are welcome. You're right, we need all those options, but I don't knwo when I'll have time to implement them.
Static compilation is disabled in the latest revision. Package size is now 10x smaller.
(In reply to comment #3) > Static compilation is disabled in the latest revision. Package size is now 10x > smaller. > Maybe that is the reason why I now see: >>> Emerging (1 of 1) sci-biology/ncbi-tools++-2009.05.15-r1 * ncbi_cxx--May_15_2009.tar.gz RMD160 SHA1 SHA256 size ;-) ... [ ok ] * checking ebuild checksums ;-) ... [ ok ] * checking auxfile checksums ;-) ... [ ok ] * checking miscfile checksums ;-) ... [ ok ] * CPV: sci-biology/ncbi-tools++-2009.05.15-r1 * REPO: gentoo * USE: amd64 elibc_glibc kernel_linux multilib userland_GNU >>> Unpacking source... >>> Unpacking ncbi_cxx--May_15_2009.tar.gz to /var/tmp/portage/sci-biology/ncbi-tools++-2009.05.15-r1/work >>> Source unpacked in /var/tmp/portage/sci-biology/ncbi-tools++-2009.05.15-r1/work >>> Preparing source in /var/tmp/portage/sci-biology/ncbi-tools++-2009.05.15-r1/work/ncbi_cxx--May_15_2009 ... * Applying ncbi-tools++-2009.05.15-gcc44.patch ... [ ok ] >>> Source prepared. >>> Configuring source in /var/tmp/portage/sci-biology/ncbi-tools++-2009.05.15-r1/work/ncbi_cxx--May_15_2009 ... configure: creating cache config.cache checking build system type... x86_64-unknown-linux-gnu checking host system type... x86_64-unknown-linux-gnu checking for a BSD-compatible install... /usr/bin/install -c checking for gcc... gcc checking for C compiler default output file name... a.out checking whether the C compiler works... yes checking whether we are cross compiling... no checking for suffix of executables... checking for suffix of object files... o checking whether we are using the GNU C compiler... yes checking whether gcc accepts -g... yes checking for gcc option to accept ANSI C... none needed checking for g++... g++ checking whether we are using the GNU C++ compiler... yes checking whether g++ accepts -g... yes adjusted C compiler: /usr/bin/gcc adjusted C++ compiler: /usr/bin/g++ checking whether /usr/bin/gcc supports -Wl,-E... yes /usr/lib/gcc/x86_64-pc-linux-gnu/4.3.4/../../../../lib64/crt1.o: In function `_start': (.text+0x20): undefined reference to `main' collect2: ld returned 1 exit status configure: WARNING: Unable to find static libstdc++ requested by --with-bin-release. A lot of configure test seem to fail badly: checking whether cross-directory hard links work... yes checking iostream usability... no checking iostream presence... yes configure: WARNING: iostream: present but cannot be compiled configure: WARNING: iostream: check for missing prerequisite headers? configure: WARNING: iostream: see the Autoconf documentation configure: WARNING: iostream: section "Present But Cannot Be Compiled" configure: WARNING: iostream: proceeding with the preprocessor's result configure: WARNING: iostream: in the future, the compiler will take precedence configure: WARNING: ## ---------------------------------------- ## configure: WARNING: ## Report this to cpp-core@ncbi.nlm.nih.gov ## configure: WARNING: ## ---------------------------------------- ## I found it is because somehow a space from my CFLAGS, actually CXXFLAGS is lost. Quoting from /var/tmp/portage/sci-biology/ncbi-tools++-2009.05.15-r1/work/ncbi_cxx--May_15_2009/config.log: configure:7926: checking whether cross-directory hard links work configure:7940: result: yes configure:8019: checking iostream usability configure:8031: /usr/bin/g++ -c -Wall -Wno-format-y2k -O2 -pipe -march=nocona -fno-strict-aliasinggdb -fPIC -DNDEBUG -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -DNCBI_WITHOUT_MT conftest.cc >&5 cc1plus: error: unrecognized command line option "-fno-strict-aliasinggdb" I do not see any obvious tricks mangling with the CXXFLAGS in the ebuild file explaining me why the config.log shows that gcc was called with properly spaced switches while g++ was not. # emerge --info Portage 2.1.7.14 (default/linux/amd64/10.0, gcc-4.4.2, glibc-2.11-r1, 2.6.32-default x86_64) ================================================================= System uname: Linux-2.6.32-default-x86_64-Intel-R-_Core-TM-2_Quad_CPU_Q9650_@_3.00GHz-with-gentoo-2.0.1 Timestamp of tree: Mon, 14 Dec 2009 11:00:01 +0000 app-shells/bash: 4.0_p35 dev-java/java-config: 2.1.10 dev-lang/python: 2.5.4-r3, 2.6.4, 3.1.1-r1 dev-util/cmake: 2.8.0 sys-apps/baselayout: 2.0.1 sys-apps/openrc: 0.5.3 sys-apps/sandbox: 2.2 sys-devel/autoconf: 2.13, 2.64 sys-devel/automake: 1.5, 1.9.6-r2, 1.10.3, 1.11.1 sys-devel/binutils: 2.20 sys-devel/gcc-config: 1.4.1 sys-devel/libtool: 2.2.6b virtual/os-headers: 2.6.30-r1 ACCEPT_KEYWORDS="amd64 ~amd64" ACCEPT_LICENSE="* -@EULA dlj-1.1 sun-bcla-java-vm" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-O2 -pipe -march=nocona -fno-strict-aliasing -ggdb" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/share/X11/xkb /usr/share/config /var/bind /var/qmail/alias /var/qmail/control /var/spool/torque" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/env.d/java/ /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo" CXXFLAGS="-O2 -pipe -march=nocona -fno-strict-aliasing -ggdb" DISTDIR="/usr/portage/distfiles" FEATURES="assume-digests distlocks fixpackages news nostrip parallel-fetch protect-owned sandbox sfperms splitdebug strict unmerge-logs unmerge-orphans userfetch" GENTOO_MIRRORS="http://distfiles.gentoo.org http://distro.ibiblio.org/pub/linux/distributions/gentoo" LDFLAGS="-Wl,-O1" LINGUAS="en cs cz" MAKEOPTS="-j5" PKGDIR="/usr/portage/packages" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" SYNC="rsync://rsync.gentoo.org/gentoo-portage" USE="X amd64 berkdb bzip2 cracklib crypt cxx dbus dri fortran gdbm gnutls hal iconv innodb java jce jpeg jpg kde laptop mmx modules mpi mpich2 multilib mysql ncurses nfs nls nptl nptlonly nsplugin opengl openssl pam pcre perl png python qt3support qt4 readline server session sse sse2 ssl svg sysfs syslog tcpd threads unicode usb xorg zlib" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" ELIBC="glibc" INPUT_DEVICES="keyboard mouse evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LINGUAS="en cs cz" RUBY_TARGETS="ruby18" USERLAND="GNU" VIDEO_CARDS="vesa" Unset: CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, FFLAGS, INSTALL_MASK, LANG, LC_ALL, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, PORTDIR_OVERLAY
I just learned a documentation is here at ftp://ftp.ncbi.nlm.nih.gov/blast/executables/blast+/LATEST/user_manual.pdf . Please get it installed under dodoc.
Created attachment 332924 [details] ncbi-tools++-9.0.0.ebuild An initial ebuild at least listing the USE flags and using some of them. It looks there are two or three types of configure option types: 1) prevent system-wiide library use and use builtin-copy in the archive (this is not interesting to gentoo, hence I just added those to dependencies (forcing them to be pre-installed) 2) disable some system-wide library usage (that is of our interest) 3) mangle type of binaries output, list of features/objects/whatever specific to ncbi-tools++ (I left them just enumerated in the ebuild in comment lines, I do no know what we want to do with them) 4) wxwidgets, ODBC, Corba etc. are not respected in the ebuild but probably should be, somehow (volunteers?) ;-) Looks that when no gnutls is available/allowed then the configure force openssl. In other words, USE="-gnutls -ssl" does not work as I would like to. I don't see --without-test in configure options and have emailed upstream how to disable testsuite compilation altogether. I see some --with-checks but that is not what I am looking for. The configure is weird and does not respect --disable-blah. Therefore, those tricks with arg passing to configure in the ebuild. Probably some eclass could be improved to handle the --with-blah --without-blah easily? I would be glad if somebody does some cleanup of the ebuild. I can help more with testing then.
Thanks Martin! Hopefully I will find some time during the next week to work on this.
Created attachment 332980 [details] ncbi-tools++-9.0.0.ebuild (v2)
Created attachment 332982 [details, diff] curl-types.patch Newer versions of curl dropped the empty header file altogether.
Created attachment 332984 [details] disable-testsuite-compilation.txt This file is used by the ebuild and configure via --projects= argument. It is to disable compilation of the testuite. See http://www.ncbi.nlm.nih.gov/books/NBK7167/ on other examples how to adjust the build system. Credits to cpp@ncbi.nlm.nih.gov mailing list.
Created attachment 332986 [details, diff] ncbi-tools++-9.0.0-fix-order-of-libs.patch This is a correct, upstream fix. However, for completeness, one could have fiddled with the --as-needed switch. In this regard, I don't know what the currently in tree, /usr/portage/sci-biology/ncbi-tools++/files/ncbi-tools++-2009.05.15-asneeded.patch does or how is it related to --as-neeeded in the filename. Its contents do not fiddle with linker setting, was the patch misnamed in the past? That patch is not needed for current, 9.0.0 version. Don't waste much time answering this philosophical question. ;-)
To continue to long story, poke briefly in the threads I have started. Then my notes in the ebuild file might become clearer. ;-) http://www.ncbi.nlm.nih.gov/mailman/pipermail/cpp/2012q4/thread.html
Created attachment 333134 [details] ncbi-tools++/ncbi-tools++-9.0.0.ebuild (v3) Works almost: # USE="-test -mysql -sqlite -ftds ssl -gnutls" CFLAGS="-O2 -pipe -march=native" CXXFLAGS="-O2 -pipe -march=native" emerge =sci-biology/ncbi-tools++-9.0.0 [cut] >>> Source compiled. >>> Test phase [not enabled]: sci-biology/ncbi-tools++-9.0.0 >>> Install ncbi-tools++-9.0.0 into /var/tmp/portage/sci-biology/ncbi-tools++-9.0.0/image/ category sci-biology make -j5 install if test -f /var/tmp/portage/sci-biology/ncbi-tools++-9.0.0/work/ncbi_cxx--9_0_0/GCC463-ReleaseMTDLL64/bin/gbench; then \ make - --jobserver-fds=3,4 -j install-gbench; \ else \ make - --jobserver-fds=3,4 -j install-toolkit; \ fi make[1]: Entering directory `/var/tmp/portage/sci-biology/ncbi-tools++-9.0.0/work/ncbi_cxx--9_0_0' /bin/rm -rf /var/tmp/portage/sci-biology/ncbi-tools++-9.0.0/image//usr/include/ncbi-tools++ /usr/bin/install -c -d /var/tmp/portage/sci-biology/ncbi-tools++-9.0.0/image//usr/bin /var/tmp/portage/sci-biology/ncbi-tools++-9.0.0/image//usr/lib64/ncbi-tools++ /var/tmp/portage/sci-biology/ncbi-tools++-9.0.0/image//usr/include/ncbi-tools++ /usr/bin/install -c /var/tmp/portage/sci-biology/ncbi-tools++-9.0.0/work/ncbi_cxx--9_0_0/GCC463-ReleaseMTDLL64/bin/* /var/tmp/portage/sci-biology/ncbi-tools++-9.0.0/image//usr/bin /usr/bin/install -c -m 644 /var/tmp/portage/sci-biology/ncbi-tools++-9.0.0/work/ncbi_cxx--9_0_0/GCC463-ReleaseMTDLL64/lib/* /var/tmp/portage/sci-biology/ncbi-tools++-9.0.0/image//usr/lib64/ncbi-tools++ /usr/bin/install: omitting directory '/var/tmp/portage/sci-biology/ncbi-tools++-9.0.0/work/ncbi_cxx--9_0_0/GCC463-ReleaseMTDLL64/lib/ncbi' make[1]: *** [install-toolkit] Error 1 make[1]: Leaving directory `/var/tmp/portage/sci-biology/ncbi-tools++-9.0.0/work/ncbi_cxx--9_0_0' make: *** [install] Error 2 emake failed * ERROR: sci-biology/ncbi-tools++-9.0.0 failed (install phase): * (no error message) * * Call stack: * ebuild.sh, line 93: Called src_install * environment, line 2522: Called die * The specific snippet of code: * emake install || die;
Created attachment 333136 [details, diff] malloc_initialize_upstream_fix.patch An upstream patch: <quote> Sorry about that; this looks like a slight difference in the <malloc.h>; our version doesn't have "volatile" in it and yours does. So you can either add "volatile" to the function declaration or to exclude "netcache" project from the compilation (e.g. by removing it from the source tree; or from 'src/app/Makefile.ini'). At any rate I don't think that you'd need it; it's expendable. Also, this whole hooky-hacky machinery has since been removed from the sources for good, in: http://www.ncbi.nlm.nih.gov/viewvc/v1/trunk/c%2B%2B/src/app/netcache/nc_memory.cpp?r1=52330&r2=53136 so it won't be a problem in future releases. </quote>
Few more patches were prepared by upstream. I put all patches and a working ebuild into sci overlay. Yes, the patch for generated configure can go away, I know. That is left for somebody else to demonstrate a good QC work. ;-) I do not intend to work more on this ebuild except testing other compilers. For me it worked with gcc-4.6.3 now. So it's about the time somebody else could polish the ebuild specific issues. I have enumerated the issues already in this bug report, so go ahead. ;-) I will not attach the new patches to this bug anymore, it is all in sci overlay.
(In reply to comment #15) > I do not intend to work more on this ebuild except testing other compilers. > For me it worked with gcc-4.6.3 now. So it's about the time somebody else > could polish the ebuild specific issues. I have enumerated the issues > already in this bug report, so go ahead. ;-) I've failed to build to build ncbi-tools++-9.0.0-r1 MAKEOPTS="-j1" LDFLAGS="" FEATURES="-ccache" USE="X berkdb bzip2 -gif -jpeg -opengl -pch -pcre -png -ssl -threads -tiff -wxwidgets xml" emerge -1 sci-biology/ncbi-tools++ /usr/lib/gcc/x86_64-pc-linux-gnu/4.6.3/../../../../x86_64-pc-linux-gnu/bin/ld: warning: libncbi_xcache_bdb.so, needed by /home/tmp/portage/sci-biology/ncbi-tools++-9.0.0-r1/work/ncbi_cxx--9_0_0_build/lib/libxalnmgr.so, not found (try using -rpath or -rpath-link) /usr/lib/gcc/x86_64-pc-linux-gnu/4.6.3/../../../../x86_64-pc-linux-gnu/bin/ld: warning: libcreaders.so, needed by /home/tmp/portage/sci-biology/ncbi-tools++-9.0.0-r1/work/ncbi_cxx--9_0_0_build/lib/libxobjread.so, not found (try using -rpath or -rpath-link) /usr/lib/gcc/x86_64-pc-linux-gnu/4.6.3/../../../../x86_64-pc-linux-gnu/bin/ld: warning: libgene_info.so, needed by /home/tmp/portage/sci-biology/ncbi-tools++-9.0.0-r1/work/ncbi_cxx--9_0_0_build/lib/libncbi_xreader_id2.so, not found (try using -rpath or -rpath-link) /home/tmp/portage/sci-biology/ncbi-tools++-9.0.0-r1/work/ncbi_cxx--9_0_0_build/lib/libxobjread.so: undefined reference to `AlignmentFileFree' /home/tmp/portage/sci-biology/ncbi-tools++-9.0.0-r1/work/ncbi_cxx--9_0_0_build/lib/libxobjread.so: undefined reference to `ReadAlignmentFile2' collect2: выполнение ld завершилось с кодом возврата 1 make[4]: *** [alnmrg] Ошибка 1 make[4]: Выход из каталога `/home/tmp/portage/sci-biology/ncbi-tools++-9.0.0-r1/work/ncbi_cxx--9_0_0_build/build/objtools/alnmgr/demo' FAILED: src/objtools/alnmgr/demo/Makefile.alnmrg.app make[4]: Вход в каталог `/home/tmp/portage/sci-biology/ncbi-tools++-9.0.0-r1/work/ncbi_cxx--9_0_0_build/build/objtools/alnmgr/demo' /bin/rm -f alnmrg .alnmrg.stamp make[4]: Выход из каталога `/home/tmp/portage/sci-biology/ncbi-tools++-9.0.0-r1/work/ncbi_cxx--9_0_0_build/build/objtools/alnmgr/demo' make[3]: *** [all.nonusr] Ошибка 2 make[3]: Выход из каталога `/home/tmp/portage/sci-biology/ncbi-tools++-9.0.0-r1/work/ncbi_cxx--9_0_0_build/build/objtools/alnmgr/demo' make[2]: *** [all_p.real] Ошибка 5 make[2]: Выход из каталога `/home/tmp/portage/sci-biology/ncbi-tools++-9.0.0-r1/work/ncbi_cxx--9_0_0_build/build/objtools/alnmgr' make[1]: *** [all_p.real] Ошибка 5 make[1]: Выход из каталога `/home/tmp/portage/sci-biology/ncbi-tools++-9.0.0-r1/work/ncbi_cxx--9_0_0_build/build/objtools' make: *** [all_p.real] Ошибка 5 make: Выход из каталога `/home/tmp/portage/sci-biology/ncbi-tools++-9.0.0-r1/work/ncbi_cxx--9_0_0_build/build'
Please re-try 9.0.0-r1 from sci overlay or even 12.0.0.
ncbi-tools++ was removed from the tree it seems, so these bugs aren't relevant anymore.
