Created attachment 364384 [details] Strace Output on failure case Weirdest problem. On one of my Gentoo systems, openbsd-netcat builds fine but I get the following output when running nc.openbsd -l (open port and listen). Other nc functions work, but not opening a socket and listening for traffic. >lfcvps portage # nc.openbsd -l 4444 >nc.openbsd: Protocol not available The nc in busybox works properly (but lacks other flags I need). Copying the binary from a ubuntu system works. Copying the binary from a different Gentoo server does not. It results in the same "Protocol not available error" Portage 2.2.7 (default/linux/amd64/13.0, gcc-4.7.3, glibc-2.16.0, 3.8.13-gentoo x86_64) ================================================================= System uname: Linux-3.8.13-gentoo-x86_64-Intel-R-_Xeon-R-_CPU_E5-2620_0_@_2.00GHz-with-gentoo-2.2 KiB Mem: 247036 total, 47400 free KiB Swap: 786420 total, 729472 free Timestamp of tree: Sun, 01 Dec 2013 08:30:01 +0000 ld GNU ld (GNU Binutils) 2.23.1 app-shells/bash: 4.2_p45 dev-lang/python: 2.7.5-r3, 3.2.5-r3, 3.3.2-r2 dev-util/cmake: 2.8.11.2 dev-util/pkgconfig: 0.28 sys-apps/baselayout: 2.2 sys-apps/openrc: 0.12.4 sys-apps/sandbox: 2.6-r1 sys-devel/autoconf: 2.69 sys-devel/automake: 1.12.6, 1.13.4 sys-devel/binutils: 2.23.1 sys-devel/gcc: 4.6.3, 4.7.3-r1 sys-devel/gcc-config: 1.7.3 sys-devel/libtool: 2.4.2 sys-devel/make: 3.82-r4 sys-kernel/linux-headers: 3.9 (virtual/os-headers) sys-libs/glibc: 2.16.0 Repositories: gentoo ACCEPT_KEYWORDS="amd64" ACCEPT_LICENSE="* -@EULA" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-O2 -pipe -fomit-frame-pointer" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc /var/bind" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/gconf /etc/gentoo-release /etc/php/apache2-php5.3/ext-active/ /etc/php/apache2-php5.4/ext-active/ /etc/php/apache2-php5.5/ext-active/ /etc/php/cgi-php5.3/ext-active/ /etc/php/cgi-php5.4/ext-active/ /etc/php/cgi-php5.5/ext-active/ /etc/php/cli-php5.3/ext-active/ /etc/php/cli-php5.4/ext-active/ /etc/php/cli-php5.5/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo" CXXFLAGS="-O2 -pipe -fomit-frame-pointer" DISTDIR="/usr/portage/distfiles" FCFLAGS="-O2 -pipe" FEATURES="assume-digests binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync news parallel-fetch preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync" FFLAGS="-O2 -pipe" GENTOO_MIRRORS="http://distfiles.gentoo.org" LDFLAGS="-Wl,-O1 -Wl,--as-needed" MAKEOPTS="-j2" PKGDIR="/usr/portage/packages" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" PORTDIR_OVERLAY="" USE="acl amd64 berkdb bzip2 caps cli cracklib crypt cxx dri fortran gdbm iconv ipv6 jpeg mmx modules mudflap multilib ncurses nls nptl openmp pam pcre png readline sasl session sse sse2 ssl tcpd threads unicode zlib" ABI_X86="64" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump author" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ublox ubx" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LINGUAS="en" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-4 php5-5" PYTHON_SINGLE_TARGET="python2_7" PYTHON_TARGETS="python2_7 python3_3" RUBY_TARGETS="ruby19 ruby18" USERLAND="GNU" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account" Unset: CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LANG, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, SYNC, USE_PYTHON
Hey Josh, thanks for reporting. I can confirm the bug on my box. No idea yet though.
Thanks Benda! This was making me nuts since mariadb-galera depends on openbsd-netcat to pass replication data between clustered databases and it took me a day to figure out the cause of replication not happening. (I'm working on moving my production DB from MySQL 5.1 to clustered MariaDB.) The weirdest thing is that I have one box that it works fine on - my home server. I've been trying to find the difference between that one and the ones it isn't. Moving the binary between the good Gentoo box to the bad ones doesn't change anything - even though the Ubuntu binary does work. Moving the binary from the bad machine to the good one does work. My current best guess is that there's something different about a flag or setting in either the kernel or one of the libraries that breaks the openbsd-netcat ebuild. I'm near the end of my usefulness at debugging further. Would the emerge --info on the working box be helpful?
(In reply to Josh G from comment #2) > Thanks Benda! This was making me nuts since mariadb-galera depends on > openbsd-netcat to pass replication data between clustered databases and it > took me a day to figure out the cause of replication not happening. (I'm > working on moving my production DB from MySQL 5.1 to clustered MariaDB.) Oh, sorry for that. > The weirdest thing is that I have one box that it works fine on - my home > server. I've been trying to find the difference between that one and the > ones it isn't. > > Moving the binary between the good Gentoo box to the bad ones doesn't change > anything - even though the Ubuntu binary does work. Moving the binary from > the bad machine to the good one does work. > > My current best guess is that there's something different about a flag or > setting in either the kernel or one of the libraries that breaks the > openbsd-netcat ebuild. I'm near the end of my usefulness at debugging > further. Yeah, I guess so, too. > Would the emerge --info on the working box be helpful? Yes please.
Working box's emerge --info: styx portage # emerge --info Portage 2.2.7 (default/linux/amd64/13.0, gcc-4.7.3, glibc-2.16.0, 3.11.4-gentoo x86_64) ================================================================= System uname: Linux-3.11.4-gentoo-x86_64-Intel-R-_Core-TM-_i5-2500_CPU_@_3.30GHz-with-gentoo-2.2 KiB Mem: 8193540 total, 65040 free KiB Swap: 2104508 total, 2040680 free Timestamp of tree: Tue, 03 Dec 2013 08:15:01 +0000 ld GNU ld (GNU Binutils) 2.23.1 distcc 3.1 x86_64-pc-linux-gnu [disabled] ccache version 3.1.9 [disabled] app-shells/bash: 4.2_p45 dev-java/java-config: 2.1.12-r1 dev-lang/python: 2.7.5-r3, 3.2.5-r3, 3.3.2-r2 dev-util/ccache: 3.1.9 dev-util/cmake: 2.8.11.2 dev-util/pkgconfig: 0.28 sys-apps/baselayout: 2.2 sys-apps/openrc: 0.12.4 sys-apps/sandbox: 2.6-r1 sys-devel/autoconf: 2.13, 2.69 sys-devel/automake: 1.11.6, 1.12.6, 1.13.4 sys-devel/binutils: 2.23.1 sys-devel/gcc: 4.7.3-r1 sys-devel/gcc-config: 1.7.3 sys-devel/libtool: 2.4.2 sys-devel/make: 3.82-r4 sys-kernel/linux-headers: 3.9 (virtual/os-headers) sys-libs/glibc: 2.16.0 Repositories: gentoo sunrise mysql x-portage ACCEPT_KEYWORDS="amd64" ACCEPT_LICENSE="*" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-O2 -pipe -fomit-frame-pointer" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/share/gnupg/qualified.txt /var/bind /var/lib/hsqldb" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php5.4/ext-active/ /etc/php/apache2-php5.5/ext-active/ /etc/php/cgi-php5.4/ext-active/ /etc/php/cgi-php5.5/ext-active/ /etc/php/cli-php5.4/ext-active/ /etc/php/cli-php5.5/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo" CXXFLAGS="-O2 -pipe -fomit-frame-pointer" DISTDIR="/usr/portage/distfiles" FCFLAGS="-O2 -pipe" FEATURES="assume-digests binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync news parallel-fetch preserve-libs protect-owned sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync" FFLAGS="-O2 -pipe" GENTOO_MIRRORS="http://distfiles.gentoo.org" LANG="en_US.UTF-8" LDFLAGS="-Wl,-O1 -Wl,--as-needed" MAKEOPTS="-j4" PKGDIR="/usr/portage/packages" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/home/portage/portage" PORTDIR_OVERLAY="/var/lib/layman/sunrise /var/lib/layman/mysql /usr/local/portage" SYNC="rsync://rsync.us.gentoo.org/gentoo-portage" USE="X a52 aac acl acpi alsa amd64 apache2 arts avi berkdb bzip2 cairo cdb cdr cli consolekit cracklib crypt cxx divx dri dvd dvdr encode exif expat extensions ffmpeg flac fortran freetype gd gdbm gmp gstreamer gtk gudev hwdb iconv icu idn ipv6 java jpeg live lua mad matroska midi mmx mmxext modules mp3 mpeg mudflap multilib ncurses nls nonfsv4 nptl nptlonly ogg oggvorbis opengl openmp oss pam pcre pdf perl png policykit pulseaudio python quicktime readline rtsp samba session sse sse2 sse3 ssl svg tcpd theora threads tiff transcode truetype unicode upcall vdpau vhosts vnc vorbis win32codecs wmp wxwindows x264 xanim xinerama xvid zlib" ABI_X86="64" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="actions alias access_compat auth_basic authn_alias authn_anon authn_dbm authn_core authn_default authn_file authz_dbm authz_core authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info lbmethod_bybusyness lbmethod_byrequests lbmethod_bytraffic lbmethod_heartbeat log_config logio mem_cache mime mime_magic negotiation proxy proxy_ajp proxy_balancer proxy_connect proxy_fcgi proxy_http rewrite setenvif slotmem_shm socache_shmcb speling status unique_id unixd userdir usertrack vhost_alias" APACHE2_MPMS="event" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump author" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="keyboard mouse" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LINGUAS="en" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-4" PYTHON_SINGLE_TARGET="python2_7" PYTHON_TARGETS="python2_7 python3_3" RUBY_TARGETS="ruby19 ruby18" USERLAND="GNU" VIDEO_CARDS="vesa vmware" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account" Unset: CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, USE_PYTHON
Second Working box's emerge --info (2 of 11 of my Gentoo boxes work - this one is i686) linux # emerge --info Portage 2.2.1 (default/linux/x86/13.0, gcc-4.7.3, glibc-2.15-r3, 3.9.3-gentoo i686) ================================================================= System uname: Linux-3.9.3-gentoo-i686-Intel-R-_Xeon-R-_CPU_L5639_@_2.13GHz-with-gentoo-2.2 KiB Mem: 121496 total, 10924 free KiB Swap: 262140 total, 257580 free Timestamp of tree: Tue, 03 Dec 2013 08:15:01 +0000 ld GNU ld (GNU Binutils) 2.23.1 app-shells/bash: 4.2_p45 dev-lang/python: 2.7.5-r3, 3.2.5-r3 dev-util/cmake: 2.8.10.2-r2 dev-util/pkgconfig: 0.28 sys-apps/baselayout: 2.2 sys-apps/openrc: 0.11.8 sys-apps/sandbox: 2.6-r1 sys-devel/autoconf: 2.69 sys-devel/automake: 1.12.6, 1.13.4 sys-devel/binutils: 2.23.1 sys-devel/gcc: 4.7.3-r1 sys-devel/gcc-config: 1.7.3 sys-devel/libtool: 2.4.2 sys-devel/make: 3.82-r4 sys-kernel/linux-headers: 3.9 (virtual/os-headers) sys-libs/glibc: 2.15-r3 Repositories: gentoo ACCEPT_KEYWORDS="x86" ACCEPT_LICENSE="* -@EULA" CBUILD="i686-pc-linux-gnu" CFLAGS="-march=native -O2 -pipe -fomit-frame-pointer" CHOST="i686-pc-linux-gnu" CONFIG_PROTECT="/etc /var/bind" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/gconf /etc/gentoo-release /etc/php/apache2-php5.4/ext-active/ /etc/php/cgi-php5.4/ext-active/ /etc/php/cli-php5.4/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo" CXXFLAGS="-march=native -O2 -pipe -fomit-frame-pointer" DISTDIR="/usr/portage/distfiles" FCFLAGS="-O2 -march=i686 -pipe" FEATURES="assume-digests binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync news parallel-fetch preserve-libs protect-owned sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync" FFLAGS="-O2 -march=i686 -pipe" GENTOO_MIRRORS="http://distfiles.gentoo.org" LDFLAGS="-Wl,-O1 -Wl,--as-needed" PKGDIR="/usr/portage/packages" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" PORTDIR_OVERLAY="" SYNC="rsync://rsync.us.gentoo.org/gentoo-portage" USE="acl berkdb bzip2 cli cracklib crypt cxx dri fortran gdbm iconv ipv6 modules mudflap ncurses nls nptl openmp pam pcre readline session ssl tcpd unicode x86 zlib" ABI_X86="32" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1 emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump author" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="keyboard mouse evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LINGUAS="en_US en" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-4 php-5-5" PYTHON_SINGLE_TARGET="python2_7" PYTHON_TARGETS="python3_2" RUBY_TARGETS="ruby19 ruby18" USERLAND="GNU" VIDEO_CARDS="fbdev glint intel mach64 mga nouveau nv r128 radeon savage sis tdfx trident vesa via vmware dummy v4l" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account" Unset: CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LANG, LC_ALL, MAKEOPTS, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, USE_PYTHON
Aha! Eureka!!! I figured it out. I was looking too specifically at the kernel version instead of a commonality... The working systems all have kernel versions of 3.9 or later. It is very likely that the Linux headers were updated and using SO_REUSEPORT which wasn't added to the kernel until 3.9. I ran into this again, but not as destructively, when building the latest version of Samba. See here: http://forums.gentoo.org/viewtopic-p-7320052.html?sid=6b6a706331bb26a1a0de68158a3eadec#7320052 http://stackoverflow.com/questions/14388706/socket-options-so-reuseaddr-and-so-reuseport-how-do-they-differ-do-they-mean-t To test, I updated the kernel on a system from 3.5.7 to 3.10.17 and the binary that didn't work beforehand, worked just fine after rebooting into the new kernel.
(In reply to Josh G from comment #6) > Aha! Eureka!!! > > I figured it out. I was looking too specifically at the kernel version > instead of a commonality... > > The working systems all have kernel versions of 3.9 or later. It is very > likely that the Linux headers were updated and using SO_REUSEPORT which > wasn't added to the kernel until 3.9. > > I ran into this again, but not as destructively, when building the latest > version of Samba. > > See here: > > http://forums.gentoo.org/viewtopic-p-7320052. > html?sid=6b6a706331bb26a1a0de68158a3eadec#7320052 > > http://stackoverflow.com/questions/14388706/socket-options-so-reuseaddr-and- > so-reuseport-how-do-they-differ-do-they-mean-t > > To test, I updated the kernel on a system from 3.5.7 to 3.10.17 and the > binary that didn't work beforehand, worked just fine after rebooting into > the new kernel. Great shot Josh! What a story! Out of curiosity, I checked the debian package, http://patch-tracker.debian.org/patch/series/view/netcat-openbsd/1.105-7/0001-port-to-linux-with-libsd.patch search for "SO_REUSEPORT" you'll see it is carefully handled. Our ebuild is stealed from RHEL6 ftp://ftp.redhat.com/pub/redhat/linux/enterprise/6Server/en/os/SRPMS/nc-1.84-22.el6.src.rpm and is similarly taken care of. So... why does our ebuild call SO_REUSEPORT even when the kernel does not support it? The kernel header and real kernel is out of sync.
Yup, that is what it was, the installed linux-headers outran our running kernel versions. -Josh
Thanks Josh, it's hard to detect kernel versions reliably. I've added a warning message and pointer to this bug in the ebuild.
Thanks Benda! If you feel like revisiting it, "uname -r" seems like a pretty safe way to detect the kernel revision.
