My system is ridiculous slow now since just about everything that calls getpwnam/getpwnam_r downloads the entire password database from NIS on every call. It should be doing a ypmatch against the name. I noticed this immediately when upgrading from glibc-2.21-r1 to 2.22-r1. jharvell@wolfhound ~ $ time strace -ffo /tmp/id-jharvell.strace id jharvell uid=4393(jharvell) gid=7000(cta) groupes=7000(cta),10(wheel),18(audio),19(cdrom),250(portage),103(vboxusers),102(wireshark),500(jharvell),35(games),78(kvm) real 0m7.734s user 0m0.532s sys 0m1.370s I have attached the strace file. it shows (and wireshark confirms) that whatever id is calling (presumably getpwnam_r or getpwnam), it is downloading the entire password database from one of the NIS servers instead of issuing a YPMATCH request to them. My system is bordering on unusable until I can get this resolved. jharvell@wolfhound ~ $ cat /etc/nsswitch.conf # /etc/nsswitch.conf: # $Header: /var/cvsroot/gentoo/src/patchsets/glibc/extra/etc/nsswitch.conf,v 1.1 2006/09/29 23:52:23 vapier Exp $ passwd: compat shadow: compat group: compat hosts: files dns networks: files dns services: db files protocols: db files rpc: db files ethers: db files netmasks: files netgroup: files nis bootparams: files automount: files nis aliases: files jharvell@wolfhound ~ $ tail -n5 /etc/passwd geoclue:x:126:974:added by portage for geoclue:/var/lib/geoclue:/sbin/nologin postgres:x:70:70:added by portage for postgresql:/var/lib/postgresql:/bin/sh +jharvell:::::/mnt/etretat/home/jharvell: +wzhang7:::::/mnt/etretat/home/wzhang7: +:::::: jharvell@wolfhound ~ $ tail -n5 /etc/group geoclue:x:974: postgres:x:70: +:::::: dialout:x:979: lock:x:978:
Created attachment 424358 [details] strace output of strace command in original bug report
Created attachment 424362 [details] wireshark TCP conversations during output of 'id jharvell' command Note all these connections are between my desktop and the NIS server
jharvell@wolfhound /tmp $ eix glibc [I] sys-libs/glibc Available versions: (2.2) 2.17^s (~)2.18-r1^s (~)2.18-r1^s[1] 2.19-r1^s 2.20-r2^s 2.21-r1^s (~)2.22-r1^s{tbz2} **9999^s {debug gd hardened multilib nscd profile selinux suid systemtap vanilla CROSSCOMPILE_OPTS="headers-only"} Installed versions: 2.22-r1(2.2)^s{tbz2}(11:24:49 31/01/2016)(multilib suid -debug -gd -hardened -nscd -profile -selinux -systemtap -vanilla CROSSCOMPILE_OPTS="-headers-only") Homepage: https://www.gnu.org/software/libc/libc.html Description: GNU libc6 (also called glibc2) C library [1] "wolfhound" /opt/portage
jharvell@wolfhound /tmp $ emerge --info Portage 2.2.27 (python 3.4.3-final-0, default/linux/amd64/13.0, gcc-5.3.0, glibc-2.22-r1, 4.4.0 x86_64) ================================================================= System uname: Linux-4.4.0-x86_64-Intel-R-_Xeon-R-_CPU_E5-2630_0_@_2.30GHz-with-gentoo-2.2 KiB Mem: 32876860 total, 10324824 free Timestamp of repository gentoo: Fri, 29 Jan 2016 16:15:01 +0000 sh bash 4.3_p42-r1 ld GNU ld (Gentoo git) 2.26.51.20160130 distcc 3.2rc1 x86_64-pc-linux-gnu [disabled] ccache version 3.2.4 [disabled] app-shells/bash: 4.3_p42-r1::gentoo dev-java/java-config: 2.2.0::gentoo dev-lang/perl: 5.22.1::gentoo dev-lang/python: 2.7.11-r2::gentoo, 3.2.5-r3::gentoo, 3.4.3-r7::gentoo dev-util/ccache: 3.2.4::gentoo dev-util/cmake: 3.4.3::gentoo dev-util/pkgconfig: 0.29::gentoo sys-apps/baselayout: 2.2::gentoo sys-apps/openrc: 0.20.4::gentoo sys-apps/sandbox: 2.10-r1::gentoo sys-devel/autoconf: 2.13::gentoo, 2.69-r1::gentoo sys-devel/automake: 1.10.3-r1::gentoo, 1.11.6-r2::gentoo, 1.12.6-r1::gentoo, 1.13.4-r1::gentoo, 1.14.1-r1::gentoo, 1.15-r1::gentoo sys-devel/binutils: 9999::gentoo sys-devel/gcc: 4.8.3::gentoo, 4.9.3::gentoo, 5.3.0::gentoo sys-devel/gcc-config: 1.8::gentoo sys-devel/libtool: 2.4.6-r1::gentoo sys-devel/make: 4.1-r1::gentoo sys-kernel/linux-headers: 4.4::gentoo (virtual/os-headers) sys-libs/glibc: 2.22-r1::gentoo Repositories: gentoo location: /usr/portage sync-type: rsync sync-uri: rsync://rsync.us.gentoo.org/gentoo-portage priority: -1000 wolfhound location: /opt/portage masters: gentoo priority: 0 gentoo-zh location: /var/lib/layman/gentoo-zh masters: gentoo priority: 50 seden location: /var/lib/layman/seden masters: gentoo priority: 50 ACCEPT_KEYWORDS="amd64 ~amd64" ACCEPT_LICENSE="* -@EULA Oracle-BCLA-JavaSE googleearth AdobeFlash-10.3 AdobeFlash-11.x" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-O3 -pipe -march=native -ggdb -flto=26 -floop-interchange -ftree-loop-distribution -floop-strip-mine -floop-block -ftree-vectorize" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc /opt/apache-zookeeper/conf /usr/lib64/libreoffice/program/sofficerc /usr/share/config /usr/share/gnupg/qualified.txt /usr/share/maven-bin-2.2/conf /usr/share/maven-bin-3.1/conf /usr/share/maven-bin-3.2/conf /usr/share/maven-bin-3.3/conf /var/lib/hsqldb" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c" CXXFLAGS="-O3 -pipe -march=native -ggdb -flto=26 -floop-interchange -ftree-loop-distribution -floop-strip-mine -floop-block -ftree-vectorize" DISTDIR="/usr/portage/distfiles" EMERGE_DEFAULT_OPTS="--jobs=30 --load-average 85.0 --accept-properties=-interactive" FCFLAGS="-O2 -pipe" FEATURES="assume-digests binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync news parallel-fetch parallel-install preserve-libs protect-owned sandbox sfperms splitdebug strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr" FFLAGS="-O2 -pipe" GENTOO_MIRRORS="http://gentoo.mirrors.easynews.com/linux/gentoo/ http://gentoo.cites.uiuc.edu/pub/gentoo/ http://gentoo.mirrors.tds.net/gentoo" LANG="fr_FR.UTF-8" LDFLAGS="-flto=26 -fuse-ld=gold -fuse-linker-plugin" MAKEOPTS="--jobs=30 --load-average=85.0" PKGDIR="/usr/portage/packages" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git" PORTAGE_TMPDIR="/var/tmp" USE="X acl acpi alsa amd64 bash-completion berkdb bidi bzip2 c++0x caps cjk cli cracklib crypt cups cxx dri exif filecaps fontconfig foomaticdb fortran ftp gdbm geoip gif gimp gnome-keyring gnuplot gnutls gtk gtk3 gzip handbook hddtemp iconv icq icu idn imap ipv6 jabber java java6 javascript jingle jit jpeg jpeg2k kde kontact lame latex ldap lm_sensors mime mms mmx mmxext mng modules mono mozilla mp3 mp4 mpeg mplayer msn multilib mysql mysqli ncurses nfsv41 nis nls nptl nsplugin odbc offensive opengl openmp oscar pam pch pcntl pcre pdf perl php plasma png posix postscript ppds python qt3support qt4 quicktime rdesktop readline rss ruby samba sasl sdl seccomp semantic-desktop session smp sound spell sse sse2 sse3 sse4 sse4_1 ssl ssse3 startup-notification subversion suid svg syslog systemd sysvipc tcmalloc tcpd threads tiff truetype udev unicode vnc win32codecs xattr xinerama xinetd xmpp yahoo zlib" ABI_X86="64" ALSA_CARDS="hda-intel" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump author" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="mmx mmxext sse sse2" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="keyboard mouse evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LINGUAS="en_US fr_FR en fr" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-5" PYTHON_SINGLE_TARGET="python2_7" PYTHON_TARGETS="python2_7 python3_4" QEMU_SOFTMMU_TARGETS="x86_64 mips64" QEMU_USER_TARGETS="x86_64 mips64" RUBY_TARGETS="ruby20 ruby21" USERLAND="GNU" VIDEO_CARDS="radeon" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account" Unset: CC, CPPFLAGS, CTARGET, CXX, INSTALL_MASK, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, USE_PYTHON
I should mention this is built with gcc-5.3.0 using O3 and lto and graphite optimizations. Also, I'm using the git version of binutils updated within the past few days I also notice that the Changelog of glibc describes changes to nss.
I'm rebuilding now with LTO disabled and also the nscd USE flag enabled. If LTO is not the problem, then at least the cache might prevent me our IT from identifying my machine as a denial of service platform.
When I ran the id command that I attached strace output for, I had the contents of /etc/default/nss as SETENT_BATCH_READ=FALSE I did not have and /etc/default/nss file at all when I first noticed the problem. And I just deleted the file and re-ran the 'time strace -ffo /tmp/blah id jharvel' command. It still took 7.7 seconds and wireshark still showed there were 1256 TCP conversations. I did not look at the strace output. Also, here is the rest of the NIS related config: jharvell@wolfhound ~ $ cat /etc/yp.conf domain inetnis server 134.64.2.60 domain inetnis server 134.64.2.63 jharvell@wolfhound ~ $ nisdomainname inetnis
I am also seeing this bug. I went from 2.21-r2 to 2.22-r4. I tried tracking this down myself with little success at figuring out what was going on. Interestingly, I didn't see the problem with ypmatch, but using 'su' or simply logging-in causes the same symptoms that the original reporter (Joe Harvell) sees. This doesn't really add anything useful to the bug report except to add that there is more than one person who sees the problem. I wanted to go back to glibc 2.21, but portage strongly advised against it. I thought it would be prudent to heed its warning.
Building glibc with nscd makes my system usable. But it's not a solution for me since I want to use environment variables controlling resolver behavior with different values for different processes. This doesn't work with nscd since in that configuration only nscd is running the code that looks at the environment variables behaves differently.
Is this bug also present in current glibc versions?
Andreas: To workaround at the time, I lived with not having NIS "compat" enabled in nsswitch.conf, and forgot about this. I tried putting everything back just now and I don't see the problem happening in glibc 2.23-r4. I'm not convinced that the glibc version change was the fix, though. This may not be related, but I'd recommend looking at this bug in glibc upstream: https://sourceware.org/bugzilla/show_bug.cgi?id=18023 It is possible that a gcc update actually fixed the problem we were seeing here. Even though extend_alloca was not removed from glibc 2.23-r4, if the new gcc is creating different code, it could be at least masking the extend_alloca problem mentioned in the upstream bug. Again, I have not looked at this enough to say that it is truly a fix, but the problem does seem to be resolved for me at this point.
(In reply to Gil Kloepfer from comment #11) > > Again, I have not looked at this enough to say that it is truly a fix, but > the problem does seem to be resolved for me at this point. OK thanks.
