Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 671080 - =sys-libs/glibc-2.27-r6: long lines in /etc/group skipped when using getent (getgrent) with "group: files" in /etc/nsswitch.conf
Summary: =sys-libs/glibc-2.27-r6: long lines in /etc/group skipped when using getent (...
Status: RESOLVED FIXED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Current packages (show other bugs)
Hardware: All Linux
: Normal normal (vote)
Assignee: Gentoo Toolchain Maintainers
URL: https://sourceware.org/PR18991
Whiteboard:
Keywords:
Depends on: glibc-2.28-stable
Blocks:
  Show dependency tree
 
Reported: 2018-11-13 14:52 UTC by Nico Rittner
Modified: 2019-06-22 14:16 UTC (History)
1 user (show)

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Nico Rittner 2018-11-13 14:52:11 UTC
when using "group: files" in /etc/nsswitch.conf as suggested at https://wiki.gentoo.org/wiki/Glibc_2.26_porting_notes/nsswitch.conf_in_glibc-2.26
the bundled "getent" utility skips long(er) lines in output when querying
/etc/group without a key ("getent group").
in my case lines with more than 762 characters (many group members) are not printed.
no error message seen.
when repeating the "oversized" line in /etc/group, it is printed N-1 times.
when using a key ("getent group $groupname") the "oversized" line is printed correctly.
after switching to "group: compat" all lines are being printed as normal.
i assume this issue is related to the underlaying "getgrent()" function since perl behaves the same when using "getgrent()".
does "compat" (beside the NIS stuff) mean that line length does not matter and "files" implies some limit?


Reproducible: Always

Steps to Reproduce:
1. put many members in /etc/group until the length of the corresponding line exceeds 762 characters; use "group: files" in /etc/nsswitch.conf
2. $ getent group;
-- or --
3. $ perl -e 'setgrent(); while( my @g = getgrent() ){ printf( "%s: %s\n", $g[0], $g[3] ); }; endgrent();';

Actual Results:  
missed lines of /etc/group with > 762 chars

Expected Results:  
all lines of /etc/group without skipping some of them silently. if there is any limit, an error message should go to stderr.

Portage 2.3.49 (python 3.6.5-final-0, default/linux/amd64/17.0/no-multilib, gcc-7.3.0, glibc-2.27-r6, 4.14.65-jason-default-x86_64-1540277370 x86_64)
=================================================================
System uname: Linux-4.14.65-jason-default-x86_64-1540277370-x86_64-Intel-R-_Pentium-R-_CPU_G6950_@_2.80GHz-with-gentoo-2.4.1
KiB Mem:    10148240 total,   1221144 free
KiB Swap:          0 total,         0 free
Timestamp of repository gentoo: Tue, 13 Nov 2018 13:00:01 +0000
Head commit of repository gentoo: 15aa69164a64d75fe7e472b1ea2c91abfc0fb444
sh bash 4.4_p12
ld GNU ld (Gentoo 2.30 p5) 2.30.0
app-shells/bash:          4.4_p12::gentoo
dev-lang/perl:            5.24.3-r1::gentoo
dev-lang/python:          2.7.15::gentoo, 3.6.5::gentoo
dev-util/cmake:           3.9.6::gentoo
dev-util/pkgconfig:       0.29.2::gentoo
sys-apps/baselayout:      2.4.1-r2::gentoo
sys-apps/openrc:          0.38.3::gentoo
sys-apps/sandbox:         2.13::gentoo
sys-devel/autoconf:       2.69-r4::gentoo
sys-devel/automake:       1.11.6-r3::gentoo, 1.15.1-r2::gentoo
sys-devel/binutils:       2.30-r4::gentoo
sys-devel/gcc:            7.3.0-r3::gentoo
sys-devel/gcc-config:     1.8-r1::gentoo
sys-devel/libtool:        2.4.6-r3::gentoo
sys-devel/make:           4.2.1-r4::gentoo
sys-kernel/linux-headers: 4.13::gentoo (virtual/os-headers)
sys-libs/glibc:           2.27-r6::gentoo
Repositories:

gentoo
    location: /var/db/portage
    sync-type: rsync
    sync-uri: rsync://rsync.gentoo.org/gentoo-portage
    priority: -1000
    sync-rsync-extra-opts:
    sync-rsync-verify-metamanifest: no
    sync-rsync-verify-jobs: 1
    sync-rsync-verify-max-age: 24

layer23
    location: /dat/local/share/portage/db
    masters: gentoo

ACCEPT_KEYWORDS="amd64"
ACCEPT_LICENSE="* -@EULA"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-O2 -fomit-frame-pointer -pipe"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/share/gnupg/qualified.txt /var/bind"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php5.6/ext-active/ /etc/php/apache2-php7.0/ext-active/ /etc/php/apache2-php7.1/ext-active/ /etc/php/apache2-php7.2/ext-active/ /etc/php/cgi-php5.6/ext-active/ /etc/php/cgi-php7.0/ext-active/ /etc/php/cgi-php7.1/ext-active/ /etc/php/cgi-php7.2/ext-active/ /etc/php/cli-php5.6/ext-active/ /etc/php/cli-php7.0/ext-active/ /etc/php/cli-php7.1/ext-active/ /etc/php/cli-php7.2/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-O2 -fomit-frame-pointer -pipe"
DISTDIR="/tmp"
ENV_UNSET="DBUS_SESSION_BUS_ADDRESS DISPLAY GOBIN PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR"
FCFLAGS="-O2 -pipe"
FEATURES="assume-digests binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync multilib-strict news parallel-fetch preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync"
FFLAGS="-O2 -pipe"
GENTOO_MIRRORS="http://distfiles.gentoo.org"
LANG="de_DE.UTF-8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
MAKEOPTS="-j3"
PKGDIR="/tmp"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"
PORTAGE_TMPDIR="/var/tmp"
USE="acl alsa amd64 berkdb bzip2 cli cxx dri fortran gd iconv imap jpeg ldap libtirpc mbox mysql ncurses nptl openmp pam pcre perl png readline sasl seccomp ssl syslog tiff truetype unicode xml zlib" ABI_X86="64" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="actions alias asis auth_basic auth_digest filter authn_core authn_file authn_anon authz_core authz_groupfile authz_host authz_owner authz_user access_compat cache cache_disk cgi deflate dumpio dav dav_fs dir env expires headers include info log_config macro mime mime_magic proxy proxy_balancer proxy_connect proxy_fcgi proxy_http ratelimit rewrite setenvif slotmem_shm so socache_shmcb status substitute userdir unique_id unixd vhost_alias" APACHE2_MPMS="prefork" CALLIGRA_FEATURES="karbon plan sheets stage words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="mmx sse sse2" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="libinput keyboard mouse" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" NGINX_MODULES_HTTP="access auth_basic charset dav fastcgi geo gzip limit_req lua map memcached proxy rewrite ssl stub_status upstream_keepalive" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-6 php7-1" POSTGRES_TARGETS="postgres9_5 postgres10" PYTHON_SINGLE_TARGET="python3_6" PYTHON_TARGETS="python2_7 python3_6" QEMU_SOFTMMU_TARGETS="x86_64" RUBY_TARGETS="ruby23 ruby24" USERLAND="GNU" VIDEO_CARDS="amdgpu fbdev intel nouveau radeon radeonsi vesa dummy v4l" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CC, CPPFLAGS, CTARGET, CXX, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LC_ALL, LINGUAS, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS
Comment 1 Sergei Trofimovich (RETIRED) gentoo-dev 2018-12-01 12:00:29 UTC
I failed to reproduce it in a chroot [1].

If you can reproduce it can you craft /etc/passwd and /etc/group files that exhibit the failure and attach it here along with /etc/nsswitch.conf?

[1]: my attempt:

# groupadd testgroup
# getent group testgroup
testgroup:x:30001:
# for i in $(seq 1 100); do useradd testuser${i} -G testgroup; done
# getent group testgroup
# getent group testgroup
testgroup:x:30001:testuser1,testuser2,testuser3,testuser4,testuser5,testuser6,testuser7,testuser8,testuser9,testuser10,testuser11,testuser12,testuser13,testuser14,testuser15,testuser16,testuser17,testuser18,testuser19,testuser20,testuser21,testuser22,testuser23,testuser24,testuser25,testuser26,testuser27,testuser28,testuser29,testuser30,testuser31,testuser32,testuser33,testuser34,testuser35,testuser36,testuser37,testuser38,testuser39,testuser40,testuser41,testuser42,testuser43,testuser44,testuser45,testuser46,testuser47,testuser48,testuser49,testuser50,testuser51,testuser52,testuser53,testuser54,testuser55,testuser56,testuser57,testuser58,testuser59,testuser60,testuser61,testuser62,testuser63,testuser64,testuser65,testuser66,testuser67,testuser68,testuser69,testuser70,testuser71,testuser72,testuser73,testuser74,testuser75,testuser76,testuser77,testuser78,testuser79,testuser80,testuser81,testuser82,testuser83,testuser84,testuser85,testuser86,testuser87,testuser88,testuser89,testuser90,testuser91,testuser92,testuser93,testuser94,testuser95,testuser96,testuser97,testuser98,testuser99,testuser100
# getent group testgroup | wc -c
1110
Comment 2 Nico Rittner 2018-12-12 14:11:20 UTC
using your example i get the same results. but as i wrote in my example ("steps to reproduce"), the issue occurs when omitting the second argument (the <key>) and only using the db-name "group" as the only argument to "getent". then the specific line is simply skipped in output. similar behaviour using the mentioned perl-oneliner. sry for replying late.

extending your example:
# getent group | grep ^testgroup: | wc -l;
0 with "group: files" in /etc/nssswitch.conf
1 with "group: compat" in /etc/nssswitch.conf
Comment 3 Andreas K. Hüttel archtester gentoo-dev 2018-12-30 20:35:57 UTC
Let's check if this is fixed in 2.28
Comment 4 Andreas K. Hüttel archtester gentoo-dev 2019-02-02 15:05:24 UTC
Not a regression in 2.28 anyway.

Did you file an upstream bug? If yes please link to it here.
Comment 5 Sergei Trofimovich (RETIRED) gentoo-dev 2019-02-06 22:32:49 UTC
(In reply to Nico Rittner from comment #2)
> using your example i get the same results. but as i wrote in my example
> ("steps to reproduce"), the issue occurs when omitting the second argument
> (the <key>) and only using the db-name "group" as the only argument to
> "getent". then the specific line is simply skipped in output. similar
> behaviour using the mentioned perl-oneliner. sry for replying late.
> 
> extending your example:
> # getent group | grep ^testgroup: | wc -l;
> 0 with "group: files" in /etc/nssswitch.conf
> 1 with "group: compat" in /etc/nssswitch.conf

Thank you! Managed to reproduce it on glibc-2.27 but not on glibc-2.29. Let's see if I can bisect it down to the fix.
Comment 6 Sergei Trofimovich (RETIRED) gentoo-dev 2019-02-07 07:22:16 UTC
Bisected it down to:

  916124ed841745b7a1e0fbc43f9909340b47d373 is the first bad commit
  commit 916124ed841745b7a1e0fbc43f9909340b47d373
  Author: Florian Weimer <fweimer@redhat.com>
  Date:   Fri Jul 6 14:23:15 2018 +0200

    nss_files: Fix re-reading of long lines [BZ #18991]
    
    Use the new __libc_readline_unlocked function to pick up
    reading at the same line in case the buffer needs to be enlarged.

  :100644 100644 51e6bdcefc5efeb34efb3f221513622cd4a92475 7a052c344a10cf3c9c89bf8ad1c8e5a535cab192 M      ChangeLog
  :040000 040000 f03d96780ae1f73923e51333dcad792ac3bfb686 a737d442aacbd3fbbb76159aa8b9535f5a0e7aa2 M      nss

Looks like a legitimate fix:
    https://sourceware.org/PR18991

Fixed in glibc-2.28.
Comment 7 Sergei Trofimovich (RETIRED) gentoo-dev 2019-06-22 14:16:51 UTC
glibc-2.28 is stable everywhere.