Created attachment 429518 [details] Patch log sys-devel/binutils-2.25.1-r1 fails to build after changing profile to hardened. Increasing limits fixes the build. i.e. * hard nofile 8192 * soft nofile 4096 emerge --info Portage 2.2.28 (python 3.4.3-final-0, hardened/linux/amd64, gcc-5.3.0, glibc-2.22-r3, 4.5.0-gentoo x86_64) ================================================================= System uname: Linux-4.5.0-gentoo-x86_64-Intel-R-_Core-TM-_i7-4770HQ_CPU_@_2.20GHz-with-gentoo-2.2 KiB Mem: 16317024 total, 11318568 free KiB Swap: 0 total, 0 free Timestamp of repository gentoo: Sat, 02 Apr 2016 20:30:01 +0000 sh bash 4.3_p42-r2 ld GNU ld (Gentoo 2.25.1 p1.1) 2.25.1 ccache version 3.2.4 [enabled] app-shells/bash: 4.3_p42-r2::gentoo dev-java/java-config: 2.2.0-r3::gentoo dev-lang/perl: 5.22.1::gentoo dev-lang/python: 2.7.11-r2::gentoo, 3.4.3-r7::gentoo dev-util/ccache: 3.2.4::gentoo dev-util/cmake: 3.5.1::gentoo dev-util/pkgconfig: 0.29.1::gentoo sys-apps/baselayout: 2.2::gentoo sys-apps/openrc: 0.20.5::gentoo sys-apps/sandbox: 2.10-r1::gentoo sys-devel/autoconf: 2.13::gentoo, 2.69-r2::gentoo sys-devel/automake: 1.11.6-r2::gentoo, 1.13.4-r1::gentoo, 1.14.1-r1::gentoo, 1.15-r2::gentoo sys-devel/binutils: 2.25.1-r1::gentoo sys-devel/gcc: 5.3.0::gentoo sys-devel/gcc-config: 1.8-r1::gentoo sys-devel/libtool: 2.4.6-r2::gentoo sys-devel/make: 4.1-r1::gentoo sys-kernel/linux-headers: 4.5::gentoo (virtual/os-headers) sys-libs/glibc: 2.22-r3::gentoo Repositories: gentoo location: /usr/portage sync-type: rsync sync-uri: rsync://rsync.gentoo.org/gentoo-portage priority: -1000 andjscott location: /usr/local/portage masters: gentoo java location: /var/lib/layman/java masters: gentoo priority: 50 megacoffee location: /var/lib/layman/megacoffee masters: gentoo priority: 50 raiagent location: /var/lib/layman/raiagent masters: gentoo priority: 50 rust location: /var/lib/layman/rust masters: gentoo priority: 50 sabayon location: /var/lib/layman/sabayon masters: gentoo priority: 50 sunrise location: /var/lib/layman/sunrise masters: gentoo priority: 50 tranquility location: /var/lib/layman/tranquility masters: gentoo priority: 50 y2kbadbug location: /var/lib/layman/y2kbadbug masters: gentoo priority: 50 ACCEPT_KEYWORDS="amd64 ~amd64" ACCEPT_LICENSE="* -@EULA" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-O2 -pipe" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/share/gnupg/qualified.txt" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo" CXXFLAGS="-O2 -pipe" DISTDIR="/usr/portage/distfiles" FCFLAGS="-O2 -pipe" FEATURES="assume-digests binpkg-logs buildpkg ccache config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync news parallel-fetch preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr" FFLAGS="-O2 -pipe" GENTOO_MIRRORS="rsync://mirror.bytemark.co.uk/gentoo/ http://mirror.qubenet.net/mirror/gentoo/ rsync://rsync.mirrorservice.org/distfiles.gentoo.org/" LANG="en_GB.utf8" LDFLAGS="-Wl,-O1 -Wl,--as-needed" MAKEOPTS="-j9" PKGDIR="/usr/portage/packages" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git" PORTAGE_TMPDIR="/var/tmp" USE="X a52 aac acl acpi alsa amd64 berkdb bindist bluetooth branding bzip2 cairo cdda cdr cli consolekit cracklib crypt cups cxx dbus dri dts dvd dvdr emboss encode exif fam firefox flac gdbm gif glamor gtk hardened iconv infinality ipv6 jpeg justify lcms ldap libnotify mad mmx mmxext mng modules mp3 mp4 mpeg multilib ncurses nls nptl ogg opengl openmp pam pango pax_kernel pcre pdf pie png policykit ppds pulseaudio qt3support qt4 readline sdl seccomp session spell sse sse2 ssl ssp startup-notification svg tcpd tiff truetype udev udisks unicode upower urandom usb vorbis wxwidgets x264 xattr xcb xml xtpax xv xvid zlib"/var/tmp/portage/sys-devel/binutils-2.25.1-r1/temp/build.log ABI_X86="64" ALSA_CARDS="hda-intel" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump author" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="mmx mmxext sse sse2" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ublox ubx" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LINGUAS="en_GB" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-5" PYTHON_SINGLE_TARGET="python2_7" PYTHON_TARGETS="python2_7 python3_4" RUBY_TARGETS="ruby20 ruby21" USERLAND="GNU" VIDEO_CARDS="intel" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account" Unset: CC, CPPFLAGS, CTARGET, CXX, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, USE_PYTHON
What did you have the limit set to before increasing it?
I hadn't set the limit before and removing those lines and returns the limit to 1024 according to 'sudo -u portage bash -c "ulimit -n"'.
doubtful this is a bug in any particular ebuild
i can't see how this is hardened related. i'm just guessing here, but if the bug is reproduceable, see what's in dmesg, if pax is killing something or grsec is presenting something because of a hard limit.
not sure how we'd manage to hit a limit of even 1024 open fd normally running maybe portage team knows of a place to put some debug statements to try and capture the failing processes open limit (like /proc/<pid>/fd/).
You can put something like this in /etc/portage/bashrc to log the open file descriptors an the beginning of src_prepare: pre_src_prepare() { elog "${FUNCNAME[0]}: max fd: $(ls -1 /proc/self/fd | tail -n1)" } That will tell us which direction to look in the process tree (open file descriptors that don't have the FD_CLOEXEC flag are typically inherited from parent processes to child processes).
(In reply to Zac Medico from comment #6) > You can put something like this in /etc/portage/bashrc to log the open file > descriptors an the beginning of src_prepare: > > pre_src_prepare() { > elog "${FUNCNAME[0]}: max fd: $(ls -1 /proc/self/fd | tail -n1)" > } > > That will tell us which direction to look in the process tree (open file > descriptors that don't have the FD_CLOEXEC flag are typically inherited from > parent processes to child processes). Actually better use wc -l like this: pre_src_prepare() { elog "${FUNCNAME[0]}: fd count: $(ls -1 /proc/self/fd | wc -l)" }
*** Bug 593092 has been marked as a duplicate of this bug. ***
I can acknowledge that this bug is not related to hardended, because I encountered it on non-hardened amd64-system (using the new x32 ABI) too. See (duplicate) bug 593092 for additional material, such as a directory listing of the working directory after the error message had been displayed. Now I wonder whether "patch" could perhaps be so stupid to open all the files mentioned in the patch at once, keeping them open until all patch chunks have been applied, and only after that close them. This would mean the "number of open files"-limit needed to be at least as large as the number of files affected by the patch. On the other hand, are there even 1024 files in the whole binutils source package? 1024 is actually a rather large count of source files. binutils is not the Linux kernel or LibreOffice, which certainly contain more files. I also severely doubt that the authors of "patch" could really have implemented this utility in such an inefficient way.
It seems "patch" is really that braindead! I just straced the "patch" invocation from bug 593092 and saved the strace output as file "out". Then: $ grep ^close out | wc -l 115 $ grep ^open out | wc -l 1139 It seems, the open file limit actually reached 1024, so it is not a bug in the ebuild infrastructure, in the sandbox, or generally anywhere in the build system. "patch" seems to require a number of open filehandles which is at least partially proportional to the number of files affected by the patch. So the real questions here are: * How to increment the resource limit for the ebuild process, avoiding this problem. * Why does not everyone who tries to emerge this package encounter the same problem. Regarding the last point, I have a suspicion: I am running a completely "systemd"-free system with no logind, consolekit or any other "Poettering" service running. I have also globally disabled the "pam" USE flag, because I hate the increased configuration complexity which comes with PAM. Perhaps most other people *do* use some of those things, and some of them interact with the resource limits and boost them, giving those people higher limits. I have no idea how I can raise the resource limits for the "portage" user right now, but I will see into it. It would be interesting whether other people who encountered the "too many files" issue have PAM and/or Poettering services disabled, too.
Which version of sys-devel/patch was this?
(In reply to Andreas K. Hüttel from comment #11) > Which version of sys-devel/patch was this? I am sorry, I can't tell any more. I stopped using Gentoo a year or so ago, when I encountered combined version+USE-flag dependency hell after making the grave mistake to skip updating for a couple of months. I am using Devuan now, and my original Gentoo installation does not exist any more. But try this: Apply some large patch affecting many files like this: $ strace -f patch -p1 < some.patch 2> plog Then grep + wc the plog file and see whether the open() and close() syscall counts are similar, which has to be expected if they are paired/balanced. But if you see a lot more open than close calls, then you are using a version of patch which obviously has the same problem as mine did.
(In reply to Guenther Brunthaler from comment #12) > (In reply to Andreas K. Hüttel from comment #11) > > Which version of sys-devel/patch was this? > > I am sorry, I can't tell any more. > > I stopped using Gentoo a year or so ago, when I encountered combined > version+USE-flag dependency hell after making the grave mistake to skip > updating for a couple of months. > > I am using Devuan now, and my original Gentoo installation does not exist > any more. > > But try this: Apply some large patch affecting many files like this: > > $ strace -f patch -p1 < some.patch 2> plog > > Then grep + wc the plog file and see whether the open() and close() syscall > counts are similar, which has to be expected if they are paired/balanced. > > But if you see a lot more open than close calls, then you are using a > version of patch which obviously has the same problem as mine did. OK
