Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 661244 - net-libs/grpc-1.13.0 with USE="tools" installs huge (>500 MiB) SOURCES.txt files for grpcio_tools
Summary: net-libs/grpc-1.13.0 with USE="tools" installs huge (>500 MiB) SOURCES.txt fi...
Status: RESOLVED FIXED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Current packages (show other bugs)
Hardware: All Linux
: Normal major (vote)
Assignee: Jason Zaman
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2018-07-15 15:18 UTC by Remy Blank
Modified: 2018-07-15 22:20 UTC (History)
2 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Remy Blank 2018-07-15 15:18:40 UTC
$ ls -lh $(qlist grpc | grep -E 'grpcio_tools.*SOURCES')
-rw-r--r-- 1 root root 557M Jul 15 15:50 /usr/lib64/python2.7/site-packages/grpcio_tools-1.13.0-py2.7.egg-info/SOURCES.txt
-rw-r--r-- 1 root root 557M Jul 15 15:51 /usr/lib64/python3.5/site-packages/grpcio_tools-1.13.0-py3.5.egg-info/SOURCES.txt
-rw-r--r-- 1 root root 557M Jul 15 15:52 /usr/lib64/python3.6/site-packages/grpcio_tools-1.13.0-py3.6.egg-info/SOURCES.txt

Looking at the content of the files, it seems that there is an "infinite" recursion (not quite infinite, probably distutils has a depth limit) and the same files are listed over and over again, at increasing directory depths.

This is due to the "grpc_root" symlink created in tools/distrib/python/grpcio_tools.

# ls -l tools/distrib/python/grpcio_tools/grpc_root
lrwxrwxrwx 1 portage portage 12 Jul 15 16:54 tools/distrib/python/grpcio_tools/grpc_root -> ../../../../

The "grpc_root" directory is listed in MANIFEST.in:

$ cat tools/distrib/python/grpcio_tools/MANIFEST.in 
include grpc_version.py
include protoc_deps.py
include protoc_lib_deps.py
include README.rst
graft grpc_tools
graft grpc_root
graft third_party

So it seems that distutils follows symlinks when processing "graft" entries. Adding the following line at the end of MANIFEST.in breaks the recursion and generates reasonable SOURCES.txt files.

prune grpc_root/tools/distrib/python/grpcio_tools
Comment 1 Remy Blank 2018-07-15 15:23:16 UTC
$ emerge --info
Portage 2.3.40 (python 2.7.14-final-0, default/linux/amd64/17.0, gcc-7.3.0, glibc-2.26-r7, 4.9.90-gentoo x86_64)
=================================================================
System uname: Linux-4.9.90-gentoo-x86_64-Intel-R-_Xeon-R-_CPU_E3-1505M_v5_@_2.80GHz-with-gentoo-2.4.1
KiB Mem:    32747712 total,  13827204 free
KiB Swap:    4194300 total,   4194300 free
Timestamp of repository gentoo: Sun, 15 Jul 2018 01:00:01 +0000
Head commit of repository gentoo: 6e465c01774203248fe78d4a6937d2dbb746f23c
sh bash 4.4_p12
ld GNU ld (Gentoo 2.30 p2) 2.30.0
app-shells/bash:          4.4_p12::gentoo
dev-java/java-config:     2.2.0-r4::gentoo
dev-lang/perl:            5.24.3-r1::gentoo
dev-lang/python:          2.7.14-r1::gentoo, 3.5.5::gentoo, 3.6.5::gentoo
dev-util/cmake:           3.9.6::gentoo
dev-util/pkgconfig:       0.29.2::gentoo
sys-apps/baselayout:      2.4.1-r2::gentoo
sys-apps/openrc:          0.34.11::gentoo
sys-apps/sandbox:         2.13::gentoo
sys-devel/autoconf:       2.13::gentoo, 2.69-r4::gentoo
sys-devel/automake:       1.11.6-r3::gentoo, 1.13.4-r2::gentoo, 1.15.1-r2::gentoo
sys-devel/binutils:       2.30-r2::gentoo
sys-devel/gcc:            7.3.0-r3::gentoo
sys-devel/gcc-config:     1.8-r1::gentoo
sys-devel/libtool:        2.4.6-r3::gentoo
sys-devel/make:           4.2.1::gentoo
sys-kernel/linux-headers: 4.13::gentoo (virtual/os-headers)
sys-libs/glibc:           2.26-r7::gentoo
Repositories:

gentoo
    location: /usr/portage
    sync-type: rsync
    sync-uri: rsync://rsync.gentoo.org/gentoo-portage
    priority: -1000
    sync-rsync-verify-jobs: 1
    sync-rsync-verify-metamanifest: yes
    sync-rsync-extra-opts: 
    sync-rsync-verify-max-age: 24

Installed sets: @c-space
ACCEPT_KEYWORDS="amd64"
ACCEPT_LICENSE="* -@EULA @EULA"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-march=native -O2 -pipe"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/.hgignore /etc /usr/share/gnupg/qualified.txt /var/bind /var/c-space.net/backup/.ssh"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php5.6/ext-active/ /etc/php/apache2-php7.1/ext-active/ /etc/php/cgi-php5.6/ext-active/ /etc/php/cgi-php7.1/ext-active/ /etc/php/cli-php5.6/ext-active/ /etc/php/cli-php7.1/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-march=native -O2 -pipe"
DISTDIR="/usr/portage/distfiles"
EMERGE_DEFAULT_OPTS="--quiet-build --keep-going -j4 --load-average 8"
ENV_UNSET="DBUS_SESSION_BUS_ADDRESS DISPLAY PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR"
FCFLAGS="-march=native -O2 -pipe"
FEATURES="assume-digests binpkg-logs collision-protect compress-build-logs config-protect-if-modified distlocks ebuild-locks fixlafiles ipc-sandbox merge-sync multilib-strict network-sandbox news parallel-fetch preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"
FFLAGS="-march=native -O2 -pipe"
GENTOO_MIRRORS="http://mirror.switch.ch/mirror/gentoo/ http://ftp-stud.hs-esslingen.de/pub/Mirrors/gentoo/ http://distfiles.gentoo.org/"
LANG="en_US.utf8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
MAKEOPTS="-j9 -l12"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"
PORTAGE_TMPDIR="/var/tmp"
USE="a52 aac acl acpi alsa amd64 apache2 audiofile avahi bash-completion bazaar berkdb bluetooth btrfs bzip2 caps cdb cdda cddb cdparanoia cdr cli consolekit crypt cryptsetup css cups curl cvs cxx dbus device-mapper dri dv dvb dvd dvdr dvdread efi encode exif expat fam ffmpeg firmware flac ftp fuse gd gdbm gif git gmp gpg http2 iconv icu id3 id3tag idn imagemagick imap imlib ipv6 java java5 java6 jbig jpeg jpeg2k lame lcms libtirpc logrotate lua lz4 lzma lzo mad maildir mbox mercurial mhash mikmod mmap mmx mmx2 mng modules mp3 mp4 mpeg mtp multilib ncurses netlink nfs nls nntp nptl nptlonly ntfs ogg openmp openssl opus oss pam pcre pcsc-lite pdf perl php pkcs11 png policykit ppds pthreads python quicktime readline rle rtc samba sasl seccomp slang smartcard smp snmp speex spell sqlite sqlite3 sse sse2 sse3 ssh ssl ssse3 subversion tcl tcpd theora threads tiff truetype udev unicode usb v4l v4l2 vcd vim-syntax vorbis vpx webp wifi wmf x264 x265 xattr xml xmp xvid zeroconf zip zlib" ABI_X86="64" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="access_compat actions alias auth_basic auth_digest authn_alias authn_anon authn_core authn_dbd authn_dbm authn_default authn_file authz_core authz_dbd authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cache_disk cgi dav dav_fs dav_lock dbd deflate dir env expires ext_filter file_cache filter headers ident imagemap include info log_config logio macro mem_cache mime mime_magic negotiation proxy proxy_ajp proxy_balancer proxy_connect proxy_http rewrite setenvif slotmem_shm socache_shmcb speling status unique_id unixd userdir usertrack vhost_alias" APACHE2_MPMS="prefork" CALLIGRA_FEATURES="karbon plan sheets stage words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="aes avx avx2 fma3 mmx mmxext mpx popcnt sse sse2 sse3 sse4_1 sse4_2 ssse3" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" GRUB_PLATFORMS="efi-64 pc" INPUT_DEVICES="evdev synaptics" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LLVM_TARGETS="AMDGPU BPF NVPTX ARM" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-6 php7-1" POSTGRES_TARGETS="postgres9_5 postgres10" PYTHON_SINGLE_TARGET="python2_7" PYTHON_TARGETS="python2_7 python3_5 python3_6" RUBY_TARGETS="ruby23" USERLAND="GNU" VIDEO_CARDS="fbdev i965 intel nouveau nvidia vesa" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CC, CPPFLAGS, CTARGET, CXX, INSTALL_MASK, LC_ALL, LINGUAS, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS
Comment 2 Georgy Yakovlev archtester gentoo-dev 2018-07-15 22:09:12 UTC
huge thanks for figuring this out!
pushing a fix for 1.12.1 and 1.13.0 soon.
Comment 3 Larry the Git Cow gentoo-dev 2018-07-15 22:20:57 UTC
The bug has been closed via the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=5d7099dca23483891bc22124c0dda25dcf7706b6

commit 5d7099dca23483891bc22124c0dda25dcf7706b6
Author:     Georgy Yakovlev <gyakovlev@gentoo.org>
AuthorDate: 2018-07-15 22:07:37 +0000
Commit:     Georgy Yakovlev <gyakovlev@gentoo.org>
CommitDate: 2018-07-15 22:13:34 +0000

    net-libs/grpc: drop old
    
    Closes: https://bugs.gentoo.org/661244
    Package-Manager: Portage-2.3.42, Repoman-2.3.9

 net-libs/grpc/grpc-1.12.1.ebuild | 160 ------------------------------------
 net-libs/grpc/grpc-1.13.0.ebuild | 171 ---------------------------------------
 2 files changed, 331 deletions(-)

Additionally, it has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=614b5aaad4f6d6c22ed92b1f99dc83d012f103ae

commit 614b5aaad4f6d6c22ed92b1f99dc83d012f103ae
Author:     Georgy Yakovlev <gyakovlev@gentoo.org>
AuthorDate: 2018-07-15 22:06:22 +0000
Commit:     Georgy Yakovlev <gyakovlev@gentoo.org>
CommitDate: 2018-07-15 22:13:24 +0000

    net-libs/grpc: revbmp 1.12.1, fix huge SOURCES.txt
    
    Bug: https://bugs.gentoo.org/661244
    Reported-by: Remy Blank <remy@c-space.org>
    Package-Manager: Portage-2.3.42, Repoman-2.3.9

 net-libs/grpc/grpc-1.12.1-r1.ebuild | 162 ++++++++++++++++++++++++++++++++++++
 1 file changed, 162 insertions(+)

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=c3028549bd420c751fed6e530e9f740445a9f7ba

commit c3028549bd420c751fed6e530e9f740445a9f7ba
Author:     Georgy Yakovlev <gyakovlev@gentoo.org>
AuthorDate: 2018-07-15 22:03:36 +0000
Commit:     Georgy Yakovlev <gyakovlev@gentoo.org>
CommitDate: 2018-07-15 22:11:59 +0000

    net-libs/grpc: revbmp 1.13.0, fix huge SOURCES.txt
    
    Bug: https://bugs.gentoo.org/661244
    Reported-by: Remy Blank <remy@c-space.org>
    Package-Manager: Portage-2.3.42, Repoman-2.3.9

 net-libs/grpc/grpc-1.13.0-r1.ebuild | 173 ++++++++++++++++++++++++++++++++++++
 1 file changed, 173 insertions(+)