Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!

Bug 375331

Summary: sys-apps/portage-2.2.0_alpha45: With FEATURES="parallel-fetch", Portage should not download more than one package at a time
Product: Portage Development Reporter: dwfreed <dwfreed>
Component: UnclassifiedAssignee: Portage team <dev-portage>
Status: RESOLVED FIXED    
Severity: enhancement Keywords: InVCS
Priority: Normal    
Version: 2.1   
Hardware: x86   
OS: All   
URL: http://i.imgur.com/OOTmV.png
Whiteboard:
Package list:
Runtime testing required: ---
Bug Depends on:    
Bug Blocks: 377365, 373933    
Attachments: use fetch queue to cap fetch threads
enable prefetch for first package

Description dwfreed 2011-07-16 05:25:43 UTC
With parallel fetch enabled, currently Portage will download one package in the foreground and one in the background at the same time if merging more than one package at a time.  This is especially harmful for dialup users whose connections are very slow anyway.  The screenshot in the URL demonstrates the current behavior.

Reproducible: Always

Steps to Reproduce:
1. Merge more than one package in the same operation.
2. Look at /var/log/emerge-fetch.log while Portage is downloading the first package.
Actual Results:  
While Portage is downloading the first package in the foreground, it is downloading the second package in the background, storing the output from the fetch command in /var/log/emerge-fetch.log

Expected Results:  
Portage should download the first package in the foreground and once finished, start downloading the second and subsequent packages in the background.

Portage 2.2.0_alpha45 (default/linux/x86/10.0, gcc-4.4.5, glibc-2.12.2-r0, 2.6.38-gentoo-r6 i686)
=================================================================
System uname: Linux-2.6.38-gentoo-r6-i686-Intel-R-_Pentium-R-_4_CPU_2.80GHz-with-gentoo-2.0.2
Timestamp of tree: Fri, 15 Jul 2011 01:15:01 +0000
app-shells/bash:          4.1_p9
dev-java/java-config:     2.1.11-r3
dev-lang/python:          2.7.1-r1, 3.1.3-r1
dev-util/cmake:           2.8.4-r1
dev-util/pkgconfig:       0.25-r2
sys-apps/baselayout:      2.0.2
sys-apps/openrc:          0.8.3-r1
sys-apps/sandbox:         2.4
sys-devel/autoconf:       2.13, 2.68
sys-devel/automake:       1.11.1
sys-devel/binutils:       2.20.1-r1
sys-devel/gcc:            4.4.5
sys-devel/gcc-config:     1.4.1-r1
sys-devel/libtool:        2.2.10
sys-devel/make:           3.82
sys-kernel/linux-headers: 2.6.36.1 (virtual/os-headers)
sys-libs/glibc:           2.12.2
Repositories:

gentoo
    location: /usr/portage
    sync: rsync://lug.mtu.edu/gentoo-portage
    priority: -1000

belak
    location: /var/lib/layman/belak
    masters: gentoo
    priority: 0

Installed sets: 
ACCEPT_KEYWORDS="x86"
ACCEPT_LICENSE="* -@EULA"
ACCEPT_PROPERTIES="*"
ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1 emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci"
ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol"
APACHE2_MODULES="actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias"
ARCH="x86"
AUTOCLEAN="yes"
CALLIGRA_FEATURES="braindump flow karbon kexi kpresenter krita tables words"
CAMERAS="ptp2"
CBUILD="i686-pc-linux-gnu"
CFLAGS="-O2 -march=native -pipe"
CHOST="i686-pc-linux-gnu"
CLEAN_DELAY="5"
COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog"
COLLISION_IGNORE="/lib/modules"
COLORFGBG="15;0"
COLORTERM="rxvt"
CONFIG_PROTECT="/etc /var/bind"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/env.d/java/ /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CVS_RSH="ssh"
CXXFLAGS="-O2 -march=native -pipe"
DISPLAY=":0"
DISTDIR="/usr/portage/distfiles"
EDITOR="/usr/bin/vim"
ELIBC="glibc"
EMERGE_DEFAULT_OPTS="-av"
EMERGE_WARNING_DELAY="10"
EPREFIX=""
EROOT="/"
FCFLAGS=""
FEATURES="assume-digests binpkg-logs candy distlocks ebuild-locks fixlafiles fixpackages news parallel-fetch preserve-libs protect-owned sandbox sfperms splitdebug strict unknown-features-warn unmerge-logs unmerge-orphans userfetch"
FETCHCOMMAND="/usr/bin/wget -t 3 -T 60 --passive-ftp -O "${DISTDIR}/${FILE}" "${URI}""
FETCHCOMMAND_RSYNC="rsync -avP "${URI}" "${DISTDIR}/${FILE}""
FETCHCOMMAND_SFTP="bash -c "x=\${2#sftp://} ; host=\${x%%/*} ; port=\${host##*:} ; host=\${host%:*} ; [[ \${host} = \${port} ]] && port=22 ; exec sftp -P \${port} \"\${host}:/\${x#*/}\" \"\$1\"" sftp "${DISTDIR}/${FILE}" "${URI}""
FETCHCOMMAND_SSH="bash -c "x=\${2#ssh://} ; host=\${x%%/*} ; port=\${host##*:} ; host=\${host%:*} ; [[ \${host} = \${port} ]] && port=22 ; exec rsync --rsh=\"ssh -p\${port}\" -avP \"\${host}:/\${x#*/}\" \"\$1\"" rsync "${DISTDIR}/${FILE}" "${URI}""
FFLAGS=""
GCC_SPECS=""
GDK_USE_XFT="1"
GENTOO_MIRRORS="http://141.219.155.230/gentoo/ http://141.219.155.230/gentoo/ http://141.219.155.230/gentoo/ http://141.219.155.230/gentoo/ http://141.219.155.230/gentoo/"
GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ubx"
GSETTINGS_BACKEND="gconf"
HG="/usr/bin/hg"
HISTCONTROL="erasedups"
HISTFILE="/home/dwfreed/.bash_history_pts_14"
HISTFILESIZE="1000000"
HISTSIZE="1000000"
HISTTIMEFORMAT="%Y:%m:%d %H:%M:%S "
HOME="/home/dwfreed"
HUSHLOGIN="FALSE"
INFOPATH="/usr/share/info:/usr/share/binutils-data/i686-pc-linux-gnu/2.20.1/info:/usr/share/gcc-data/i686-pc-linux-gnu/4.4.5/info"
INPUT_DEVICES="evdev void"
JAVAC="/etc/java-config-2/current-system-vm/bin/javac"
JAVA_HOME="/etc/java-config-2/current-system-vm"
JDK_HOME="/etc/java-config-2/current-system-vm"
KERNEL="linux"
LANG="en_US.UTF-8"
LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
LESS="-R -M --shift 5"
LESSOPEN="|lesspipe %s"
LINGUAS="en"
LOGNAME="dwfreed"
LS_COLORS="rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=01;05;37;41:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.pdf=00;32:*.ps=00;32:*.txt=00;32:*.patch=00;32:*.diff=00;32:*.log=00;32:*.tex=00;32:*.doc=00;32:*.aac=00;36:*.au=00;36:*.flac=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.axa=00;36:*.oga=00;36:*.spx=00;36:*.xspf=00;36:"
MAIL="/var/mail/dwfreed"
MAKEOPTS="-j2"
MANPATH="/etc/java-config-2/current-system-vm/man:/usr/local/share/man:/usr/share/man:/usr/share/binutils-data/i686-pc-linux-gnu/2.20.1/man:/usr/share/gcc-data/i686-pc-linux-gnu/4.4.5/man:/etc/java-config/system-vm/man/"
NETBEANS="apisupport cnd groovy gsf harness ide identity j2ee java mobility nb php profiler soa visualweb webcommon websvccommon xml"
NOCOLOR="true"
OPENGL_PROFILE="xorg-x11"
PAGER="/usr/bin/less"
PATH="/home/dwfreed/bin:/usr/local/bin:/usr/bin:/bin:/opt/bin:/usr/i686-pc-linux-gnu/gcc-bin/4.4.5:/usr/games/bin"
PHP_TARGETS="php5-3"
PKGDIR="/usr/portage/packages"
PORTAGE_ARCHLIST="ppc sparc64-freebsd ppc-openbsd x86-openbsd ppc64 x86-winnt x86-fbsd ppc-aix alpha arm x86-freebsd s390 amd64 arm-linux x86-macos x64-openbsd ia64-hpux hppa x86-netbsd x86-cygwin amd64-linux ia64-linux x86 sparc-solaris x64-freebsd sparc64-solaris x86-linux x64-macos sparc m68k-mint ia64 mips ppc-macos x86-interix hppa-hpux amd64-fbsd x64-solaris mips-irix m68k sh x86-solaris sparc-fbsd"
PORTAGE_BINHOST_CHUNKSIZE="3000"
PORTAGE_BIN_PATH="/usr/lib/portage/bin"
PORTAGE_COMPRESS_EXCLUDE_SUFFIXES="css gif htm[l]? jp[e]?g js pdf png"
PORTAGE_CONFIGROOT="/"
PORTAGE_DEBUG="0"
PORTAGE_DEPCACHEDIR="/var/cache/edb/dep"
PORTAGE_ELOG_CLASSES="log warn error"
PORTAGE_ELOG_MAILFROM="portage@localhost"
PORTAGE_ELOG_MAILSUBJECT="[portage] ebuild log for ${PACKAGE} on ${HOST}"
PORTAGE_ELOG_MAILURI="root"
PORTAGE_ELOG_SYSTEM="save_summary echo"
PORTAGE_FETCH_CHECKSUM_TRY_MIRRORS="5"
PORTAGE_FETCH_RESUME_MIN_SIZE="350K"
PORTAGE_GID="250"
PORTAGE_GPG_SIGNING_COMMAND="gpg --sign --clearsign --yes --default-key "${PORTAGE_GPG_KEY}" --homedir "${PORTAGE_GPG_DIR}" "${FILE}""
PORTAGE_INST_GID="0"
PORTAGE_INST_UID="0"
PORTAGE_PYM_PATH="/usr/lib/portage/pym"
PORTAGE_RSYNC_INITIAL_TIMEOUT="0"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --delete-delay --fuzzy --delay-updates --stats --timeout=0 --contimeout=60 --progress --exclude='/distfiles' --exclude='/local' --exclude='/packages'"
PORTAGE_RSYNC_RETRIES="10"
PORTAGE_SANDBOX_COMPAT_LEVEL="22"
PORTAGE_SYNC_STALE="30"
PORTAGE_TMPDIR="/var/tmp"
PORTAGE_VERBOSE="1"
PORTAGE_WORKDIR_MODE="0700"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY="/var/lib/layman/belak"
PRELINK_PATH_MASK="/usr/lib/libfreebl3.so:/usr/lib/libnssdbm3.so:/usr/lib/libsoftokn3.so"
PROFILE_ONLY_VARIABLES="ARCH ELIBC KERNEL USERLAND"
PWD="/home/dwfreed"
PYTHONDONTWRITEBYTECODE="1"
RESUMECOMMAND="/usr/bin/wget -c -t 3 -T 60 --passive-ftp -O "${DISTDIR}/${FILE}" "${URI}""
RESUMECOMMAND_RSYNC="rsync -avP "${URI}" "${DISTDIR}/${FILE}""
RESUMECOMMAND_SSH="bash -c "x=\${2#ssh://} ; host=\${x%%/*} ; port=\${host##*:} ; host=\${host%:*} ; [[ \${host} = \${port} ]] && port=22 ; exec rsync --rsh=\"ssh -p\${port}\" -avP \"\${host}:/\${x#*/}\" \"\$1\"" rsync "${DISTDIR}/${FILE}" "${URI}""
ROOT="/"
ROOTPATH="/opt/bin:/usr/i686-pc-linux-gnu/gcc-bin/4.4.5"
RPMDIR="/usr/portage/rpm"
RUBYOPT="-rauto_gem"
RUBY_TARGETS="ruby19"
SHELL="/bin/bash"
SHLVL="3"
SYNC="rsync://lug.mtu.edu/gentoo-portage"
TERM="rxvt-unicode-256color"
USE="X acl alsa bash-completion berkdb bzip2 cli cracklib crypt cups curl cxx dri drm fortran gallium gdbm gif gpm iconv ipv6 jpeg jpeg2k mmx modules mudflap ncurses nls nptl nptlonly nsplugin offensive opengl openmp pam pcre perl png pppd python qt3support qt4 readline session sqlite sqlite3 sse sse2 ssl sysfs tcpd tiff udev unicode vim-syntax x86 xft xinerama xorg xpm zlib" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1 emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="braindump flow karbon kexi kpresenter krita tables words" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ubx" INPUT_DEVICES="evdev void" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LINGUAS="en" PHP_TARGETS="php5-3" RUBY_TARGETS="ruby19" USERLAND="GNU" VIDEO_CARDS="nouveau intel dummy" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account" 
USER="dwfreed"
USERLAND="GNU"
USE_EXPAND="ALSA_CARDS ALSA_PCM_PLUGINS APACHE2_MODULES APACHE2_MPMS CALLIGRA_FEATURES CAMERAS COLLECTD_PLUGINS CROSSCOMPILE_OPTS DRACUT_MODULES DVB_CARDS ELIBC FCDSL_CARDS FOO2ZJS_DEVICES FRITZCAPI_CARDS GPSD_PROTOCOLS INPUT_DEVICES KERNEL LCD_DEVICES LINGUAS LIRC_DEVICES MISDN_CARDS NETBEANS_MODULES NGINX_MODULES_HTTP NGINX_MODULES_MAIL OFED_DRIVERS PHP_TARGETS QEMU_SOFTMMU_TARGETS QEMU_USER_TARGETS RUBY_TARGETS SANE_BACKENDS USERLAND VIDEO_CARDS XFCE_PLUGINS XTABLES_ADDONS"
USE_EXPAND_HIDDEN="CROSSCOMPILE_OPTS ELIBC KERNEL USERLAND"
USE_ORDER="env:pkg:conf:defaults:pkginternal:repo:env.d"
VIDEO_CARDS="nouveau intel dummy"
WINDOWID="48234502"
WINDOWPATH="7"
XDG_CONFIG_DIRS="/etc/xdg"
XDG_DATA_DIRS="/usr/local/share:/usr/share"
XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
_="/usr/bin/emerge"
Comment 1 Zac Medico gentoo-dev 2011-07-16 06:19:22 UTC
Created attachment 280161 [details, diff]
use fetch queue to cap fetch threads

Save as /tmp/fetch.patch and apply as follows:

  patch /usr/lib/portage/pym/_emerge/EbuildBuild.py /tmp/fetch.patch
Comment 2 Zac Medico gentoo-dev 2011-07-16 06:31:27 UTC
(In reply to comment #1)
> Created attachment 280161 [details, diff]
> use fetch queue to cap fetch threads

This patch can cause waiting on the fetch queue even when everything is already fetched. We could optimize it to check if the files are already fetched and skip the fetch queue in that case.
Comment 3 dwfreed 2011-07-16 07:07:09 UTC
(In reply to comment #1)
> Created attachment 280161 [details, diff]
> use fetch queue to cap fetch threads

Oddly, no matter how many packages are to be merged, this patch grabs the second package first.  In the mean time, Portage is doing nothing because it hasn't grabbed the first package yet.
Comment 4 Zac Medico gentoo-dev 2011-07-16 07:37:52 UTC
Created attachment 280167 [details, diff]
enable prefetch for first package

Save as /tmp/first_fetch.patch and apply as follows:

  patch /usr/lib/portage/pym/_emerge/Scheduler.py /tmp/first_fetch.patch

This patch applies on top of the previous patch.

(In reply to comment #3)
> Oddly, no matter how many packages are to be merged, this patch grabs the
> second package first.  In the mean time, Portage is doing nothing because it
> hasn't grabbed the first package yet.

This additional patch will make it fetch the first package first. The output will go to /var/log/emerge-fetch.log though, so you'll have to do something like this if you want to monitor the progress:

  tail -f /var/log/emerge-fetch.log

We can tweak it to show the first fetch in the foreground, though the code will be a little more complex that way.
Comment 5 Zac Medico gentoo-dev 2011-07-16 23:33:54 UTC
(In reply to comment #2)
> (In reply to comment #1)
> > Created attachment 280161 [details, diff]
> > use fetch queue to cap fetch threads
> 
> This patch can cause waiting on the fetch queue even when everything is already
> fetched. We could optimize it to check if the files are already fetched and
> skip the fetch queue in that case.

This above issue is fixed in git:

http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=785fe0b42a66b1d137955a4b816b3c254a1b785d
Comment 6 Zac Medico gentoo-dev 2011-07-16 23:52:19 UTC
(In reply to comment #4)
> This additional patch will make it fetch the first package first. The output
> will go to /var/log/emerge-fetch.log though, so you'll have to do something
> like this if you want to monitor the progress:
> 
>   tail -f /var/log/emerge-fetch.log
> 
> We can tweak it to show the first fetch in the foreground, though the code will
> be a little more complex that way.

Since I don't see a clean and easy way to make the first fetch show in the foreground, I'm not going to bother with it. People can always disable FEATURES=parallel-fetch if they really want to see the output and they can't be bothered to run tail -f.
Comment 7 Zac Medico gentoo-dev 2011-07-17 00:07:13 UTC
Here's a patch for fetching binary packages (PORTAGE_BINHOST):

http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=25d00e7bc361a659a729d5165fe580afcbd39664
Comment 8 Zac Medico gentoo-dev 2011-07-17 18:19:21 UTC
For now, I've make --jobs > 1 enable concurrent fetch:

http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=ef58bc7573ddce5e3a5466eea50160b81de8edf4

We can add a --fetch-jobs option later, but if we enable that for parallel-fetch then that's going to require logging changes since parallel-fetch shares emerge-fetch.log.
Comment 9 Zac Medico gentoo-dev 2011-07-19 22:13:52 UTC
This is fixed in 2.1.10.7 and 2.2.0_alpha46.