Bug 338190 – sys-apps/portage-2.1.8.3 dies with stacktrace on binhost FTP errors

• Product: Portage Development
• Status: RESOLVED FIXED

[Description Christian Kauhaus 2010-09-20 13:06:23 UTC]

When emerging binary packages from PORTAGE_BINHOST, FTP protocol errors are not being catched. This leads to a ugly stack trace and emerge fails immediately instead of proceeding with the next package.

The problem is the "except" line in /usr/lib64/portage/pym/portage/getbinpkg.py +504. socket.error exceptions are catched, but protocol errors are not catched. 
I think that ftplib.Error should be catched as well. All exceptions which are raised from ftplib can be found in ftplib.all_errors.

Reproducible: Always

Steps to Reproduce:
export PORTAGE_BINHOST="ftp://anonymous:portage@services03/pub/packages/d9b4d6a87b064ce0" # binary server in our network
emerge eix # any ebuild for which a binpkg exists
Actual Results:  
# With a defective FTP server (e.g., chmod -x pure-ftpd), I get the following stack trace:

!!! Error fetching binhost package info from 'ftp://anonymous:portage@services03/pub/packages/d9b4d6a87b064ce0'
!!! [Errno ftp error]


Fetching bininfo from ftp://anonymous:*password*@services03/pub/packages/d9b4d6a87b064ce0
Traceback (most recent call last):
  File "/usr/bin/emerge", line 42, in <module>
    retval = emerge_main()
  File "/usr/lib/portage/pym/_emerge/main.py", line 1119, in emerge_main
    getbinpkgs="--getbinpkg" in myopts)
  File "/usr/lib/portage/pym/portage/dbapi/bintree.py", line 482, in populate
    self._populate(getbinpkgs)
  File "/usr/lib/portage/pym/portage/dbapi/bintree.py", line 835, in _populate
    self.settings["PORTAGE_BINHOST"], chunk_size=chunk_size)
  File "/usr/lib/portage/pym/portage/getbinpkg.py", line 503, in dir_get_metadata 
    conn, protocol, address, params, headers = create_conn(baseurl, conn)
  File "/usr/lib/portage/pym/portage/getbinpkg.py", line 163, in create_conn
    conn = ftplib.FTP(host)
  File "/usr/lib/python2.6/ftplib.py", line 116, in __init__
    self.connect(host)
  File "/usr/lib/python2.6/ftplib.py", line 134, in connect
    self.welcome = self.getresp()
  File "/usr/lib/python2.6/ftplib.py", line 209, in getresp
    resp = self.getmultiline()
  File "/usr/lib/python2.6/ftplib.py", line 195, in getmultiline
    line = self.getline()
  File "/usr/lib/python2.6/ftplib.py", line 185, in getline
    if not line: raise EOFError
EOFError   


Expected Results:  
# With a working FTP server, I get the normal binpkg installation output:

Calculating dependencies... done!

>>> Emerging binary (1 of 1) app-portage/eix-0.20.5
--2010-09-20 14:44:04--  ftp://anonymous:*password*@services03/pub/packages/d9b4d6a87b064ce0/app-portage/eix-0.20.5.tbz2
           => “/tmp/portage/packages/d9b4d6a87b064ce0/app-portage/eix-0.20.5.tbz2”
Resolving services03... 172.20.3.4
Connecting to services03|172.20.3.4|:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done.    ==> PWD ... done.
==> TYPE I ... done.  ==> CWD (1) /pub/packages/d9b4d6a87b064ce0/app-portage ... done.
==> SIZE eix-0.20.5.tbz2 ... 528553
==> PASV ... done.    ==> RETR eix-0.20.5.tbz2 ... done.

     0K .......... .......... .......... .......... ..........  796K
    50K .......... .......... .......... .......... .......... 76.2M
   100K .......... .......... .......... .......... .......... 91.9M
   150K .......... .......... .......... .......... ..........  110M
   200K .......... .......... .......... .......... ..........  101M
   250K .......... .......... .......... .......... .......... 1.82M
   300K .......... .......... .......... .......... .......... 88.7M
   350K .......... .......... .......... .......... .......... 49.8M
   400K .......... .......... .......... .......... .......... 41.0M
   450K .......... .......... .......... .......... .......... 25.7M
   500K .......... ......                                      18.9M=0.1s

2010-09-20 14:44:04 (5.18 MB/s) - “/tmp/portage/packages/d9b4d6a87b064ce0/app-portage/eix-0.20.5.tbz2” saved [528553]

 * eix-0.20.5.tbz2 MD5 SHA1 size ;-) ...                                 [ ok ]
>>> Extracting info
 * CPV:  app-portage/eix-0.20.5
 * REPO: gentoo
 * USE:  bzip2 elibc_glibc kernel_linux nls userland_GNU x86
>>> Extracting app-portage/eix-0.20.5

>>> Installing (1 of 1) app-portage/eix-0.20.5
>>> Auto-cleaning packages...

>>> No outdated packages were found on your system.

 * GNU info directory index is up-to-date.

Portage 2.1.8.3 (default/linux/x86/10.0, gcc-4.4.3, glibc-2.11.2-r0, 2.6.34-gentoo-r1 i686)
=================================================================
System uname: Linux-2.6.34-gentoo-r1-i686-QEMU_Virtual_CPU_version_0.12.5-with-gentoo-1.12.13
Timestamp of tree: Fri, 13 Aug 2010 07:00:01 +0000
ccache version 2.4 [enabled]
app-shells/bash:     4.0_p37
dev-java/java-config: 2.1.11
dev-lang/python:     2.4.6, 2.5.4-r4, 2.6.4-r1, 3.1.2-r4
dev-util/ccache:     2.4-r7
dev-util/cmake:      2.8.1-r2
sys-apps/baselayout: 1.12.13
sys-apps/sandbox:    1.6-r2
sys-devel/autoconf:  2.65
sys-devel/automake:  1.11.1
sys-devel/binutils:  2.20.1-r1
sys-devel/gcc:       4.4.3-r2
sys-devel/gcc-config: 1.4.1
sys-devel/libtool:   2.2.6b
virtual/os-headers:  2.6.30-r1
ACCEPT_KEYWORDS="x86"
ACCEPT_LICENSE="* -@EULA dlj-1.1"
CBUILD="i686-pc-linux-gnu"
CFLAGS="-O2 -pipe -march=i686"
CHOST="i686-pc-linux-gnu"
CONFIG_PROTECT="/etc"
services03/pub/packages/d9b4d6aCONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/conf.d/apache2 /etc/eclean /etc/env.d /etc/env.d/java/ /etc/eselect/postgresql /etc/fonts /etc/fonts/fonts.conf /etc/gconf /etc/init.d /etc/openldap/schema /etc/portage/savedconfig /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-O2 -pipe -march=i686"
DISTDIR="/tmp/portage/distfiles"
EMERGE_DEFAULT_OPTS="--binpkg-respect-use y --with-bdeps y --jobs=1 --load-average=1 --getbinpkg"
FEATURES="assume-digests buildpkg ccache distlocks fixpackages news parallel-fetch protect-owned sandbox sfperms strict unmerge-logs unmerge-orphans userfetch usersandbox"
GENTOO_MIRRORS="http://download.gocept.com/gentoo/mirror/ ftp://de-mirror.org/distro/gentoo ftp://ftp.wh2.tu-dresden.de/pub/mirrors/gentoo http://mirrors.sec.informatik.tu-darmstadt.de/gentoo/ "
LANG="en_US.UTF-8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
LINGUAS="de en"
MAKEOPTS="-j1 -l1"
PKGDIR="/tmp/portage/packages/d9b4d6a87b064ce0"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"
PORTAGE_TMPDIR="/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY="/usr/local/portage"
SYNC="rsync://rsync.europe.gentoo.org/gentoo-portage"
USE="acl acpi apache2 bash-completion berkdb bzip2 cli cracklib crypt ctype curl cxx device-mapper dri filter fortran gd gdbm geoip gnutls iconv ipv6 jpeg json kvm ldap logrotate mercurial mhash mudflap nagios-dns nagios-ntp nagios-ping nagios-ssh ncurses nls nptl nptlonly openmp pam pcre perl png pppd python readline reflection sasl session simplexml snmp spl sqlite3 sse sse2 ssl symlink sysfs syslog tcpd truetype unicode utils vhosts vim-syntax x86 xinetd xml xorg zlib" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1 emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias proxy proxy_balancer proxy_connect proxy_ftp proxy_http" ELIBC="glibc" INPUT_DEVICES="keyboard mouse evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LINGUAS="de en" RUBY_TARGETS="ruby18" USERLAND="GNU" VIDEO_CARDS="fbdev glint intel mach64 mga neomagic nv r128 radeon savage sis tdfx trident vesa via vmware voodoo" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CPPFLAGS, CTARGET, FFLAGS, INSTALL_MASK, LC_ALL, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS

[Comment 1 Zac Medico 2010-09-21 09:24:56 UTC]

The exception indicates that you're triggering the old protocol, which happens whenever $PORTAGE_BINHOST/Packages can't be fetched. Make sure that you have $PORTAGE_BINHOST set correctly so that the Packages file can be fetched. You can run `emaint --fix binhost` on the server in order to generate or update the Packages file. This file is updated automatically with new packages when they are built, but you need to call `emaint --fix binhost` if it is out of sync with the packages in $PKGDIR for any reason. For example, eclean-pkg calls `emaint --fix binhost` after it removes old binary packages.

[Comment 2 Christian Kauhaus 2010-09-21 10:07:43 UTC]

(In reply to comment #1)
> The exception indicates that you're triggering the old protocol.

I'm trying to clarify. The bug is triggered when a defective FTP server gives *no* sensible answers at all. I think that emerge should not die in such cases.
Instead, it should either try to compile the package or proceed with the next one.

[Comment 3 Zac Medico 2010-09-21 15:45:09 UTC]

Thanks, there's a fix in git now:

http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=bf5877484cbca0b59c1b9d3688d338465083bcf9

[Comment 4 Zac Medico 2010-09-22 03:45:09 UTC]

This is fixed in 2.1.9.10 and 2.2_rc86.