emerge freezes using PORTAGE_BINHOST="ftp://...". Related pure-ftpd log:
Dec 30 12:05:48 <host> pure-ftpd: (<user>@<client>) [DEBUG] 150 145.9 kbytes to download
Dec 30 12:05:48 <host> pure-ftpd: (<user>@<client>) [NOTICE] /usr/portage//packages/Packages downloaded (0 bytes, 0.00KB/sec)
Steps to Reproduce:
1. run emerge -gupvDN world
2. run emerge -gupvDN world
emerge freezes w. no output when running two times in a row.
# rm /var/cache/edb/binhost/<user>\:<passwd>\@<host>/packages/Packages
In order to create a backtrace, open a separate shell from the frozen emerge and run `kill -s SIGUSR1 <pid>` for the emerge pid. That will make the frozen emerge start the debugger (Pdb). At the (Pdb) prompt, type the "bt" command and press Enter. Please copy the backtrace and post it here.
Created attachment 258487 [details]
Created attachment 258490 [details, diff]
add timeout when closing binhost connection
Save as /tmp/binhost_timeout.patch and apply as follows:
patch /usr/lib/portage/pym/portage/dbapi/bintree.py /tmp/binhost_timeout.patch
Timeout works. Download still fails unless the local copy of Packages is deleted. I'll post ftp logs..
Created attachment 258492 [details]
download failed session log
Created attachment 258493 [details]
successful download session log
After the timeout, does emerge output some more error info?
It seems like the ftp server and/or python's ftplib are behaving incorrectly. Maybe try another ftp server program? It seems the person in the other forums thread was having the same problem with pure-ftp.
(In reply to comment #7)
> After the timeout, does emerge output some more error info?
Nope. Just '!!! Timed out while closing connection to binhost'
I'll try an another software. Might take some time. Meanwhile I tested with a simple python script. Worked fine.
Created attachment 258496 [details]
A simple python script for ftp testing
Did ran with python2.7:
$ python2.7 ftptest.py
Created attachment 258497 [details]
three repeated successes with ftptest.py
(In reply to comment #4)
> Timeout works. Download still fails unless the local copy of Packages is
> deleted. I'll post ftp logs..
It's supposed to use the cached copy of Packages. Does it contain the correct content or not?
(In reply to comment #11)
> It's supposed to use the cached copy of Packages. Does it contain the correct
> content or not?
Yes, diff returns empty when compared to server's copy. Emerge updates cached copy correctly (not freezing) when server side gets updated.
I've tested locally with pure-ftpd-1.0.29-r2 and an anonymous ftp account, and I didn't have any freeze. Please post full details about your server configuration.
Created attachment 258504 [details]
(In reply to comment #7)
> Maybe try another ftp server program?
I tried with an another server and net-ftp/oftpd. Works fine.
Pure-fptd seems to work with small files but freezes on files >100k when using urllib or urllib2. Ftplib works ok.
I'd say that your patch fixes the problem fine.
(In reply to comment #15)
> I tried with an another server and net-ftp/oftpd. Works fine.
> Pure-fptd seems to work with small files but freezes on files >100k when using
> urllib or urllib2. Ftplib works ok.
My guess is that the interaction between urllib and pure-fptd is triggering some sort of deadlock in the communication protocol. It could be either one, both, or the protocol spec that's at fault.
> I'd say that your patch fixes the problem fine.
Great, then I'll mark this bug as fixed. The issue between urllib and pure-fptd is really beyond the scope of what I want to handle on the portage side.
*** Bug 353189 has been marked as a duplicate of this bug. ***
The ftp hang is supposed to be fixed in Python 3.3:
That fix is not included in Python 2.7.3 or 3.2.3, since their NEWS files do not mention it.
I am seeing this issue re-occur when using python-2.7.5, while python-2.7.3 and python-3.2.5 work fine. Can anyone verify? Or is it just me.