ncftp /pentium-m > ls -l Packages -rw-r--r-- 1 0 0 1439665 Jan 30 02:30 Packages pentium-m-test ~ # ls -l /var/cache/edb/binhost/wald.local/pentium-m/Packages -rw-r--r-- 1 root root 1394660 Jan 29 15:57 /var/cache/edb/binhost/wald.local/pentium-m/Packages pentium-m-test ~ # ls -l /usr/portage/packages/Packages -rw-r--r-- 1 root root 1467657 Jan 29 23:00 /usr/portage/packages/Packages what's the problem? Reproducible: Always Steps to Reproduce:
It's supposed to use the existing file if it has the same TIMESTAMP entry near the top of the file. Is it the same?
Nope. Timestamp on server: TIMESTAMP: 1296354620 vs. local timestamp: TIMESTAMP: 1296338446 Downgrading to stable portage also doesn't help.
Try this: wget $(portageq envvar PORTAGE_BINHOST)/Packages Does it download?
# wget $(portageq envvar PORTAGE_BINHOST)/Packages --2011-01-30 04:01:41-- ftp://wald.local/pentium-m/Packages => `Packages' Resolving wald.local... 169.254.243.195 Connecting to wald.local|169.254.243.195|:21... connected. Logging in as anonymous ... Logged in! ==> SYST ... done. ==> PWD ... done. ==> TYPE I ... done. ==> CWD (1) /pentium-m ... done. ==> SIZE Packages ... 1439665 ==> PASV ... done. ==> RETR Packages ... done. Length: 1439665 (1.4M) (unauthoritative) 100%[=====================================================>] 1,439,665 --.-K/s in 0.07s 2011-01-30 04:01:41 (20.2 MB/s) - `Packages' saved [1439665] pentium-m-test ~ # ll Packages -rw-r--r-- 1 root root 1439665 Jan 30 04:01 Packages pentium-m-test ~ # grep ^TIMESTAMP Packages TIMESTAMP: 1296354620
When you try to use emerge -g/-G, does it show any error message(s)?
This seems similar to bug 350139.
No errors: # emerge -guDavN world These are the packages that would be merged, in order: Calculating dependencies... done! [binary U ] sys-apps/portage-2.2.0_alpha19 [2.1.9.25] USE="(ipc) -build -doc -epydoc -python3 (-selinux)" LINGUAS="-pl" [binary N ] x11-proto/bigreqsproto-1.1.1 USE="-doc" [binary N ] x11-proto/xcmiscproto-1.2.1 USE="-doc" [binary U ] x11-apps/xrdb-1.0.7 [1.0.6] USE="(-debug%)" [binary N ] x11-apps/xinit-1.3.0-r1 USE="minimal" Total: 5 packages (2 upgrades, 3 new, 5 binaries), Size of downloads: 0 kB Would you like to merge these packages? [Yes/No] n
Try this: rm -rf /var/cache/edb/binhost Does it download now?
right, vsftpd says: Sun Jan 30 04:15:48 2011 [pid 13558] CONNECT: Client "169.254.64.12" Sun Jan 30 04:15:48 2011 [pid 13558] FTP response: Client "169.254.64.12", "220 (vsFTPd 2.3.2)" Sun Jan 30 04:15:48 2011 [pid 13558] FTP command: Client "169.254.64.12", "USER anonymous" Sun Jan 30 04:15:48 2011 [pid 13558] [anonymous] FTP response: Client "169.254.64.12", "331 Please specify the password." Sun Jan 30 04:15:48 2011 [pid 13558] [anonymous] FTP command: Client "169.254.64.12", "PASS <password>" Sun Jan 30 04:15:48 2011 [pid 13557] [ftp] OK LOGIN: Client "169.254.64.12", anon password "anonymous@" Sun Jan 30 04:15:48 2011 [pid 13559] [ftp] FTP response: Client "169.254.64.12", "230 Login successful." Sun Jan 30 04:15:48 2011 [pid 13559] [ftp] FTP command: Client "169.254.64.12", "CWD pentium-m" Sun Jan 30 04:15:48 2011 [pid 13559] [ftp] FTP response: Client "169.254.64.12", "250 Directory successfully changed." Sun Jan 30 04:15:48 2011 [pid 13559] [ftp] FTP command: Client "169.254.64.12", "TYPE I" Sun Jan 30 04:15:48 2011 [pid 13559] [ftp] FTP response: Client "169.254.64.12", "200 Switching to Binary mode." Sun Jan 30 04:15:48 2011 [pid 13559] [ftp] FTP command: Client "169.254.64.12", "PASV" Sun Jan 30 04:15:48 2011 [pid 13559] [ftp] FTP response: Client "169.254.64.12", "227 Entering Passive Mode (169,254,243,195,21,101)." Sun Jan 30 04:15:48 2011 [pid 13559] [ftp] FTP command: Client "169.254.64.12", "RETR Packages" Sun Jan 30 04:15:48 2011 [pid 13559] [ftp] FTP response: Client "169.254.64.12", "150 Opening BINARY mode data connection for Packages (1439665 bytes)." Sun Jan 30 04:15:48 2011 [pid 13559] [ftp] FTP response: Client "169.254.64.12", "426 Failure writing network stream." Sun Jan 30 04:15:48 2011 [pid 13559] [ftp] FAIL DOWNLOAD: Client "169.254.64.12", "/pentium-m/Packages", 83984 bytes, 4582.65Kbyte/sec
I guess it's something like bug 350139. Can you try another ftp server program. The person in bug 350139 said that net-ftp/oftpd worked fine.
yup. Now: Sun Jan 30 04:18:18 2011 [pid 14375] CONNECT: Client "169.254.64.12" Sun Jan 30 04:18:18 2011 [pid 14375] FTP response: Client "169.254.64.12", "220 (vsFTPd 2.3.2)" Sun Jan 30 04:18:18 2011 [pid 14375] FTP command: Client "169.254.64.12", "USER anonymous" Sun Jan 30 04:18:18 2011 [pid 14375] [anonymous] FTP response: Client "169.254.64.12", "331 Please specify the password." Sun Jan 30 04:18:18 2011 [pid 14375] [anonymous] FTP command: Client "169.254.64.12", "PASS <password>" Sun Jan 30 04:18:18 2011 [pid 14374] [ftp] OK LOGIN: Client "169.254.64.12", anon password "anonymous@" Sun Jan 30 04:18:18 2011 [pid 14376] [ftp] FTP response: Client "169.254.64.12", "230 Login successful." Sun Jan 30 04:18:18 2011 [pid 14376] [ftp] FTP command: Client "169.254.64.12", "CWD pentium-m" Sun Jan 30 04:18:18 2011 [pid 14376] [ftp] FTP response: Client "169.254.64.12", "250 Directory successfully changed." Sun Jan 30 04:18:18 2011 [pid 14376] [ftp] FTP command: Client "169.254.64.12", "TYPE I" Sun Jan 30 04:18:18 2011 [pid 14376] [ftp] FTP response: Client "169.254.64.12", "200 Switching to Binary mode." Sun Jan 30 04:18:18 2011 [pid 14376] [ftp] FTP command: Client "169.254.64.12", "PASV" Sun Jan 30 04:18:18 2011 [pid 14376] [ftp] FTP response: Client "169.254.64.12", "227 Entering Passive Mode (169,254,243,195,26,228)." Sun Jan 30 04:18:18 2011 [pid 14376] [ftp] FTP command: Client "169.254.64.12", "RETR Packages" Sun Jan 30 04:18:18 2011 [pid 14376] [ftp] FTP response: Client "169.254.64.12", "150 Opening BINARY mode data connection for Packages (1439665 bytes)." Sun Jan 30 04:18:20 2011 [pid 14376] [ftp] OK DOWNLOAD: Client "169.254.64.12", "/pentium-m/Packages", 1439665 bytes, 1040.68Kbyte/sec Sun Jan 30 04:18:20 2011 [pid 14376] [ftp] FTP response: Client "169.254.64.12", "226 Transfer complete." CONNECT: Client "169.254.64.12"
Ugh. I like vsftpd. What is it doing wrong?
Like I said in bug #350139, comment #16, it's some kind of interaction between python's urllib and your ftp server.
Bleh. I hate this kinda stuff. Now do I want to investigate this? Argh. Stupid ftp. Any other option besides ftp or http?
But the thing is, why did it work after deleting /var/cache/edb/binhost? That shouldn't make any difference to the ftp transfer, should it?
ssh and sftp protocols also work. I don't know why it behaves this way with some ftp servers. When I tried to reproduce it with pure-ftpd, I couldn't reproduce it (bug #350139, comment #13).
Ok, this stuff gives me nightmares, but I guess there's nothing that can be done right now...
I'm going to mark this as a duplicate of bug 350139 since the symptoms are almost identical.
*** This bug has been marked as a duplicate of bug 350139 ***
Do I read the code right in that you do ssh/sftp yourself, instead of using crappy urllib? BTW, wouldn't it be nice to throw out a warning if getting the TIMESTAMP header failed? The comment "no timestamp in the header, something's wrong" in bintree.py kind of suggests it.
(In reply to comment #20) > Do I read the code right in that you do ssh/sftp yourself, instead of using > crappy urllib? I suppose we could spawn FETCHCOMMAND in order to handle the ftp protocol, or at least use it as a fallback if urllib fails somehow. > BTW, wouldn't it be nice to throw out a warning if getting the TIMESTAMP header > failed? The comment "no timestamp in the header, something's wrong" in > bintree.py kind of suggests it. I guess that's the one that's being triggered here. Now I've added a warning in git: http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=a607079b3b556de243e955cda7ae9a3defd26750
(In reply to comment #21) > (In reply to comment #20) > > Do I read the code right in that you do ssh/sftp yourself, instead of using > > crappy urllib? > > I suppose we could spawn FETCHCOMMAND in order to handle the ftp protocol, or > at least use it as a fallback if urllib fails somehow. Actually, ftplib may be worth a try, since it was reported to work in bug 350139.
I'm planning to merge this patch, for using FETCHCOMMAND instead of urllib: http://codereview.chromium.org/6458015/