Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 353189 - portage isn't downloading package index (vsftpd)
Summary: portage isn't downloading package index (vsftpd)
Status: RESOLVED DUPLICATE of bug 350139
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Binary packages support (show other bugs)
Hardware: All Linux
: High major (vote)
Assignee: Portage team
URL: http://codereview.chromium.org/6458015/
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2011-01-30 02:43 UTC by Victor Mataré
Modified: 2011-02-11 05:30 UTC (History)
0 users

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Victor Mataré 2011-01-30 02:43:06 UTC
ncftp /pentium-m > ls -l Packages 
-rw-r--r--    1 0        0      1439665   Jan 30 02:30   Packages

pentium-m-test ~ # ls -l /var/cache/edb/binhost/wald.local/pentium-m/Packages
-rw-r--r-- 1 root root 1394660 Jan 29 15:57 /var/cache/edb/binhost/wald.local/pentium-m/Packages

pentium-m-test ~ # ls -l /usr/portage/packages/Packages 
-rw-r--r-- 1 root root 1467657 Jan 29 23:00 /usr/portage/packages/Packages

what's the problem?

Reproducible: Always

Steps to Reproduce:
Comment 1 Zac Medico gentoo-dev 2011-01-30 02:48:16 UTC
It's supposed to use the existing file if it has the same TIMESTAMP entry near the top of the file. Is it the same?
Comment 2 Victor Mataré 2011-01-30 02:56:24 UTC
Nope. Timestamp on server:
TIMESTAMP: 1296354620

vs. local timestamp:
TIMESTAMP: 1296338446

Downgrading to stable portage also doesn't help.
Comment 3 Zac Medico gentoo-dev 2011-01-30 03:00:13 UTC
Try this:

  wget $(portageq envvar PORTAGE_BINHOST)/Packages

Does it download?
Comment 4 Victor Mataré 2011-01-30 03:02:42 UTC
# wget $(portageq envvar PORTAGE_BINHOST)/Packages
--2011-01-30 04:01:41--  ftp://wald.local/pentium-m/Packages
           => `Packages'
Resolving wald.local... 169.254.243.195
Connecting to wald.local|169.254.243.195|:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done.    ==> PWD ... done.
==> TYPE I ... done.  ==> CWD (1) /pentium-m ... done.
==> SIZE Packages ... 1439665
==> PASV ... done.    ==> RETR Packages ... done.
Length: 1439665 (1.4M) (unauthoritative)

100%[=====================================================>] 1,439,665   --.-K/s   in 0.07s   

2011-01-30 04:01:41 (20.2 MB/s) - `Packages' saved [1439665]

pentium-m-test ~ # ll Packages 
-rw-r--r-- 1 root root 1439665 Jan 30 04:01 Packages
pentium-m-test ~ # grep ^TIMESTAMP Packages
TIMESTAMP: 1296354620
Comment 5 Zac Medico gentoo-dev 2011-01-30 03:05:32 UTC
When you try to use emerge -g/-G, does it show any error message(s)?
Comment 6 Zac Medico gentoo-dev 2011-01-30 03:06:59 UTC
This seems similar to bug 350139.
Comment 7 Victor Mataré 2011-01-30 03:09:53 UTC
No errors:

# emerge -guDavN world

These are the packages that would be merged, in order:

Calculating dependencies... done!
[binary     U ] sys-apps/portage-2.2.0_alpha19 [2.1.9.25] USE="(ipc) -build
-doc -epydoc -python3 (-selinux)" LINGUAS="-pl"                                 
[binary  N    ] x11-proto/bigreqsproto-1.1.1  USE="-doc" 
[binary  N    ] x11-proto/xcmiscproto-1.2.1  USE="-doc" 
[binary     U ] x11-apps/xrdb-1.0.7 [1.0.6] USE="(-debug%)" 
[binary  N    ] x11-apps/xinit-1.3.0-r1  USE="minimal" 

Total: 5 packages (2 upgrades, 3 new, 5 binaries), Size of downloads: 0 kB

Would you like to merge these packages? [Yes/No] n
Comment 8 Zac Medico gentoo-dev 2011-01-30 03:16:49 UTC
Try this:

  rm -rf /var/cache/edb/binhost

Does it download now?
Comment 9 Victor Mataré 2011-01-30 03:17:08 UTC
right, vsftpd says:

Sun Jan 30 04:15:48 2011 [pid 13558] CONNECT: Client "169.254.64.12"
Sun Jan 30 04:15:48 2011 [pid 13558] FTP response: Client "169.254.64.12", "220 (vsFTPd 2.3.2)"
Sun Jan 30 04:15:48 2011 [pid 13558] FTP command: Client "169.254.64.12", "USER anonymous"
Sun Jan 30 04:15:48 2011 [pid 13558] [anonymous] FTP response: Client "169.254.64.12", "331 Please specify the password."
Sun Jan 30 04:15:48 2011 [pid 13558] [anonymous] FTP command: Client "169.254.64.12", "PASS <password>"
Sun Jan 30 04:15:48 2011 [pid 13557] [ftp] OK LOGIN: Client "169.254.64.12", anon password "anonymous@"
Sun Jan 30 04:15:48 2011 [pid 13559] [ftp] FTP response: Client "169.254.64.12", "230 Login successful."
Sun Jan 30 04:15:48 2011 [pid 13559] [ftp] FTP command: Client "169.254.64.12", "CWD pentium-m"
Sun Jan 30 04:15:48 2011 [pid 13559] [ftp] FTP response: Client "169.254.64.12", "250 Directory successfully changed."
Sun Jan 30 04:15:48 2011 [pid 13559] [ftp] FTP command: Client "169.254.64.12", "TYPE I"
Sun Jan 30 04:15:48 2011 [pid 13559] [ftp] FTP response: Client "169.254.64.12", "200 Switching to Binary mode."
Sun Jan 30 04:15:48 2011 [pid 13559] [ftp] FTP command: Client "169.254.64.12", "PASV"
Sun Jan 30 04:15:48 2011 [pid 13559] [ftp] FTP response: Client "169.254.64.12", "227 Entering Passive Mode (169,254,243,195,21,101)."
Sun Jan 30 04:15:48 2011 [pid 13559] [ftp] FTP command: Client "169.254.64.12", "RETR Packages"
Sun Jan 30 04:15:48 2011 [pid 13559] [ftp] FTP response: Client "169.254.64.12", "150 Opening BINARY mode data connection for Packages (1439665 bytes)."
Sun Jan 30 04:15:48 2011 [pid 13559] [ftp] FTP response: Client "169.254.64.12", "426 Failure writing network stream."
Sun Jan 30 04:15:48 2011 [pid 13559] [ftp] FAIL DOWNLOAD: Client "169.254.64.12", "/pentium-m/Packages", 83984 bytes, 4582.65Kbyte/sec
Comment 10 Zac Medico gentoo-dev 2011-01-30 03:19:56 UTC
I guess it's something like bug 350139. Can you try another ftp server program. The person in bug 350139 said that net-ftp/oftpd worked fine.
Comment 11 Victor Mataré 2011-01-30 03:22:30 UTC
yup. Now:

Sun Jan 30 04:18:18 2011 [pid 14375] CONNECT: Client "169.254.64.12"
Sun Jan 30 04:18:18 2011 [pid 14375] FTP response: Client "169.254.64.12", "220 (vsFTPd 2.3.2)"
Sun Jan 30 04:18:18 2011 [pid 14375] FTP command: Client "169.254.64.12", "USER anonymous"
Sun Jan 30 04:18:18 2011 [pid 14375] [anonymous] FTP response: Client "169.254.64.12", "331 Please specify the password."
Sun Jan 30 04:18:18 2011 [pid 14375] [anonymous] FTP command: Client "169.254.64.12", "PASS <password>"
Sun Jan 30 04:18:18 2011 [pid 14374] [ftp] OK LOGIN: Client "169.254.64.12", anon password "anonymous@"
Sun Jan 30 04:18:18 2011 [pid 14376] [ftp] FTP response: Client "169.254.64.12", "230 Login successful."
Sun Jan 30 04:18:18 2011 [pid 14376] [ftp] FTP command: Client "169.254.64.12", "CWD pentium-m"
Sun Jan 30 04:18:18 2011 [pid 14376] [ftp] FTP response: Client "169.254.64.12", "250 Directory successfully changed."
Sun Jan 30 04:18:18 2011 [pid 14376] [ftp] FTP command: Client "169.254.64.12", "TYPE I"
Sun Jan 30 04:18:18 2011 [pid 14376] [ftp] FTP response: Client "169.254.64.12", "200 Switching to Binary mode."
Sun Jan 30 04:18:18 2011 [pid 14376] [ftp] FTP command: Client "169.254.64.12", "PASV"
Sun Jan 30 04:18:18 2011 [pid 14376] [ftp] FTP response: Client "169.254.64.12", "227 Entering Passive Mode (169,254,243,195,26,228)."
Sun Jan 30 04:18:18 2011 [pid 14376] [ftp] FTP command: Client "169.254.64.12", "RETR Packages"
Sun Jan 30 04:18:18 2011 [pid 14376] [ftp] FTP response: Client "169.254.64.12", "150 Opening BINARY mode data connection for Packages (1439665 bytes)."
Sun Jan 30 04:18:20 2011 [pid 14376] [ftp] OK DOWNLOAD: Client "169.254.64.12", "/pentium-m/Packages", 1439665 bytes, 1040.68Kbyte/sec
Sun Jan 30 04:18:20 2011 [pid 14376] [ftp] FTP response: Client "169.254.64.12", "226 Transfer complete." CONNECT: Client "169.254.64.12"
Comment 12 Victor Mataré 2011-01-30 03:23:13 UTC
Ugh. I like vsftpd. What is it doing wrong?
Comment 13 Zac Medico gentoo-dev 2011-01-30 03:25:36 UTC
Like I said in bug #350139, comment #16, it's some kind of interaction between python's urllib and your ftp server.
Comment 14 Victor Mataré 2011-01-30 03:29:29 UTC
Bleh. I hate this kinda stuff. Now do I want to investigate this? Argh. Stupid ftp. Any other option besides ftp or http?
Comment 15 Victor Mataré 2011-01-30 03:31:46 UTC
But the thing is, why did it work after deleting /var/cache/edb/binhost? That shouldn't make any difference to the ftp transfer, should it?
Comment 16 Zac Medico gentoo-dev 2011-01-30 03:37:22 UTC
ssh and sftp protocols also work.

I don't know why it behaves this way with some ftp servers. When I tried to reproduce it with pure-ftpd, I couldn't reproduce it (bug #350139, comment #13).
Comment 17 Victor Mataré 2011-01-30 03:47:58 UTC
Ok, this stuff gives me nightmares, but I guess there's nothing that can be done right now...
Comment 18 Zac Medico gentoo-dev 2011-01-30 03:54:26 UTC
I'm going to mark this as a duplicate of bug 350139 since the symptoms are almost identical.
Comment 19 Zac Medico gentoo-dev 2011-01-30 03:56:24 UTC

*** This bug has been marked as a duplicate of bug 350139 ***
Comment 20 Victor Mataré 2011-01-30 04:02:01 UTC
Do I read the code right in that you do ssh/sftp yourself, instead of using crappy urllib?

BTW, wouldn't it be nice to throw out a warning if getting the TIMESTAMP header failed? The comment "no timestamp in the header, something's wrong" in bintree.py kind of suggests it.
Comment 21 Zac Medico gentoo-dev 2011-01-30 04:14:30 UTC
(In reply to comment #20)
> Do I read the code right in that you do ssh/sftp yourself, instead of using
> crappy urllib?

I suppose we could spawn FETCHCOMMAND in order to handle the ftp protocol, or at least use it as a fallback if urllib fails somehow.

> BTW, wouldn't it be nice to throw out a warning if getting the TIMESTAMP header
> failed? The comment "no timestamp in the header, something's wrong" in
> bintree.py kind of suggests it.

I guess that's the one that's being triggered here. Now I've added a warning in git:

http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=a607079b3b556de243e955cda7ae9a3defd26750
Comment 22 Zac Medico gentoo-dev 2011-01-30 04:38:04 UTC
(In reply to comment #21)
> (In reply to comment #20)
> > Do I read the code right in that you do ssh/sftp yourself, instead of using
> > crappy urllib?
> 
> I suppose we could spawn FETCHCOMMAND in order to handle the ftp protocol, or
> at least use it as a fallback if urllib fails somehow.

Actually, ftplib may be worth a try, since it was reported to work in bug 350139.
Comment 23 Zac Medico gentoo-dev 2011-02-11 05:30:33 UTC
I'm planning to merge this patch, for using FETCHCOMMAND instead of urllib:

  http://codereview.chromium.org/6458015/