Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 354555 - sys-apps/portage: fetch should fallback to default mirror (distfiles.gentoo.org) as the last fetch try if everything else fails (including ebuild's fetch URI)
Summary: sys-apps/portage: fetch should fallback to default mirror (distfiles.gentoo.o...
Status: IN_PROGRESS
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Current packages (show other bugs)
Hardware: All Linux
: High normal (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords:
Depends on:
Blocks: 377365
  Show dependency tree
 
Reported: 2011-02-12 01:29 UTC by David Carlos Manuelda
Modified: 2016-10-02 09:29 UTC (History)
2 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
emerge--info output (emergeinfo.txt,13.54 KB, text/plain)
2011-02-15 15:50 UTC, David Carlos Manuelda
Details

Note You need to log in before you can comment on or make changes to this bug.
Description David Carlos Manuelda 2011-02-12 01:29:01 UTC
Any of versions in current portage tree of dev-python/paramiko can't be fetched, nor from mirrors nor from upstream website.

From upstream website, there is only a maximum version of 1.7.4, while in portage there are 1.7.5 and 1.7.6.

Where do they come from if can't be fetched and are not listed upstream?

It is a requirement for other important software, like mysql-workbench.

Reproducible: Always
Comment 1 David Carlos Manuelda 2011-02-12 13:21:32 UTC
I've only located paramiko 1.7.6 in http://pypi.python.org/packages/source/p/paramiko/paramiko-1.7.6.zip#md5=b1cfe0cd55772115f808a11c1baba8a0 as a zip package.
Comment 2 David Carlos Manuelda 2011-02-13 02:28:45 UTC
Unzipped, tarred and forcing manifest of paramiko-1.7.6 ebuild worked.
Comment 3 Jeroen Roovers gentoo-dev 2011-02-15 14:00:48 UTC
You need to fix your GENTOO_MIRRORS in /etc/make.conf - our mirrors host those files quite nicely for you. Other than that you have a probably valid point about the availability of versions which the HOMEPAGE doesn't list - maybe the HOMEPAGE variable should also list <https://github.com/robey/paramiko/> since that seems to be the best source of downloads mentioned at <http://www.lag.net/paramiko/>

gentoo-x86/dev-python/paramiko $ for i in paramiko-1.7.*; do ebuild $i fetch; done                Appending /newaches/gentoo/cvs/gentoo-x86 to PORTDIR_OVERLAY...
>>> Downloading 'http://mirror.cambrium.nl/pub/os/linux/gentoo/distfiles/paramiko-1.7.5.tar.gz'
--2011-02-15 14:55:01--  http://mirror.cambrium.nl/pub/os/linux/gentoo/distfiles/paramiko-1.7.5.tar.gz
Resolving mirror.cambrium.nl... 217.19.16.188, 2a02:58:3:2:80::1
Connecting to mirror.cambrium.nl|217.19.16.188|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2011-02-15 14:55:01 ERROR 404: Not Found.

>>> Downloading 'http://mirror.leaseweb.com/gentoo/distfiles/paramiko-1.7.5.tar.gz'
--2011-02-15 14:55:01--  http://mirror.leaseweb.com/gentoo/distfiles/paramiko-1.7.5.tar.gz
Resolving mirror.leaseweb.com... 94.75.223.121, 2001:1af8:4030:1:215:17ff:fecf:3668
Connecting to mirror.leaseweb.com|94.75.223.121|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 843046 (823K) [application/octet-stream]
Saving to: `/aches/distfiles/paramiko-1.7.5.tar.gz'

100%[==========================================================================================>] 843,046     1.11M/s   in 0.7s

2011-02-15 14:55:02 (1.11 MB/s) - `/aches/distfiles/paramiko-1.7.5.tar.gz' saved [843046/843046]

 * paramiko-1.7.5.tar.gz RMD160 SHA1 SHA256 size ;-) ...                                                                     [ ok ]
 * checking ebuild checksums ;-) ...                                                                                         [ ok ]
 * checking auxfile checksums ;-) ...                                                                                        [ ok ]
 * checking miscfile checksums ;-) ...                                                                                       [ ok ]
Appending /newaches/gentoo/cvs/gentoo-x86 to PORTDIR_OVERLAY...
>>> Downloading 'http://mirror.cambrium.nl/pub/os/linux/gentoo/distfiles/paramiko-1.7.6.tar.gz'
--2011-02-15 14:55:03--  http://mirror.cambrium.nl/pub/os/linux/gentoo/distfiles/paramiko-1.7.6.tar.gz
Resolving mirror.cambrium.nl... 217.19.16.188, 2a02:58:3:2:80::1
Connecting to mirror.cambrium.nl|217.19.16.188|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2011-02-15 14:55:03 ERROR 404: Not Found.

>>> Downloading 'http://mirror.leaseweb.com/gentoo/distfiles/paramiko-1.7.6.tar.gz'
--2011-02-15 14:55:03--  http://mirror.leaseweb.com/gentoo/distfiles/paramiko-1.7.6.tar.gz
Resolving mirror.leaseweb.com... 94.75.223.121, 2001:1af8:4030:1:215:17ff:fecf:3668
Connecting to mirror.leaseweb.com|94.75.223.121|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 843748 (824K) [application/octet-stream]
Saving to: `/aches/distfiles/paramiko-1.7.6.tar.gz'

100%[==========================================================================================>] 843,748      795K/s   in 1.0s

2011-02-15 14:55:04 (795 KB/s) - `/aches/distfiles/paramiko-1.7.6.tar.gz' saved [843748/843748]

 * paramiko-1.7.6.tar.gz RMD160 SHA1 SHA256 size ;-) ...                                                                     [ ok ]
 * checking ebuild checksums ;-) ...                                                                                         [ ok ]
 * checking auxfile checksums ;-) ...                                                                                        [ ok ]
 * checking miscfile checksums ;-) ...                                                                                       [ ok ]
Comment 4 David Carlos Manuelda 2011-02-15 15:49:44 UTC
I've already have setup GENTOO_MIRRORS, and I get this:

LC_ALL="C" emerge -f paramiko

These are the packages that would be fetched, in order:

Calculating dependencies... done!
[ebuild   R    ] dev-python/paramiko-1.7.6  USE="-doc -examples" 824 kB

Total: 1 package (1 reinstall), Size of downloads: 824 kB

Would you like to fetch the source files for these packages? [Yes/No] 

>>> Fetching (1 of 1) dev-python/paramiko-1.7.6
>>> Downloading 'http://ftp.udc.es/gentoo/distfiles/distfiles/paramiko-1.7.6.tar.gz'
--2011-02-15 16:48:42--  http://ftp.udc.es/gentoo/distfiles/distfiles/paramiko-1.7.6.tar.gz
Resolving ftp.udc.es... 193.144.61.75
Connecting to ftp.udc.es|193.144.61.75|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2011-02-15 16:48:43 ERROR 404: Not Found.

>>> Downloading 'http://gentoo-euetib.upc.es/mirror/gentoo/distfiles/distfiles/paramiko-1.7.6.tar.gz'
--2011-02-15 16:48:43--  http://gentoo-euetib.upc.es/mirror/gentoo/distfiles/distfiles/paramiko-1.7.6.tar.gz
Resolving gentoo-euetib.upc.es... 147.83.169.250
Connecting to gentoo-euetib.upc.es|147.83.169.250|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2011-02-15 16:48:43 ERROR 404: Not Found.

>>> Downloading 'http://www.lag.net/paramiko/download/paramiko-1.7.6.tar.gz'
--2011-02-15 16:48:43--  http://www.lag.net/paramiko/download/paramiko-1.7.6.tar.gz
Resolving www.lag.net... 173.255.213.140
Connecting to www.lag.net|173.255.213.140|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2011-02-15 16:48:43 ERROR 404: Not Found.

!!! Couldn't download 'paramiko-1.7.6.tar.gz'. Aborting.
 * Fetch failed for 'dev-python/paramiko-1.7.6'

>>> Failed to emerge dev-python/paramiko-1.7.6

I attach a emerge --info containing my GENTOO_MIRRORS too.
Comment 5 David Carlos Manuelda 2011-02-15 15:50:26 UTC
Created attachment 262613 [details]
emerge--info output
Comment 6 David Carlos Manuelda 2011-03-02 22:24:42 UTC
By the way, as GENTOO-MIRROR, I have one of listed in http://www.gentoo.org/main/en/mirrors.xml:
GENTOO_MIRRORS="http://ftp.udc.es/gentoo/distfiles/  http://gentoo-euetib.upc.es/mirror/gentoo/distfiles/"
Isn't it valid?
Comment 7 Matt Summers (RETIRED) gentoo-dev 2011-03-02 22:45:34 UTC
Just so everyone knows, there have been issues with upstream for quite awhile regarding release distribution, see [1]. It looks like upstream cleaned up his PYPI page [2], buts its well known that his VPS lag.net is unreliable. It might be nice to have the SRC_URI go to PYPI for this reason, and set the HOMEPAGE to the Github page [1] or his Launchpad page [3].


[1] https://github.com/robey/paramiko/issues#issue/3
[2] https://pypi.python.org/pypi/paramiko
[3] https://launchpad.net/paramiko
Comment 8 Arfrever Frehtes Taifersar Arahesis (RETIRED) gentoo-dev 2011-03-03 03:34:54 UTC
The only difference between original paramiko-1.7.6.tar.gz and new paramiko-1.7.6.zip is that paramiko-1.7.6.tar.gz additionally contains "docs" directory (with documentation)...
Comment 9 David Carlos Manuelda 2011-03-30 10:46:08 UTC
It seems caused by a misconfiguration of GENTOO_MIRRORS, but still a bug, because if not found on specified mirrors, it should be downloaded elsewhere.

If user specified mirrors are invalid, and since fetch URI specified in ebuild is invalid, it can't be downloaded.

To reproduce:
Change GENTOO_MIRRORS="foo bar" (invalid ones) and you'll get:
emerge -f paramiko

These are the packages that would be fetched, in order:

Calculating dependencies... done!
[ebuild   R    ] dev-python/paramiko-1.7.6  USE="-doc -examples" 824 kB

Total: 1 package (1 reinstall), Size of downloads: 824 kB

Would you like to fetch the source files for these packages? [Yes/No] 

>>> Fetching (1 of 1) dev-python/paramiko-1.7.6
>>> Downloading 'foo/distfiles/paramiko-1.7.6.tar.gz'
--2011-03-30 12:44:28--  http://foo/distfiles/paramiko-1.7.6.tar.gz
Resolviendo foo... falló: Unknown host.
wget: unable to resolve host address `foo'
>>> Downloading 'bar/distfiles/paramiko-1.7.6.tar.gz'
--2011-03-30 12:44:28--  http://bar/distfiles/paramiko-1.7.6.tar.gz
Resolviendo bar... falló: Unknown host.
wget: unable to resolve host address `bar'
>>> Downloading 'http://www.lag.net/paramiko/download/paramiko-1.7.6.tar.gz'
--2011-03-30 12:44:28--  http://www.lag.net/paramiko/download/paramiko-1.7.6.tar.gz
Resolviendo www.lag.net... 173.255.213.140
Connecting to www.lag.net|173.255.213.140|:80... conectado.
Petición HTTP enviada, esperando respuesta... 404 Not Found
2011-03-30 12:44:29 ERROR 404: Not Found.

!!! Couldn't download 'paramiko-1.7.6.tar.gz'. Aborting.
 * Fetch failed for 'dev-python/paramiko-1.7.6'

>>> Failed to emerge dev-python/paramiko-1.7.6

However, if GENTOO_MIRRORS is NOT set, then portage will download it from distfiles.gentoo.org.

Then, I think it is a bug in portage, since the last fallback URI should be always distfiles.gentoo.org, instead of fetch URI of ebuild.
Comment 10 David Carlos Manuelda 2011-03-30 10:49:32 UTC
I suggest to CC portage team.
Since external fetch URI can change anytime and can be no longer valid at any moment, this is the reason I suggest to have the last callback a generic mirror (like distfiles.gentoo.org), instead of having to change ebuild's fetch URI when neccesary.
So I'll change bug subject accordingly.
Comment 11 Zac Medico gentoo-dev 2011-03-30 18:49:23 UTC
Sounds good to me. Maybe the primary URI should still be last though, to ensure that we use a mirror whenever possible.
Comment 12 Zac Medico gentoo-dev 2011-03-30 20:36:31 UTC
We can use settings.configdict["globals"]["GENTOO_MIRRORS"] to grab the default http://distfiles.gentoo.org setting from make.globals.
Comment 13 SpanKY gentoo-dev 2011-03-30 20:58:23 UTC
while in general i like this idea (because a lot of people screw up their configs with dumb settings that lack proper mirrors), i feel like there are people who strictly control their mirror settings for various reasons.  like needing to stick to internal machines, or they have metered access and want to only hit specific mirrors that are internal to their network (i have one such machine myself, although the limit is high enough that i dont mind).  this seems like it could be a very nasty surprise indeed.

i'm not sure how to address this though whilst satisfying everyone.
Comment 14 Zac Medico gentoo-dev 2011-03-31 23:57:00 UTC
How about if we add a GENTOO_MIRRORS_FALLBACK variable, with a default setting like this in make.globals:

# Default distfiles mirrors fallback. Override with an empty string in
# make.conf if you never want to use the default GENTOO_MIRRORS setting.
GENTOO_MIRRORS_FALLBACK=${GENTOO_MIRRORS}
Comment 15 SpanKY gentoo-dev 2011-04-01 00:04:53 UTC
if we accompany it with a news entry, i think that should be sufficient
Comment 16 Zac Medico gentoo-dev 2011-04-01 00:12:51 UTC
Alternatively, we could modify mirrorselect to add the fallback to GENTOO_MIRRORS by default, and also add a note in the make.conf manpage that it's recommended to have a generic fallback for the last element in GENTOO_MIRRORS. That way we don't really need a news item, and we can get by with one less variable.
Comment 17 SpanKY gentoo-dev 2011-04-01 01:08:02 UTC
as long as we can start marking bugs invalid when people dont set up their mirrors correctly ;)

the policy tweak should be announced on gentoo-dev mailing list
Comment 18 Jan Kundrát (RETIRED) gentoo-dev 2011-05-09 20:31:25 UTC
(In reply to comment #16)
> Alternatively, we could modify mirrorselect to add the fallback to
> GENTOO_MIRRORS by default, and also add a note in the make.conf manpage that
> it's recommended to have a generic fallback for the last element in
> GENTOO_MIRRORS. That way we don't really need a news item, and we can get by
> with one less variable.

I'd argue that default settings should work for majority of people. Running mirrorselect has been always marked as an optional step in our docs. I can file in the URL of my nearby mirror much faster than I could look up what arguments to pass to mirrorselect. Even if that wasn't relevant at all, there'd still the question "why should people set a variable even though it could be handled by default?".

Sure, by all means provide some override for people who absolutely *don't* want to hit distfiles.g.o, but most of them shouldn't care, so having a reasonable settings out of the box is a good idea.