Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!

Bug 367479

Summary: emerge fetch function should be mirror-selective depending on ebuild location
Product: Portage Development Reporter: Sergey S. Starikoff <Ikonta>
Component: Enhancement/Feature RequestsAssignee: Portage team <dev-portage>
Status: UNCONFIRMED ---    
Severity: enhancement    
Priority: Normal    
Version: unspecified   
Hardware: All   
OS: Linux   
Whiteboard:
Package list:
Runtime testing required: ---
Bug Depends on:    
Bug Blocks: 377365    

Description Sergey S. Starikoff 2011-05-16 07:15:19 UTC
If the GENTOO_MIRRORS variable is set in /etc/make.conf emerge always starts fetching from mirrors listed in GENTOO_MIRRORS.
But many (most up to almost all) ebuilds not in mail tree (layman or local overlays, excluding custom builds of software from main tree) contains packages, which distfiles should not predent (and don't present) on official Gentoo mirrors.

Reproducible: Always

Steps to Reproduce:
1. Set up some (3-4 or above) GENTOO_MIRRORS;
2. Install layman, add an overlay, unmask and try to fetch distfiles for some package (for example myrulib in rion).
Actual Results:  
See emerge attemps to fetch distfiles from GENTOO_MIRRORS (alhough they are not distributed on GENTOO_MIRRORS).

Expected Results:  
emerge should start fetch from SRC_URI and do not make requests to the mirrors doesn't contains requested distfiles.

The issue was discussed on gentoo.ru forum.

We think, that the best, but long-strategy solution is the adding to ebuild skel in all editors the RESTRICT="mirror" line.
Updating documentation and update request for maintainers of all ebuild (at least in layman overlays).

$ emerge  --info
Portage 2.1.9.42 (default/linux/x86/10.0, gcc-4.4.5, libc-0-r0, 2.6.36-gentoo-r5 i686)
=================================================================
System uname: Linux-2.6.36-gentoo-r5-i686-Intel-R-_Pentium-R-_4_CPU_3.00GHz-with-gentoo-2.0.2
Timestamp of tree: Tue, 10 May 2011 00:45:01 +0000
app-shells/bash:     4.1_p9
dev-java/java-config: 2.1.11-r3
dev-lang/python:     2.7.1-r1, 3.1.3-r1
dev-util/cmake:      2.8.1-r2
sys-apps/baselayout: 2.0.2
sys-apps/openrc:     0.8.2-r1
sys-apps/sandbox:    2.4
sys-devel/autoconf:  2.13, 2.65-r1
sys-devel/automake:  1.10.3, 1.11.1
sys-devel/binutils:  2.20.1-r1
sys-devel/gcc:       4.4.5
sys-devel/gcc-config: 1.4.1-r1
sys-devel/libtool:   2.2.10
sys-devel/make:      3.81-r2
sys-kernel/linux-headers: 2.6.36.1
sys-libs/glibc:      2.11.3
virtual/os-headers:  0
ACCEPT_KEYWORDS="x86"
ACCEPT_LICENSE="* -@EULA"
CBUILD="i686-pc-linux-gnu"
CFLAGS="-O2 -march=native -pipe"
CHOST="i686-pc-linux-gnu"
CONFIG_PROTECT="/etc /etc/env.d/50glib2 /usr/share/gnupg/qualified.txt /var/bind"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/env.d/java/ /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php5.2/ext-active/ /etc/php/cgi-php5.2/ext-active/ /etc/php/cli-php5.2/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c"
CXXFLAGS="-O2 -march=native -pipe"
DISTDIR="/usr/portage/distfiles"
FEATURES="assume-digests binpkg-logs buildsyspkg collision-protect distlocks fixlafiles fixpackages news parallel-fetch protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch"
FFLAGS=""
GENTOO_MIRRORS="http://mirror.yandex.ru/gentoo-distfiles/ 		ftp://mirror.yandex.ru/gentoo-distfiles/                 http://ftp.chg.ru/pub/Linux/distributions/gentoo/                 ftp://ftp.chg.ru/pub/Linux/distributions/gentoo/ 		http://ftp.corbina.net/pub/Linux/gentoo/ 		ftp://ftp.corbina.net/pub/Linux/gentoo/"
LANG="ru_RU.KOI8-R"
LC_ALL=""
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
LINGUAS="ru en"
MAKEOPTS="-j2"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY="/var/lib/layman/rion /usr/local/portage-ftn"
SYNC="rsync://rsync.gentoo.org/gentoo-portage"
USE="X a52 ac3 acl alsa avi berkdb bold bzip2 cdr cli cracklib crypt cups cxx dbus djvu dri dvd flac fortran gdbm gif gtk iconv jpeg jpeg2k modules mp3 mudflap ncurses nls nptl nptlonly ogg openmp pam pcre pdf perl png pppd python qt3support readline session ssl sysfs tcpd tiff unicode utf8 vorbis x86 xcb xorg xulrunner zlib" ALSA_CARDS="intel8x0" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ubx" INPUT_DEVICES="keyboard mouse" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LINGUAS="ru en" PHP_TARGETS="php5-3" RUBY_TARGETS="ruby18" USERLAND="GNU" VIDEO_CARDS="sis" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account" 
Unset:  CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS
Comment 1 Zac Medico gentoo-dev 2011-05-16 07:28:58 UTC
This seem like a possible duplicate of either bug 205709 or bug 299181 (see comments about RESTRICT=primaryuri).
Comment 2 Sergey S. Starikoff 2011-05-16 10:47:01 UTC
(In reply to comment #1)
> This seem like a possible duplicate of either bug 205709 or bug 299181 (see
> comments about RESTRICT=primaryuri).

Yes, the issue is exactly like described in bug 205709 only suggested solution differs.
Asking primary URLs/mirrors first I think is incorrect.
Putting RESTRICT="mirror" into ebuilds in non-maintree overlays (+ update documentation and ebuilds update maintainer-addresed request) and updating repoman adding check of distfiles on GENTOO_MIRRORS if restriction is not present in ebuilds in additional overlays seems to be better solution.
Comment 3 Zac Medico gentoo-dev 2011-05-18 14:22:25 UTC
(In reply to comment #2)
> Putting RESTRICT="mirror" into ebuilds in non-maintree overlays (+ update
> documentation and ebuilds update maintainer-addresed request)

I think using RESTRICT="mirror" for this is going to be confusing, since the common meaning of this setting is that the file(s) should not be mirrored due to license requirements. If RESTRICT="primaryuri" is also unsuitable for some reason, then I'd suggest the we invent some new metadata for the specific use case that we are considering.
Comment 4 Sergey S. Starikoff 2011-05-20 06:26:03 UTC
(In reply to comment #3)
> (In reply to comment #2)
> > Putting RESTRICT="mirror" into ebuilds in non-maintree overlays (+ update
> > documentation and ebuilds update maintainer-addresed request)
> 
> I think using RESTRICT="mirror" for this is going to be confusing, since the
> common meaning of this setting is that the file(s) should not be mirrored due
> to license requirements. If RESTRICT="primaryuri" is also unsuitable for some
> reason, then I'd suggest the we invent some new metadata for the specific use
> case that we are considering.

RESTRICH="primaryurl" looks for me too high-weight.

There may be another solution: search deistfiles on GENTOO_MIRRORS only for not-restricted ebuilds in main tree.
If it is possible depending on package metadata identify custom (overlay) builds of main tree packages (which distfiles could and should be fetched from GENTOO_MIRRORS)...


Otherwise I think that functionally RESTRICT="mirror" solves issue.
For escaping confusing may be ebough to create an alias (restriction with the same functionality) for "mirror", named, for example "overlay".
Which should be used to prevent packages in overlays to search missed on official mirrors distfiles.
With adding this restriction in ebuild skels in all editors and describing issue in documentation.
Comment 5 Sergey S. Starikoff 2012-12-03 08:44:00 UTC
Find in unofficial wiki key:
RESTRICT="NOMIRROR"
seems to be a solution.
Comment 6 Zac Medico gentoo-dev 2012-12-03 21:10:53 UTC
(In reply to comment #5)
> Find in unofficial wiki key:
> RESTRICT="NOMIRROR"
> seems to be a solution.

RESTRICT="mirror"/nomirror has a subtly different meaning (it means that it's not legal to mirror the files). Typically, RESTRICT="primaryuri" is really what you want. It means that emerge shouldn't try to fetch the file from mirrors.
Comment 7 Sergey S. Starikoff 2012-12-07 05:07:03 UTC
(In reply to comment #6)
> Typically, RESTRICT="primaryuri" is really what you want.

Thank you!
It is.
But I can solve this issue only in my local overlay.

For complete solution of this issue string
RESTRICT="primaryuri"
is to be added into skel.ebuild (opened by vim while creating ebuild file) and added to ebuilds in overlays, which uses not distributed on official mirrors distfiles.
Comment 8 Sergey S. Starikoff 2015-07-24 06:33:15 UTC
For now (=sys-apps/portage-2.2.20) this bug has a solution.
It's enough to move GENTOO_MIRRORS parametr into /etc/portage/repos.conf/${REPONAME}.conf after sync-uri.
And the task could be forwarder on overlay maintainer's level.

Also it should provide permitting mirror requests for overlays, which have no mirror infrastructure, most typical example — the local overlay.
Comment 9 Brian Dolbec gentoo-dev 2015-07-24 14:31:24 UTC
(In reply to Sergey S. Starikoff from comment #8)
> For now (=sys-apps/portage-2.2.20) this bug has a solution.
> It's enough to move GENTOO_MIRRORS parametr into
> /etc/portage/repos.conf/${REPONAME}.conf after sync-uri.
> And the task could be forwarder on overlay maintainer's level.
> 
> Also it should provide permitting mirror requests for overlays, which have
> no mirror infrastructure, most typical example — the local overlay.

No, your wrong.  2.2.20 has a fixed repos.conf usage, it does not allow to move a global setting into a repo specific one.

There is code in git now (since 2.2.20) that will allow repos specific settings for the sync modules.  But that does not include settings like you are wanting.

The config validation system still needs some updating/re-writing to enhance it's capabilities.  But that is not a high priority item at this time.  There are other tasks in greater need of completion.
Comment 10 Sergey S. Starikoff 2015-07-25 08:47:49 UTC
(In reply to Brian Dolbec from comment #9)
> (In reply to Sergey S. Starikoff from comment #8)
> > For now (=sys-apps/portage-2.2.20) this bug has a solution.
> > It's enough to move GENTOO_MIRRORS parametr into
> > /etc/portage/repos.conf/${REPONAME}.conf after sync-uri.
> > And the task could be forwarder on overlay maintainer's level.
> > 
> > Also it should provide permitting mirror requests for overlays, which have
> > no mirror infrastructure, most typical example — the local overlay.
> 
> No, your wrong.  2.2.20 has a fixed repos.conf usage, it does not allow to
> move a global setting into a repo specific one.
> 
> There is code in git now (since 2.2.20) that will allow repos specific
> settings for the sync modules.  But that does not include settings like you
> are wanting.

No.
I'm right. I just wasn't clear enough.
I've don't made any non-standard config changes, I just suggest to move mirror settings from top-level make.conf to per-repository config in some future release (>sys-apps/portage-2.2.20).


> The config validation system still needs some updating/re-writing to enhance
> it's capabilities.  But that is not a high priority item at this time. 
> There are other tasks in greater need of completion.

I agree with importance of config validation system, but it is another question.
Maybe it will be reasonable to provide special interface for this function (an utility in app-portage/gentoolkit or sys-apps/portage itself).