Bug 365033 – setting SYNC variable in /etc/make.conf brokes emerge-webrsync utility

• Product: Portage Development
• Status: RESOLVED FIXED

[Description Sergey S. Starikoff 2011-04-27 06:18:23 UTC]

With default value of SYNC variable (doesn't set in /etc/make.conf) both emerge --sync and emerge-webrsync works normally.

After setting in /etc/make.conf custom portage sync mirror:
SYNC="\ \ \ rsync://mirror.yandex.ru/gentoo-portage/"

emerge-webrsync falls into error

Reproducible: Always

Steps to Reproduce:
1. Set up into /etc/make.conf the SYNC variable (for example the string listed above);
2. Ensure it't operable one (emerge --sync works fine);
3. Remove /usr/portage/metadata/timestamp.x
4. Try to sync portage tree using emerge-webrsync
Actual Results:  
# emerge-webrsync 
The current SYNC variable setting does not refer to an rsync URI:

  SYNC= rsync://mirror.yandex.ru/gentoo-portage/

If you intend to use emerge-webrsync then please
adjust SYNC to refer to an rsync URI.
emerge-webrsync exiting due to abnormal SYNC setting.

Expected Results:  
Normal operating of emerge-webrsync (as in the case, where default sync servers are used).

# emerge --info
Portage 2.1.9.42 (default/linux/x86/10.0, gcc-4.4.5, libc-0-r0, 2.6.36-gentoo-r5 i686)
=================================================================
System uname: Linux-2.6.36-gentoo-r5-i686-Intel-R-_Pentium-R-_4_CPU_3.00GHz-with-gentoo-1.12.14
Timestamp of tree: Tue, 26 Apr 2011 00:45:01 +0000
app-shells/bash:     4.1_p9
dev-java/java-config: 2.1.11-r3
dev-lang/python:     2.7.1-r1, 3.1.3-r1
dev-util/cmake:      2.8.1-r2
sys-apps/baselayout: 1.12.14-r1
sys-apps/sandbox:    2.4
sys-devel/autoconf:  2.13, 2.65-r1
sys-devel/automake:  1.10.3, 1.11.1
sys-devel/binutils:  2.20.1-r1
sys-devel/gcc:       4.4.5
sys-devel/gcc-config: 1.4.1
sys-devel/libtool:   2.2.10
sys-devel/make:      3.81-r2
sys-kernel/linux-headers: 2.6.36.1
virtual/os-headers:  0
ACCEPT_KEYWORDS="x86"
ACCEPT_LICENSE="* -@EULA"
CBUILD="i686-pc-linux-gnu"
CFLAGS="-O2 -march=native -pipe"
CHOST="i686-pc-linux-gnu"
CONFIG_PROTECT="/etc /etc/env.d/50glib2 /usr/share/gnupg/qualified.txt /var/bind"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/env.d/java/ /etc/fonts/fonts.conf /etc/gconf /etc/php/apache2-php5.2/ext-active/ /etc/php/cgi-php5.2/ext-active/ /etc/php/cli-php5.2/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c"
CXXFLAGS="-O2 -march=native -pipe"
DISTDIR="/usr/portage/distfiles"
FEATURES="assume-digests binpkg-logs buildsyspkg collision-protect distlocks fixlafiles fixpackages news parallel-fetch protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch"
FFLAGS=""
GENTOO_MIRRORS="http://mirror.yandex.ru/gentoo-distfiles/ 		ftp://mirror.yandex.ru/gentoo-distfiles/                 http://ftp.chg.ru/pub/Linux/distributions/gentoo/                 ftp://ftp.chg.ru/pub/Linux/distributions/gentoo/ 		http://ftp.corbina.net/pub/Linux/gentoo/ 		ftp://ftp.corbina.net/pub/Linux/gentoo/"
LANG="ru_RU.KOI8-R"
LC_ALL=""
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
LINGUAS="ru en"
MAKEOPTS="-j2"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY="/var/lib/layman/rion /usr/local/portage"
SYNC="rsync://rsync.gentoo.org/gentoo-portage"
USE="X a52 ac3 acl alsa avi berkdb bold bzip2 cdr cli cracklib crypt cups cxx dbus djvu dri dvd flac fortran gdbm gif gtk iconv jpeg jpeg2k modules mp3 mudflap ncurses nls nptl nptlonly ogg openmp pam pcre pdf perl png pppd python qt3support readline session ssl sysfs tcpd tiff unicode utf8 vorbis x86 xcb xorg xulrunner zlib" ALSA_CARDS="intel8x0" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ubx" INPUT_DEVICES="keyboard mouse" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LINGUAS="ru en" PHP_TARGETS="php5-3" RUBY_TARGETS="ruby18" USERLAND="GNU" VIDEO_CARDS="sis" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account" 
Unset:  CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS

[Comment 1 Jeroen Roovers (RETIRED) 2011-04-28 20:35:24 UTC]

Are you arguing that emerge/emerge-webrsync should sanitise that variable's value?

[Comment 2 Sergey S. Starikoff 2011-04-29 05:31:02 UTC]

(In reply to comment #1)
> Are you arguing that emerge/emerge-webrsync should sanitise that variable's
> value?

In the default configuration (SYNC variable is not set in /etc/make.conf) both emerge --sync and emerge-webrsync are operable (emerge --sync uses default mirrors).

If I set in make.conf custom (local) SYNC mirror, emerge-webrsync becomes unoperable.
With rather strange error message (SYNC variable set in make.conf is good and worsk fine with emerge --sync).

To my mind this behaviour is incorrect.

Perhaps (I'm not familiar with portage development) by sanitizing SYNC variable from make.conf.

[Comment 3 Zac Medico 2011-06-09 14:45:11 UTC]

(In reply to comment #2)
> If I set in make.conf custom (local) SYNC mirror, emerge-webrsync becomes
> unoperable.

It checks for rsync protocol:

http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=fa651f44178fa2d7a6e10f6ce2be7b36217022ca

Are you using some other protocol for your SYNC setting?

[Comment 4 Sergey S. Starikoff 2011-06-10 09:54:42 UTC]

(In reply to comment #3)
> Are you using some other protocol for your SYNC setting?
No.
I'm using Gentoo GNU/Linux.
With stardart (rsync) portage tree access protocol.

$ grep SYNC /etc/make.conf
#SYNC="\ \ \ rsync://mirror.yandex.ru/gentoo-portage/"

(commented because now I have to use emerge-webrsync)

[Comment 5 Zac Medico 2011-06-10 10:01:57 UTC]

(In reply to comment #4)
> #SYNC="\ \ \ rsync://mirror.yandex.ru/gentoo-portage/"

Why does your SYNC variable start with "\ \ \ "? I guess that's why the emerge-webrsync code doesn't recognize that you are using the rsync protocol.

[Comment 6 Sergey S. Starikoff 2011-06-25 11:09:59 UTC]

(In reply to comment #5)
> (In reply to comment #4)
> > #SYNC="\ \ \ rsync://mirror.yandex.ru/gentoo-portage/"
> 
> Why does your SYNC variable start with "\ \ \ "? I guess that's why the
> emerge-webrsync code doesn't recognize that you are using the rsync protocol.

To my mind it also seems strange :)

But Handbook (mirrorselect -i -r -o >> /mnt/gentoo/etc/make.conf command) tells that it must be so.
And my own experience prompts the fact.

[Comment 7 Zac Medico 2011-06-25 11:21:22 UTC]

I guess it's a mirrorselect bug. You should be able to safely remove the backslashes and spaces.

[Comment 8 Sergey S. Starikoff 2011-06-27 05:03:32 UTC]

(In reply to comment #7)
> I guess it's a mirrorselect bug. You should be able to safely remove the
> backslashes and spaces.

I've removed backslashes and spaces and check operability with updated make.conf.
Everything works as it should.

Am I to open a new bug about mirrorselect SYNC variable format?

[Comment 9 Zac Medico 2011-06-27 05:47:34 UTC]

(In reply to comment #8)
> Am I to open a new bug about mirrorselect SYNC variable format?

Yes, please do. That "\ \ \ " seems like an obvious bug. There's no reason for it to do that.

[Comment 10 Sergey S. Starikoff 2011-06-28 07:00:21 UTC]

(In reply to comment #9)
> (In reply to comment #8)
> > Am I to open a new bug about mirrorselect SYNC variable format?
> 
> Yes, please do. That "\ \ \ " seems like an obvious bug. There's no reason for
> it to do that.


While exploreing mirrorselect behaviour (bug #373195) I've find that it reproduces prefix from actual /etc/make.conf (if SYNC variable in make.conf contains prefix "\ \ \ " mirror select reproduces it, if not --- it display correct string).

The issue is completely explored. The bug should be closed.
But I'm nos shure about resolution (because it contains some, hope useful, improvement).
Could you do it?

[Comment 11 Zac Medico 2011-06-28 08:07:58 UTC]

The reason that your odd SYNC setting doesn't cause emerge --sync to fail is that the code which portage uses to parse make.conf (varexpand function) removes the backslashes. This behavior is not compatible with bash, so I'd like to fix to preserve the backslashes like bash does.

[Comment 12 Zac Medico 2011-06-28 09:14:59 UTC]

It's fixed for bash compatibility in git:

http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=e63823dd8358f50559fa616313cdde3ceaf104ed

[Comment 13 Zac Medico 2011-07-01 05:06:04 UTC]

(In reply to comment #12)
> It's fixed for bash compatibility in git:
> 
> http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=e63823dd8358f50559fa616313cdde3ceaf104ed

This is in 2.2.0_alpha42, but I'll leave this bug open until it's in an unmasked release.

[Comment 14 Martin Väth 2011-07-01 08:23:09 UTC]

With the new behavior the comment in make.conf.example is false:
The FEATURES variable now must not contain "\" at the end of lines.

[Comment 15 Zac Medico 2011-07-01 08:49:00 UTC]

(In reply to comment #14)
> With the new behavior the comment in make.conf.example is false:
> The FEATURES variable now must not contain "\" at the end of lines.

That's a regression, fixed in git now:

http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=799c576de2afb31cea67cb2b186051bcdae29b1e

[Comment 16 Zac Medico 2011-07-01 13:39:54 UTC]

(In reply to comment #15)
> (In reply to comment #14)
> > With the new behavior the comment in make.conf.example is false:
> > The FEATURES variable now must not contain "\" at the end of lines.
> 
> That's a regression, fixed in git now:
> 
> http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=799c576de2afb31cea67cb2b186051bcdae29b1e

That's fixed in 2.2.0_alpha43 (bug 373703).

[Comment 17 Zac Medico 2011-07-03 17:31:50 UTC]

This is fixed in 2.1.10.4.