Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 388345 - prefix rsync tree infrastructure is faulty
Summary: prefix rsync tree infrastructure is faulty
Status: RESOLVED FIXED
Alias: None
Product: Gentoo/Alt
Classification: Unclassified
Component: Prefix Support (show other bugs)
Hardware: All Solaris
: Normal normal (vote)
Assignee: Gentoo Prefix
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2011-10-24 15:17 UTC by Jens
Modified: 2011-10-26 15:30 UTC (History)
0 users

See Also:
Package list:
Runtime testing required: ---


Attachments
build.log (build.log,6.86 KB, text/plain)
2011-10-24 15:17 UTC, Jens
Details
environment (environment,98.46 KB, text/plain)
2011-10-24 15:17 UTC, Jens
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Jens 2011-10-24 15:17:22 UTC
Created attachment 290719 [details]
build.log

emerge -pqv =sys-process/procps-3.2.8-r2
[ebuild  N    ] sys-process/procps-3.2.8-r2  USE="unicode (-n32)" 



Compiling on Solaris 11, the include <sys/dir.h> won't be found. After adding the include found at [1] to $EPREFIX/usr/include/sys I get a bit further and stumble upon

...
proc/sysinfo.c: In function 'find_elf_note':
proc/sysinfo.c:191: error: 'environ' undeclared (first use in this function)
...
In file included from proc/ksym.c:24:
proc/sysinfo.h:65: error: conflicting types for 'meminfo'
/usr/include/sys/mman.h:253: error: previous declaration of 'meminfo' was here
make: *** [proc/ksym.o] Error 1


See the attached build output for more info.

[1]
See <http://www.linuxquestions.org/questions/solaris-opensolaris-20/hwere-can-i-find-sys-dir-h-in-solaris-11-express-866677/>
Comment 1 Jens 2011-10-24 15:17:44 UTC
Created attachment 290721 [details]
environment
Comment 2 Jens 2011-10-24 15:19:38 UTC
# emerge --info =sys-process/procps-3.2.8-r2
Portage 2.2.01.19572-prefix (prefix/sunos/solaris/5.11/x86, gcc-4.2.4, unavailable, 5.11 i86pc)
=================================================================
                        System Settings
=================================================================
System uname: Solaris-2.11-i86pc-i386-32bit-ELF
Timestamp of tree: Sun, 23 Oct 2011 17:39:46 +0000
app-shells/bash:      4.2_p10
dev-lang/python:      2.7.2
dev-util/pkgconfig:   0.25-r2
sys-devel/autoconf:   2.68
sys-devel/automake:   1.11.1
sys-devel/binutils:   2.20.1-r1
sys-devel/gcc:        4.2.4-r01.4
sys-devel/gcc-config: 1.4.1-r00.2
sys-devel/libtool:    2.4-r01.1
sys-devel/make:       3.82
Repositories: gentoo_prefixInstalled sets: 
ACCEPT_KEYWORDS="~x86-solaris"
ACCEPT_LICENSE="* -@EULA"
CBUILD="i386-pc-solaris2.11"
CFLAGS="-O2 -pipe"
CHOST="i386-pc-solaris2.11"
CONFIG_PROTECT="/etc"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/gconf /etc/portage /etc/revdep-rebuild /etc/terminfo"CXXFLAGS="-O2 -pipe"
DISTDIR="/gentoo/usr/portage/distfiles"
FEATURES="assume-digests binpkg-logs collision-protect distlocks ebuild-locks fixlafiles news nostrip parallel-fetch preserve-libs protect-owned sfperms strict unknown-features-warn unmerge
-logs unmerge-orphans userfetch"
FFLAGS=""
GENTOO_MIRRORS="http://distfiles.gentoo.org"LDFLAGS=""MAKEOPTS="-j3"PKGDIR="/gentoo/usr/portage/packages"PORTAGE_CONFIGROOT="/gentoo/"PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"PORTAGE_TMPDIR="/gentoo/var/tmp"
PORTDIR="/gentoo/usr/portage"
PORTDIR_OVERLAY=""
SYNC="rsync://rsync.prefix.freens.org/gentoo-portage-prefix"

USE="aac cracklib cxx flac lame modules ncurses nls ogg prefix readline ssl unicode x86-solaris zlib" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan stage tables krita karbon braindump" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="SunOS" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ubx" INPUT_DEVICES="keyboard mouse" KERNEL="SunOS" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" PHP_TARGETS="php5-3" RUBY_TARGETS="ruby18" USERLAND="GNU" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LANG, LC_ALL, LINGUAS, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS
Comment 3 Fabian Groffen gentoo-dev 2011-10-24 16:49:53 UTC
Isn't procps highly linux specific?  I doubt whether it's worth investing any time in it.  If you need top, use unixtop.
Comment 4 Jens 2011-10-24 18:18:19 UTC
(In reply to comment #3)

Fabian,

> Isn't procps highly linux specific?  I doubt whether it's worth investing any
> time in it.  If you need top, use unixtop.

It is a dependency for dev-db/mysql-5.1.59
Comment 5 Fabian Groffen gentoo-dev 2011-10-24 18:21:17 UTC
ah, that means the problem with this dep is still not solved :/

*** This bug has been marked as a duplicate of bug 385001 ***
Comment 6 Jens 2011-10-24 18:23:26 UTC
Fabian,

thanks for he pointer

Jens
Comment 7 Jeremy Olexa (darkside) (RETIRED) archtester gentoo-dev Security 2011-10-24 18:50:03 UTC
metadata cache is flapping WHILE I'm looking at it...

This file: http://rsync1.prefix.freens.org/metadata/cache/dev-db/mysql-5.1.59

flaps between kernel_linux and userland_GNU deps. Please look onto the health of rsync0, grobian.
Comment 8 Fabian Groffen gentoo-dev 2011-10-24 20:39:13 UTC
Seems we were hit by files with the same size, same modification dates, but different contents.  All files should be zapped now, which means in an hour or two, the rsync slaves should catch this up, and this dreaded bug should be gone.
Comment 9 Jeremy Olexa (darkside) (RETIRED) archtester gentoo-dev Security 2011-10-25 18:29:33 UTC
(In reply to comment #8)
> Seems we were hit by files with the same size, same modification dates, but
> different contents.  All files should be zapped now, which means in an hour or
> two, the rsync slaves should catch this up, and this dreaded bug should be
> gone.

And now, similarily, you need to get all the nodes to use rsync --checksum too.

Example:
http://rsync1.prefix.freens.org/metadata/cache/dev-db/mysql-5.1.59
http://tinderbox.dev.gentoo.org/portage-prefix/metadata/cache/dev-db/mysql-5.1.59
... are different
Comment 10 Fabian Groffen gentoo-dev 2011-10-25 18:43:13 UTC
The problem is actually caused by egencache, because it changes the file, but restores mtime back to the ebuild, when the eclass changed.  Because size is equal, rsync sees no difference.

rsync -c is too slow to use all the time
Comment 11 Jeremy Olexa (darkside) (RETIRED) archtester gentoo-dev Security 2011-10-25 18:48:44 UTC
Take the broken nodes out of the rotation then?

(In reply to comment #10)
> rsync -c is too slow to use all the time

Why? All official rsync.g.o nodes use it all the time.
Comment 12 Fabian Groffen gentoo-dev 2011-10-25 18:53:46 UTC
hmmm, really? nodes synced manually now.  My default PORTAGE_RSYNC_OPTS does not seem to include --checksum, so how would a fix ever get to the clients even though the rsync slaves would get the correct file?
Comment 13 Zac Medico gentoo-dev 2011-10-25 19:21:23 UTC
(In reply to comment #10)
> The problem is actually caused by egencache, because it changes the file, but
> restores mtime back to the ebuild, when the eclass changed.  Because size is
> equal, rsync sees no difference.

This is bug 139134 and it is the reason for the egencache --rsync option (see `man egencache`).
Comment 14 Jeremy Olexa (darkside) (RETIRED) archtester gentoo-dev Security 2011-10-25 19:22:23 UTC
(In reply to comment #12)
> My default PORTAGE_RSYNC_OPTS does
> not seem to include --checksum, so how would a fix ever get to the clients even
> though the rsync slaves would get the correct file?

Interesting question/comment. By using --checksum on ALL internal infra we
validate that the data is always correct until it leaves our infra. Since it
causes more load/slowness we prefer that users do not use it, but they always
can to validate themselves.
Comment 15 Fabian Groffen gentoo-dev 2011-10-25 20:05:48 UTC
hmmm, I use --rsync with egencache, but it keeps the mtime of the ebuilds
Comment 16 Fabian Groffen gentoo-dev 2011-10-25 20:47:03 UTC
(In reply to comment #14)
> (In reply to comment #12)
> > My default PORTAGE_RSYNC_OPTS does
> > not seem to include --checksum, so how would a fix ever get to the clients even
> > though the rsync slaves would get the correct file?
> 
> Interesting question/comment. By using --checksum on ALL internal infra we
> validate that the data is always correct until it leaves our infra. Since it
> causes more load/slowness we prefer that users do not use it, but they always
> can to validate themselves.

maybe --whole-file saves the day here, or nobody notices, because userland_GNU vs kernel_linux is of no change to most
Comment 17 Fabian Groffen gentoo-dev 2011-10-26 15:30:38 UTC
Up till rsync1 rsync --checksum is now used, the rsync slaves use the OPTS from the gentoo rsync mirror docs.  Should fix this bug, apart from that it doesn't convince me that the problem won't reoccur now if the same conditions hold.