Hi, Today I'm syncing the portage tree : > 1543567134: Started emerge on: nov. 30, 2018 09:38:53 > 1543567134: *** emerge --keep-going --load-average=3.0 --jobs=3 --with-bdeps=y --sync > 1543567134: === sync > 1543567134: >>> Syncing repository 'gentoo' into '/portage/trees/gentoo'... > 1543567136: >>> Starting rsync with rsync://178.210.128.3/gentoo-portage > 1543567614: === Sync completed for gentoo > 1543567631: *** terminating. Before portage run its own checks using gemato, I got a warning (and it's seems not logged anywhere) saying something like that : > warning : Manifest is over 24 days old, please report this mirror And once the gemato check process finished, I'm having an old portage tree installed (lot of potential packages downgrade). So : 1. 178.210.128.3 is an outdated mirror 2. this important warning should be logged somewhere, but I did not find it. 3. portage should simply abort when trying to sync in such situation. Reproducible: Always $ emerge --info portage Portage 2.3.51 (python 2.7.15-final-0, default/linux/amd64/17.0/desktop, gcc-7.3.0, glibc-2.27-r6, 4.14.83-gentoo x86_64) ================================================================= System Settings ================================================================= System uname: Linux-4.14.83-gentoo-x86_64-Intel-R-_Core-TM-_i5-2500K_CPU_@_3.30GHz-with-gentoo-2.6 KiB Mem: 8138060 total, 4896436 free KiB Swap: 6291444 total, 6291444 free Timestamp of repository gentoo: Fri, 12 Oct 2018 07:45:01 +0000 Head commit of repository gentoo: 473d4aa8ee4032857045d5ae37b8d2caef209eb7 Head commit of repository gnome15: 6c8c18706db1218e532d2c6f29a25f4c32bbb56d Head commit of repository java: 22b1d35cc4be3dc33bbb6592c5989a15e5d8c4b2 Head commit of repository obs-studio-overlay: 986b6a026efbf0cfe46d4cc0d850a80ff79766f8 Head commit of repository steam-overlay: 0db8f021e8c769344a1fdb5c17b914353b03abba Head commit of repository voyageur: f6d7db581213a167d9d09bb8ab184ee56e34b3d3 Head commit of repository perso: 25dcd492932e34a6255c8542019e8cf92cf8bcdd Head commit of repository netfab: 5e4abf3a2b06d95e9b99b5348c8f05f4a3934172 sh bash 4.4_p12 ld GNU ld (Gentoo 2.30 p5) 2.30.0 distcc 3.2rc1 x86_64-pc-linux-gnu [disabled] app-shells/bash: 4.4_p12::gentoo dev-java/java-config: 2.2.0-r4::gentoo dev-lang/perl: 5.24.3-r1::gentoo dev-lang/python: 2.7.15::gentoo, 3.6.5::gentoo dev-util/cmake: 3.9.6::gentoo dev-util/pkgconfig: 0.29.2::gentoo sys-apps/baselayout: 2.6-r1::gentoo sys-apps/openrc: 0.38.3::gentoo sys-apps/sandbox: 2.13::gentoo sys-devel/autoconf: 2.13::gentoo, 2.69-r4::gentoo sys-devel/automake: 1.11.6-r3::gentoo, 1.13.4-r2::gentoo, 1.15.1-r2::gentoo sys-devel/binutils: 2.30-r4::gentoo sys-devel/gcc: 7.3.0-r3::gentoo sys-devel/gcc-config: 1.8-r1::gentoo sys-devel/libtool: 2.4.6-r3::gentoo sys-devel/make: 4.2.1-r4::gentoo sys-kernel/linux-headers: 4.13::gentoo (virtual/os-headers) sys-libs/glibc: 2.27-r6::gentoo Repositories: gentoo location: /portage/trees/gentoo sync-type: rsync sync-uri: rsync://rsync.europe.gentoo.org/gentoo-portage priority: -1000 sync-rsync-verify-jobs: 1 sync-rsync-verify-metamanifest: yes sync-rsync-extra-opts: sync-rsync-verify-max-age: 24 gnome15 location: /portage/trees/gnome15 sync-type: git sync-uri: https://github.com/CMoH/gnome15-overlay.git masters: gentoo priority: 0 java location: /portage/trees/java sync-type: git sync-uri: git://anongit.gentoo.org/proj/java.git masters: gentoo priority: 0 obs-studio-overlay location: /portage/trees/obs-studio sync-type: git sync-uri: https://github.com/saintdev/obs-studio-overlay masters: gentoo priority: 0 steam-overlay location: /portage/trees/steam sync-type: git sync-uri: git://github.com/anyc/steam-overlay.git masters: gentoo priority: 0 voyageur location: /portage/trees/voyageur sync-type: git sync-uri: https://cafarelli.fr/git/voyageur-overlay/ masters: gentoo priority: 0 perso location: /portage/trees/perso sync-type: git sync-uri: /home/depots/gentoo/overlays/perso.git masters: gentoo priority: 9998 netfab location: /portage/trees/netfab sync-type: git sync-uri: /home/depots/gentoo/overlays/netfab.git masters: gentoo priority: 9999 Installed sets: @common-apps, @dev-apps, @games-apps, @kernel-apps, @portage-apps, @qemukvm, @system-apps, @xfce ACCEPT_KEYWORDS="amd64" ACCEPT_LICENSE="* -@EULA" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-march=sandybridge -mtune=generic -O2 -pipe -mmmx -msse -msse2 -msse3 -mssse3 -mcx16 -msahf -mpclmul -mpopcnt -mavx -msse4.2 -msse4.1 -mfxsr -mxsave -mxsaveopt -fstack-protector-strong -mno-3dnow -mno-sse4a -mno-movbe -mno-aes -mno-sha -mno-abm -mno-lwp -mno-fma -mno-fma4 -mno-xop -mno-bmi -mno-bmi2 -mno-tbm -mno-avx2 -mno-lzcnt -mno-rtm -mno-hle -mno-rdrnd -mno-f16c -mno-fsgsbase -mno-rdseed -mno-prfchw -mno-adx -mno-avx512f -mno-avx512er -mno-avx512cd -mno-avx512pf -mno-prefetchwt1 -mno-clflushopt -mno-xsavec -mno-xsaves -mno-avx512dq -mno-avx512bw -mno-avx512vl -mno-avx512ifma -mno-avx512vbmi -mno-clwb -mno-mwaitx -mno-clzero -mno-pku" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/share/gnupg/qualified.txt" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo" CXXFLAGS="-march=sandybridge -mtune=generic -O2 -pipe -mmmx -msse -msse2 -msse3 -mssse3 -mcx16 -msahf -mpclmul -mpopcnt -mavx -msse4.2 -msse4.1 -mfxsr -mxsave -mxsaveopt -fstack-protector-strong -mno-3dnow -mno-sse4a -mno-movbe -mno-aes -mno-sha -mno-abm -mno-lwp -mno-fma -mno-fma4 -mno-xop -mno-bmi -mno-bmi2 -mno-tbm -mno-avx2 -mno-lzcnt -mno-rtm -mno-hle -mno-rdrnd -mno-f16c -mno-fsgsbase -mno-rdseed -mno-prfchw -mno-adx -mno-avx512f -mno-avx512er -mno-avx512cd -mno-avx512pf -mno-prefetchwt1 -mno-clflushopt -mno-xsavec -mno-xsaves -mno-avx512dq -mno-avx512bw -mno-avx512vl -mno-avx512ifma -mno-avx512vbmi -mno-clwb -mno-mwaitx -mno-clzero -mno-pku" DISTDIR="/portage/distpack/distfiles" EMERGE_DEFAULT_OPTS="--with-bdeps=y --jobs 3 --load-average 3 --keep-going" ENV_UNSET="DBUS_SESSION_BUS_ADDRESS DISPLAY PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR" FCFLAGS="-O2 -pipe" FEATURES="assume-digests binpkg-logs buildpkg clean-logs config-protect-if-modified distlocks ebuild-locks fail-clean fixlafiles merge-sync multilib-strict news parallel-fetch preserve-libs protect-owned sandbox sfperms split-log strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr" FFLAGS="-O2 -pipe" GENTOO_MIRRORS="ftp://ftp.free.fr/mirrors/ftp.gentoo.org/ ftp://mirror.ovh.net/gentoo-distfiles/ http://mirror.ovh.net/gentoo-distfiles/" LANG="fr_FR.utf8" LDFLAGS="-Wl,-O1 -Wl,--as-needed" LINGUAS="en fr" MAKEOPTS="-j5" PKGDIR="/portage/distpack/packages/coreIx" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git" PORTAGE_TMPDIR="/var/tmp" USE="X a52 aac acl acpi alsa amd64 berkdb bluetooth branding bzip2 cairo cdda cdr cli consolekit crypt cups cxx dbus dri dts dvd dvdr emboss encode exif fam ffmpeg flac fortran gdbm gif glamor gnutls gstreamer gtk gtk3 hddtemp iconv id3tag ipv6 java jpeg lcms libnotify libtirpc lm_sensors mad matroska mng mp3 mp4 mpeg multilib ncurses nls nptl nsplugin ogg opengl openmp pam pango pcre pdf png policykit ppds pulseaudio qt5 readline sdl seccomp sox spell ssl startup-notification steamruntime svg tcpd theora tiff truetype udev udisks unicode upower usb v4l vaapi vdpau vorbis wxwidgets x264 xattr xcb xml xv xvid zlib" ABI_X86="64 32" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon plan sheets stage words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="avx mmx mmxext popcnt sse sse2 sse3 sse4_1 sse4_2 ssse3" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" GRUB_PLATFORMS="pc" INPUT_DEVICES="evdev" KERNEL="linux" L10N="en fr" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-6 php7-1" POSTGRES_TARGETS="postgres9_5 postgres10" PYTHON_SINGLE_TARGET="python3_6" PYTHON_TARGETS="python2_7 python3_6" QEMU_SOFTMMU_TARGETS="x86_64 i386" QEMU_USER_TARGETS="x86_64 i386" RUBY_TARGETS="ruby23" USERLAND="GNU" VIDEO_CARDS="nouveau nvidia" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account" Unset: CC, CPPFLAGS, CTARGET, CXX, INSTALL_MASK, LC_ALL, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS ================================================================= Package Settings ================================================================= sys-apps/portage-2.3.51-r1::gentoo was built with the following: USE="(ipc) native-extensions rsync-verify xattr -build -doc -epydoc -gentoo-dev (-selinux)" ABI_X86="(64)" PYTHON_TARGETS="python2_7 python3_6 (-pypy) -python3_4 -python3_5 (-python3_7)"
I also ran into this once again. Was wondering about the many downgrades. According to the mirrorstats https://mirrorstats.gentoo.org/rsync/ there are quite some mirrors behind. I also think, that portage should reject to downgrade the tree and print a prominent error, also mentioning the used mirror.
(In reply to Markus from comment #1) > I also think, that portage should reject to downgrade the tree and print a > prominent error, also mentioning the used mirror. Normally, comparison of the local and remote metdata/timestamp.chk files should prevent syncing with an older version of the repository. However, the if either the local or remote metdata/timestamp.chk file is missing the there's no protection. Could that be the cause of your problem? For reference, the timestamp comparison logic is here: https://gitweb.gentoo.org/proj/portage.git/tree/lib/portage/sync/modules/rsync/rsync.py?h=portage-2.3.51#n657
Created attachment 557386 [details] emerge --sync from old mirror Sync is not prevented. Any ideas?
I added a debug print: servertimestamp and timestamp are both always 0 at that point. For a good and for a bad mirror. outdated: # dir /usr/portage/metadata/timestamp* -rw-r--r-- 1 root root 29 2018-12-02 13:09 /usr/portage/metadata/timestamp -rw-r--r-- 1 root root 32 2018-12-02 13:15 /usr/portage/metadata/timestamp.chk -rw-r--r-- 1 root root 78 2018-12-02 13:09 /usr/portage/metadata/timestamp.commit -rw-r--r-- 1 root root 43 2018-12-02 13:09 /usr/portage/metadata/timestamp.x current: # dir /usr/portage/metadata/timestamp* -rw-r--r-- 1 root root 29 2018-12-09 10:39 /usr/portage/metadata/timestamp -rw-r--r-- 1 root root 32 2018-12-09 10:45 /usr/portage/metadata/timestamp.chk -rw-r--r-- 1 root root 78 2018-12-09 10:09 /usr/portage/metadata/timestamp.commit -rw-r--r-- 1 root root 43 2018-12-09 10:39 /usr/portage/metadata/timestamp.x
Catching the exception in line 652: ValueError: time data 'Sun, 02 Dec 2018 12:15:01 +0000' does not match format '%a, %d %b %Y %H:%M:%S +0000' That being said, the follwing works as expected: LC_ALL=C emerge --sync
So it needs to be worked around the locale dependency. Different ideas come to my mind: 1) Set locale to "C" before all "strptime" calls and restore afterwards. (Can can messy to catch every case that is locale dependend but shouldnt. Threadsafe?) 2) Get timestamp from metadata of file. (Dont know if this is guaranteed?) 3) Use an universial timestamp without a locale dependency. E.g. "%Y-%m-%d %H:%M:%S" or just "seconds since the epoch". (More work, as the whole interface around the timestamp file changes.)
this one seems to have hit me during the past week, as rsync5.de.gentoo.org is outdated since about early Nov. 14.. (<- yes, I've already contacted the maintainer about the mirror issue) however, is this bug being actually worked on to globally fix this in Portage..?
We certainly want to skip sync with the mirror if timestamp.chk is corrupt. If timestamp.chk is missing entirely, we might need a repos.conf flag to control the behavior, for example we could use a setting like sync-rsync-check-timestamp = false to make it ignore the timestamp.chk file.
