Every time there is a package move portage runs all package moves that are in the repo: Performing Global Updates (Could take a couple of minutes if you have a lot of binary packages.) .='update pass' *='binary update' #='/var/db update' @='/var/db move' s='/var/db SLOT move' %='binary move' S='binary SLOT move' p='update /etc/portage/package.*' /usr/portage/profiles/updates/1Q-2010....................................................................... /usr/portage/profiles/updates/2Q-2010....................................... /usr/portage/profiles/updates/3Q-2010............ /usr/portage/profiles/updates/4Q-2010............................ /usr/portage/profiles/updates/1Q-2011........................................................................... /usr/portage/profiles/updates/2Q-2011................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................. ........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................... /usr/portage/profiles/updates/3Q-2011.......................... /usr/portage/profiles/updates/4Q-2011............................................................................................................. /usr/portage/profiles/updates/1Q-2012....................... /usr/portage/profiles/updates/2Q-2012........................ /usr/portage/profiles/updates/3Q-2012..................................................... /usr/portage/profiles/updates/4Q-2012............................................................. /usr/portage/profiles/updates/1Q-2013.................................................................................................. /usr/portage/profiles/updates/2Q-2013............... /usr/portage/profiles/updates/3Q-2013............ /usr/portage/profiles/updates/4Q-2013...... /usr/portage/profiles/updates/1Q-2014....... /usr/portage/profiles/updates/2Q-2014.. /usr/portage/profiles/updates/3Q-2014................ /usr/portage/profiles/updates/4Q-2014............. /usr/portage/profiles/updates/1Q-2015................................................................... /usr/portage/profiles/updates/2Q-2015................................................................................................................................................................................................................................................. /usr/portage/profiles/updates/3Q-2015........................... /usr/portage/profiles/updates/4Q-2015............................................. /usr/portage/profiles/updates/1Q-2016.......................... Portage 2.2.27 (python 3.4.3-final-0, default/linux/amd64/13.0, gcc-5.3.0, glibc-2.22-r2, 4.1.15-gentoo-r1 x86_64) ================================================================= System uname: Linux-4.1.15-gentoo-r1-x86_64-Intel-R-_Xeon-R-_CPU_E31275_@_3.40GHz-with-gentoo-2.2 KiB Mem: 32799736 total, 2073660 free KiB Swap: 0 total, 0 free Timestamp of repository gentoo: Tue, 08 Mar 2016 08:00:01 +0000 sh bash 4.3_p42-r2 ld GNU ld (Gentoo 2.25.1 p1.1) 2.25.1 app-shells/bash: 4.3_p42-r2::gentoo dev-lang/perl: 5.22.1::gentoo dev-lang/python: 2.7.11-r2::gentoo, 3.4.3-r7::gentoo dev-util/pkgconfig: 0.29.1::gentoo sys-apps/baselayout: 2.2::gentoo sys-apps/openrc: 0.20.5::gentoo sys-apps/sandbox: 2.10-r1::gentoo sys-devel/autoconf: 2.69-r1::gentoo sys-devel/automake: 1.11.6::gentoo, 1.12.6::gentoo, 1.13.4-r1::gentoo, 1.14.1-r1::gentoo, 1.15-r2::gentoo sys-devel/binutils: 2.25.1-r1::gentoo sys-devel/gcc: 4.9.3::gentoo, 5.3.0::gentoo sys-devel/gcc-config: 1.8-r1::gentoo sys-devel/libtool: 2.4.6-r2::gentoo sys-devel/make: 4.1-r1::gentoo sys-kernel/linux-headers: 4.4::gentoo (virtual/os-headers) sys-libs/glibc: 2.22-r2::gentoo Repositories: gentoo location: /usr/portage sync-type: rsync sync-uri: rsync://rsync.gentoo.org/gentoo-portage priority: -1000 x-overlay location: /overlay masters: gentoo priority: 0 ACCEPT_KEYWORDS="amd64 ~amd64" ACCEPT_LICENSE="* -@EULA" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-O2 -pipe" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/share/gnupg/qualified.txt /var/bind" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo" CXXFLAGS="-O2 -pipe" DISTDIR="/usr/portage/distfiles" EMERGE_DEFAULT_OPTS="--with-bdeps=y" FCFLAGS="-O2 -pipe" FEATURES="assume-digests binpkg-logs buildpkg config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync parallel-fetch preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr" FFLAGS="-O2 -pipe" GENTOO_MIRRORS="http://distfiles.gentoo.org" INSTALL_MASK="/lib/systemd /lib32/systemd /lib64/systemd /usr/lib/systemd /usr/lib32/systemd /usr/lib64/systemd /etc/systemd" LANG="en_US.utf8" LDFLAGS="-Wl,-O1 -Wl,--as-needed" MAKEOPTS="-j4" PKGDIR="/usr/portage/packages" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git" PORTAGE_TMPDIR="/var/tmp" USE="acl amd64 berkdb bindist bzip2 cli cracklib crypt cxx dri fortran gdbm iconv ipv6 mmx mmxext modules multilib ncurses nls nptl openmp pam pcre readline seccomp session sse sse2 ssl tcpd udev unicode xattr zlib" ABI_X86="64" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump author" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="mmx mmxext sse sse2" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="keyboard mouse evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-5" PYTHON_SINGLE_TARGET="python2_7" PYTHON_TARGETS="python2_7 python3_4" RUBY_TARGETS="ruby20 ruby21" USERLAND="GNU" VIDEO_CARDS="fbdev glint intel mach64 mga nouveau nv r128 radeon savage sis tdfx trident vesa via vmware dummy v4l" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account" Unset: CC, CPPFLAGS, CTARGET, CXX, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, USE_PYTHON
When something moves, the moved package might be matched by a later move, so portage checks all later moves. The 1Q-2010 file triggered this case recently: https://gitweb.gentoo.org/repo/gentoo.git/commit/profiles/updates?id=e204fe58c26c1fefdbc60e5db92acecf0fdd90df
Well, it repeated the whole thing again last night for me. Which is when Patrick made this bug. So, it is likely something changed again either infra changed the rsync generation scripts for or something... Patrick you know enough about portage to know that this is/was not due to some change in portage code. So why in the blazes did you assign it to us? Also, Those updates haven't been trimmed in a few years either since I did the first time it a few years ago... Time to nuke a couple years off of them. This should be done by the tree cleaners...
(In reply to Brian Dolbec from comment #2) > Well, it repeated the whole thing again last night for me. Which is when > Patrick made this bug. So, it is likely something changed again either > infra changed the rsync generation scripts for or something... > > Patrick you know enough about portage to know that this is/was not due to > some change in portage code. So why in the blazes did you assign it to us? Patrick's description of the problem is inaccurate, but portage's behavior is kind of non-obvious here. Zac's explanation helps.
Well, anyway, I've posted to the -dev list [1] to remove the 2010, 2011 files from the active tree. So in a few days that should be 8 less files to potentially re-process. [1] https://archives.gentoo.org/gentoo-dev/message/3f7da6288430319943095ddf678f768e
I just did another sync. <24hrs since my last one... and rsync downloaded these files as part of the sync again: profiles/updates/1Q-2010 profiles/updates/1Q-2016 profiles/updates/4Q-2015 but gitweb shows that the 1Q-2010 file has not changed since Ulrich changed it 6 days ago. https://gitweb.gentoo.org/repo/gentoo.git/log/profiles/updates/1Q-2010 So, what is causing this?
I just nuked the timestamp.chk file to force a re-sync. It did not transfer any files, nor process the updates again. Which to me indicates this problem is not a portage bug, but likely something in the rsync tree generation's handling of the mtimes.
# grep . gentoo-x86-final*/metadata/timestamp ; ls gentoo-x86-final*/profiles/updates/ -lat gentoo-x86-final/metadata/timestamp:Tue Mar 8 23:08:33 UTC 2016 gentoo-x86-final~old~/metadata/timestamp:Tue Mar 8 22:39:39 UTC 2016 gentoo-x86-final~old~/profiles/updates/: total 174 drwxr-xr-x 14 gmirror gmirror 1000 Mar 8 22:38 .. drwxr-xr-x 2 gmirror gmirror 648 Mar 8 22:12 . -rw-r--r-- 1 gmirror gmirror 3231 Mar 8 22:12 1Q-2010 -rw-r--r-- 1 gmirror gmirror 1135 Mar 8 22:12 1Q-2016 -rw-r--r-- 2 gmirror gmirror 1980 Mar 8 22:12 4Q-2015 -rw-r--r-- 2 gmirror gmirror 1253 Oct 5 17:09 3Q-2015 -rw-r--r-- 2 gmirror gmirror 3036 Aug 9 2015 1Q-2011 -rw-r--r-- 2 gmirror gmirror 1036 Aug 9 2015 1Q-2012 -rw-r--r-- 2 gmirror gmirror 5397 Aug 9 2015 1Q-2013 -rw-r--r-- 2 gmirror gmirror 314 Aug 9 2015 1Q-2014 -rw-r--r-- 2 gmirror gmirror 3664 Aug 9 2015 1Q-2015 -rw-r--r-- 2 gmirror gmirror 1626 Aug 9 2015 2Q-2010 -rw-r--r-- 2 gmirror gmirror 58987 Aug 9 2015 2Q-2011 -rw-r--r-- 2 gmirror gmirror 1205 Aug 9 2015 2Q-2012 -rw-r--r-- 2 gmirror gmirror 631 Aug 9 2015 2Q-2013 -rw-r--r-- 2 gmirror gmirror 74 Aug 9 2015 2Q-2014 -rw-r--r-- 2 gmirror gmirror 10752 Aug 9 2015 2Q-2015 -rw-r--r-- 2 gmirror gmirror 496 Aug 9 2015 3Q-2010 -rw-r--r-- 2 gmirror gmirror 1173 Aug 9 2015 3Q-2011 -rw-r--r-- 2 gmirror gmirror 2278 Aug 9 2015 3Q-2012 -rw-r--r-- 2 gmirror gmirror 548 Aug 9 2015 3Q-2013 -rw-r--r-- 2 gmirror gmirror 745 Aug 9 2015 3Q-2014 -rw-r--r-- 2 gmirror gmirror 1200 Aug 9 2015 4Q-2010 -rw-r--r-- 2 gmirror gmirror 5554 Aug 9 2015 4Q-2011 -rw-r--r-- 2 gmirror gmirror 2529 Aug 9 2015 4Q-2012 -rw-r--r-- 2 gmirror gmirror 211 Aug 9 2015 4Q-2013 -rw-r--r-- 2 gmirror gmirror 636 Aug 9 2015 4Q-2014 gentoo-x86-final/profiles/updates/: total 174 drwxr-xr-x 14 gmirror gmirror 1000 Mar 8 23:07 .. -rw-r--r-- 1 gmirror gmirror 3231 Mar 8 22:27 1Q-2010 -rw-r--r-- 1 gmirror gmirror 1135 Mar 8 22:27 1Q-2016 drwxr-xr-x 2 gmirror gmirror 648 Mar 8 22:12 . -rw-r--r-- 2 gmirror gmirror 1980 Mar 8 22:12 4Q-2015 -rw-r--r-- 2 gmirror gmirror 1253 Oct 5 17:09 3Q-2015 -rw-r--r-- 2 gmirror gmirror 3036 Aug 9 2015 1Q-2011 -rw-r--r-- 2 gmirror gmirror 1036 Aug 9 2015 1Q-2012 -rw-r--r-- 2 gmirror gmirror 5397 Aug 9 2015 1Q-2013 -rw-r--r-- 2 gmirror gmirror 314 Aug 9 2015 1Q-2014 -rw-r--r-- 2 gmirror gmirror 3664 Aug 9 2015 1Q-2015 -rw-r--r-- 2 gmirror gmirror 1626 Aug 9 2015 2Q-2010 -rw-r--r-- 2 gmirror gmirror 58987 Aug 9 2015 2Q-2011 -rw-r--r-- 2 gmirror gmirror 1205 Aug 9 2015 2Q-2012 -rw-r--r-- 2 gmirror gmirror 631 Aug 9 2015 2Q-2013 -rw-r--r-- 2 gmirror gmirror 74 Aug 9 2015 2Q-2014 -rw-r--r-- 2 gmirror gmirror 10752 Aug 9 2015 2Q-2015 -rw-r--r-- 2 gmirror gmirror 496 Aug 9 2015 3Q-2010 -rw-r--r-- 2 gmirror gmirror 1173 Aug 9 2015 3Q-2011 -rw-r--r-- 2 gmirror gmirror 2278 Aug 9 2015 3Q-2012 -rw-r--r-- 2 gmirror gmirror 548 Aug 9 2015 3Q-2013 -rw-r--r-- 2 gmirror gmirror 745 Aug 9 2015 3Q-2014 -rw-r--r-- 2 gmirror gmirror 1200 Aug 9 2015 4Q-2010 -rw-r--r-- 2 gmirror gmirror 5554 Aug 9 2015 4Q-2011 -rw-r--r-- 2 gmirror gmirror 2529 Aug 9 2015 4Q-2012 -rw-r--r-- 2 gmirror gmirror 211 Aug 9 2015 4Q-2013 -rw-r--r-- 2 gmirror gmirror 636 Aug 9 2015 4Q-2014
No changes to the scripts for nearly 2 weeks now: https://gitweb.gentoo.org/infra/mastermirror-scripts.git/
It appears to be the result of more github PR's being merged by devs without checking, rebasing and/or having the user rebase before being merged. This causes the timestamps to change causing a re-download of unchanged files. and also a re-processing of the updates. If we change portage to perform a checksum on the updates files instead of checking mtimes with the recorded value of the last run of them. It will prevent them from being re-processed, but not from being needlessly re-downloaded.
Marking this as IN_PROGRESS because we're working on fixing it from the infra side. Git's behavior when showing merges using the raw diff format can result in really weird output when a user contribution is based on a really old checkout. This weird output then results in the mtime of files getting bumped that really shouldn't be. Fix should be completely in place within a day or so, but will trigger one more round of portage processing all the package update files as a result of changing mtimes.
Infra change is in place; but I do support keeping this bug open until Portage verifies checksums to see if there was a content change in the files.
