Since generation of /var/tmp/gmirror/metadata/md5-cache was enabled yesterday on kookaburra for bug #409445, I've received a report from Jeff Horelick that the files in this directory are being transferred unnecessarily by emerge --sync. This would suggest that /var/tmp/gmirror/metadata/md5-cache is being deleted and regenerated unecessarily, causing the timestamps to change when it's regenerated. This is likely due to an rsync job that syncs the tree from flycatcher to kookaburra. In order to fix this, the rsync job should have --exclude=/metdata/md5-cache added to its arguments.
Does egencache remove entries for which the ebuilds have been removed? Prefix rsync0 seems to suffer from the same problem.
kookaburra ~ # grep rsync /usr/local/bin/mastermirror/rsync-gen.sh # 1) Update rsync checkouts or exports from cvs.g.o::mastermirror-staging # 1b) rsync EXPORTS/gentoo-x86 to STAGEDIR # 7) rsync from STAGEDIR to FINALDIR source /usr/local/bin/mastermirror/rsync-gen.vars TIMESLOG="${LOGDIR}/rsync-gen-times.log" # 1) Update rsync checkouts or exports from cvs.g.o::mastermirror-staging PASSWD_FILE="/etc/mastermirror-fetch/gcvsd-rsync.rsync.passwd" RSYNC="/usr/bin/rsync" # 1b) rsync EXPORTS/gentoo-x86 to STAGEDIR # Keep /metadata/cache around so the --rsync switch of egencache will work as rsync -Wqa --exclude=CVS --exclude=/metadata/timestamp.x \ egencache --update --rsync --jobs=4 --tolerant --cache-dir=${BASE}/tmp/ \ rsync -Wqa --exclude=CVS ${EXPORTS}/dtd ${STAGEDIR}/metadata/ rsync -Wqa --exclude=CVS ${EXPORTS}/glsa ${STAGEDIR}/metadata/ rsync -Wqa --exclude '.git*' --delete "${news_item}" ${STAGEDIR}/metadata/news/ rsync -Wqa ${EXPORTS}/herds/herds.xml ${STAGEDIR}/metadata/herds.xml # 7) rsync from STAGEDIR to FINALDIR # where all the rsync nodes pull from. rsync -Wqa --exclude=/metadata/timestamp.chk --delete ${STAGEDIR}/ ${FINALDIR}/ which rsync call should get it added?
Created attachment 307453 [details] rsync-gen.sh
(In reply to comment #1) > Does egencache remove entries for which the ebuilds have been removed? Yes. It also does it for the metadata/cache directory. Also, it's worth noting that if you regenerate the metadata/cache directory then the prevents the egencache --rsync option from working. So, all together, the relevant rsync options for flycatcher to kookaburra sync are: --exclude=/metdata/md5-cache --exclude=/metdata/cache (In reply to comment #2) > which rsync call should get it added? This one: rsync -Wqa --exclude=CVS --exclude=/metadata/timestamp.x \ --exclude=/metadata/cache --delete \ ${EXPORTS}/gentoo-x86/ ${STAGEDIR}/ || exit
InGit, will auto-deploy in 15 mins
This bug is not fixed; the unnesessary re-syncing of every file under md5-cache continues. Please re-open.
(In reply to comment #6) > This bug is not fixed; the unnesessary re-syncing of every file under > md5-cache continues. An easy way to verify is with a command like this after sync: find $PORTDIR/metadata/md5-cache -type f -mtime +1 | wc -l If it's regenerating every cache entry every 30 minutes, then none of those files will be matched by the above find command. I've tried it here on my local rsync tree and there seem to be a good number of files that have older mtimes.
