There are a few rsync server in Japan. So I build new rsync server. server name :rsync.gentoo.gr.jp IP address :210.138.60.45 administrator :MATSUDA Kazuo e-mail address:egg@lostbutterfly.net or num-o@bp.iij4u.or.jp Reproducible: Always Steps to Reproduce: 1. 2. 3.
Just did a check on ::gentoo-portage/metadata/timestamp.chk, which reports the last sync at 16:01 UTC. This is almost 2 hours ago. Can you verify that you're syncing every 30 minutes at :00 and :30 intervals?
I run rsync script every 30 minutes at :00 and :30. And result messages of executing the script have no failure. The script is ------- BEGIN -------- #!/usr/local/bin/bash RSYNC="/usr/local/bin/rsync" OPTS="--quiet --recursive --links --perms --times --compress --delete --timeout=600" SRC="rsync://rsync2.de.gentoo.org/gentoo-portage" DST="/home/dist/rsync/portage" LOCKFILE="/home/dist/rsync/lock/portage.lock" if [ -f $LOCKFILE ] then logger -t rsync "re-rsyncing the gentoo-portage tree already started!" exit 0 fi touch $LOCKFILE echo "Started update at" `date` >> /home/dist/log/rsync-portage.log 2>&1 logger -t rsync "start re-rsyncing the gentoo-portage tree" ${RSYNC} ${OPTS} ${SRC} ${DST} >> /home/dist/log/rsync-portage.log 2>&1 echo "End: "`date` >> /home/dist/log/rsync-portage.log 2>&1 rm $LOCKFILE logger -t rsync "end re-rsyncing the gentoo-portage tree" exit 0 ------- END -------- Is this script wrong?
Looks like this server is missing the :30 rsync. Have you timed to see how long an rsync normally takes? If it takes more than just a few minutes, you might be overlapping rsyncs... Here's the output of the latest test we ran: kurtl@x5 trashme2 $ rsync -avz 210.138.60.45::gentoo-portage/metadata/timestamp.chk ./ This rsync server locate rsync.gentoo.gr.jp(210.138.60.45). It's first server in Tokyo Japan! ____ (____) O O _ _ _ ____ _ _____ ___ _ | || || | _________________ (____) | | (_ _) (___)| | |_||_|| | (_G_e_n_t_o_o_J_P_) ______| | _| |_ ____| | ____| | (________| (_____) (______| (______| totemo omoshiroi. mailto:MATSUDA Kazuo<matsuda@gentoo.gr.jp> receiving file list ... done timestamp.chk wrote 125 bytes read 570 bytes 278.00 bytes/sec total size is 32 speedup is 0.05 kurtl@x5 trashme2 $ less timestamp.chk Wed, 09 Apr 2003 16:01:00 +0000 kurtl@x5 trashme2 $ date -u Wed Apr 9 16:45:57 UTC 2003 ----------------- Can you double-check your cron job to make sure it's running every 30 minutes? Thanks.
I corrected the rsync script to get a debug log. ----- BEGIN ----- #!/usr/local/bin/bash RSYNC="/usr/local/bin/rsync" #OPTS="--quiet --recursive --links --perms --times --devices --compress --delete --timeout=600" #OPTS="--quiet --recursive --links --perms --times --compress --delete --timeout=600" #for Debug OPTS="--recursive --links --perms --times --compress --delete --timeout=600 --stats --progress --verbose" #Uncomment the following line only if you have been granted access to rsync1.us.gentoo.org #SRC="rsync://rsync1.us.gentoo.org/gentoo-portage" #If you are waiting for access to our master mirror, select one of our mirrors to mirror from: SRC="rsync://rsync2.de.gentoo.org/gentoo-portage" DST="/home/gentoosync/rsync/portage" LOCKFILE="/home/gentoosync/rsync/lock/portage.lock" DEBUG_DATE=`date "+%Y%m%d_%H%M"` if [ -f $LOCKFILE ] then logger -t rsync "re-rsyncing the gentoo-portage tree already started!" exit 0 fi touch $LOCKFILE echo "Started update at" `date` >> /home/gentoosync/log/rsync-portage.log.$DEBUG_DATE 2>&1 logger -t rsync "start re-rsyncing the gentoo-portage tree" ${RSYNC} ${OPTS} ${SRC} ${DST} >> /home/gentoosync/log/rsync-portage.log.$DEBUG_DATE 2>&1 echo "End: "`date` >> /home/gentoosync/log/rsync-portage.log.$DEBUG_DATE 2>&1 rm $LOCKFILE logger -t rsync "end re-rsyncing the gentoo-portage tree" exit 0 ----- END ----- Result. ----- BEGIN ----- hoge@ns:~/log$ cat rsync-portage.log.20030410_1230 Started update at Thu Apr 10 12:30:00 JST 2003 receiving file list ... 40165 files to consider ./0100 files... app-admin/ app-admin/gentoolkit/files/ app-arch/ app-cdr/ app-crypt/ app-crypt/heimdal/files/ app-doc/ app-editors/ app-emulation/ app-emulation/vmware-workstation/files/ app-games/ app-games/freecraft/files/ app-i18n/ app-i18n/canna/files/ app-i18n/kde-i18n-ga/ app-i18n/kinput2/files/ app-misc/ app-office/ app-office/openoffice-bin/files/ app-office/openoffice/files/ app-pda/ app-sci/ app-text/ app-text/dictd/files/ app-xemacs/ dev-db/ dev-java/ dev-java/ant/files/ dev-java/java-config/files/ dev-lang/ dev-lang/icc/files/ dev-libs/ dev-lisp/ dev-perl/ dev-python/ dev-ruby/ dev-tcltk/ dev-util/ eclass/ gnome-base/ gnome-base/gdm/files/ gnome-extra/ kde-base/ kde-base/kdebase/files/ media-gfx/ media-libs/ media-libs/netpbm/files/ media-plugins/ media-sound/ media-video/ metadata/ metadata/cache/media-video/ net-analyzer/ net-analyzer/nagios/ net-dialup/ net-dialup/isdn4k-utils/files/ net-fs/ net-ftp/ net-im/ net-irc/ net-libs/ net-mail/ net-mail/qmail/files/ net-mail/sendmail/files/ net-misc/ net-misc/pump/files/ net-p2p/ net-print/ net-wireless/ net-www/ net-www/apache/files/ net-www/orion/files/ net-www/squid/files/ net-www/tomcat/files/ profiles/ profiles/default-sparc64-1.4/ sys-apps/ sys-apps/baselayout/files/ sys-apps/grub/files/ sys-apps/lilo/files/ sys-apps/portage/files/ sys-apps/shadow/files/ sys-apps/tcp-wrappers/files/ sys-apps/yard/files/ sys-cluster/ sys-devel/ sys-devel/gcc/files/ sys-devel/libtool/files/ sys-kernel/ sys-kernel/linux-headers/files/ sys-kernel/mjc-sources/ sys-libs/ sys-libs/cracklib/files/ sys-libs/pam/files/ x11-base/ x11-base/xfree/files/ x11-libs/ x11-libs/ecore/files/ x11-misc/ x11-plugins/ x11-terms/ x11-themes/ x11-wm/ metadata/timestamp 29 100% 0.00kB/s 0:00:00 metadata/timestamp.chk 32 100% 0.00kB/s 0:00:00 metadata/timestamp.x 40 100% 0.00kB/s 0:00:00 Number of files: 40165 Number of files transferred: 3 Total file size: 32989938 bytes Total transferred file size: 101 bytes Literal data: 101 bytes Matched data: 0 bytes File list size: 899945 Total bytes written: 172 Total bytes read: 900193 wrote 172 bytes read 900193 bytes 43920.24 bytes/sec total size is 32989938 speedup is 36.64 End: Thu Apr 10 12:30:20 JST 2003 ----- END ----- ----- BEGIN ----- hoge@ns:~/log$ cat rsync-portage.log.20030410_1300 Started update at Thu Apr 10 13:00:00 JST 2003 receiving file list ... 40165 files to consider ./0100 files... app-admin/ app-admin/gentoolkit/files/ app-arch/ app-cdr/ app-crypt/ app-crypt/heimdal/files/ app-doc/ app-editors/ app-emulation/ app-emulation/vmware-workstation/files/ app-games/ app-games/freecraft/files/ app-i18n/ app-i18n/canna/files/ app-i18n/kde-i18n-ga/ app-i18n/kinput2/files/ app-misc/ app-office/ app-office/openoffice-bin/files/ app-office/openoffice/files/ app-pda/ app-sci/ app-text/ app-text/dictd/files/ app-xemacs/ dev-db/ dev-java/ dev-java/ant/files/ dev-java/java-config/files/ dev-lang/ dev-lang/icc/files/ dev-libs/ dev-lisp/ dev-perl/ dev-python/ dev-ruby/ dev-tcltk/ dev-util/ eclass/ gnome-base/ gnome-base/gdm/files/ gnome-extra/ kde-base/ kde-base/kdebase/files/ media-gfx/ media-libs/ media-libs/netpbm/files/ media-plugins/ media-sound/ media-video/ metadata/ metadata/cache/media-video/ net-analyzer/ net-analyzer/nagios/ net-dialup/ net-dialup/isdn4k-utils/files/ net-fs/ net-ftp/ net-im/ net-irc/ net-libs/ net-mail/ net-mail/qmail/files/ net-mail/sendmail/files/ net-misc/ net-misc/pump/files/ net-p2p/ net-print/ net-wireless/ net-www/ net-www/apache/files/ net-www/orion/files/ net-www/squid/files/ net-www/tomcat/files/ profiles/ profiles/default-sparc64-1.4/ sys-apps/ sys-apps/baselayout/files/ sys-apps/grub/files/ sys-apps/lilo/files/ sys-apps/portage/files/ sys-apps/shadow/files/ sys-apps/tcp-wrappers/files/ sys-apps/yard/files/ sys-cluster/ sys-devel/ sys-devel/gcc/files/ sys-devel/libtool/files/ sys-kernel/ sys-kernel/linux-headers/files/ sys-kernel/mjc-sources/ sys-libs/ sys-libs/cracklib/files/ sys-libs/pam/files/ x11-base/ x11-base/xfree/files/ x11-libs/ x11-libs/ecore/files/ x11-misc/ x11-plugins/ x11-terms/ x11-themes/ x11-wm/ metadata/timestamp 29 100% 0.00kB/s 0:00:00 metadata/timestamp.chk 32 100% 0.00kB/s 0:00:00 metadata/timestamp.x 40 100% 0.00kB/s 0:00:00 Number of files: 40165 Number of files transferred: 3 Total file size: 32989938 bytes Total transferred file size: 101 bytes Literal data: 101 bytes Matched data: 0 bytes File list size: 899945 Total bytes written: 172 Total bytes read: 900191 wrote 172 bytes read 900191 bytes 51449.31 bytes/sec total size is 32989938 speedup is 36.64 End: Thu Apr 10 13:00:17 JST 2003 ----- END ----- I think that the script runs every 30 minutes. What do I correct?
There are a few time lags between rsync2.de.gentoo.org and rsync.gentoo.gr.jp. I ran the rsync script at :10. Then I got timestamp.chk generated at :00. There is timestamp.chk generated at :00 in rsync2.de.gentoo.org when the script run at :30. There is timestamp.chk generated at :30 in rsync2.de.gentoo.org when the script run at :00. >kurtl@x5 trashme2 $ less timestamp.chk >Wed, 09 Apr 2003 16:01:00 +0000 >kurtl@x5 trashme2 $ date -u >Wed Apr 9 16:45:57 UTC 2003 So, The problem happened. The script ran at 16:30 and got timestamp.chk generated at 16:00. #But the machine synchronized by ntp.
The timestamp.chk generated at every :01 and :31 on rsync://rsync2.de.gentoo.org/gentoo-portage. I run the script at every :00 and :30. Then the result is propriety.
I've added 210.138.60.45 to the access list for rsync1.us.gentoo.org to see if that makes a difference. Please update your rsync cron jobs to point to that server and post back here when you've done so. If that seems to fix the problem, we'll monitor the server for a couple days to make sure things are working fine and then add it to the rotation.
I've added 210.138.60.45 to the access list for rsync1.us.gentoo.org to see if that makes a difference. Please update your rsync cron jobs to point to that server and post back here when you've done so. If that seems to fix the problem, we'll monitor the server for a couple days to make sure things are working fine and then add it to the rotation. I did it. And the problem fixed. Thanks.
>I've added 210.138.60.45 to the access list for rsync1.us.gentoo.org to see >if that makes a difference. Please update your rsync cron jobs to point to >that server and post back here when you've done so. If that seems to fix >the problem, we'll monitor the server for a couple days to make sure things >are working fine and then add it to the rotation. I did it. And the problem fixed. Thanks.
Added this server to the rotation. It is now rsync2.jp.gentoo.org and is part of the rsync.asia.gentoo.org rotation. Additionally, I've subscribed egg@lostbutterfly.net to our low-volume gentoo-mirrors mailing list, which is where we discuss mirror-related issues and announcements. Thanks for helping to support Gentoo Linux.
