On Tue, 15 Apr 2003 13:52:31 -0400 Kurt Lieber <klieber@gentoo.org> wrote: > On Wed, Apr 16, 2003 at 01:34:56AM +0800 or thereabouts, mirror-maintainer@mirror.averse.net wrote: > > On Mon, 14 Apr 2003, Erwin _ wrote: > > > hi, > > > i can't emerge sync using "rsync.sg.gentoo.org" anymore, but "mirror.averse.net/gentoo-portage" still works.. > > > does it mean, it's not an official gentoo portage tree mirror anymore? > > > -Erwin- > > > > I guess so. > > > > I never got direct notification when I was originally removed from the > > rsync.gentoo.org round robin back in december, and never received any > > suitable response to my replies on that. Now, it appears that I've been > > removed from rsync.sg too, again without notification. I guess this is an > > indication that I should just remove my mirror. > > rsync1.sg.gentoo.org started having problems around March 6th at which > point is was removed from the rotation. I sent an > email to "mirror-maintainer@mirror.averse.net" on March 7th. (copy > attached) I never received a response to that email so it has never been > re-enabled in our rotation. > > We will monitor your server over the next few days and, as long as it > appears to be working correctly, we'll go ahead and add it back into the > rotation. > > If there is another, more suitable contact address we should use for any > future outages, please let me know. > > --kurt
Please add a gentoo-x86-portage mirror for backwards compatibility. 210.193.6.237 has been added to the access list for rsync1.us.gentoo.org. Please update your rsync cron jobs to point to that server and post back here when you've done so. We'll monitor the server for a couple days to make sure things are working fine and then add it to the rotation.
From: Peter Penkala <pjp@gentoo.org> To: mirror-maintainer@mirror.averse.net Subject: Re: gentoo rsync mirror is not official anymore??? Date: Tue, 15 Apr 2003 13:32:37 -0600 On Wed, 16 Apr 2003 01:34:56 +0800 (SGT) mirror-maintainer@mirror.averse.net wrote: > On Mon, 14 Apr 2003, Erwin _ wrote: > > > hi, > > i can't emerge sync using "rsync.sg.gentoo.org" anymore, but "mirror.averse.net/gentoo-portage" still works.. > > does it mean, it's not an official gentoo portage tree mirror anymore? > > -Erwin- > > I guess so. > > I never got direct notification when I was originally removed from the > rsync.gentoo.org round robin back in december, and never received any > suitable response to my replies on that. Now, it appears that I've been > removed from rsync.sg too, again without notification. I guess this is an > indication that I should just remove my mirror. > > P.S. rsync.sg.gentoo.org seems to be a cname for rsync.gentoo.org, so you > should still be able to sync with the global round robin pool. > Please followup at http://bugs.gentoo.org/show_bug.cgi?id=19374 Thanks.
From mirror-maintainer@mirror.averse.net Fri Apr 18 11:06:09 2003 Date: Sat, 8 Mar 2003 09:40:19 +0800 (SGT) From: mirror-maintainer@mirror.averse.net To: Kurt Lieber <klieber@gentoo.org> Subject: Re: difficulties with gentoo-portage rsync module On Fri, 7 Mar 2003, Kurt Lieber wrote: > Hello -- > > We've been noticing some problems with the gentoo-portage rsync directory > on mirror.averse.net. It does not seem to complete full rsyncs, failing > with error messages similar to the following: > > receiving file list ... done > app-admin/addpatches/ChangeLog > write failed on app-admin/addpatches/ChangeLog : Bad address > rsync error: error in file IO (code 11) at receiver.c(271) > rsync: connection unexpectedly closed (891963 bytes read so far) > rsync error: error in rsync protocol data stream (code 12) at io.c(165) Yes, I've been having what seem like random problems in rsync'ing and it's driven me half crazy. It terminates some of the time, and completes other times. I'd also replied to the thread about disabling compression, mentioning my problem. That didn't seem to help, though. > Could you try deleting your entire gentoo-portage directory and re-syncing > from rsync1.us.gentoo.org to see if that resolves the problem? Will try this out now. From mirror-maintainer@mirror.averse.net Fri Apr 18 11:06:18 2003 Date: Sat, 8 Mar 2003 15:19:57 +0800 (SGT) From: mirror-maintainer@mirror.averse.net To: Kurt Lieber <klieber@gentoo.org> Subject: Re: difficulties with gentoo-portage rsync module On Sat, 8 Mar 2003 mirror-maintainer@mirror.averse.net wrote: > > Could you try deleting your entire gentoo-portage directory and re-syncing > > from rsync1.us.gentoo.org to see if that resolves the problem? > Will try this out now. Okay, the last 6 syncs have all completed successfully. This is much better than normal here! I'll try to monitor this where I have available time. In the meantime, please feel free to contact me if you spot any problems. From klieber@gentoo.org Fri Apr 18 11:05:12 2003 Date: Sat, 8 Mar 2003 06:59:57 -0500 From: Kurt Lieber <klieber@gentoo.org> To: mirror-maintainer@mirror.averse.net Subject: Re: difficulties with gentoo-portage rsync module On Sat, Mar 08, 2003 at 03:19:57PM +0800 or thereabouts, mirror-maintainer@mirror.averse.net wrote: > Okay, the last 6 syncs have all completed successfully. This is much > better than normal here! I'll try to monitor this where I have available > time. In the meantime, please feel free to contact me if you spot any > problems. Sounds good -- thanks for the update and I'll keep an eye on it as well. --kurt
> Please add a gentoo-x86-portage mirror for backwards compatibility. I think gentoo-x86-portage is identical to gentoo-portage, right? If so, that works and has been my configuration since I started mirroring gentoo portage in June last year. It's not advertised when doing rsync mirror.averse.net:: because there's no point advertising legacy stuff. > 210.193.6.237 has been added to the access list for rsync1.us.gentoo.org. > Please update your rsync cron jobs to point to that server and post back > here when you've done so. We'll monitor the server for a couple days to > make sure things are working fine and then add it to the rotation. Ok; I've been syncing from rsync1.us since William McArthur added me last year.
Here's a log of timestamp checks. Both times are UTC, with the left column being when I checked. When checked at ??:20, the timestamp should be close to the hour, and when checked at ??:50, close to the 1/2 hour. Are you seeing any rsync errors? IP: 210.193.6.237 BUG: Checked (UTC) Timestamp ================================================ 16:20:10 Fri, 18 Apr 2003 16:01:00 +0000 16:50:05 Fri, 18 Apr 2003 16:32:00 +0000 17:20:04 Fri, 18 Apr 2003 16:32:00 +0000 17:50:06 Fri, 18 Apr 2003 17:32:00 +0000 18:20:04 Fri, 18 Apr 2003 17:32:00 +0000 18:50:05 Fri, 18 Apr 2003 17:32:00 +0000 19:20:05 Fri, 18 Apr 2003 17:32:00 +0000 19:50:06 Fri, 18 Apr 2003 17:32:00 +0000 20:20:06 Fri, 18 Apr 2003 17:32:00 +0000 20:50:05 Fri, 18 Apr 2003 17:32:00 +0000 21:20:05 Fri, 18 Apr 2003 17:32:00 +0000 21:50:05 Fri, 18 Apr 2003 21:33:00 +0000 22:20:05 Fri, 18 Apr 2003 22:02:00 +0000 22:50:10 Fri, 18 Apr 2003 22:02:00 +0000 23:20:07 Fri, 18 Apr 2003 22:02:00 +0000 23:50:04 Fri, 18 Apr 2003 22:02:00 +0000 00:20:05 Fri, 18 Apr 2003 22:02:00 +0000 00:50:04 Sat, 19 Apr 2003 00:33:00 +0000 01:20:05 Sat, 19 Apr 2003 01:03:00 +0000 01:50:05 Sat, 19 Apr 2003 01:32:00 +0000 02:20:05 Sat, 19 Apr 2003 02:03:00 +0000 02:50:04 Sat, 19 Apr 2003 02:33:00 +0000 03:20:05 Sat, 19 Apr 2003 03:02:00 +0000 03:50:05 Sat, 19 Apr 2003 03:33:00 +0000 04:20:04 Sat, 19 Apr 2003 04:03:00 +0000 04:50:06 Sat, 19 Apr 2003 04:03:00 +0000 05:20:04 Sat, 19 Apr 2003 05:02:00 +0000 05:50:05 Sat, 19 Apr 2003 05:31:00 +0000 06:20:05 Sat, 19 Apr 2003 06:03:00 +0000 06:50:05 Sat, 19 Apr 2003 06:32:00 +0000 07:20:04 Sat, 19 Apr 2003 06:32:00 +0000 07:50:05 Sat, 19 Apr 2003 06:32:00 +0000 08:20:04 Sat, 19 Apr 2003 06:32:00 +0000 08:50:04 Sat, 19 Apr 2003 08:33:00 +0000 09:20:05 Sat, 19 Apr 2003 09:01:00 +0000 09:50:05 Sat, 19 Apr 2003 09:32:00 +0000 10:20:05 Sat, 19 Apr 2003 10:02:00 +0000 10:50:38 Sat, 19 Apr 2003 10:32:00 +0000 11:20:04 Sat, 19 Apr 2003 11:03:00 +0000 11:50:05 Sat, 19 Apr 2003 11:31:00 +0000 12:20:05 Sat, 19 Apr 2003 12:01:00 +0000 12:50:06 Sat, 19 Apr 2003 12:01:00 +0000 13:20:04 Sat, 19 Apr 2003 13:01:00 +0000
Yes, I see occasional errors (the random ones I'd mentioned previously. I normally only keep 7 cycles of logs; that's usually sufficient for a week's worth of logs since most sites are mirrored daily. The errors look similar to: ---cut-here--- Welcome to the rsync1.us.gentoo.org rsync server. This is a restricted master rsync mirror for the Gentoo Linux free software project. Please direct any issues relating to this server to klieber@gentoo.org. receiving file list ... done rsync error: timeout in data send/receive (code 30) at io.c(85) rsync: connection unexpectedly closed (1234079 bytes read so far) rsync error: error in rsync protocol data stream (code 12) at io.c(165) ./ app-admin/ app-admin/gentoolkit/files/ app-arch/ app-arch/bsdsfv/files/ [snip] x11-misc/ x11-plugins/ x11-terms/ x11-themes/ x11-wm/ rsync: connection unexpectedly closed (1234079 bytes read so far) rsync error: error in rsync protocol data stream (code 12) at io.c(165) ---cut-here--- I'm currently running rsync 2.5.6 from the debian package. I'll try using another build of rsync, as well as increasing the number of logfiles I retain to 48, so that I get 1 day of logs for portage.
Have you noticed any improvement with the rsync errors?
There have been a few variables in this period of time: 1. I switched to rsync 2.5.6 from redhat to check if there is some sort of problem in the debian rsync. Didn't see any improvement, so I do not suspect any rsync/protocol incompatibility. 2. I changed my script to have it retry the sync if there is a failure due to "connection unexpectedly closed". This seems to fix the problem except for certain periods where I can persistently obtain this error for a couple of hours. Might this suggest that my ISP occasionally experiences issues? I can't think of any other possibilities, since other mirrors don't seem to experience this connection unexpectedly closed problem. 3. I switched colo providers on May 23, changing IPs. Sent e-mail to gentoo-mirrors@, but my new IP has not been added to rsync1.us's access list yet. Have been syncing from rsync.au since then, and it appears to be working okay, except for periods of time where rsync.au is busy (max 50 connections reached). Will forward request to mirror-admin@, since that seems to be the correct place to send these requests?
On 3), I just removed the old IP and added the new one. Using bugzilla is a much better way to manage these issues, especially when existing bugs are open, such as this one. Otherwise, it is difficult to track emails and bugs that they are related to. Post back here when you can sync with rsync1.us again (or receive errors trying). Once that is working, I'll try monitoring again. Thanks.
Just to make sure, 203.208.241.235 is the new IP address? It has been added to the rsync1.us access list.
okay, i can sync with rsync1.us again. I'll wait a day or two to see if it seems stable
You've probably noticed, but there are alot of missed syncs.
Yes indeed, there were a lot of missed syncs. :( I thought it seemed good mirroring from rsync.au (didn't get the dreaded "rsync: connection unexpectedly closed") errors, so upon switching to rsync1.us I attempted to remove the script checking for that string and retrying the sync. That didn't seem to work so well (as you noticed). I'd then put the retries back in, and that seemed to be okay judging from the times that my rsync processes completed. Following that, I thought that perhaps using the time my rsync process completed isn't a good gauge, so wrote a script to poll the metadata/timestamp* files to see if they were being updated regularly. Found that my script wasn't generating the logs I desired (I mistakenly used > instead of >>, clobbering my log file). Fixed that, then found that I can't connect to rsync1.us any more since yesterday.
couple of thoughts: 1) Which is the right file to track to determine if it's up-to-date? There are 3 metadata/timestamp* files, with differing timestamps. 2) I'm starting to thinking that there may be some kind of incompatibility between gentoo's rsync and the one I use. Probably most gentoo mirrors use gentoo; I don't seeem to see much problems sync'ing with rsync.au which, I believe, doesn't run gentoo. I used to think that it might be caused by some kind of network problem, but I can't believe that a network problem would have followed me to another colocation provider. If the problem is indeed due to some sort of rsync incompatibility, it may be best for me to withdraw from mirroring gentoo - wouldn't make sense to give users the same connection closed errors that I'm facing.
Am asking gentoo users to try syncing against my mirror, and report problems (especially "rsync: connection unexpectedly closed" errors) to me. If I get a high percentage of errors, it would seem that something is incompatible.
Okay, it's been about 10 days and I've not had any problems reported by the handful of users I asked to attempt sync'ing with my mirror (currently syncing against rsync.kr pretty successfully). I'd noticed that I sync against rsync.kr and rsync.au well, but I have very poor records syncing against rsync1.us. I tried out syncing against rsync1.jp as well, and found that I was also getting virtually 100% failed syncs. With lots of help from the rsync1.jp maintainer (thanks Georgi!) and the rsync mailing list, we found that rsync1.jp ran the rsync daemon with the --compress option on the command line, and that clients which do not specify the --compress option will not be able to sync to a server that has specified in. Quoting Georgi, ---begin-quote--- Well... it seems that the transfer will fail only if the server is running with --compress, and the client without. I just made a few tests, and I will leave my rsync server running without the --compress option. Here is the output if you're interested. The filename nomenclature was "client_has_-z_or_not"-"server_has_-z_or_not".log As you can see, the transfer fails in the client - no compression, server - compression case. Since the transferred data was the same in both comp-* cases, I feel content leaving my server running without --compress. $ tail -2 *.log ==> comp-comp.log <== wrote 9802 bytes read 2490106 bytes 555535.11 bytes/sec total size is 4573465 speedup is 1.83 ==> comp-nocomp.log <== wrote 9802 bytes read 2490106 bytes 555535.11 bytes/sec total size is 4573465 speedup is 1.83 ==> nocomp-comp.log <== x11-wm/windowmaker/files/ app-admin/openca/Manifest ==> nocomp-nocomp.log <== wrote 9801 bytes read 4608065 bytes 710440.92 bytes/sec total size is 4573465 speedup is 0.99 ---end-quote--- I checked with Jason, the rsync.au administrator, and he confirmed that his daemon runs without the --compress option. Now that Georgi is running rsync1.jp without --compress, I've been able to sync from him (to a seperate test portage tree) without problems too. Can a mirror admin check whether rsync1.us runs with/without --compress, and if possible, remove that option?
rsync1.us is running without --compress.. I'll start monitoring your rsync mirror again for a couple of days, and if all goes well we'll throw you in rotation. Cheers!
Also, please start mirroring off of rsync1.us.gentoo.org again. We need you to mirror off of that server and fix any problems there before we can throw you into the list of mirrors. Cheers! -Corey
I've gone back to mirroring from rsync1.us since about 20 hours ago, although my script also falls back to rsync1.kr or rsync1.jp if there is an error in the mirroring process. Is there something wrong with monitor...indiana? I've been using that to gauge how up-to-date my mirror is, but it's currently showing "no time", red and black bars, and apparently thinks that 50% of mirrors are outdated.
Great! I'll start watching and hopefully get you in to DNS soon. As for the monitor, yes there seems to be a problem, thanks for pointing that out! I was messing with the script yesterday due to another problem that I had noticed, and it looks like I have created more of a problem than I fixed. More on this later.. Cheers!
Everything looks good here, so I've setup mirror.averse.net as rsync1.sg.gentoo.org which is also in the rsync.sg.gentoo.org set. closing this bug Thanks for your time and resources! Cheers!
