I am intending to set up a rsync mirror/source file server at mirror.cs.wisc.edu (128.105.103.11). Technically the machine is mirror3.cs.wisc.edu but it will not make a difference, unless you guys care the real machine name as supposed to the CNAME. I hope to set it up tomorrow (6/6/2003) sometime, and I will post when it is ready. It will be hosted on a Dual P4 Xeon machine which is also serving several other mirrors, including OpenBSD and Redhat, as well as a few others, with the files hosted via one of our AFS servers. The connection is rate-limited to 20Mbps/up during most of the day, but it is higher at off-peak times. How many concurrent users would you suggest that I limit the rsync/source mirror to, before I set it up? I hope to have this set up sometime on friday, but it may not get done until next week depending on how busy I am tomorrow. Any ideas that any of you have would be great, thanks. -stefan (easykill on gentoo forums) Reproducible: Always Steps to Reproduce: 1. 2. 3.
Oh, just to clear it up, we would prefer that this not turn into the primary gentoo mirror, because if the bandwith usage gets *too* high, we may remove it. That's not too likely, since we have plenty of bandwith, but it could turn out to be an issue, heh. Thanks -stefan (stefan@cs.wisc.edu)
Sorry for the delay on this -- we've been short-staffed of late and bugs have been an unfortunate casualty... As for concurrent users, I'd suggest limiting the rsync connections to 15-20 and leaving the HTTP limit uncapped unless bandwidth starts to be a problem. Most of our source mirrors use 3Mbps - 5Mbps on average, and that number is decreasing as we significantly add to our mirror system. Since you said you're capped at 20Mbps during peak times, I don't think we'll really impact you guys at all. Let me know if we can be of further assistance and again, my apologies for the delay.
This has sort of been on hiatus at the moment, mostly because we are using an AFS file system for this, and the fact that the /distfiles directory is all one huge directory is causing problems with size. We would prefer to separate it out into volumes similar to the portage tree, e.g. /distfiles/net-www/mozilla/mozilla-whatever.tgz having one big volume is kind of messing up our AFS stuff...AFS wasn't really designed for that. It would be great if you guys would make a change to the /distfiles directory and separate it out so it's not a flat tree. That would make maintenance a lot easier as well, I would think. We are trying to work around this, but it hasn't been that high of a priority at the moment. I'll keep you updated on this. -stefan
There have been thoughts along those lines in the past - and it would make my job as distfiles janitor a hell of a lot easier - but the big showstopper is that there is no decent upgrade path for it without breaking Portage for a lot of people.
Not being a gentoo maintainer, but having used it on numerous machines for a while, I would think it would be worth it. I could deal with needing to wipe my /usr/portage/distfiles directory...worse things have happened. It would probably bother dial-up users more. Maybe there could be a script to rearrange everybody's distfiles for them. Probably wouldn't be too tough to do. If you want my help on that, let me know. I should be able to find some time for it. Of course, there may be a number of other issues here that I'm either not thinking of in the 20 seconds taht I've thought about this, or that I just don't know about. I guess one reason to do this is that as time goes on, /distfiles is just going to get more obnoxious as more packages/versions are added. I don't know how often you clean out old packages, but I can see it getting insane in the not-so-distant future. Then again, if you do it soon, you get a nice fat mirror from us, hehe. -stefan
moving /distfiles away from a flat tree is something we will be doing, but right now, we simply don't have the time or resources. marking this as later.
