just made 'emerge rsync' 2 times and both didn't work: - 1st mirror has a maximum of only 5 connections (don't think this is really much) - 2nd mirror fails with '@ERROR: chroot failed' To avoid people switching to the "good" mirrors (and thus circumvent round-robin balancing) I think it's important to a) have some minimum requirements regarding max.conn. b) test them from time to time (maybe can be automated) (See also the message <1026313682.4997.41.camel@belkelel.dyndns.org> from Dar@Dar-Klajid.de on gentoo-dev) -Markus- --- output of 'emerge rsync' ---- gandalf root # emerge rsync >>> starting rsync with rsync://rsync.gentoo.org/gentoo-portage... ,---------------- | Welcome to rsync1.uk.gentoo.org! | | For info on Gentoo Linux, see <http://www.gentoo.org/>. | | For more on this server, see <http://vegetable.org.uk/>. | If you're not doing so already, please consider compressing the connection here by using rsync -z ; you can edit /usr/sbin/emerge (or /usr/bin/emerge in newer versions of portage) to assert this. @ERROR: max connections (5) reached - try again later rsync: connection unexpectedly closed (461 bytes read so far) rsync error: error in rsync protocol data stream (code 12) at io.c(150) gandalf root # emerge rsync >>> starting rsync with rsync://rsync.gentoo.org/gentoo-portage... Welcome to SunSITE.dk ===================== SunSITE Denmark is located at Aalborg University, Institute of Electronic Systems, Denmark, and is running on a Sun Enterprise E3500 Server with 2 CPUs, 2 GB Memory and approximately 240 GB storage. The server was kindly donated by Sun Microsystems. Aalborg University, Mortens Recipe Collection, 3Com Nordic, Silcon Group and CLARiiON are sponsoring the project. DISCLAIMER: Neither Aalborg University nor Sun Microsystems are liable for any use, storage or transmission of any files stored on this archive. Please email suggestions and questions to staff@sunsite.dk Access our ftp area by anonymous rsync like rsync -avz sunsite.dk::ftp/pub/<path> Section Content ====================================================================== @ERROR: chroot failed rsync: connection unexpectedly closed (867 bytes read so far) rsync error: error in rsync protocol data stream (code 12) at io.c(150)
Thank you very much for this information. I do test all the mirrors on pretty much a weekly basis but that is really quite limited because it doesn't always test at peak usage hours. I'm mostly testing that they have a current mirror of the content. Because I don't ahve root on all the boxes that host mirros I have no way of knowing what exactly how each is set up. I'll see if I can come up with some more tests to test things like the max connection settings.
Looks like they upped their number of connections, but only by two. Just received the following: >>> starting rsync with rsync://rsync.gentoo.org/gentoo-portage... ,---------------- | Welcome to rsync1.uk.gentoo.org! | | For info on Gentoo Linux, see <http://www.gentoo.org/>. | | For more on this server, see <http://vegetable.org.uk/>. | If you're not doing so already, please consider compressing the connection here by using rsync -z ; you can edit /usr/sbin/emerge (or /usr/bin/emerge in newer versions of portage) to assert this. @ERROR: max connections (7) reached - try again later
Here's another '5 user' site.. >>> starting rsync with rsync://rsync.gentoo.org/gentoo-portage... ____ _ _ _ __ __ _ | __ )| | | | | | \/ | / \ | _ \| | | | | | |\/| | / _ \ | |_) | |_| | |___| | | |/ ___ \ |____/ \___/|_____|_| |_/_/ \_\ Bienvenido a rsync1.es.gentoo.org Este servidor esta situado en Palma de Mallorca, Espa
Here's another '5 user' site.. >>> starting rsync with rsync://rsync.gentoo.org/gentoo-portage... ____ _ _ _ __ __ _ | __ )| | | | | | \/ | / \ | _ \| | | | | | |\/| | / _ \ | |_) | |_| | |___| | | |/ ___ \ |____/ \___/|_____|_| |_/_/ \_\ Bienvenido a rsync1.es.gentoo.org Este servidor esta situado en Palma de Mallorca, España Hospedado por BULMA, el LUG de las Baleares Bisoños Usuarios de Linux de Mallorca y Alrededores Bergantells Usuaris de Linux de Mallorca i Afegitons http://bulmalug.net Email: gentoo-mirror@bulma.net @ERROR: max connections (5) reached - try again later rsync: connection unexpectedly closed (567 bytes read so far) rsync error: error in rsync protocol data stream (code 12) at io.c(150)
Here's a 10 user site. (both tried just before the bug comments were posted.) Question: How many users is too few? >>> starting rsync with rsync://rsync.gentoo.org/gentoo-portage... --------------------------------------------------------------------------- F T P . O R S T . E D U Oregon State University Unauthorized use is prohibited - violators will be prosecuted --------------------------------------------------------------------------- For more information on accessing these archives visit: http://www.net.oregonstate.edu/ftp/ --------------------------------------------------------------------------- @ERROR: max connections (10) reached - try again later
I've encountered this a few times (the limited # of users) and it is a little annoying, often requiring repeated rsync attempts until the round-robin dns spits out a different server. Perhaps the rsync functionality of portage/emerge could be modified to allow users to specify a number of retries if there is an rsync failure eg: an env var RSYNC_RETRIES=3 Then when a mirror returns an error portage could attempt connection again. I don't think portage should allow unlimited retries though, the user setting should be checked to ensure it is within sane limits (say 5-10 retries max). There is no point in having all the gentoo users performing the equivelant of a DDoS because they have set their retries to 100000. -- Troy Dack.
*** Bug 5697 has been marked as a duplicate of this bug. ***
*** Bug 5047 has been marked as a duplicate of this bug. ***
*** Bug 6213 has been marked as a duplicate of this bug. ***
I think all mirrors should decide for themselves how many users they want max, but to reduce load of the master server it should be at least 2. emerge just should notice errors, and try the next server. About retrying: maybe there could be made a difference in what kind of error: obviously, the max users error is very likely to be solved at the next server, but if it can't reach the mirror due to network probles, it doesn't solve itself automatically. I suggest endless retries (or a very large number) if it's 'cause of max users, and only a couple retries if there is some other kind of error.
i'd agree, that a 'good' solution would be for emege to walk the list of rsync mirrors until it either hits a good one or hits a repeat address
Not being an rsync fanatic, is there a way for rsync to indicate the failure was due to a concurrent user limitation? There doesn't appear to be an apparent exit value in the man page for rsync. If it can't, while I don't think we should be turning away rsync mirrors (especially as Gentoo grows), I don't think we should be setting an ubsurdly large number of retries either. Perhaps the best balance can be struck in the 'local-balancing' DNS calls. Perhaps require the mirrors to, as Tom suggested (in comment 5), require >= 2 concurrent connections, and keep the gentoo-rsync-DNS-guy apprised of the current value. Then use those numbers to 'tweak' how often the DNS server(s) spit out that IP address. I.E. A server w/ 500 connections will get hit ~10x more often then one with 50. Obviously this will need to be worked on as the ones with 2 connections would rarely see activity, while the large mirrors would get pounded. ... Just a thought.
How about this horrible message:

>>> starting rsync with rsync://rsync.gentoo.org/gentoo-portage...
rsync: failed to connect to rsync.gentoo.org: Connection refused
rsync error: error in socket IO (code 10) at clientserver.c(97)

>>> Updating Portage cache... 
thats not really horrible, it just timed out ive seen it once or twice
No... it happens instantly and has been the only response I've gotten over 30-40 tries over the last few days. That's what makes it horrible.
Network problems? New firewall installed?
I'm not too concerned about number-of-user limits, although more graceful handling of these failures would be a Good Idea. My concern is with mirrors that simply don't work at all. For example, there's one server that's not responding to connections at all, and which I got on three consecutive tries this afternoon, and a couple of times yeterday too. It looks like someone's set up a mirror on their own DSL line, and has simply switched the machine off. Other comments here suggest that misconfigured servers are also a problem. Back to the subject of user limits and load-balancing, proportional DNS is actually a good idea. Servers with high user limits will get hammered, sure, but the high user limit is supposed to denote a server that's capable of taking that hammering. If it's too much, they can lower the limit and accept less users. Servers with low user limits won't be left in the dark - they'll get a proportionate segment of what is now a very large userbase, which in turn works out to be quite a good number. If they want more users, they can again adjust their user limit.
We have a minimum requirement of 5 connections, so that uk server is ok.. and the sunsite.dk server is fixed and up again.. regards
