Here in NZ the typical home "broadband" connection is ratelimited to 128kbit incoming, and recent changes to the queueing mechanism has resulted in terrible performance when using full line rate. Portage has the ability to rate limit its downloads, via a custom FETCHCOMMAND, but emerge sync still runs at full rate, making everything else, including SSH, unusuable. Fixing our ISPs aren't an answer, however adding rate limiting support to portage is. rsync supports a --bwlimit command line option to limit its i/o speed. Add a RSYNC_RATELIMIT option to /etc/make.conf. It is in KB (eg, RSYNC_RATELIMIT =12 will limit rsyncs i/o usabe to 12KB/sec). The following patch is to 'emerge' in portage (2.0.49-r18), I have ACCEPT_KEYWORDS=~x86 --- emerge 2003-11-29 13:21:10.369246000 +1300 +++ emerge.dhtrl1 2003-11-29 13:20:51.909052376 +1300 @@ -1754,6 +1754,8 @@ mycommand=mycommand+" --exclude-from "+portage.settings["RSYNC_EXCLUDEFROM"] else: print "!!! RSYNC_EXCLUDEFROM specified, but file does not exist." + if portage.settings.has_key("RSYNC_RATELIMIT"): + mycommand = mycommand+ " --bwlimit="+portage.settings["RSYNC_RATELIMIT"] servertimestampdir=portage.settings["PORTAGE_TMPDIR"]+"/sync/"
*sigh*. Sorry bout this. The attached patch doesn't quite work, although the theory is fine. I'll upload a working patch ASAP.
Created attachment 21428 [details, diff] Patch to emerge to add support for bandwidth limiting emerge rsync
Portage uses tabs. Do not submit patches using spaces.
Created attachment 22686 [details, diff] Patch to emerge to add support for bandwidth limiting emerge rsync Fixes tab issue
Comment on attachment 22686 [details, diff] Patch to emerge to add support for bandwidth limiting emerge rsync --- /usr/bin/emerge 2003-12-27 12:21:40.000000000 +1300 +++ /usr/bin/emerge.dhtrl1 2003-12-27 12:21:05.000000000 +1300 @@ -1755,6 +1755,9 @@ else: print "!!! RSYNC_EXCLUDEFROM specified, but file does not exist." + if portage.settings.has_key("RSYNC_RATELIMIT"): + rsynccommand = rsynccommand + " --bwlimit="+portage.settings["RSYNC_RATELIMIT"] + servertimestampdir=portage.settings["PORTAGE_TMPDIR"]+"/sync/" content=portage.grabfile(myportdir+"/metadata/timestamp.chk")
Created attachment 23091 [details, diff] Patch to emerge to add support for bandwidth limiting emerge rsync
since rsync can keep adding more options, we could keep doing this, or we could add a catch all variable like RSYNC_EXTRA_OPTS to make.conf
full ack with Spanky
Created attachment 23254 [details, diff] allows arbitrary extra parameters to 'rsync' when called by emerge sync If RSYNC_EXTRA_OPTS is set in /etc/make.conf, the contents of this variable are applied to the rsync command line when performing an emerge [r]sync. The full option must be used of course, and any incorrect options in this field will cause rsync to fail. Any failures get reported to stdout however, and emerge fails cleanly although at this time suggests an incorrect fix: >>> starting rsync with rsync://ftp.citylink.co.nz/gentoo-portage... >>> checking server timestamp ... rsync: --bwlimi=12: unknown option rsync error: syntax or usage error (code 1) at main.c(1013) !!! Rsync has reported that there is a syntax error. Please ensure !!! that your SYNC statement is proper. !!! SYNC=rsync://ftp.citylink.co.nz/gentoo-portage This is working fine on my system. As a side note, I'm not 100% sure that --bwlimit=12 actually limits rsync to 12k, and not some smaller value. However, for my purposes this is ok - I just want to retain interactive comms with my home machine when doing emerge sync/emerge world (see earlier comment about braindead queuing by telco)
supposed to be fixed in 2.0.50 which is stable now. If this bug is not fixed please reopen.
