emerge-webrsync uses wget to download things. I did not find an option to configure proxy user and password. (for emerge it's described in the FAQ). I changed the script to add the necessary parameters to the wget command and it worked. So I suggest add that to the FAQ or implementing a central configuration option for proxy stuff. Reproducible: Always Steps to Reproduce: 1. Be behind a proxy which needs authentification 2. call emerge-webrsync Actual Results: It will fail after some time. Expected Results: Syncing the portage tree.
please attach your script or a patch for it
I'm currently rewriting and integrating emerge-webrsync into emerge sync, it will be changed to use FETCH_COMMAND to download snapshots. Will this solve your problem ?
I changed the line if (wget $wgetopts $url) && [ -s $file ] ; then to this: if (wget --proxy-user=myUser --proxy-passwd=myPassword $wgetopts $url) && [ -s $file ] ; then And yes, changing to use FETCHCOMMAND would solve it. I added the proxy settings there as well.
Created attachment 19028 [details, diff] patch to make emerge --sync recognize SYNC="http://" urls This is a first and untested patch, it should do what emerge-webrync did if you set SYNC="$GENTOO_MIRRORS" in make.conf and run emerge --sync, but you can set a specific mirror for snapshots as well. It checks the protocol part of the url to decide if it should use rsync, cvs, http or ftp. It's using whatever you specified in FETCHCOMMAND to get the snapshots. Please report errors and success here (as I said, it's untested and I need feedback).
Finally I got around to try it (and I had some trouble doing so). First of all I did not find the original emerge file to apply the patch to (I looked here http://www.gentoo.org/cgi-bin/viewcvs.cgi/portage/bin/emerge?cvsroot=gentoo-src). The patch accesses the mysyncuri variable which does not exist in my emerge script. I'm using Portage 2.0.48-r5 (default-x86-1.4, gcc-3.2.3, glibc-2.3.2-r1). Now I inserted the patched code into my local emerge script and changed a few things (I'll attach the file): 1. mysyncuri to syncuri 2. mytime = time.localtime() - (attempts*24*60*60) mysnapshot = time.strftime("%d%m%Y", mytime)+".tar.bz2" to mytime = time.localtime( time.time() - (attempts*24*60*60)) mysnapshot = "portage-" + time.strftime("%Y%m%d", mytime)+".tar.bz2" 3. if portage.spawn(mycmd, free=1) == 0: to if portage.spawn(myfetchcmd, free=1) == 0: It downloaded the file but os.path.exists(mysnapshot) returns 0 Here are some values: syncdir: /var/tmp/websync myfetchcmd: /usr/bin/wget --proxy-user=<proxyuser> --proxy-passwd=<proxypw> -t 5 --passive-ftp -P ${DISTDIR} ${URI} http://gentoo.oregonstate.edu/snapshots/portage-20031023.tar.bz2 wget writes: --15:07:22-- http://gentoo.oregonstate.edu/snapshots/portage-20031023.tar.bz2 => `/usr/portage/distfiles/portage-20031023.tar.bz2.2' Interestinlgy it doesn't store the file in the syncdir even though the script changes to that dir (I also printed os.getcwd() which is /var/tmp/websync). Now I have no more ideas. I think it should be possible for you to test the script just using http (no proxy, no authentication).
Created attachment 19737 [details] Patch integrated into emerge script
Thanks for testing it and sorry for the broken patch, it had even more bugs than you found :( I've fixed it and tested it myself this time, it seems to work (I never used emerge-webrsync, so I don't know if there are any differences in behavior). Btw, it's normal that the snapshot didn't show up in syncdir, FETCHCOMMAND stores it in DISTDIR and I forgot to copy it over :(
Created attachment 19939 [details, diff] patch to fix all known bugs This patch should fix all the bugs pointed out earlier. It's also including my overlay-sync code as seperating the two patches is a lot of work and will likely introduce other errors.
*** Bug 34956 has been marked as a duplicate of this bug. ***
*** This bug has been marked as a duplicate of 35535 ***
