I have added a small functionality in the emerge command (changed emerge and portage.py), which, by invoking emerge with parameter -w or --wget will create a /tmp/wget_me file (a bash script), which when run will fetch the required packages from an available mirror. The bash script, contains the needed wget commands. The whole point of this is to be able to get the packages needed from another machine with a faster internet connection. I know the code isnt the best it can be, and have never done anything in python before, so be polite ;-) Reproducible: Always Steps to Reproduce: 1. patch /usr/bin/emerge emerge.diff 2. patch /usr/lib/python2.2/site-packages/portage.py portage.py.diff 3. emerge -w system will create the /tmp/wget_me file. 4. ./wget_me will get the files. Actual Results: Got the files ;-) The emerge.diff 58c58,59 < "--verbose", "--version" --- > "--verbose", "--version", > "--wget" 78c79,80 < "v":"--verbose", "V":"--version" --- > "v":"--verbose", "V":"--version", > "w":"--wget" 198d199 < 1263c1264 < if ("--fetchonly" not in myopts) and ("--buildpkgonly" not in myopts): --- > if ("--fetchonly" not in myopts) and ("--buildpkgonly" not in myopts) and ("--wget" not in myopts): 1323c1324,1332 < if ("--fetchonly" in myopts) or (x[0]=="blocks"): --- > if ("--wget" in myopts) or (x[0]=="blocks"): > retval=portage.doebuild(y,"wget",myroot,edebug,1,fetchonly=1) > if retval: > print > print "!!! Wget for",y,"failed, continuing..." > print > returnme=1 > continue > elif ("--fetchonly" in myopts) or (x[0]=="blocks"): 1425d1433 < 1437,1438c1445,1447 < if (mergecount>0): < portage.env_update() --- > if ("--wget" not in myopts): > if (mergecount>0): > portage.env_update() 1442c1451 < if "--fetchonly" in myopts: --- > if "--fetchonly" in myopts or "--wget" in myopts: The portage.py.diff 1206a1207,1313 > def wget(myuris): > print ">>> Printing appropriate wget command" > listonly=1 > fetchonly=0 > "fetch files. Will use digest file if available." > if ("mirror" in features) and ("nomirror" in settings["RESTRICT"].split()): > print ">>> \"mirror\" mode and \"nomirror\" restriction enabled; skipping fetch." > return 1 > global thirdpartymirrors > > if ("nomirror" in settings["RESTRICT"].split()): > mymirrors=[] > else: > mymirrors=settings["GENTOO_MIRRORS"].split() > > fetchcommand=settings["FETCHCOMMAND"] > resumecommand=settings["RESUMECOMMAND"] > fetchcommand=string.replace(fetchcommand,"${DISTDIR}",settings["DISTDIR"]) > resumecommand=string.replace(resumecommand,"${DISTDIR}",settings["DISTDIR"]) > mydigests=None > digestfn=settings["FILESDIR"]+"/digest-"+settings["PF"] > if os.path.exists(digestfn): > myfile=open(digestfn,"r") > mylines=myfile.readlines() > mydigests={} > for x in mylines: > myline=string.split(x) > if len(myline)<4: > #invalid line > print "!!! The digest",digestfn,"appears to be corrupt. Aborting." > return 0 > try: > mydigests[myline[2]]={"md5":myline[1],"size":string.atol(myline[3])} > except ValueError: > print "!!! The digest",digestfn,"appears to be corrupt. Aborting." > if "fetch" in settings["RESTRICT"].split(): > # fetch is restricted. Ensure all files have already been downloaded; otherwise, > # print message and exit. > gotit=1 > for myuri in myuris: > myfile=os.path.basename(myuri) > try: > mystat=os.stat(settings["DISTDIR"]+"/"+myfile) > except (OSError,IOError),e: > # file does not exist > print "!!!",myfile,"not found in",settings["DISTDIR"]+"." > gotit=0 > if not gotit: > print > print "!!!",settings["CATEGORY"]+"/"+settings["PF"],"has fetch restriction turned on." > print "!!! This probably means that this ebuild's files must be downloaded" > print "!!! manually. See the comments in the ebuild for more information." > print > spawn("/usr/sbin/ebuild.sh nofetch") > return 0 > return 1 > locations=mymirrors[:] > filedict={} > > for myuri in myuris: > myfile=os.path.basename(myuri) > print ">>> for file " + myfile > if not filedict.has_key(myfile): > filedict[myfile]=[] > for y in range(0,len(locations)): > filedict[myfile].append(locations[y]+"/distfiles/"+myfile) > if myuri[:9]=="mirror://": > eidx = myuri.find("/", 9) > if eidx != -1: > mirrorname = myuri[9:eidx] > if thirdpartymirrors.has_key(mirrorname): > try: > shuffle(thirdpartymirrors[mirrorname]) > except: > writemsg(red("!!! YOU HAVE A BROKEN PYTHON/GLIBC.\n")) > writemsg( "!!! You are most likely on a pentium4 box and have specified -march=pentium4\n") > writemsg( "!!! or -fpmath=sse2. GCC was generating invalid sse2 instructions in versions\n") > writemsg( "!!! prior to 3.2.3. Please merge the latest gcc or rebuid python with either\n") > writemsg( "!!! -march=pentium3 or set -mno-sse2 in your cflags.\n\n\n") > time.sleep(10) > > for locmirr in thirdpartymirrors[mirrorname]: > filedict[myfile].append(locmirr+"/"+myuri[eidx+1:]) > else: > filedict[myfile].append(myuri) > if os.path.isfile("/tmp/wget_me"): > data = open("/tmp/wget_me", "a") > else: > data = open("/tmp/wget_me", "a") > data.writelines("#!/bin/bash\nif [ -z \"$1\" ]\nthen\nrate=0\nelse\nrate=\"$1\"\nfi\n") > for myfile in filedict.keys(): > if listonly: > fetched=0 > writemsg("\n") > for loc in filedict[myfile]: > if listonly: > data.writelines("wget -c -t 5 --limit-rate=\"$rate\"k "+loc) > if loc != filedict[myfile][-1]: > data.writelines(" || ") > continue > > data.writelines("\n\n"); > data.close() > spawn("chmod +x /tmp/wget_me") > print(">>> wgets written, check and run file /tmp/wget_me") > return 1 > 1647c1754 < "package","unmerge", "manifest"]: --- > "package","unmerge", "manifest", "wget"]: 1879a1987,1991 > > if mydo=="wget": > wget(fetchme) > return 0 >
please attach the patch ... pasting as a comment ruins it also, i dont see the point to the patch ... could you elaborate a bit more ? :)
Created attachment 22619 [details, diff] emerge.diff
Created attachment 22620 [details, diff] portage.py.diff
Ok, here's the deal. I've got 2 gentoo boxes, one at home on a dialup line, and one at work at a much faster one. I needed an easy way to use my fast connection, to get files to my home machine. What this thing does, is that I can do an emerge -w system at home, get the /tmp/wget_me file and run in in the box at work. I can then transfer the downloaded files with a usb HD to home in /usr/portage/distfiles. Sorry about the description.
yeah, thats what i thought :) we have syntax for this kind of thing already: `emerge world -upf` basically when you put -p with -f portage will print out all the SRC_URI's for you to pass to whatever you want in this case, generating a bash script is a pretty specific task that could be handled by just parsing the output of -pf
Yeah, I know about -fp, the thing was that I wanted an easy way to do it ;-) and it seemed nice, that I didnt have to parse the whole thing (especially in a world or system update), or worry about getting the output to a file, etc. Anyway, thanks and marry christmas to all. Regards, Emmanuel
well here's a quick script: #!/bin/bash savedir=/mnt/usbdrive/ fetchlist=`tempfile` emerge world -upf > ${fetchlist} for url in $(<${fetchlist}) ; do wget -nc ${url} -P ${savedir} done rm ${fetchlist}
(In reply to comment #7) > well here's a quick script: > #!/bin/bash > savedir=/mnt/usbdrive/ > fetchlist=`tempfile` > emerge world -upf > ${fetchlist} > for url in $(<${fetchlist}) ; do > wget -nc ${url} -P ${savedir} > done > rm ${fetchlist} How about adding this to the man page of emerge? ;-)
(In reply to comment #8) > (In reply to comment #7) > > well here's a quick script: > > #!/bin/bash > > savedir=/mnt/usbdrive/ > > fetchlist=`tempfile` > > emerge world -upf > ${fetchlist} > > for url in $(<${fetchlist}) ; do > > wget -nc ${url} -P ${savedir} > > done > > rm ${fetchlist} > > How about adding this to the man page of emerge? ;-) SRC_URI arrows, available since EAPI 2, complicate matters. See discussion in bug 358923.
