Hello. Recently, because I thought it would be useful, both to myself and to other people, I decided to write a script that would enable someone to perform an emerge --emptytree and resume from wherever an error occurred in the process. I posted this in the tips and tricks forum (http://forums.gentoo.org/viewtopic.php?t=9638), however, someone suggested that this script was useful enough that I should submit it for inclusion in the portage tree. So I wrote an ebuild for it. The script supports emerging the entire world, or one or more packages, and if multiple packages are specified they can be emerged as a group or individually. The script stores the information it needs to resume the merges in /var/tmp/rmerge by default, but the script can easily be changed by editing a variable at the top of the file to store the information somewhere else. It's written in bash. To the best of my knowledge, it performs as intended, but I'm not Don Knuth, so I can't claim to have proven its correctness. Attached you'll find rmerge-1.0.tar.gz, which should contain rmerge, the actual shell script, rmerge-1.0.ebuild, the ebuild (of course), and rmerge.8, which is a man page for the script. I would suggest the app-admin category, because that's the category the likes of portagemaster and kmerge have been put in, and this seems like a similar type of application. Also, this application obviously depends on portage, but that probably won't be a problem for most users. Thanks for your consideration.
Created attachment 2638 [details] rmerge-1.0.ebuild (New Package) This is the file referenced in the bug submission
I forgot to mention: the ebuild is written assuming that the script and the man page are in the files directory under the ebuild directory (I took a cue from the superadduser ebuild). The files are very small, so I don't think this should be a problem.
clever :)
This ebuild has been selected as a testing candidate for the prototype Gentoo User Ebuild Submission System. This means it'll initially be processed by an automated validator. I'll get back to you with more details once the validator has been properly set up.
I hate to be a pain, but since I've posted this bug, I've some work, and I've pretty much re-written the entire script to be more modular (broke stuff down into functions), which, I think, makes it easier to understand if you're looking at it, and also makes it easier to fit new functionality in. Also, I added some new features, and fixed a minor bug/problem or two, and I think that the script I currently have is quite a bit better than the one that's posted for addition in this bug report (one important feature is the ability to resume an emerge that was cut off by killing the process in some way (like your system losing power)). So, my question is, should I post the update here? Or do I wait until this package is added to portage, and then file an update bug? I imagine that posting the stuff here might cause problems for your automated handler. Sorry about this, and thanks for your time.
Created attachment 2711 [details] rmerge-2.0.ebuild (Update) This is the update referenced in the previous post. Includes several new options and some minor bug fixes.
I've run your submission through the submission system. It found a few problems: 1) - (E) Missing Copyright statement - (E) Missing $Header:$ - (E) Missing GPL license 2) - (E) Missing HOMEPAGE= This can be empty, but it must be in there. - (E) Missing DEPEND= - (E) Missing LICENSE= - (E) Missing KEYWORDS= You may want to test this locally by installing dev-util/lintool and doing lintool --show-details --ebuild <yourebuildfile>
Created attachment 3024 [details] Fixed rmerge ebuild + files Sorry about that. I fixed the problems listed, and ran the ebuild through lintool, so I think it's correct now. The tarball has the updated ebuild, and the man page and script in the files subdirectory (as they should be in the ebuild directory).
emerge -pe world says: [ebuild N ] sys-libs/db-1.85-r1 to / [ebuild N ] sys-libs/db-3.2.9-r1 to / rmerge does: >>> sys-libs/db-3.2.9-r1 merged. >>> sys-libs/db-3.2.9-r1 merged. i wouldnt be surprised if it would do the same with qt2 and qt3 etc.
Vinz: Hmm. At first, I thought that the --fix option fixed that problem, but it just occurred to me that it doesn't... I'm going to have to think about this for a little while... Note: This bug would only occur if you have two versions of a library installed on your system as you do. Looks like I'm going to have to take the nasty way out and hop around to all the different directories... A big re-write indeed (perhaps warranting version 3.0 :)). Thanks.
Created attachment 3156 [details] rmerge 2.2 (bugfix + added features) Okay, this version should fix the bug referenced above. It also includes some extra options (BCFLAGS and --skip). Read the docs to find out about those. If it's not too much trouble to test this (I know it emerges regular packages correctly, so it should theoretically work for doubled packages, but I don't have any on my system), I'd appreciate it. An easy way would to create an appropriate file (like world.rmerge) with just the two packages inside in the form: category/package-version And then resume that merge. It _should_ work. (Didn't make it version 3.0. The re-write was really easy :)).
i didn't know where to put the ebuild to install so i installed maually by just copying the script to /sbin and the manpage to /usr/man/man8. the first time i tried to run it, however, it errored out saying 'mkdir /var/tmp/rmerge/tmp/ no such directory'. there was no /var/tmp/rmerge dir to make tmp/ in, so i mkdired it and then ran it again and it worked. it is compiling everything right now, good job.
The script doesn't seem to respect the value of the PORTDIR_OVERLAY variable. This is quite annoying, and basically means that "hobbyist" package builders like me cannot get it to properly build everything. It should be trivial to add, so consider this a bugfix/feature request... ;)
Created attachment 3212 [details] rmerge-2.2.1 (PORTDIR_OVERLAY fix) My apologies on the PORTDIR_OVERLAY thing. I remember reading about it, but it totally slipped my mind (since I'd never used it before). Now, however, I have a place to put my ebuilds without having them killed by an rsync. The script checks to see if the ebuild is in $PORTDIR_OVERLAY first, and then falls back to $PORTDIR. I hope this is the correct behavior, since I'd think that whatever ebuild you're working on would be more up to date than whatever is in the main portage tree. You can get the fix here (2.2.1) or at the rmerge homepage (the address of which is above, I believe). Cheers.
Created attachment 3239 [details] rmerge2-0.9.tar.gz Hi there, I've taken the liberty of re-implementing most (all?) of the functionality of your script in Python. The motivation was the addition of the following features: - It now possible to give the "--pretend" option to the script, which shows which packages *would* be emerged. - I've added the "--empty" (-e) option, which allows one to emulate the functionality of your original rmerge script. However -- and here's where the "new feature" part comes in -- if the flag is *absent*, only packages which were actually mentioned on the command line will be emerged. This is very useful to merge a list of packages with proper ordering (emerge doesn't do this, at least not yet). - The state is saved in /var/lib/rmerge2/MD5HASH where MD5HASH is the MD5 hash of all the names of the packages specified on the command line. Btw, there seem to be some issues with this in your rmerge script (I haven't bothered to pinpoint them, but I couldn't get it working properly with multiple packages on the command line). The file contains a pickled representation of the state of the process (written to file in a crashproof way, of course). The /var/lib/rmerge2 directory must be created before running the script. I haven't bothered to add coloured output since it's not exactly a vital piece of functionality. Run the script without arguments to get a usage message. The script is about one third the size of the original rmerge (in bytes) and slightly less than half the number of lines. I also (subjectively) think it's much more readable. I've tested the script quite thoroughly on my own machine, but there are of course no guarantees. Please test and comment away. :) (Btw, a little side remark on the PORTDIR_OVERLAY issue: It seems that it is not necessary to actually find the .ebuild files, it is possible (in newer versions at least) to specify a specific package version to emerge like so: emerge =category/pkgname-1.2.3 This neatly avoids having to know about PORTDIR_OVERLAY. Just thought you'd like to know :))
Heh. Wow. Well, I guess we can discontinue adding to my script. :) I'm not sure what was wrong with entering multiple package names when you were using it (It seemed to work fine for me when I was testing stuff, but I might have been careless). I noticed a problem with having names in different orders, and thought I might MD5 them, but I had to sort them to use that anyway, so I figured I'd just leave them as-is. I guess my code must be faulty there. I wasn't aware of the =category/package-version feature. I remember trying to emerge specific versions in the past (not with =), and having it fail unless I specified the exact location of the ebuild. Knowing that would have probably made the script more bug-free from the start. :) Have you told people about your implementation on the forums (specifically, in the old Tips and Tricks post I originally made, and, more importantly, in the rebuilding with GCC 3.2 thread stuck at the top of Portage and Programming)? I'm sure they'd like to have it. It's probably better to have it written in Python, too, since it's more likely to actually be included in Portage that way (which is where this really belongs, because I think recompiling all your stuff is a somewhat common occurence). I'd have done it in Python myself, but I never finished the book I bought on it (bad book, plus I was already spoiled with the book on Ruby I'd just read :)). Anyway, I'll be unavailable for the next couple days, so you couldn't have announced this at a better time. :) Now I don't have to worry about any bugs in my script while I'm gone. Cheers.
nice tool :) i tested rmerge-2.2.1 and it works fine here (resuming etc.). for better usability i'd just suggest to make rmerge's options a bit more consistent with those of emerge. i.e. -p should preferrably be used for --pretend only, not --previous. etc. rmerge showed to be great in conjunction with spider's tips for upgrading gentoo (e.g. from default-1.0-gcc3 to default-x86-2.0): quoting spider from "[gentoo-dev] Gentoo 1.2 to 1.4 update, HowTo do it and not break everything": /* I think we can / should be able to cut the procedure down : original profile: emerge --buildpkg gcc glibc binutils move aside change profile export USE="-* build" emerge gcc texinfo binutils gettext unset USE emerge --buildpkg glibc texinfo gettext binutils gcc Now, the original USE="build" gcc replaced /usr/bin/gcc, and this second "emerge gcc" then doesn't realize theres a different version of gcc there, and overwrites gcc, c++ and all.. Neato. no more emerge -C gcc anywhere. After this, re-add the /etc/env.d/05gcc-old with the gcc 2.95.3 path, run env-update and test by launching... mozilla? */ then i just run 'rmerge world' (which could be further optimized if we could force-exclude specific ebuilds from rmerg'ing). later, i will test rmerge 2.
Created attachment 3255 [details] rmerge2-0.9.2.tar.gz Fixed listing of failed packages.
Created attachment 3266 [details] rmerge2-0.9.3.tar.gz - Fixes a typo in handling of --pretend. - Fixes environment propagation from rmerge2 to rmerge2.py
Created attachment 3300 [details] rmerge2-0.9.4.tar.gz Grrr... never re-factor without re-testing... This version fixes: - Proper handling of "--force" - Fixed typo which prevented proper "skipping" behavior
Created attachment 3302 [details] rmerge2-0.9.4.tar.gz (Get this one!) *Sigh*... never commit anything to bugzilla while drunk ;)
portage # rmerge2 -p -e world rmerge2: emerge returned bad exit status. cannot continue. portage # Boohoo, why does this alway happens to me ;) !
Created attachment 3343 [details] rmerge2-0.9.5.tar.gz > portage # rmerge2 -p -e world > rmerge2: emerge returned bad exit status. cannot continue. > portage # > > Boohoo, why does this alway happens to me ;) ! This is usually indicative of some sort of (possibly quite minor) inconsistency in the portage database. I've changed the rmerge2 script to actually display the output from emerge in this case, so that it is actually possible to see what's wrong. :) Also fixed a minor issue with not flushing stdout (which would cause a bit of weirdness in the progress messages).
Ah, see below (=local ebuild errors), much better thank you ;) Rigo These are the packages that I would merge, in order. Calculating world dependencies \ !!! Error: couldn't find match for "media-libs/sndlib" in media-sound/ceres3-0.27 rmerge2: emerge returned bad exit status. cannot continue.
lappy portage # rmerge2 -pe world|wc -l 886 No wonder I'm so busy ;) Kewl script...You made my day ! Rigo
Created attachment 3406 [details] rmerge2-0.9.6.tar.gz I've updated the package handling to allow for input package names to contain version numbers. This allows one to do something like the following: rmerge2 -p -e world > package.list [...edit "package.list" to remove/add packages...] rmerge2 `package.list` This can come in handy if you wish to recompile almost everything, but don't want to have to edit the "package.list" to remove version numbers in addition to packages. Barring bugs, this is most probably the final version.
On first use had to manually create /var/lib/rmerge2 directory. error message was a bit cryptic, but its ticking over nicely now. BillK
>./rmerge2 -p -e world I would merge the following packages (in order): >
Okay, this one will go in if we can manage to fix a few problems with it: 1) rmerge needs a --version switch 2) rmerge2.py should be installed either in /usr/lib/rmerge2/ or /usr/share/rmerge2/ (probably the latter since the script is platform-independent). 3) rmerge should call rmerge2.py from its new location, not assume it's in the path. 4) perhaps update the man page with --version Could you bundle all of this, with the ChangeLog and the ebuild into a .tar.gz and attach it ?
I ran rmerge2 from /usr/share/rmerge2 and I got the same error: >./rmerge2 -p -e world I would merge the following packages (in order): >
> > Okay, this one will go in if we can manage to fix a few problems with it: > > 1) rmerge needs a --version switch Done. > 2) rmerge2.py should be installed either in /usr/lib/rmerge2/ or > /usr/share/rmerge2/ (probably the latter since the script is > platform-independent). > 3) rmerge should call rmerge2.py from its new location, not assume > it's in the path. I've gone one better and eliminated the wrapper script completely. Instead of the wrapper, the "portage" python module is used to get the CFLAGS/CXXFLAGS/BCFLAGS/BCXXFLAGS variables from the make.globals/make.conf/etc. The ebuild simply installs the rmerge2 script in /usr/bin which is the same location that emerge is installed in. > 4) perhaps update the man page with --version There is no man page for rmerge2, but the usage message has been updated. > > Could you bundle all of this, with the ChangeLog > and the ebuild into a .tar.gz and attach it ? Unfortunately, the only real changelog I have is the notes of changes on these pages which is of limited use to anyone. I've tarred the updated rmerge2 script (sans Changelog) and the ebuild into an attachment (rmerge2-0.9.7.tar.gz). P.S. Oh, and I also fixed the (regexp) bug which prevented rmerge2 working with newer versions of portage. P.P.S. I added an option to allow the user to specify the state file explicitly ("--statefile").
Created attachment 5211 [details] rmerge2-0.9.7.tar.gz
Fixed. Available for testing as app-admin/rmerge2-0.9.7
This is what is happening: rmerge2 -f -s -e world rmerge2: must set CFLAGS in /etc/make.conf or environment In make.conf I have this line: CFLAGS="-march=athlon-tbird -O3 -pipe" and I have tried typing that in the console before I ran rmerge2... What's wrong?
See bug #11495.
