Recently, because I thought it would be useful, both to myself and to other
people, I decided to write a script that would enable someone to perform an
emerge --emptytree and resume from wherever an error occurred in the process. I
posted this in the tips and tricks forum
(http://forums.gentoo.org/viewtopic.php?t=9638), however, someone suggested that
this script was useful enough that I should submit it for inclusion in the
portage tree. So I wrote an ebuild for it.
The script supports emerging the entire world, or one or more packages, and if
multiple packages are specified they can be emerged as a group or individually.
The script stores the information it needs to resume the merges in
/var/tmp/rmerge by default, but the script can easily be changed by editing a
variable at the top of the file to store the information somewhere else. It's
written in bash.
To the best of my knowledge, it performs as intended, but I'm not Don Knuth, so
I can't claim to have proven its correctness.
Attached you'll find rmerge-1.0.tar.gz, which should contain rmerge, the actual
shell script, rmerge-1.0.ebuild, the ebuild (of course), and rmerge.8, which is
a man page for the script. I would suggest the app-admin category, because
that's the category the likes of portagemaster and kmerge have been put in, and
this seems like a similar type of application.
Also, this application obviously depends on portage, but that probably won't be
a problem for most users.
Thanks for your consideration.
Created attachment 2638 [details]
rmerge-1.0.ebuild (New Package)
This is the file referenced in the bug submission
I forgot to mention: the ebuild is written assuming that the script and the man
page are in the files directory under the ebuild directory (I took a cue from
the superadduser ebuild). The files are very small, so I don't think this should
be a problem.
This ebuild has been selected as a testing candidate for the prototype Gentoo
User Ebuild Submission System. This means it'll initially be processed by an
I'll get back to you with more details once the validator has been properly set up.
I hate to be a pain, but since I've posted this bug, I've some work, and I've
pretty much re-written the entire script to be more modular (broke stuff down
into functions), which, I think, makes it easier to understand if you're looking
at it, and also makes it easier to fit new functionality in. Also, I added some
new features, and fixed a minor bug/problem or two, and I think that the script
I currently have is quite a bit better than the one that's posted for addition
in this bug report (one important feature is the ability to resume an emerge
that was cut off by killing the process in some way (like your system losing
So, my question is, should I post the update here? Or do I wait until this
package is added to portage, and then file an update bug? I imagine that posting
the stuff here might cause problems for your automated handler.
Sorry about this, and thanks for your time.
Created attachment 2711 [details]
This is the update referenced in the previous post. Includes several new
options and some minor bug fixes.
I've run your submission through the submission system. It found a few problems:
- (E) Missing Copyright statement
- (E) Missing $Header:$
- (E) Missing GPL license
- (E) Missing HOMEPAGE=
This can be empty, but it must be in there.
- (E) Missing DEPEND=
- (E) Missing LICENSE=
- (E) Missing KEYWORDS=
You may want to test this locally by installing dev-util/lintool and doing
lintool --show-details --ebuild <yourebuildfile>
Created attachment 3024 [details]
Fixed rmerge ebuild + files
Sorry about that. I fixed the problems listed, and ran the ebuild through
lintool, so I think it's correct now. The tarball has the updated ebuild, and
the man page and script in the files subdirectory (as they should be in the
emerge -pe world says:
[ebuild N ] sys-libs/db-1.85-r1 to /
[ebuild N ] sys-libs/db-3.2.9-r1 to /
>>> sys-libs/db-3.2.9-r1 merged.
>>> sys-libs/db-3.2.9-r1 merged.
i wouldnt be surprised if it would do the same with qt2 and qt3 etc.
Vinz: Hmm. At first, I thought that the --fix option fixed that problem, but it
just occurred to me that it doesn't...
I'm going to have to think about this for a little while...
Note: This bug would only occur if you have two versions of a library installed
on your system as you do.
Looks like I'm going to have to take the nasty way out and hop around to all the
A big re-write indeed (perhaps warranting version 3.0 :)).
Created attachment 3156 [details]
rmerge 2.2 (bugfix + added features)
Okay, this version should fix the bug referenced above. It also includes some
extra options (BCFLAGS and --skip). Read the docs to find out about those.
If it's not too much trouble to test this (I know it emerges regular packages
correctly, so it should theoretically work for doubled packages, but I don't
have any on my system), I'd appreciate it. An easy way would to create an
appropriate file (like world.rmerge) with just the two packages inside in the
And then resume that merge. It _should_ work.
(Didn't make it version 3.0. The re-write was really easy :)).
i didn't know where to put the ebuild to install so i installed maually by
just copying the script to /sbin and the manpage to
/usr/man/man8. the first time i tried to run it, however, it errored out
saying 'mkdir /var/tmp/rmerge/tmp/ no such directory'. there was no
/var/tmp/rmerge dir to make tmp/ in, so i mkdired it and then ran it again and
it worked. it is compiling everything right now, good job.
The script doesn't seem to respect the value of the PORTDIR_OVERLAY variable.
This is quite annoying, and basically means that "hobbyist" package builders
like me cannot get it to properly build everything.
It should be trivial to add, so consider this a bugfix/feature request... ;)
Created attachment 3212 [details]
rmerge-2.2.1 (PORTDIR_OVERLAY fix)
My apologies on the PORTDIR_OVERLAY thing. I remember reading about it, but it
totally slipped my mind (since I'd never used it before).
Now, however, I have a place to put my ebuilds without having them killed by an
The script checks to see if the ebuild is in $PORTDIR_OVERLAY first, and then
falls back to $PORTDIR. I hope this is the correct behavior, since I'd think
that whatever ebuild you're working on would be more up to date than whatever
is in the main portage tree.
You can get the fix here (2.2.1) or at the rmerge homepage (the address of
which is above, I believe).
Created attachment 3239 [details]
I've taken the liberty of re-implementing most (all?) of the functionality of
your script in Python. The motivation was the addition of the following
- It now possible to give the "--pretend" option to the script, which shows
which packages *would* be emerged.
- I've added the "--empty" (-e) option, which allows one to emulate the
functionality of your original rmerge script. However -- and here's where the
"new feature" part comes in -- if the flag is *absent*, only packages which
were actually mentioned on the command line will be emerged. This is very
useful to merge a list of packages with proper ordering (emerge doesn't do
this, at least not yet).
- The state is saved in /var/lib/rmerge2/MD5HASH where MD5HASH is the MD5 hash
of all the names of the packages specified on the command line. Btw, there seem
to be some issues with this in your rmerge script (I haven't bothered to
pinpoint them, but I couldn't get it working properly with multiple packages on
the command line). The file contains a pickled representation of the state of
the process (written to file in a crashproof way, of course). The
/var/lib/rmerge2 directory must be created before running the script.
I haven't bothered to add coloured output since it's not exactly a vital piece
of functionality. Run the script without arguments to get a usage message.
The script is about one third the size of the original rmerge (in bytes) and
slightly less than half the number of lines. I also (subjectively) think it's
much more readable.
I've tested the script quite thoroughly on my own machine, but there are of
course no guarantees. Please test and comment away. :)
(Btw, a little side remark on the PORTDIR_OVERLAY issue: It seems that it is
not necessary to actually find the .ebuild files, it is possible (in newer
versions at least) to specify a specific package version to emerge like so:
This neatly avoids having to know about PORTDIR_OVERLAY. Just thought you'd
like to know :))
Well, I guess we can discontinue adding to my script. :)
I'm not sure what was wrong with entering multiple package names when you were
using it (It seemed to work fine for me when I was testing stuff, but I might
have been careless). I noticed a problem with having names in different orders,
and thought I might MD5 them, but I had to sort them to use that anyway, so I
figured I'd just leave them as-is. I guess my code must be faulty there.
I wasn't aware of the =category/package-version feature. I remember trying to
emerge specific versions in the past (not with =), and having it fail unless I
specified the exact location of the ebuild. Knowing that would have probably
made the script more bug-free from the start. :)
Have you told people about your implementation on the forums (specifically, in
the old Tips and Tricks post I originally made, and, more importantly, in the
rebuilding with GCC 3.2 thread stuck at the top of Portage and Programming)? I'm
sure they'd like to have it.
It's probably better to have it written in Python, too, since it's more likely
to actually be included in Portage that way (which is where this really belongs,
because I think recompiling all your stuff is a somewhat common occurence). I'd
have done it in Python myself, but I never finished the book I bought on it (bad
book, plus I was already spoiled with the book on Ruby I'd just read :)).
Anyway, I'll be unavailable for the next couple days, so you couldn't have
announced this at a better time. :) Now I don't have to worry about any bugs in
my script while I'm gone.
nice tool :) i tested rmerge-2.2.1 and it works fine here (resuming etc.). for
better usability i'd just suggest to make rmerge's options a bit more
consistent with those of emerge. i.e. -p should preferrably be used for
--pretend only, not --previous. etc.
rmerge showed to be great in conjunction with spider's tips for upgrading
gentoo (e.g. from default-1.0-gcc3 to default-x86-2.0):
quoting spider from "[gentoo-dev] Gentoo 1.2 to 1.4 update, HowTo do it and
not break everything":
I think we can / should be able to cut the procedure down :
emerge --buildpkg gcc glibc binutils
export USE="-* build"
emerge gcc texinfo binutils gettext
emerge --buildpkg glibc texinfo gettext binutils gcc
Now, the original USE="build" gcc replaced /usr/bin/gcc, and this second
"emerge gcc" then doesn't realize theres a different version of gcc
there, and overwrites gcc, c++ and all.. Neato. no more emerge -C gcc
After this, re-add the /etc/env.d/05gcc-old with the gcc 2.95.3 path,
run env-update and test by launching... mozilla?
then i just run 'rmerge world' (which could be further optimized if we could
force-exclude specific ebuilds from rmerg'ing).
later, i will test rmerge 2.
Created attachment 3255 [details]
Fixed listing of failed packages.
Created attachment 3266 [details]
- Fixes a typo in handling of --pretend.
- Fixes environment propagation from rmerge2 to rmerge2.py
Created attachment 3300 [details]
Grrr... never re-factor without re-testing... This version fixes:
- Proper handling of "--force"
- Fixed typo which prevented proper "skipping" behavior
Created attachment 3302 [details]
rmerge2-0.9.4.tar.gz (Get this one!)
*Sigh*... never commit anything to bugzilla while drunk ;)
portage # rmerge2 -p -e world
rmerge2: emerge returned bad exit status. cannot continue.
Boohoo, why does this alway happens to me ;) !
Created attachment 3343 [details]
> portage # rmerge2 -p -e world
> rmerge2: emerge returned bad exit status. cannot continue.
> portage #
> Boohoo, why does this alway happens to me ;) !
This is usually indicative of some sort of (possibly quite minor) inconsistency
in the portage database. I've changed the rmerge2 script to actually display
the output from emerge in this case, so that it is actually possible to see
what's wrong. :)
Also fixed a minor issue with not flushing stdout (which would cause a bit of
weirdness in the progress messages).
Ah, see below (=local ebuild errors), much better thank you ;)
These are the packages that I would merge, in order.
Calculating world dependencies \
!!! Error: couldn't find match for "media-libs/sndlib" in media-sound/ceres3-0.27
rmerge2: emerge returned bad exit status. cannot continue.
lappy portage # rmerge2 -pe world|wc -l
No wonder I'm so busy ;)
Kewl script...You made my day !
Created attachment 3406 [details]
I've updated the package handling to allow for input package names to contain
This allows one to do something like the following:
rmerge2 -p -e world > package.list
[...edit "package.list" to remove/add packages...]
This can come in handy if you wish to recompile almost everything, but don't
want to have to edit the "package.list" to remove version numbers in addition
Barring bugs, this is most probably the final version.
On first use had to manually create /var/lib/rmerge2 directory. error message
was a bit cryptic, but its ticking over nicely now.
>./rmerge2 -p -e world
I would merge the following packages (in order):
Okay, this one will go in if we can manage to fix a few problems with it:
1) rmerge needs a --version switch
2) rmerge2.py should be installed either in /usr/lib/rmerge2/ or
/usr/share/rmerge2/ (probably the latter since the script is platform-independent).
3) rmerge should call rmerge2.py from its new location, not assume it's in the path.
4) perhaps update the man page with --version
Could you bundle all of this, with the ChangeLog and the ebuild into a .tar.gz
and attach it ?
I ran rmerge2 from /usr/share/rmerge2 and I got the same error:
>./rmerge2 -p -e world
would merge the following packages (in order):
> Okay, this one will go in if we can manage to fix a few problems with it:
> 1) rmerge needs a --version switch
> 2) rmerge2.py should be installed either in /usr/lib/rmerge2/ or
> /usr/share/rmerge2/ (probably the latter since the script is
> 3) rmerge should call rmerge2.py from its new location, not assume
> it's in the path.
I've gone one better and eliminated the wrapper script completely. Instead of
the wrapper, the "portage" python module is used to get the
CFLAGS/CXXFLAGS/BCFLAGS/BCXXFLAGS variables from the
The ebuild simply installs the rmerge2 script in /usr/bin which is the same
location that emerge is installed in.
> 4) perhaps update the man page with --version
There is no man page for rmerge2, but the usage message has been updated.
> Could you bundle all of this, with the ChangeLog
> and the ebuild into a .tar.gz and attach it ?
Unfortunately, the only real changelog I have is the notes of changes on these
pages which is of limited use to anyone.
I've tarred the updated rmerge2 script (sans Changelog) and the ebuild into an
P.S. Oh, and I also fixed the (regexp) bug which prevented rmerge2 working
with newer versions of portage.
P.P.S. I added an option to allow the user to specify the state file
Created attachment 5211 [details]
Fixed. Available for testing as app-admin/rmerge2-0.9.7
This is what is happening:
rmerge2 -f -s -e world
rmerge2: must set CFLAGS in /etc/make.conf or environment
In make.conf I have this line: CFLAGS="-march=athlon-tbird -O3 -pipe"
and I have tried typing that in the console before I ran rmerge2...
See bug #11495.