Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!

Bug 5667

Summary: rmerge-1.0.ebuild (New Package)
Product: Gentoo Linux Reporter: Dan Doel <dolio>
Component: New packagesAssignee: Karl Trygve Kalleberg (RETIRED) <karltk>
Severity: enhancement CC:, dolio, k, kevin, m.debruijne, mettlerd, mkrainer, rigo
Priority: High    
Version: unspecified   
Hardware: All   
OS: Linux   
Package list:
Runtime testing required: ---
Attachments: rmerge-1.0.ebuild (New Package)
rmerge-2.0.ebuild (Update)
Fixed rmerge ebuild + files
rmerge 2.2 (bugfix + added features)
rmerge-2.2.1 (PORTDIR_OVERLAY fix)
rmerge2-0.9.4.tar.gz (Get this one!)

Description Dan Doel 2002-07-28 01:38:30 UTC

Recently, because I thought it would be useful, both to myself and to other
people, I decided to write a script that would enable someone to perform an
emerge --emptytree and resume from wherever an error occurred in the process. I
posted this in the tips and tricks forum
(, however, someone suggested that
this script was useful enough that I should submit it for inclusion in the
portage tree. So I wrote an ebuild for it.

The script supports emerging the entire world, or one or more packages, and if
multiple packages are specified they can be emerged as a group or individually.
The script stores the information it needs to resume the merges in
/var/tmp/rmerge by default, but the script can easily be changed by editing a
variable at the top of the file to store the information somewhere else. It's
written in bash.

To the best of my knowledge, it performs as intended, but I'm not Don Knuth, so
I can't claim to have proven its correctness.

Attached you'll find rmerge-1.0.tar.gz, which should contain rmerge, the actual
shell script, rmerge-1.0.ebuild, the ebuild (of course), and rmerge.8, which is
a man page for the script.  I would suggest the app-admin category, because
that's the category the likes of portagemaster and kmerge have been put in, and
this seems like a similar type of application.

Also, this application obviously depends on portage, but that probably won't be
a problem for most users.

Thanks for your consideration.
Comment 1 Dan Doel 2002-07-28 01:42:15 UTC
Created attachment 2638 [details]
rmerge-1.0.ebuild (New Package)

This is the file referenced in the bug submission
Comment 2 Dan Doel 2002-07-28 01:47:10 UTC
I forgot to mention: the ebuild is written assuming that the script and the man
page are in the files directory under the ebuild directory (I took a cue from
the superadduser ebuild). The files are very small, so I don't think this should
be a problem.
Comment 3 Seemant Kulleen (RETIRED) gentoo-dev 2002-07-28 01:56:35 UTC
clever :)
Comment 4 Karl Trygve Kalleberg (RETIRED) gentoo-dev 2002-07-28 14:31:58 UTC
This ebuild has been selected as a testing candidate for the prototype Gentoo
User Ebuild Submission System. This means it'll initially be processed by an
automated validator. 

I'll get back to you with more details once the validator has been properly set up.
Comment 5 Dan Doel 2002-07-30 01:41:42 UTC
I hate to be a pain, but since I've posted this bug, I've some work, and I've
pretty much re-written the entire script to be more modular (broke stuff down
into functions), which, I think, makes it easier to understand if you're looking
at it, and also makes it easier to fit new functionality in. Also, I added some
new features, and fixed a minor bug/problem or two, and I think that the script
I currently have is quite a bit better than the one that's posted for addition
in this bug report (one important feature is the ability to resume an emerge
that was cut off by killing the process in some way (like your system losing

So, my question is, should I post the update here? Or do I wait until this
package is added to portage, and then file an update bug? I imagine that posting
the stuff here might cause problems for your automated handler.

Sorry about this, and thanks for your time.
Comment 6 Dan Doel 2002-07-31 13:43:53 UTC
Created attachment 2711 [details]
rmerge-2.0.ebuild (Update)

This is the update referenced in the previous post. Includes several new
options and some minor bug fixes.
Comment 7 Karl Trygve Kalleberg (RETIRED) gentoo-dev 2002-08-12 08:59:13 UTC
I've run your submission through the submission system. It found a few problems:
- (E) Missing Copyright statement
- (E) Missing $Header:$
- (E) Missing GPL license
- (E) Missing HOMEPAGE=
This can be empty, but it must be in there.
- (E) Missing DEPEND=
- (E) Missing LICENSE=
- (E) Missing KEYWORDS=

You may want to test this locally by installing dev-util/lintool and doing
lintool --show-details --ebuild <yourebuildfile>
Comment 8 Dan Doel 2002-08-12 13:14:26 UTC
Created attachment 3024 [details]
Fixed rmerge ebuild + files

Sorry about that. I fixed the problems listed, and ran the ebuild through
lintool, so I think it's correct now. The tarball has the updated ebuild, and
the man page and script in the files subdirectory (as they should be in the
ebuild directory).
Comment 9 vinz meier 2002-08-16 16:29:28 UTC
emerge -pe world says:
[ebuild  N   ] sys-libs/db-1.85-r1 to /
[ebuild  N   ] sys-libs/db-3.2.9-r1 to /

rmerge does:
>>> sys-libs/db-3.2.9-r1 merged.
>>> sys-libs/db-3.2.9-r1 merged.

i wouldnt be surprised if it would do the same with qt2 and qt3 etc.
Comment 10 Dan Doel 2002-08-17 00:40:15 UTC
Vinz: Hmm. At first, I thought that the --fix option fixed that problem, but it
just occurred to me that it doesn't...

I'm going to have to think about this for a little while...

Note: This bug would only occur if you have two versions of a library installed
on your system as you do.

Looks like I'm going to have to take the nasty way out and hop around to all the
different directories...

A big re-write indeed (perhaps warranting version 3.0 :)).

Comment 11 Dan Doel 2002-08-17 03:01:46 UTC
Created attachment 3156 [details]
rmerge 2.2 (bugfix + added features)

Okay, this version should fix the bug referenced above. It also includes some
extra options (BCFLAGS and --skip). Read the docs to find out about those.

If it's not too much trouble to test this (I know it emerges regular packages
correctly, so it should theoretically work for doubled packages, but I don't
have any on my system), I'd appreciate it. An easy way would to create an
appropriate file (like world.rmerge) with just the two packages inside in the


And then resume that merge. It _should_ work.

(Didn't make it version 3.0. The re-write was really easy :)).
Comment 12 burzmali 2002-08-17 13:00:40 UTC
i didn't know where to put the ebuild to install so i installed maually by 
just copying the script to /sbin and the manpage to   
/usr/man/man8.  the first time i tried to run it, however, it errored out   
saying 'mkdir /var/tmp/rmerge/tmp/  no such directory'.  there was no  
/var/tmp/rmerge dir to make tmp/ in, so i mkdired it and then ran it again and  
it worked.  it is compiling everything right now, good job.   
Comment 13 Bardur Arantsson 2002-08-19 00:56:46 UTC
The script doesn't seem to respect the value of the PORTDIR_OVERLAY variable. 
This is quite annoying, and basically means that "hobbyist" package builders 
like me cannot get it to properly build everything. 
It should be trivial to add, so consider this a bugfix/feature request... ;) 
Comment 14 Dan Doel 2002-08-19 13:28:55 UTC
Created attachment 3212 [details]
rmerge-2.2.1 (PORTDIR_OVERLAY fix)

My apologies on the PORTDIR_OVERLAY thing. I remember reading about it, but it
totally slipped my mind (since I'd never used it before).

Now, however, I have a place to put my ebuilds without having them killed by an

The script checks to see if the ebuild is in $PORTDIR_OVERLAY first, and then
falls back to $PORTDIR. I hope this is the correct behavior, since I'd think
that whatever ebuild you're working on would be more up to date than whatever
is in the main portage tree.

You can get the fix here (2.2.1) or at the rmerge homepage (the address of
which is above, I believe).

Comment 15 Bardur Arantsson 2002-08-20 15:09:59 UTC
Created attachment 3239 [details]

Hi there,

I've taken the liberty of re-implementing most (all?) of the functionality of
your script in Python. The motivation was the addition of the following

- It now possible to give the "--pretend" option to the script, which shows
which packages *would* be emerged.

- I've added the "--empty" (-e) option, which allows one to emulate the
functionality of your original rmerge script. However -- and here's where the
"new feature" part comes in -- if the flag is *absent*, only packages which
were actually mentioned on the command line will be emerged. This is very
useful to merge a list of packages with proper ordering (emerge doesn't do
this, at least not yet).

- The state is saved in /var/lib/rmerge2/MD5HASH where MD5HASH is the MD5 hash
of all the names of the packages specified on the command line. Btw, there seem
to be some issues with this in your rmerge script (I haven't bothered to
pinpoint them, but I couldn't get it working properly with multiple packages on
the command line). The file contains a pickled representation of the state of
the process (written to file in a crashproof way, of course). The
/var/lib/rmerge2 directory must be created before running the script.

I haven't bothered to add coloured output since it's not exactly a vital piece
of functionality. Run the script without arguments to get a usage message.

The script is about one third the size of the original rmerge (in bytes) and
slightly less than half the number of lines. I also (subjectively) think it's
much more readable.

I've tested the script quite thoroughly on my own machine, but there are of
course no guarantees. Please test and comment away. :)

(Btw, a little side remark on the PORTDIR_OVERLAY issue: It seems that it is
not necessary to actually find the .ebuild files, it is possible (in newer
versions at least) to specify a specific package version to emerge like so: 

   emerge =category/pkgname-1.2.3

This neatly avoids having to know about PORTDIR_OVERLAY. Just thought you'd
like to know :))
Comment 16 Dan Doel 2002-08-20 16:39:40 UTC
Heh. Wow.

Well, I guess we can discontinue adding to my script. :)

I'm not sure what was wrong with entering multiple package names when you were
using it (It seemed to work fine for me when I was testing stuff, but I might
have been careless). I noticed a problem with having names in different orders,
and thought I might MD5 them, but I had to sort them to use that anyway, so I
figured I'd just leave them as-is. I guess my code must be faulty there.

I wasn't aware of the =category/package-version feature. I remember trying to
emerge specific versions in the past (not with =), and having it fail unless I
specified the exact location of the ebuild. Knowing that would have probably
made the script more bug-free from the start. :)

Have you told people about your implementation on the forums (specifically, in
the old Tips and Tricks post I originally made, and, more importantly, in the
rebuilding with GCC 3.2 thread stuck at the top of Portage and Programming)? I'm
sure they'd like to have it.

It's probably better to have it written in Python, too, since it's more likely
to actually be included in Portage that way (which is where this really belongs,
because I think recompiling all your stuff is a somewhat common occurence). I'd
have done it in Python myself, but I never finished the book I bought on it (bad
book, plus I was already spoiled with the book on Ruby I'd just read :)).

Anyway, I'll be unavailable for the next couple days, so you couldn't have
announced this at a better time. :) Now I don't have to worry about any bugs in
my script while I'm gone.

Comment 17 Daniel Mettler 2002-08-20 19:48:53 UTC
nice tool :) i tested rmerge-2.2.1 and it works fine here (resuming etc.). for 
better usability i'd just suggest to make rmerge's options a bit more 
consistent with those of emerge. i.e. -p should preferrably be used for 
--pretend only, not --previous. etc. 
rmerge showed to be great in conjunction with spider's tips for upgrading 
gentoo (e.g. from default-1.0-gcc3 to default-x86-2.0):     
quoting spider from "[gentoo-dev] Gentoo 1.2 to 1.4 update, HowTo do it and 
not break everything":    
I think we can / should be able to cut the procedure down : 
original profile: 
emerge --buildpkg gcc glibc binutils 
move aside 
change profile 
export USE="-* build" 
emerge gcc texinfo binutils gettext 
unset USE 
emerge --buildpkg glibc texinfo gettext binutils gcc 
Now, the original USE="build" gcc replaced /usr/bin/gcc, and this second 
"emerge gcc" then doesn't realize theres a different version of gcc 
there, and overwrites gcc, c++ and all.. Neato. no more emerge -C gcc 
After this, re-add the /etc/env.d/05gcc-old with the gcc 2.95.3 path, 
run env-update and test by launching... mozilla? 
then i just run 'rmerge world' (which could be further optimized if we could 
force-exclude specific ebuilds from rmerg'ing). 
later, i will test rmerge 2. 
Comment 18 Bardur Arantsson 2002-08-21 08:53:35 UTC
Created attachment 3255 [details]

Fixed listing of failed packages.
Comment 19 Bardur Arantsson 2002-08-21 12:16:53 UTC
Created attachment 3266 [details]

- Fixes a typo in handling of --pretend.
- Fixes environment propagation from rmerge2 to
Comment 20 Bardur Arantsson 2002-08-22 03:38:54 UTC
Created attachment 3300 [details]

Grrr... never re-factor without re-testing... This version fixes:

- Proper handling of "--force"
- Fixed typo which prevented proper "skipping" behavior
Comment 21 Bardur Arantsson 2002-08-22 03:44:18 UTC
Created attachment 3302 [details]
rmerge2-0.9.4.tar.gz (Get this one!)

*Sigh*... never commit anything to bugzilla while drunk ;)
Comment 22 Rigo 2002-08-23 04:25:53 UTC
portage # rmerge2 -p -e world
rmerge2: emerge returned bad exit status. cannot continue.
portage # 

Boohoo, why does this alway happens to me ;) !
Comment 23 Rigo 2002-08-23 04:30:57 UTC
portage # rmerge2 -p -e world
rmerge2: emerge returned bad exit status. cannot continue.
portage # 

Boohoo, why does this alway happens to me ;) !
Comment 24 Bardur Arantsson 2002-08-23 04:46:46 UTC
Created attachment 3343 [details]

> portage # rmerge2 -p -e world
> rmerge2: emerge returned bad exit status. cannot continue.
> portage #
> Boohoo, why does this alway happens to me ;) !

This is usually indicative of some sort of (possibly quite minor) inconsistency
in the portage database. I've changed the rmerge2 script to actually display
the output from emerge in this case, so that it is actually possible to see
what's wrong. :)

Also fixed a minor issue with not flushing stdout (which would cause a bit of
weirdness in the progress messages).
Comment 25 Rigo 2002-08-23 05:59:53 UTC
Ah, see below (=local ebuild errors), much better thank you ;)


These are the packages that I would merge, in order.

Calculating world dependencies \
!!! Error: couldn't find match for "media-libs/sndlib" in media-sound/ceres3-0.27

rmerge2: emerge returned bad exit status. cannot continue.
Comment 26 Rigo 2002-08-23 06:50:25 UTC
lappy portage # rmerge2 -pe world|wc -l

No wonder I'm so busy ;)

Kewl script...You made my day !

Comment 27 Bardur Arantsson 2002-08-25 06:45:53 UTC
Created attachment 3406 [details]

I've updated the package handling to allow for input package names to contain
version numbers.

This allows one to do something like the following:

rmerge2 -p -e world > package.list
[...edit "package.list" to remove/add packages...]
rmerge2 `package.list`

This can come in handy if you wish to recompile almost everything, but don't
want to have to edit the "package.list" to remove version numbers in addition
to packages.

Barring bugs, this is most probably the final version.
Comment 28 Bill Kenworthy 2002-08-28 20:37:59 UTC
On first use had to manually create /var/lib/rmerge2 directory.  error message
was a bit cryptic, but its ticking over nicely now.

Comment 29 Aniruddha Shankar 2002-10-17 17:05:16 UTC
>./rmerge2 -p -e world
I would merge the following packages (in order):


Comment 30 Karl Trygve Kalleberg (RETIRED) gentoo-dev 2002-10-18 17:23:13 UTC
Okay, this one will go in if we can manage to fix a few problems with it:

1) rmerge needs a --version switch
2) should be installed either in /usr/lib/rmerge2/ or
/usr/share/rmerge2/ (probably the latter since the script is platform-independent).
3) rmerge should call from its new location, not assume it's in the path.
4) perhaps update the man page with --version

Could you bundle all of this, with the ChangeLog and the ebuild into a .tar.gz
and attach it ?
Comment 31 Aniruddha Shankar 2002-10-18 19:16:01 UTC
I ran rmerge2 from /usr/share/rmerge2 and I got the same error: 

>./rmerge2 -p -e world
would merge the following packages (in order):

Comment 32 Bardur Arantsson 2002-10-30 08:38:45 UTC
> Okay, this one will go in if we can manage to fix a few problems with it:   
> 1) rmerge needs a --version switch   
> 2) should be installed either in /usr/lib/rmerge2/ or   
> /usr/share/rmerge2/ (probably the latter since the script is   
> platform-independent).   
> 3) rmerge should call from its new location, not assume   
> it's in the path.   
I've gone one better and eliminated the wrapper script completely. Instead of   
the wrapper, the "portage" python module is used to get the   
The ebuild simply installs the rmerge2 script in /usr/bin which is the same  
location that emerge is installed in.  
> 4) perhaps update the man page with --version   
There is no man page for rmerge2, but the usage message has been updated. 
> Could you bundle all of this, with the ChangeLog   
> and the ebuild into a .tar.gz and attach it ?   
Unfortunately, the only real changelog I have is the notes of changes on these 
pages which is of limited use to anyone.  
I've tarred the updated rmerge2 script (sans Changelog) and the ebuild into an 
attachment (rmerge2-0.9.7.tar.gz). 
P.S. Oh, and I also fixed the (regexp) bug which prevented rmerge2 working  
with newer versions of portage.  
P.P.S. I added an option to allow the user to specify the state file  
explicitly ("--statefile").  
Comment 33 Bardur Arantsson 2002-10-30 08:39:26 UTC
Created attachment 5211 [details]
Comment 34 Karl Trygve Kalleberg (RETIRED) gentoo-dev 2002-10-31 12:10:34 UTC
Fixed. Available for testing as app-admin/rmerge2-0.9.7
Comment 35 Art "fREW" Schmidt 2003-03-12 16:46:21 UTC
This is what is happening:

rmerge2 -f -s -e world          
rmerge2: must set CFLAGS in /etc/make.conf or environment

In make.conf I have this line:  CFLAGS="-march=athlon-tbird -O3 -pipe"
and I have tried typing that in the console before I ran rmerge2...

What's wrong?
Comment 36 Bardur Arantsson 2003-03-12 17:07:14 UTC
See bug #11495.