emerge appears to be modifying the system's world file rather than the one under ${ROOT}. In the following log, ROOT is set for a package merge, but /var/lib/portage/world is replaced rather than ${ROOT}/var/lib/portage/world # grep kde /var/lib/portage/world # ROOT="/alternaroot/" emerge --nodeps --ask kde These are the packages that would be merged, in order: Calculating dependencies... done! [ebuild N ] kde-base/kde-3.5.1 to /alternaroot/ Do you want me to merge these packages? [Yes/No] >>> Emerging (1 of 1) kde-base/kde-3.5.1 to /alternaroot/ ... >>> Merging kde-base/kde-3.5.1 to /alternaroot/ ... # grep kde /var/lib/portage/world kde-base/kde # grep kde /alternaroot/var/lib/portage/world # emerge --info Portage 2.1_pre5-r1 (default-linux/x86/2005.0, gcc-3.4.5, glibc-2.3.6-r3, 2.6.15-gentoo-r1 i686) ================================================================= System uname: 2.6.15-gentoo-r1 i686 Pentium III (Coppermine) Gentoo Base System version 1.12.0_pre16 distcc 2.18.3 i686-pc-linux-gnu (protocols 1 and 2) (default port 3632) [enabled] ccache version 2.4 [enabled] dev-lang/python: 2.3.5-r2, 2.4.2-r1 sys-apps/sandbox: 1.2.17 sys-devel/autoconf: 2.13, 2.59-r7 sys-devel/automake: 1.4_p6, 1.5, 1.6.3, 1.7.9-r1, 1.8.5-r3, 1.9.6-r1 sys-devel/binutils: 2.16.1-r1 sys-devel/libtool: 1.5.22 virtual/os-headers: 2.6.11-r3 ACCEPT_KEYWORDS="x86 ~x86" AUTOCLEAN="yes" CBUILD="i686-pc-linux-gnu" CFLAGS="-O2 -march=pentium3 -fomit-frame-pointer -pipe" CHOST="i686-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/kde/2/share/config /usr/kde/3.5/env /usr/kde/3.5/share/config /usr/kde/3.5/shutdown /usr/kde/3/share/config /usr/lib/X11/xkb /usr/share/config /var/qmail/control" CONFIG_PROTECT_MASK="/etc/gconf /etc/terminfo /etc/env.d" CXXFLAGS="-O2 -march=pentium3 -fomit-frame-pointer -pipe -fvisibility-inlines-hidden" DISTDIR="/usr/portage/distfiles" FEATURES="autoconfig ccache distcc sandbox sfperms strict" GENTOO_MIRRORS="http://distfiles.gentoo.org http://distro.ibiblio.org/pub/linux/distributions/gentoo" MAKEOPTS="-j7" PKGDIR="/scratch/pkgdir/" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" PORTDIR_OVERLAY="/usr/local/portage" SYNC="rsync://rsync.gentoo.org/gentoo-portage" USE="x86 X alsa apache2 apm audiofile avi bash-completion berkdb bitmap-fonts bzip2 crypt cups curl dbus emboss encode esd fam firefox foomaticdb gd gdbm gif glut gpm gstreamer gtk gtk2 gtkhtml hal idn imagemagick imlib ipv6 jpeg kde lcms ldap libg++ libwww mikmod mmx mng motif mp3 mpeg ncurses network nls nptl ogg oggvorbis opengl pam pdflib perl php png postgres python qt quicktime readline sasl sdl spell sse ssl tcpd tiff truetype truetype-fonts type1-fonts usb vhosts vorbis xml2 xv zlib elibc_glibc kernel_linux userland_GNU" Unset: ASFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, LANG, LC_ALL, LDFLAGS, LINGUAS
Recently I changed all the WORLD_FILE writes to use write_atomic and I noticed this bug while making those changes, but I didn't want to fix this bug in the same patch, so I left the bugs in place while I made the write_atomic changes. So, while I acknowledge touching the code and seeing the bug, afaik I didn't cause a regression. Jason, did you add the REGRESSION keyword because of my changes?
After looking through the code, I see that the problem is due to the wrong root being passed around somewhere (like bug 124334). On a related note, the global updates code doesn't do the world file from ${ROOT} even though the mtimedb from ${ROOT} is used for the update logic.
I'm not sure when or where it happened, but this did work correctly some time in the past.
Problem is the use of os.path.join() for concatenation of the root and portage.WORLD_FILE, because portage.WORLD_FILE is an absolute path, with leading "/". I will attach a patch which replaces the join() calls with a "+", like it is already at various other places in the code. I've also adding a missing trailing "\n". Oh, and just to make it clear, this bug is really a critical one, since it empties world file as soon as you emerge something in a new $ROOT. At least that's what just happened to me :-/ I think it would deserve a revbump ASAP.
Created attachment 81293 [details, diff] portage-2.1_pre5--root+WORLDFILE.patch
Created attachment 81331 [details, diff] patch that uses WORLDFILE.lstrip(os.sep) (In reply to comment #4) > Problem is the use of os.path.join() for concatenation of the root and > portage.WORLD_FILE, because portage.WORLD_FILE is an absolute path, with > leading "/". Thanks for discovering this. I've been bitten by this before and I should have known better. > I will attach a patch which replaces the join() calls with a "+", like it is > already at various other places in the code. I've also adding a missing > trailing "\n". We've already begun a migration toward using os.path.join, and I'd rather not reverse the trend. Therefore, I'd prefer to patch this with WORLDFILE.lstrip(os.sep) until we've migrated more code to use the os.path.join approach. As for the trailing "\n", it's not really needed because the supported method for adding atoms to the world file is `emerge --noreplace <atom>` (not `echo <atom> >> /var/lib/portage/world`). > Oh, and just to make it clear, this bug is really a critical one, since it > empties world file as soon as you emerge something in a new $ROOT. At least > that's what just happened to me :-/ > I think it would deserve a revbump ASAP. Agreed. I'll probably do it today.
(In reply to comment #6) > We've already begun a migration toward using os.path.join, and I'd rather not > reverse the trend. Therefore, I'd prefer to patch this with > WORLDFILE.lstrip(os.sep) until we've migrated more code to use the > os.path.join approach. Ok, it makes sense. Do you want a modified patch, or will you do it yourself? Oh, and just a suggestion: there are so many occurences of "root+foopath" that i think replacing them all with "os.path.join(portage.root,foopath.lstrip(os.sep))" will just make the code longer, and less readable. What about a shortcut function, like this?: def root_prefix(mypath,myroot=portage.root): return os.path.join(myroot, mypath.lstrip(os.sep)) Sure, it's really just a matter of taste, but imho "root_prefix(foopath)" looks better than its expanded equivalent. > As for the trailing "\n", it's not really needed because the > supported method for adding atoms to the world file is `emerge --noreplace > <atom>` (not `echo <atom> >> /var/lib/portage/world`). Please, could you reconsider this position? The world file format has been well known enough, for enough years, to be considered by many people (including me) as a standard interface to portage, almost a config file. It's really more than an internal, hidden, implementation detail. Btw, there is a whole lot of useful oneliners or scripts around which do touch this file the echo way (or other ways, like using some diff merging commands, etc.), and they are widely used. So, i may be wrong, but i would expect a significant number of world files corruptions if you keep this format change. Imho, it's just not "nice" for users, but if you have a good technical reason to do so. As for --noreplace, well, as far as i remember there has been no such thing in emerge during years, which is part of the explaination of why people are used to do it their own way. I think that if you really want them (us) to lose this habits, it can only go with a filename change. Thanks.
(In reply to comment #7) > Ok, it makes sense. Do you want a modified patch, or will you do it yourself? I've already attached it along with comment #6. > Oh, and just a suggestion: there are so many occurences of "root+foopath" that > i think replacing them all with > "os.path.join(portage.root,foopath.lstrip(os.sep))" will just make the code > longer, and less readable. > What about a shortcut function, like this?: > def root_prefix(mypath,myroot=portage.root): > return os.path.join(myroot, mypath.lstrip(os.sep)) > Sure, it's really just a matter of taste, but imho "root_prefix(foopath)" looks > better than its expanded equivalent. Long term, the lstrip won't be necessary because the leading path separator shouldn't be there to begin with. > > As for the trailing "\n", it's not really needed because the > > supported method for adding atoms to the world file is `emerge --noreplace > > <atom>` (not `echo <atom> >> /var/lib/portage/world`). > > Please, could you reconsider this position? > > The world file format has been well known enough, for enough years, to be > considered by many people (including me) as a standard interface to portage, > almost a config file. It's really more than an internal, hidden, implementation > detail. > Btw, there is a whole lot of useful oneliners or scripts around which do touch > this file the echo way (or other ways, like using some diff merging commands, > etc.), and they are widely used. So, i may be wrong, but i would expect a > significant number of world files corruptions if you keep this format change. > Imho, it's just not "nice" for users, but if you have a good technical reason > to do so. > As for --noreplace, well, as far as i remember there has been no such thing in > emerge during years, which is part of the explaination of why people are used > to do it their own way. I think that if you really want them (us) to lose this > habits, it can only go with a filename change. I understand your concern, and this type of change concerns me too, but we really don't want to encourage direct access to the world file. In addition, emaint has already written the world file without a trailing "\n" since portage-2.0.53, so to backtrack now would be even more questionable.
(In reply to comment #8) > > I've already attached it along with comment #6. Oops, sorry, i must be blind or something. > we really don't want to encourage direct access to the world file. I can understand that. I spend enough time on forums to know that some people do stupid things on their world file, like adding all installed packages, etc. :) That said, I'm not talking about encouraging anything, but simply about not breaking some existing legitimate practices without offering good alternatives. I mean, "regenworld" and "--noreplace" are simply not enough to cover all the situations where one needs to hack his world file. Hence the need to keep a proper text file format... > In addition, emaint has already written the world file without a > trailing "\n" since portage-2.0.53, so to backtrack now would be > even more questionable. Ok then, i guess this closes the discussion. Well, at least i've tried...
Released in 2.1_pre5-r4. (In reply to comment #9) > That said, I'm not talking about encouraging anything, but simply about not > breaking some existing legitimate practices without offering good alternatives. > I mean, "regenworld" and "--noreplace" are simply not enough to cover all the > situations where one needs to hack his world file. Hence the need to keep a > proper text file format... External programs can still use the world file directly if they must but they can't assume that there will be a trailing "\n". Perhaps a portageq query that lists the world packages would be helpful? This would allow the user to read and write the world file via portage commands, without needing to access /var/lib/portage/world directly.
