I ran into this problem while using catalyst. I disabled the 'minimal' USE flag in my livecd-stage1.spec and rebuilt it expecting xorg-x11 to be recompiled. Instead, the existing binary (from pkgcache) was used. I got around it by just removing the binary, but it was still annoying. It would make more sense for the --newuse flag to override the --usepkg flag.
*** Bug 81872 has been marked as a duplicate of this bug. ***
I am trying to patch emerge to handle this problem, (actually bug 81872, but should be the same fix?) but I am having some problems with figuring out how to get a) System applied USE flags b) binary package applied USE flags. Is there portage programming documentation anywhere? Right now emerge looks like a bunch of spaghetti to me!
Created attachment 51244 [details, diff] patches emerge 2.0.51 so that --usepkg recompiles from source if binary package USE flags do not match system USE flags WFM, but an experienced emerge programmer should check how proper the code is, especially how the various USE flags are retrieved. There are some debug "print" left in to make it easier to sanity check. This is my first time hacking emerge (and programming in python) so be expect some ugliness! :)
Discovered a bug today with my patch above: USE flags set in /etc/portage/package.use are not gotten correctly. The solution would be to get the global and package.use USE flags, not just the global. If someone who knew portage/emerge could make this fix I would be grateful.
Created attachment 53049 [details, diff] (an attempt) to cause emerge to use source rather than binary pkg if binary pkg USE flags different than system USE flags I think the following patch does a better job of closing bug 81182 ("--usepkg overrides --newuse")and bug 81872 ("emerge -k (or --usepkg) ignores USE flags") The function "useflagchanged" which is central to the working of the patch is basically a copy and modification of the --verbose USE flag code. There could easily be little bugs throughout the code given that it has been moved out of its original context without much understanding on my part. 1. Changes behavior of --usepkg to only use a binary package if the package applied USE flags and system applied USE flags match. (This seems to be correct behavior according to the man page.) 2. fixes problem in change() --newuse detection in which USE flags from /etc/portage/package.use are not included in calculations.
ping to self.
It was just a temporary case of blindness but I'm all better now.
Created attachment 58658 [details, diff] Simpler patch Gah.. Commented on the wrong bug. Anyway, here's the patch that I'll use to fix this assuming there are no problems with it. It only scans once and localizes the changes.
Your patch got cut off. There is a long line near the end that ends with $ like you copy and pasted from nano.
Created attachment 58673 [details, diff] fixed "simpler patch" I think is this a correct edit to the jstubbs "Simpler patch"
Created attachment 58676 [details, diff] even simpler patch? NO. only difference from revised simpler patch is I've changed "(old_use.count(x) and not now_use.count(x)) or (not old_use.count(x) and now_use.count(x))" to "(old_use.count(x) is not now_use.count(x))" which I am fairly certain is the same, but easier to understand. The first logic test is also used when displaying USE flags in verbose mode and could be replaced with the second form there as well.
fixed "simpler patch" doesn't work for me. In the following I turn off the kde USE flag. openoffice binary package with USE kde is incorrectly listed. The previous patch labeled "(an attempt) to cause emerge to use source rather than binary pkg if binary pkg USE flags different than system USE flags" does correctly not use the openoffice binary package. Should I post "debug" output (a big load of text)? _____fixed "simpler patch" output______________________ USE="-kde" emerge.jstubbs.orig -vp --usepkg --newuse --debug world These are the packages that I would merge, in order: [binary R ] app-office/lyx-1.3.5 +cups -debug -gnome +nls +qt [binary R ] net-misc/unison-2.9.1-r1 +gtk [binary R ] media-gfx/gimp-2.0.4 -aalib (-altivec) -debug -doc -gimpprint +jpeg +mmx -mng +png +python -sse +svg +tiff +wmf [binary R ] mail-client/mozilla-thunderbird-1.0.2 +crypt -debug -gnome -java -ldap -moznoxft -xinerama +xprint [binary R ] media-video/mplayer-1.0_pre6-r4 -3dfx +3dnow -3dnowext +X -aalib -alsa (-altivec) -arts +avi -bidi -cdparanoia -debug -dga -directfb -divx4linux -doc -dts -dv -dvb +dvd +dvdread -edl -encode -esd -fbcon -ggi +gif +gtk -i8x0 -ipv6 -jack -joystick +jpeg -libcaca -lirc +live -lzo -mad -matroska -matrox +mmx +mmxext +mpeg -mythtv -nas +nls +nvidia +oggvorbis +opengl +oss +png +real -rtc -samba -sdl -sse -sse2 -svga -tga -theora +truetype -v4l -v4l2 -xanim -xinerama -xmms +xv -xvid -xvmc [binary R ] app-editors/scite-1.6.2 +gtk2 -lua [binary R ] x11-base/xorg-x11-6.8.2-r1 -3dfx +3dnow +bitmap-fonts -cjk -debug -dlloader -dmx -doc +font-server -hardened -insecure-drivers -ipv6 -minimal +mmx +nls +opengl +pam -sdk -sse -static +truetype-fonts +type1-fonts (-uclibc) +xprint +xv [binary R ] media-video/kplayer-0.5.2 -arts +debug -kdeenablefinal -xinerama [binary R ] app-office/openoffice-1.1.4-r1 -curl -debug -hardened -java +kde -nptl +zlib Total size of downloads: 0 kB ________________________________________________________ __"(an attempt) to cause emerge to use source rather than binary pkg if binary pkg USE flags different than system USE flags" output ________________ USE="-kde" emerge.newusepkg -vp --usepkg --newuse world These are the packages that I would merge, in order: Calculating world dependencies ...done! [ebuild R ] mail-client/mozilla-thunderbird-1.0.2 +crypt -debug -gnome -java -ldap -moznoxft -xinerama -xprint* 0 kB [ebuild R ] app-editors/scite-1.6.2 -gtk2* -lua 0 kB [ebuild R ] x11-base/xorg-x11-6.8.2-r1 -3dfx +3dnow +bitmap-fonts -cjk -debug -dlloader -dmx -doc -font-server* -hardened -insecure-drivers -ipv6 -minimal +mmx +nls +opengl +pam -sdk -sse -static +truetype-fonts +type1-fonts (-uclibc) -xprint* +xv 0 kB [ebuild R ] media-video/kplayer-0.5.2 -arts -debug* -kdeenablefinal -xinerama 0 kB [ebuild R ] app-office/openoffice-1.1.4-r1 -curl -debug -hardened -java -kde* -nptl +zlib 0 kB Total size of downloads: 0 kB ___________________________________________________
"(old_use.count(x) and not now_use.count(x)) or (not old_use.count(x) and now_use.count(x))" It's not the same. I don't know if the case ever occurs anymore, but this way accounts for duplicate USE flags whereas testing for equality doesn't. Either way, that is what got cut off. Ok, I'll go back to the drawing-board and actually test this time. You're patch is good, but makes changes in a lot of places. Whatever patch goes in to stable should fix the case of --usepkg --newuse only and (if the patch is broken for some small case) can't affect any other case at all.
Created attachment 58757 [details, diff] Working "simpler patch" ;) Not sure how anybody actually got results with the previous patch. When I got around to actually trying it, it tracebacked due to undefined globals. Once I fixed that, it worked without issue. jason@linux ~ $ emerge -kvNp bzip2 These are the packages that I would merge, in order: Calculating dependencies ...done! [binary R ] app-arch/bzip2-1.0.3 -build -debug -static Total size of downloads: 0 kB jason@linux ~ $ USE="static" emerge -kvNp bzip2 These are the packages that I would merge, in order: Calculating dependencies ...done! [ebuild R ] app-arch/bzip2-1.0.3 -build -debug +static* 0 kB Total size of downloads: 0 kB
A few comments: 1)____________________ I am using what looks like a different emerge than you (I've been manually patching emerge): Mine (from emerge header): # $Header: /var/cvsroot/gentoo-src/portage/bin/emerge,v 1.345.2.18 2005/03/01 15:00:41 jstubbs Exp $ from sys-apps/portage Latest version available: 2.0.51.19 Latest version installed: 2.0.51.19 Yours (from patch info): --- emerge 12 May 2005 15:20:22 -0000 1.345.2.29 2)_________________________ I have no idea why the first "simpler patch" worked: It doesn't now! (Thought maybe it was "world" versus package name, but world also gives a traceback for me now. 3)____________________ Try running the new patch on "world" rather than a single package. I get the same results as earlier. (Specifically, note that openoffice has 'kde' while the command line is '-kde'. I am fairly certain (I've said that before!) that the problem arises when the package is in the world file and whether an ebuild or binary is used is determined in xcreate(), not create(). My patch has a check for USE flags in xcreate() because of this problem. It might be nicer to have only the decision "source or binary?" made at only one point in the code. Instead of having the decision take place in both create() (for command line package names and everything but world file packages) and xcreate() (for world file packages) it might be nicer to have xcreate() pass only the package name to create() (kind of like when a single package is named at command line) and have create() decide "source or binary?". I played with this, but it got complicated. :( #USE="-kde" emerge.jstubbs.2 -pkv --newuse --usepkg world These are the packages that I would merge, in order: Calculating world dependencies ...done! [binary R ] app-office/lyx-1.3.5 +cups -debug -gnome +nls +qt [binary R ] net-misc/unison-2.9.1-r1 +gtk [binary R ] media-gfx/gimp-2.2.6-r1 -aalib (-altivec) -debug -doc -gimpprint -gtkhtml +jpeg +mmx +mng +png +python -sse +svg +tiff +wmf [binary R ] mail-client/mozilla-thunderbird-1.0.2 +crypt -debug -gnome -java -ldap -moznoxft -xinerama +xprint [binary R ] media-video/mplayer-1.0_pre6-r4 -3dfx +3dnow -3dnowext +X -aalib -alsa (-altivec) -arts +avi -bidi -cdparanoia -debug -dga -directfb -divx4linux -doc -dts -dv -dvb +dvd +dvdread -edl -encode -esd -fbcon -ggi +gif +gtk -i8x0 -ipv6 -jack -joystick +jpeg -libcaca -lirc +live -lzo -mad -matroska -matrox +mmx +mmxext +mpeg -mythtv -nas +nls +nvidia +oggvorbis +opengl +oss +png +real -rtc -samba -sdl -sse -sse2 -svga -tga -theora +truetype -v4l -v4l2 -xanim -xinerama -xmms +xv -xvid -xvmc [binary R ] app-editors/scite-1.6.2 +gtk2 -lua [binary R ] x11-base/xorg-x11-6.8.2-r1 -3dfx +3dnow +bitmap-fonts -cjk -debug -dlloader -dmx -doc +font-server -hardened -insecure-drivers -ipv6 -minimal +mmx +nls +opengl +pam -sdk -sse -static +truetype-fonts +type1-fonts (-uclibc) +xprint +xv [binary R ] media-video/kplayer-0.5.2 -arts +debug -kdeenablefinal -xinerama [binary R ] app-office/openoffice-1.1.4-r1 -curl -debug -hardened -java +kde -nptl +zlib
Created attachment 58820 [details, diff] Getting messy... Yep. Emerge is pretty messy. I'm starting to think that it would be better to separate it out, but went ahead and just copied the code in the previous patch to xcreate instead. Fortunately, pretty much all of emerge and most of the binpkg-specific code in the rest of portage will be thrown out and done in a much nicer fashion in the next big version.
Created attachment 58846 [details, diff] getting Messy++, for emerge in portage-2.0.51.19 I've tested "getting messy..." and it still fails the "USE="-kde" emerge -pv --newuse --usepkg world" test (above). Looking at the "(an attempt) to cause emerge to use source rather than binary pkg if binary pkg USE flags different than system USE flags" patch which doesn't fail this test, the fix seems to be to set binpkguseflags = None in xcreate() if it decides to use an ebuild rather than a binary. The other change made to "getting messy" was to cut and paste the USE flag detection code to the --newuse part of create(). The replaced code did not detect /etc/portage/package.use USE flags correctly. Yeah, like you said, might make sense to break the USE flag detection to a different function. Besides the three places it is now, it might also be used for the --verbose printout. Hope it works as expected now! :) p.s. instead of create() in my previous post, I meant select_dep() please search and replace :)
package.use was being checked. The only issue I found was that the USE flags from the binary were being used incorrectly. Just killing binpkguseflags made it all work properly here.
Hi, The reason why I thought the --newuse part of create() was not detecting /etc/portage/package.use USE flags correctly was that without modification emerge wants to emerge a bunch of binaries, all of which are in package.use, and none of which have changed USE flags (correctly reported in their verbose USE flag output, no *s). With the addition of the modification to --newuse of create(), emerge no longer feels the urge to merge these binaries. # cat /etc/portage/package.use net-misc/unison gtk app-office/lyx qt media-gfx/gimp wmf mng media-video/mplayer dvdread live gtk avi app-office/openoffice kde x11-libs/wxGTK gtk2 _______________________________________________________________ #emerge.noModCreate -pv --usepkg --newuse world These are the packages that I would merge, in order: Calculating world dependencies ...done! [binary R ] app-office/lyx-1.3.5 +cups -debug -gnome +nls +qt [binary R ] net-misc/unison-2.9.1-r1 +gtk [binary R ] media-gfx/gimp-2.2.6-r1 -aalib (-altivec) -debug -doc -gimpprint -gtkhtml +jpeg +mmx +mng +png +python -sse +svg +tiff +wmf [ebuild R ] mail-client/mozilla-thunderbird-1.0.2 +crypt -debug -gnome -java -ldap -moznoxft -xinerama -xprint* 0 kB [binary R ] media-video/mplayer-1.0_pre6-r4 -3dfx +3dnow -3dnowext +X -aalib -alsa (-altivec) -arts +avi -bidi -cdparanoia -debug -dga -directfb -divx4linux -doc -dts -dv -dvb +dvd +dvdread -edl -encode -esd -fbcon -ggi +gif +gtk -i8x0 -ipv6 -jack -joystick +jpeg -libcaca -lirc +live -lzo -mad -matroska -matrox +mmx +mmxext +mpeg -mythtv -nas +nls +nvidia +oggvorbis +opengl +oss +png +real -rtc -samba -sdl -sse -sse2 -svga -tga -theora +truetype -v4l -v4l2 -xanim -xinerama -xmms +xv -xvid -xvmc [ebuild R ] app-editors/scite-1.6.2 -gtk2* -lua 0 kB [ebuild R ] x11-base/xorg-x11-6.8.2-r1 -3dfx +3dnow +bitmap-fonts -cjk -debug -dlloader -dmx -doc -font-server* -hardened -insecure-drivers -ipv6 -minimal +mmx +nls +opengl +pam -sdk -sse -static +truetype-fonts +type1-fonts (-uclibc) -xprint* +xv 0 kB [ebuild R ] media-video/kplayer-0.5.2 -arts -debug* -kdeenablefinal -xinerama 0 kB [ebuild U ] sys-devel/gnuconfig-20050324 [20050223] 35 kB [ebuild R ] app-cdr/k3b-0.11.23-r2 -arts -debug +dvdr -encode -flac +kde* -kdeenablefinal -mad +oggvorbis -xinerama 0 kB Total size of downloads: 35 kB _______________________________________________________________ (getting Messy++) # emerge.jstubbs.3_5 -pv --usepkg --newuse world These are the packages that I would merge, in order: Calculating world dependencies ...done! [ebuild R ] mail-client/mozilla-thunderbird-1.0.2 +crypt -debug -gnome -java -ldap -moznoxft -xinerama -xprint* 0 kB [ebuild R ] app-editors/scite-1.6.2 -gtk2* -lua 0 kB [ebuild R ] x11-base/xorg-x11-6.8.2-r1 -3dfx +3dnow +bitmap-fonts -cjk -debug -dlloader -dmx -doc -font-server* -hardened -insecure-drivers -ipv6 -minimal +mmx +nls +opengl +pam -sdk -sse -static +truetype-fonts +type1-fonts (-uclibc) -xprint* +xv 0 kB [ebuild R ] media-video/kplayer-0.5.2 -arts -debug* -kdeenablefinal -xinerama 0 kB [ebuild U ] sys-devel/gnuconfig-20050324 [20050223] 35 kB [ebuild R ] app-cdr/k3b-0.11.23-r2 -arts -debug +dvdr -encode -flac +kde* -kdeenablefinal -mad +oggvorbis -xinerama 0 kB Total size of downloads: 35 kB
Not sure what you are referring to, but adding binpkguseflags=None just before the break made everything work fine here.
>Not sure what you are referring to, but adding binpkguseflags=None just before >the break made everything work fine here. From default emerge, if I only modify select_dep() and xcreate() including the binpkguseflag=None, emerge returns a list of binary packages that it wants to reemerge even though no USE flags have been changed. If I additionally replace the current --newuse code in create() with the USE flag change detection code (with the used in xcreate() and select_dep() ) then these binaries are (correctly) not reemerged. Why? I don't know. The only correlation I noticed was that all of the binaries that were to be incorrectly reemerged were listed in /etc/portage/package.use My previous post gives the contents of my /etc/portage/package.use, the output of incorrect emerge behavior with only xcreate() and select_dep() changed (including binpkguseflag=None) in xcreate() ), and output of correct emerge behavior with create() also modified. The patch "getting Messy++, ..." has select_dep(), xcreate(), and create() modified, and so far, works for me. Thanks!
--newuse implementation is horrible. :( Anyway, after looking at create() it would seem that the situation you describe would be possible if the flags in a binary's IUSE haven't changed but it's corresponding (that which has identical name and version) ebuild's IUSE has changed. However, that doesn't seem to be the case here. Nor can I reproduce this at all locally. Can you try 2.0.51.22 and see what happens with that?
>--newuse implementation is horrible. :( :) Looking forward to the next major version of portage! I get the same no-*-but-reemerge-anyway problem with unmodified portage 2.0.51.22 . Are you interested in --debug output? Appears that the same "fix" in create() for 2.0.51.19 works for 2.0.51.22 . ________________________________________________________________ # emerge --version Portage 2.0.51.22 (default-linux/x86/2005.0/2.4, gcc-3.3.5-20050130, glibc-2.3.4.20041102-r1, 2.4.24-om2 i686) # emerge -pv --usepkg --newuse world These are the packages that I would merge, in order: Calculating world dependencies ...done! [binary R ] app-office/lyx-1.3.5 +cups -debug -gnome +nls +qt [binary R ] net-misc/unison-2.9.1-r1 +gtk [ebuild U ] net-misc/wget-1.9.1-r5 [1.9.1-r3] -build -debug -ipv6 +nls -socks5 +ssl -static 36 kB [binary R ] media-gfx/gimp-2.2.6-r1 -aalib (-altivec) -debug -doc -gimpprint -gtkhtml +jpeg +mmx +mng +png +python -sse +svg +tiff +wmf [ebuild R ] mail-client/mozilla-thunderbird-1.0.2 +crypt -debug -gnome -java -ldap -moznoxft -xinerama -xprint* 0 kB [ebuild UD] app-cdr/graveman-0.3.10 [0.3.11] -debug -doc +dvdr -flac -mad +nls +oggvorbis -sox 0 kB [binary R ] media-video/mplayer-1.0_pre6-r4 -3dfx +3dnow -3dnowext +X -aalib -alsa (-altivec) -arts +avi -bidi -cdparanoia -debug -dga -directfb -divx4linux -doc -dts -dv -dvb +dvd +dvdread -edl -encode -esd -fbcon -ggi +gif +gtk -i8x0 -ipv6 -jack -joystick +jpeg -libcaca -lirc +live -lzo -mad -matroska -matrox +mmx +mmxext +mpeg -mythtv -nas +nls +nvidia +oggvorbis +opengl +oss +png +real -rtc -samba -sdl -sse -sse2 -svga -tga -theora +truetype -v4l -v4l2 -xanim -xinerama -xmms +xv -xvid -xvmc [ebuild R ] app-editors/scite-1.6.2 -gtk2* -lua 0 kB [ebuild R ] x11-base/xorg-x11-6.8.2-r1 -3dfx +3dnow +bitmap-fonts -cjk -debug -dlloader -dmx -doc -font-server* -hardened -insecure-drivers -ipv6 -minimal +mmx +nls +opengl +pam -sdk -sse -static +truetype-fonts +type1-fonts (-uclibc) -xprint* +xv 0 kB [ebuild R ] media-video/kplayer-0.5.2 -arts -debug* -kdeenablefinal -xinerama 0 kB [ebuild R ] app-cdr/k3b-0.11.23-r2 -arts -debug +dvdr -encode -flac +kde* -kdeenablefinal -mad +oggvorbis -xinerama 0 kB Total size of downloads: 36 kB
Created attachment 59217 [details, diff] Fix the binpkg + create() case Against 2.0.51.22. Instead of rechecking the binary package's USE flags which are guaranteed to have been checked, only do the checking if an ebuild is passed and the checks are known to have not been run. Please confirm ASAP :)
(In reply to comment #24) > Fix the binpkg + create() case This patch works for me: emerge gives the same output as "getting Messy++" (for 2.0.51.19). No binaries are reemerged with unchanged USE flags. As a sanity check, not using the --usepkg flag gives same output as using it. On to the next bug? ;) _____________________________________________________ # emerge.22.createCase -pv --newuse --usepkg world These are the packages that I would merge, in order: Calculating world dependencies ...done! [ebuild U ] net-misc/wget-1.9.1-r5 [1.9.1-r3] -build -debug -ipv6 +nls -socks5 +ssl -static 36 kB [ebuild R ] mail-client/mozilla-thunderbird-1.0.2 +crypt -debug -gnome -java -ldap -moznoxft -xinerama -xprint* 0 kB [ebuild UD] app-cdr/graveman-0.3.10 [0.3.11] -debug -doc +dvdr -flac -mad +nls +oggvorbis -sox 0 kB [ebuild R ] app-editors/scite-1.6.2 -gtk2* -lua 0 kB [ebuild R ] x11-base/xorg-x11-6.8.2-r1 -3dfx +3dnow +bitmap-fonts -cjk -debug -dlloader -dmx -doc -font-server* -hardened -insecure-drivers -ipv6 -minimal +mmx +nls +opengl +pam -sdk -sse -static +truetype-fonts +type1-fonts (-uclibc) -xprint* +xv 0 kB [ebuild R ] media-video/kplayer-0.5.2 -arts -debug* -kdeenablefinal -xinerama 0 kB [ebuild R ] app-cdr/k3b-0.11.23-r2 -arts -debug +dvdr -encode -flac +kde* -kdeenablefinal -mad +oggvorbis -xinerama 0 kB Total size of downloads: 36 kB
*** Bug 94104 has been marked as a duplicate of this bug. ***
Fixed on or before 2.0.51.22-r1
Is 2.0.51.22-r1 stable on all arches now?
Looking through the batch of bugs, I'm not sure that some of these are actually fixed in stable. Others, the requirements have possibly changed after the initial fix was committed. If you think this bug has been closed incorrectly, please reopen or ask that it be reopened.
Stable on amd64, ppc64, and x86. Still marked testing everywhere else.
Yep. Need sandbox to be stabled first. I'll get a bug open about that and CC the remaining archs.
