If the @installed set is used to update all packages, it tends to trigger unsolvable blockers since emerge will not automatically uninstall packages that have been pulled into the dependency graph. As a solution, atoms pulled in by @installed could be marked with attribute which indicates that they are "optional", and then the blocker resolution code could remove them from the conflicting packages from the graph if it happens that they only got pulled in by an "optional" atom.
Well, people just shouldn't use @installed to update their system (who recommends that anyway?). What are the use cases for this set? Maybe it should just be removed.
(In reply to comment #1) > Well, people just shouldn't use @installed to update their system (who > recommends that anyway?). Well, it's common for people who aren't aware of the caveats to suggest it to other people. For example: http://archives.gentoo.org/gentoo-dev/msg_fd355f2fb8d05712401cf8fcd426257e.xml Only people who are intimately familiar with the blocker resolving code would know the caveats, so I've had to explain it many times. > What are the use cases for this set? Maybe it should just be removed. I don't use it for anything. I can't speak for others though.
(In reply to comment #2) > (In reply to comment #1) > http://archives.gentoo.org/gentoo-dev/msg_fd355f2fb8d05712401cf8fcd426257e.xml > > Only people who are intimately familiar with the blocker resolving code would > know the caveats, so I've had to explain it many times. > There is more about that than blockers. It's @installed vs @world +--depclean and people should be using the latter. > > What are the use cases for this set? Maybe it should just be removed. > > I don't use it for anything. I can't speak for others though. Even more I can't think of a good reason. The only useful case I knew was quickpkg @installed, but you can do quickpkg */* now. I'd say lets remove it.
To make it clear, what we loose is nothing and what we gain is less confusion for the user as to what they should use to update. On top of that it's only in portage-2.2, which lowers the possible impact of broken work flows and scripts.
Okay, let's do a deprecation warning first, and then if we don't hear any good arguments against removing it then we'll go ahead and do it.
(In reply to comment #5) > Okay, let's do a deprecation warning first, and then if we don't hear any good > arguments against removing it then we'll go ahead and do it. Sounds good.
The deprecation warning is in git: http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=12c142c529909acf1b755690404cef51d8de3fe4
This is fixed in 2.2.0_alpha125.
I apologize for asking in the closed bug. I'm using a set @instelled to check the consistency of the binary packages on my binhost as follows emerge -pvK @installed Whether I would do something like that after removing the set?
(In reply to comment #9) > I'm using a set @instelled to check the consistency of the binary packages > on my binhost as follows > > emerge -pvK @installed > > Whether I would do something like that after removing the set? This should be close enough: emerge -pvKe --with-bdeps=y @world
Ok. This work. Thanks.
(In reply to comment #10) > (In reply to comment #9) > > I'm using a set @instelled to check the consistency of the binary packages > > on my binhost as follows > > > > emerge -pvK @installed > > > > Whether I would do something like that after removing the set? > > This should be close enough: > > emerge -pvKe --with-bdeps=y @world Sometimes it's useful to know exactly which packages ARE installed, not which the package manager think should be, portage keep a database of them in /var/db/pkg, the following command create a list of these, it's ordered by time of install (ls -tr), changing "-ctime +0" to any number of days skip the packages installed in that period. find /var/db/pkg -name BUILD_TIME -ctime +0 -exec ls -1tr {} + 2>/dev/null \ | sed -e 's:^/var/db/pkg/:~:' -e 's:/BUILD_TIME$::' -e 's:-r[0-9][0-9]*$::' \ > /etc/portage/world-rebuild for the record I do use this list to rebuild world forcing emerges without dependancies in parallel: parallel -j 5 --timeout $(( 60 * 120 )) \ emerge -1 --color=n --nodeps {} ::: $(< /etc/portage/world-rebuild) It can succeed only if *all* the packages are up to date, i.e. `emerge -uDpN --with-bdeps` give an empty list
I just used emerge -pv @installed > /tmp/installed on two machines to find the differences in the installed packages, along with scp and diff -wu <( awk '/kB$/ {$(NF-1)=""; print}' tmp/installed-vdr |sort) \ <( awk '/kB$/ {$(NF-1)=""; print}' tmp/installed-vdr-vb |sort) Are there other options to find the differences? The above solution includes packages and their USE flags. Kind regards, mdbuerkle
(In reply to Zac Medico from comment #8) > This is fixed in 2.2.0_alpha125. It possible to remove deprecation warning from git? It is a bit confusing.
(In reply to Mark Dominik Bürkle from comment #13) > I just used > emerge -pv @installed > /tmp/installed > on two machines to find the differences in the installed packages, along > with scp and > diff -wu <( awk '/kB$/ {$(NF-1)=""; print}' tmp/installed-vdr |sort) \ > <( awk '/kB$/ {$(NF-1)=""; print}' tmp/installed-vdr-vb |sort) > > Are there other options to find the differences? > The above solution includes packages and their USE flags. This command works, but doesn't show USE flags: portageq match / '*/*' (In reply to Daniel Wlasow from comment #14) > It possible to remove deprecation warning from git? It is a bit confusing. But @installed is still deprecated. Why do you want to use @installed?
I'm using it to fetch all distfiles required to re-install: emerge -f @installed
(In reply to Alex Efros from comment #16) > I'm using it to fetch all distfiles required to re-install: > > emerge -f @installed Similar to the command suggested in comment #10, the following command should give you the desired result: emerge --fetchonly --emptytree @world
(In reply to Zac Medico from comment #15) > ... > This command works, but doesn't show USE flags: > > portageq match / '*/*' Thanks Zac, that'll do it for me. Kind regards.
I wish to write in defense of the @installed set. I do so for many reasons. Here are just a few. 1. When using -1 (one-shot), the world file shrinks. This is good for "unresolvable" conflicts, but using emerge -ave1 world means packages NOT compiled because they don't show up in the world file. 2. The use of the term, "unresolvable" seems a bit wrong. I have been in the process of upgrading gcc, and that means I should recompile everything. This is especially true when you consider the news item about 4.x and 4.y versions of gcc not getting along well if x!-y. That means I've been having to use the @installed set quite a bit, if only to hunt down STALE packages without ebuilds. I've nuked hundreds, literally HUNDREDS of stale packages of this type from my machines. After that work, yes, I can say that the issues are resolvable, if you're willing to put in the effort to make it happen. Sure, that's not for the average user, but then again, Gentoo is not your average Linux distro. 3. Is there another way to recompile everything if you remove this? If not, then, don't remove it. I've had one of my installs going since 2009, and it's stayed stable partly because I recompile everything, or at least thought I was until a few days ago. 4. If portage did a better job of cleaning dead packages with dead ebuilds, the problems that arise with the use of the @installed set would be negligible. When I went through my /var/db/pkg directory, I was stunned at the amount of files that had not been recompiled since 2009. 2009! We're talking at least three hundred packages. On a system with 1400 packages, that's a pretty significant number of unclean files cluttering up my hard drive. 5. --depclean can get the job done, but more often than not, it misses crap and nukes good. Perhaps a bit of thought should be given to actually making it a bit easier to at least get rid of dead packages attached to dead ebuilds. Maybe make it so that auto-cleaning packages is more than a question I get asked when there's nothing new to emerge. I'm betting if that's actually done, your issues with the @installed set will melt away like snow on a Texas highway in August. 6. If you can give proceed at your own rish warnings for other things, why not for the @installed set? That way, anyone who whines gets to get treated to a dose of "talk to the hand". 7. Maybe make working with the @installed set a little more forgiving, or perhaps integrate stale package removal as a part of the set itself.
(In reply to Bob Raitz from comment #19) > 1. When using -1 (one-shot), the world file shrinks. This is good for > "unresolvable" conflicts, but using emerge -ave1 world means packages NOT > compiled because they don't show up in the world file. Generally, it's best practice to add anything to your world file that you don't want 'emerge --depclean' to remove. > 2. The use of the term, "unresolvable" seems a bit wrong. I have been in the > process of upgrading gcc, and that means I should recompile everything. This > is especially true when you consider the news item about 4.x and 4.y > versions of gcc not getting along well if x!-y. That means I've been having > to use the @installed set quite a bit, if only to hunt down STALE packages > without ebuilds. I've nuked hundreds, literally HUNDREDS of stale packages > of this type from my machines. The recommended practice is to use these two commands: emerge -auDN --with-bdeps=y @world emerge -a --depclean The result should be an optimal state, as far as specified dependencies are concerned. > After that work, yes, I can say that the issues are resolvable, if you're > willing to put in the effort to make it happen. Sure, that's not for the > average user, but then again, Gentoo is not your average Linux distro. > > 3. Is there another way to recompile everything if you remove this? If you use your world file to protect anything that you don't want emerge --depclean to remove, then you can recompile everything with this command: emerge -e @world > 4. If portage did a better job of cleaning dead packages with dead ebuilds, > the problems that arise with the use of the @installed set would be > negligible. The recommended practice is to use emerge --depclean to clean dead packages. > > 5. --depclean can get the job done, but more often than not, it misses crap > and nukes good. You should use --ask or --pretend with --depclean, and if it 'nukes good', then you should add 'good' to your world file like this: emerge --noreplace good > 6. If you can give proceed at your own rish warnings for other things, why > not for the @installed set? That way, anyone who whines gets to get treated > to a dose of "talk to the hand". Sure, we could do that. > 7. Maybe make working with the @installed set a little more forgiving, or > perhaps integrate stale package removal as a part of the set itself. That's a possibility.
As a way of illustrating my point about being able to compile everything, here are some results I tried once I got the machine cleaned out enough to not fail when emerging the @installed set. BEGIN___________________ results with emerge -aev world --keep-going: Total: 1055 packages (3 upgrades, 4 new, 1048 reinstalls), Size of downloads: 0 KiB Fetch Restriction: 2 packages results with emerge -aev1 @installed --keep-going: Total: 1328 packages (111 upgrades, 1 downgrade, 26 new, 16 in new slots, 1174 reinstalls), Size of downloads: 147,934 KiB Fetch Restriction: 2 packages END____________________ If only for this reason, getting rid of the @installed set makes no sense. Considering that Gentoo has the reputation of being for people who have actual brains and ability to work with Linux at the source code level, I think we need to have tools that can clean up the messes that portage sometimes leaves. There is always Ubuntu and it's derivatives for those who don't quite get source-based Linux. I do get it, and I'd like as many tools as possible in my kit when it comes to keeping my OS running like it ought to.
One final suggestion, then it's bed time. Perhaps, when doing the @installed set, the first thing it does is find all the dead ebuilds. It would then allow you to remove the packages, or to continue on ignoring the problematic packages. For better or worse, I enjoyed untying all the knots that kept my machines from recompiling everything. When it's all done, the systems in question will be cleaner and more up to date than they've been in a while. Without the @installed and the information it gave me on the condition of my machines, they'd probably still have lots of dead packages hanging out. Thank GNU for gcc updates. Cheers.
(In reply to Bob Raitz from comment #21) > As a way of illustrating my point about being able to compile everything, > here are some results I tried once I got the machine cleaned out enough to > not fail when emerging the @installed set. > > BEGIN___________________ > > results with emerge -aev world --keep-going: > > Total: 1055 packages (3 upgrades, 4 new, 1048 reinstalls), Size of > downloads: 0 KiB > Fetch Restriction: 2 packages > > results with emerge -aev1 @installed --keep-going: > > Total: 1328 packages (111 upgrades, 1 downgrade, 26 new, 16 in new slots, > 1174 reinstalls), Size of downloads: 147,934 KiB > Fetch Restriction: 2 packages It would be worthwhile to investigate the reasons for the difference you see here. Ideally, emerge -e @world should be sufficient.
"Ideally, emerge -e @world should be sufficient." Yes, but we don't live in an ideal world. The point remains that saying the @install set should be sent away to exile is shortsighted, if only because it is apparently a one of a kind tool that can and WILL find every possible problem with your portage system. I sure hope it isn't being sent to the chopping block because it does that. It should be praised and modified so it's a bit tamer, but still has it's bloodhound ability to find portage issues. I think that with some of my suggestions, the @installed set could become an incredibly powerful tool for keeping portage in tip-top shape. Since it seems so able to find every possible hiccup in the portage system, it would be an absolute shame to throw it away just because it points up some weaknesses in an otherwise stable and reliable installation system. I know I'd feel better knowing that a gcc update is actually going to be a true update with all packages being recompiled. Since emerge -e world can't get the job done apparently, we do need something that can. As for --depclean, I think the idea is a little off. It should be able to find stale packages with dead ebuilds without having to consult the @world set. It may need the @world set for dependencies, but somehow, I don't see dead ebuilds having live dependencies.
(In reply to Bob Raitz from comment #24) > "Ideally, emerge -e @world should be sufficient." > > Yes, but we don't live in an ideal world. I have 2550 packages installed, and emerge -e @world pulls in all 2550 packages perfectly. No more, and no less. As I said earlier, if it's not working for you then it would be worthwhile to investigate the reasons for it. > The point remains that saying the @install set should be sent away to exile > is shortsighted, if only because it is apparently a one of a kind tool that > can and WILL find every possible problem with your portage system. I sure > hope it isn't being sent to the chopping block because it does that. It > should be praised and modified so it's a bit tamer, but still has it's > bloodhound ability to find portage issues. I've already said that we could change the deprecation warning to a "proceed at your own risk warning" as you've suggested. > I think that with some of my suggestions, the @installed set could become an > incredibly powerful tool for keeping portage in tip-top shape. Since it > seems so able to find every possible hiccup in the portage system, it would > be an absolute shame to throw it away just because it points up some > weaknesses in an otherwise stable and reliable installation system. I know > I'd feel better knowing that a gcc update is actually going to be a true > update with all packages being recompiled. Since emerge -e world can't get > the job done apparently, we do need something that can. Again, if emerge -e @world doesn't work for you, then it would be worthwhile to investigate the reasons for it. > As for --depclean, I think the idea is a little off. It should be able to > find stale packages with dead ebuilds without having to consult the @world > set. It may need the @world set for dependencies, but somehow, I don't see > dead ebuilds having live dependencies. It has to consult world in order to obtain the roots of the dependency tree. Without roots, every package is eligible for removal!
(In reply to Zac Medico from comment #25) > I've already said that we could change the deprecation warning to a "proceed > at your own risk warning" as you've suggested. I've posted a patch for review here: http://thread.gmane.org/gmane.linux.gentoo.portage.devel/4848
It does so on three machines. If there is anything wrong with my portage setups, it's not because of lack of effort. I just got the last one cleaned up to the point where it can actually run the @installed set. It was the oldest install, and therefore, had the most leftovers from dead packages with equally dead ebuilds. It was about three days of intense work, but I finally cleared all that stuff out. Before that, I was under the impression my portage was pretty stable and kept clean. I can't help but detect a slight bit of defensiveness. It was not my intent to point fingers or whatever is causing this reaction. It was my intent to keep a decent tool that can actually help fix errors in portage that nothing else seems able to detect, much less fix. If I have given offense in any way, I can only apologize and assure you that was not my intent. Cheers.
(In reply to Bob Raitz from comment #27) > I can't help but detect a slight bit of defensiveness. It was not my intent > to point fingers or whatever is causing this reaction. I have worked very hard to make emerge update, depclean, and emptytree actions be consistent with each other. This makes it annoying is someone says that they are inconsistent while not taking steps to help fix any issues that may exist > It was my intent to > keep a decent tool that can actually help fix errors in portage that nothing > else seems able to detect, much less fix. Sure, @installed may have some legitimate uses. Like Sebastian (see comment #3), I don't use @installed for anything, so it's important for other people to speak up if they think they need it for some reason. > If I have given offense in any > way, I can only apologize and assure you that was not my intent. No offense taken. However, I reserve the right to be annoyed by anyone who claims that 'emerge -e @world' doesn't work, without taking steps to help fix the issues that they have.
(In reply to Zac Medico from comment #26) > (In reply to Zac Medico from comment #25) > > I've already said that we could change the deprecation warning to a "proceed > > at your own risk warning" as you've suggested. > > I've posted a patch for review here: > > http://thread.gmane.org/gmane.linux.gentoo.portage.devel/4848 This is in the master branch now: https://github.com/gentoo/portage/commit/b5b0d117433843da6a8d695140a68c7f13cf72ac
Unless your name is Daniel Robbins, I'm not really all that impressed. "No offense taken. However, I reserve the right to be annoyed by anyone who claims that 'emerge -e @world' doesn't work, without taking steps to help fix the issues that they have." I reserve the right to tell someone their program isn't working if it isn't working. The fact that I showed a weakness in your system should be praiseworthy, not a reason for you to have histrionics. Open source is about everybody adding their two cents. I helped troubleshoot your code, and you get "annoyed". What kind of garbage is that? I expect that from a Redmondite, not from a member of the open source side of things. Instead, strangely, you have chosen to get upset because I showed a flaw in your code. Perhaps you should think about not taking things so personally. I said your program didn't work. I did NOT call your knowledge, ability, or anything else into question. For you to whine like this makes me wonder what you're doing being a developer if you're going to take bugs in your code as a personal affronts or commentaries on your abilities. I did neither. I appreciate your efforts. I don't appreciate your snarky attitude.
(In reply to Bob Raitz from comment #30) > Instead, strangely, you have chosen to get upset because I showed a flaw in > your code. Not upset, just a little disappointed that you didn't file a new bug in order to help us correct that flaw.
(In reply to Zac Medico from comment #31) > (In reply to Bob Raitz from comment #30) > > > Instead, strangely, you have chosen to get upset because I showed a flaw in > > your code. > > Not upset, just a little disappointed that you didn't file a new bug in > order to help us correct that flaw. As far as I can tell, that's hogwash. I "annoyed" you, in your own words, and you let me know in a way I can only describe as curt and rude. You need only read the flow to see this. That's immaterial at this point. It seems as if the @installed set is going to stay around. That's all I wanted and cared about. Perhaps some of my suggestions might find their way into the code. If so, awesome. If not, as long as the @installed set is there to help keep portage humming along cleanly, then my work here is done. At least until I move up to gcc 4.9.x
(In reply to Bob Raitz from comment #32) > Perhaps some of my suggestions might find their way into the code. I tried to explain that 'emerge -e @world' works perfectly for me (Sebastian too, apparently). If we can't reproduce your problem locally, then there is little chance of it getting fixed. This makes us dependent on you filing a bug in order to fix the flaw that you claim to have found.
Pardon me for being confused here, but as far as I could read, my mere suggestion that the two operations (@installed vs. aev @world) did not work the same was enough to raise ire to the point that I thought I had somehow been swept into a Facebook rage quit thread. My initial intent was just making sure the @installed set wasn't thrown away because some people who shouldn't have been messing around with it were messing around with it. If you want me to write a bug report on the above situation, you're going to have to cool your jets when it comes to attitude. I am not going to spend a day stressing out in Southern California traffic to hop on my machine helping you to fix your code and in the process, have to put up with Prima Donna antics. If there is enough patience in the world to deal with that kind of nonsense, I'm not the person possessing it. If that's agreeable, then I would be happy to write up a bug report on that very issue. As a long-time user of Gentoo, I'm all for ironing out as many bugs as possible. Now, do you want reports on all the installs that do this? That would include one that I keep updating on the off chance I will run into another machine like or close to it with a functional motherboard. I also have a Funtoo install that I'm setting up for the eventual upgrade of my laptop. I have three that are fully functional, and running on their own individual machines. All of these installations will show the same thing, even if two only live under chroot at the moment.
(In reply to Bob Raitz from comment #34) > Now, do you want reports on all the installs that do this? Just a single bug report would be fine. We could focus on just one of your systems first. If we have a patch that works for the first system, then you can test in on the others to see if it fixes them too. If it turns out that the others are not fixed by the same patch, then you should file a new bug for one of those, and we'll repeat the same process.
Before filing a new bug, you might have a look at bug 350230, since it is the only bug that I know of which prevents 'emerge -e @world' from pulling in all of the appropriate packages.
(In reply to Bob Raitz from comment #34) > Pardon me for being confused here, but as far as I could read, my mere > suggestion that the two operations (@installed vs. aev @world) did not work > the same was enough to raise ire to the point that I thought I had somehow > been swept into a Facebook rage quit thread. My initial intent was just > making sure the @installed set wasn't thrown away because some people who > shouldn't have been messing around with it were messing around with it. > > If you want me to write a bug report on the above situation, you're going to > have to cool your jets when it comes to attitude. I am not going to spend a > day stressing out in Southern California traffic to hop on my machine > helping you to fix your code and in the process, have to put up with Prima > Donna antics. If there is enough patience in the world to deal with that > kind of nonsense, I'm not the person possessing it. If that's agreeable, > then I would be happy to write up a bug report on that very issue. As a > long-time user of Gentoo, I'm all for ironing out as many bugs as possible. > > Now, do you want reports on all the installs that do this? That would > include one that I keep updating on the off chance I will run into another > machine like or close to it with a functional motherboard. I also have a > Funtoo install that I'm setting up for the eventual upgrade of my laptop. I > have three that are fully functional, and running on their own individual > machines. All of these installations will show the same thing, even if two > only live under chroot at the moment. Bob, I'm sorry to have to say this, but the only real attitude I see, is coming from you. Things have been quite civil up to this point, so I was staying out of it. I have had a difficult time trying to see how Zac was giving you attitude and being rude. I did see some areas where you may have been in a wrong frame of mind and mistakenly taken them the wrong way, it happens. Not the way they were meant. As such your responses here have been escalating. I have not seen his posts here escalate in any way in response. Please step back and allow yourself to cool down.
From my point of view what Zac is asking is simply to get a bug report with your concrete case that shows that "world" is not working as it should as probably that could be a bug that should be fixed instead of using "installed" as a workaround. I don't understand how that willing from Zac to fix the root cause can be seen as an attitude issue from him that is only offering his help to you, I would instead appreciate that he offers his help to fix a possible bug :) I would calm down a bit, and simply open a new bug report with your concrete setup to see if that bug can be solved finally instead of needing to keep using @installed to overcome it Thanks a lot
Created attachment 391046 [details] results of 'emerge --debug -eptv portage --color' n (bug 360230) These are the results of 'emerge --debug -eptv portage --color' n in reference to bug 350230. Is presence of bug shown, or is my issue something else?
It does happen on that newest setup. I will open the new bug, then post it here for review and reference.
Bug number 531770. Will be populating soon.
I'm sorry posting here in this bug report, but I just realised now the warning message when I was using emerge -uDNav @installed Well the reason I do this is because it's usefull for a test environment where I'm using many paralel desktop environments. I only add to world those that are crucial for my environment, all the others are installed just as oneshot. This way, when updating all packages, conflicting packages are automatically uninstalled. My update procedures goes in two phases emerge -uDNav @world and then for the remaining emerge -uDNav @installed The problem mentioned in this bug never happen to me. So I'll ask you not to remove @installed, as it is usefull and afterwards is only used for who need it. This is a very interesting feature, similar to other package managers, and I don't understand the reason why you pretend to remove it.
