| Summary: | Enhancement request for EAPI 5 migration with old portage | ||
|---|---|---|---|
| Product: | Gentoo Release Media | Reporter: | Kai Krakow <hurikhan77+bgo> |
| Component: | Other | Assignee: | The Gentoo Linux Hardened Team <hardened> |
| Status: | RESOLVED OBSOLETE | ||
| Severity: | normal | CC: | dev-portage, docs-team, sam, zerochaos |
| Priority: | Normal | ||
| Version: | unspecified | ||
| Hardware: | All | ||
| OS: | Linux | ||
| URL: | https://plus.google.com/u/0/100249555891135899031/posts/NJY7rLBnDoo | ||
| Whiteboard: | |||
| Package list: | Runtime testing required: | --- | |
|
Description
Kai Krakow
2013-03-25 15:30:21 UTC
Meanwhile, I discovered that allmost all my machines are affected by this. How is this supposed to work? The upgrade path is seriously broken for Hardened Gentoo... I don't want to tinker around with production installations by unpacking tarballs and crossing fingers that it works. @portage: opinions? (In reply to comment #1) > Meanwhile, I discovered that allmost all my machines are affected by this. > How is this supposed to work? The upgrade path is seriously broken for > Hardened Gentoo... The hardened team could have made the upgrade path easier by creating separate 13.0 profiles, like we did for the other non-hardened profiles. > I don't want to tinker around with production installations by unpacking > tarballs and crossing fingers that it works. If you have at least python-2.6 then you can use this guide to manually upgrade portage to the latest version: http://www.gentoo.org/proj/en/portage/doc/manually-fixing-portage.xml Otherwise, you should use the approach described in the "Updating old systems" section of the Upgrading Guide: http://www.gentoo.org/doc/en/gentoo-upgrading.xml#doc_chap4 This problem hit me even harder with another machine because it also had an outdated glibc and expat, and python versions from tinderbox could not work due to some incompatibility with the hardened-kernel and some execmap problems. And even then, tinderbox had been no help because it misses glibc and some other libs (probably for good reasons). I got around it by manually editing some essential ebuilds to use EAPI=3 and which suprisingly worked (so maybe portage devs should only raise the EAPI level of an ebuild which really requires it), I also had to use the "ebuild" command to get around some checks which emerge did and failed. After this I was able to compile some essentials like pkgbuild, expat, ... - then upgrade python after rebuilding it, then switch the profile, upgrade portage (which I could not do with old python version because it failed some ctype assertions), switch profile back, and rebuild world with "-e" (which required multiple retries because it failed at different points even with --keep-going, I had to rebuild some other packages first, mainly due to some binary feature collisions with glibc upgrades). If I remember correctly I even injected some packages into provided because some packages deps I could not compile at that time - but they compiled fine even with older versions of the deps (so maybe portage devs should also only raise dep version requirements if absolutely required). Hell, that was dirty! But the system is up and running with most current stable packages now, and migrated to systemd. I could only try that dirty work because it was a machine abandoned some time ago and had to be put back into production now and I was able to take a VM snapshot before starting. But I learned a lot of doing such dirty work without completely destroying the system. So what did I learn and maybe helps with getting out of the trouble next time: 1. Not all ebuilds make use of the EAPI version they are claiming to use :-( 2. Not all deps are needed in the version claimed to be needed :-( 3. Be careful with tinderbox packages, they may require deps not downloadable/available there (big pita) 4. Use ebuild command to manually merge packages when official merge/install function from ebuild fails 5. Don't forget to "emerge -e world" after done (maybe "emerge -e system" first) 6. Do NOT reboot or logout before everything is fixed, take VM snapshots if applicable, keep at least one shell open (In reply to comment #3) > Otherwise, you should use the approach described in the "Updating old > systems" section of the Upgrading Guide: > > http://www.gentoo.org/doc/en/gentoo-upgrading.xml#doc_chap4 Ahh, I've already thought about "ROOT=/mnt/host emerge -1v portage", too. But I wasn't sure if it would really work. Thanks for the pointer. I will try that next time. There are still some systems in the queue. ;-) So, actually, from that description I deduce I could just mount the VM image of an outdated machine to a current system and then use the ROOT-emerge-method from above without any disturbance of the current system? (In reply to comment #4) > 1. Not all ebuilds make use of the EAPI version they are claiming to use :-( We added automatic die support to ebuild helpers in EAPI 4, so changing the the EAPI to 3 means that the ebuild may still work but it won't die if the helper functions fail unexpectedly. > 2. Not all deps are needed in the version claimed to be needed :-( It could be that those deps are needed in order to avoid subtle bugs that you haven't noticed. (In reply to comment #5) > So, actually, from that description I deduce I could just mount the VM image > of an outdated machine to a current system and then use the > ROOT-emerge-method from above without any disturbance of the current system? More or less, yes. If there are bugs in the ebuilds, the it's possible for them to disturb the host system. If you stick to system packages though, they should be fairly reliable, since they get more testing with ROOT than other packages. (In reply to comment #6) > (In reply to comment #4) > > 1. Not all ebuilds make use of the EAPI version they are claiming to use :-( > > We added automatic die support to ebuild helpers in EAPI 4, so changing the > the EAPI to 3 means that the ebuild may still work but it won't die if the > helper functions fail unexpectedly. > > > 2. Not all deps are needed in the version claimed to be needed :-( > > It could be that those deps are needed in order to avoid subtle bugs that > you haven't noticed. While I'm sure you are correct, that was a non-issue in this case as I rebuilt the affected packages directly afterwards with a clean portage tree, then rebuilt the complete system. > (In reply to comment #5) > > So, actually, from that description I deduce I could just mount the VM image > > of an outdated machine to a current system and then use the > > ROOT-emerge-method from above without any disturbance of the current system? > > More or less, yes. If there are bugs in the ebuilds, the it's possible for > them to disturb the host system. If you stick to system packages though, > they should be fairly reliable, since they get more testing with ROOT than > other packages. Is there a way to sandbox writes done by emerge with ROOT= into the mounted root? Maybe by using the sandbox utility that's used by emerge itself? I don't want buggy ebuilds / build scripts write outside of this mounted root. Chroot won't work in this case. (In reply to comment #7) > Is there a way to sandbox writes done by emerge with ROOT= into the mounted > root? Maybe by using the sandbox utility that's used by emerge itself? I > don't want buggy ebuilds / build scripts write outside of this mounted root. There's a patch attached to bug 138388 that you could test. Also, keep in mind that several phases always run outside of the sandbox: pkg_pretend pkg_setup pkg_preinst pkg_postinst pkg_prerm pkg_postrm > Chroot won't work in this case. Why don't you just unpack a stage3 as suggested in the "Updating old systems" section of the upgrading guide, and throw it away when you're done? (In reply to comment #4) > 1. Not all ebuilds make use of the EAPI version they are claiming to use :-( This is not really like this. I'm sorry but we're supposed to use newest EAPI whenever possible. This has two significant advantages: 1) making use of fixes which avoid a terribly huge bugs you don't usually notice. 2) making the learning curve easier -- we're going to get EAPI=6 soon and we don't really want new developers to memorize what 7 EAPIs do differently. > 2. Not all deps are needed in the version claimed to be needed :-( I'd say that needs to be discussed per-package. But in the general case, it's better to require a newer version than one being too old. Also think of small bugs that you may not notice on your system, e.g. which may be USE-flag or arch-dependent. I don't think we're supposed to make devs do deps like: '>=foo-1 amd64? ( >=foo-2 )'... (In reply to comment #9) > (In reply to comment #4) > > 1. Not all ebuilds make use of the EAPI version they are claiming to use :-( > > This is not really like this. I'm sorry but we're supposed to use newest > EAPI whenever possible. This has two significant advantages: > > 1) making use of fixes which avoid a terribly huge bugs you don't usually > notice. > > 2) making the learning curve easier -- we're going to get EAPI=6 soon and we > don't really want new developers to memorize what 7 EAPIs do differently. > > > 2. Not all deps are needed in the version claimed to be needed :-( > > I'd say that needs to be discussed per-package. But in the general case, > it's better to require a newer version than one being too old. Also think of > small bugs that you may not notice on your system, e.g. which may be > USE-flag or arch-dependent. I don't think we're supposed to make devs do > deps like: '>=foo-1 amd64? ( >=foo-2 )'... Okay, you guys convinced me: I should have prefixed that with "In the context of solving this problem"... ;-) Still, exactly these points make upgrading an old installation a real pita, and I don't think there is an easy way out of it despite Gentoo having one of the best dependency systems I've ever came across. Even when everything is okay with the EAPIs you sometimes have broken dependencies because specific versions of packages need to be rebuilt first but are missing from portage. It's not always possible to keep a production machine up-to-date with the latest packages besides security updates (what is where glsa-check comes in as a handy tool). But at some point even these can no longer be installed because it starts to pull in updates you don't want yet. Now that this bug has been sitting for over 6 months, I believe it should either be closed or that it should be assigned to another team. This issue wasn't caused nor is it related to release, so there's no point for us to stick in here. Unless someone is ready to do the work to support this request, I also suggest we close it as INVALID as this wasn't a "bug", but the result of a conscious decision of the hardened team. (In reply to Jorge Manuel B. S. Vicetto from comment #11) > Now that this bug has been sitting for over 6 months, I believe it should > either be closed or that it should be assigned to another team. > This issue wasn't caused nor is it related to release, so there's no point > for us to stick in here. > Unless someone is ready to do the work to support this request, I also > suggest we close it as INVALID as this wasn't a "bug", but the result of a > conscious decision of the hardened team. +1 I could get around this problem on most systems I manage. But I wish that there's documentation somewhere on which date EAPIs had been switched so I could checkout exactly these portage tree snapshots and do successive updates. This would make it much more easy to transition old systems to current portage implementation in a clean way. So maybe move to the documentation team. Otherwise simply close. @docs-team: do you want this bug or shall we close it? Switching EAPIs is done gradually and by the various profiles afaics. Most profile changes of this extend will put in some timeline before the older profiles are removed (even hardened did a while with new profiles before switching eventually). Another solution you might want to look at is find a portage snapshot "in between" and do gradual upgrades, but you're not guaranteed to have all software still available (the sources, that is). Perhaps the Gentoo Upgrading guide might need some more love to document things (it's on the wiki now, so by all means, be my guest) but I don't think for this particular bug itself we can do much more. (In reply to Sven Vermeulen from comment #15) > Switching EAPIs is done gradually and by the various profiles afaics. Most > profile changes of this extend will put in some timeline before the older > profiles are removed (even hardened did a while with new profiles before > switching eventually). > > Another solution you might want to look at is find a portage snapshot "in > between" and do gradual upgrades, but you're not guaranteed to have all > software still available (the sources, that is). > > Perhaps the Gentoo Upgrading guide might need some more love to document > things (it's on the wiki now, so by all means, be my guest) but I don't > think for this particular bug itself we can do much more. So actually you are suggesting to simply checkout more or less random snapshots of the portage tree and hope, it works? (In reply to Kai Krakow from comment #16) > So actually you are suggesting to simply checkout more or less random > snapshots of the portage tree and hope, it works? That would probably be the simplest option. If you're lucky, the removal of the 10.0 profiles is in a ChangeLog file in the tree, which'll give you a date that they were removed, and then you can hunt down a portage snapshot from before that time. From my understanding, CVS has no unified version number, and versions every file separately, which would make using the CVS tree a rather large pain. There is also a somewhat old conversion of the tree to git, which lives here: http://git-exp.overlays.gentoo.org/gitweb/?p=exp/gentoo-x86.git;a=summary This appears to have the 10.0 profiles, and a new enough portage for EAPI 5 support. Once the main tree has migrated to git, it'll be much easier to support incremental migrations, as you can then simply checkout a version from git that is old enough to be able to complete the migration. Moving forward, however, I would recommend that you keep up with updates, as this specific case aside, it's loads easier to update a system that is 1 month out of date than it is to update a system that is 10 months out of date. (In reply to Kai Krakow from comment #16) > (In reply to Sven Vermeulen from comment #15) > > Switching EAPIs is done gradually and by the various profiles afaics. Most > > profile changes of this extend will put in some timeline before the older > > profiles are removed (even hardened did a while with new profiles before > > switching eventually). > > > > Another solution you might want to look at is find a portage snapshot "in > > between" and do gradual upgrades, but you're not guaranteed to have all > > software still available (the sources, that is). > > > > Perhaps the Gentoo Upgrading guide might need some more love to document > > things (it's on the wiki now, so by all means, be my guest) but I don't > > think for this particular bug itself we can do much more. > > So actually you are suggesting to simply checkout more or less random > snapshots of the portage tree and hope, it works? The upgrade from EAPI 4 to 5 was tested over a 2 month period and went smoothly. We perhaps could have sent out a news item alerting people to upgrade to the latest portage during that test phase, but I do recall sending out emails asking for the testing of the new 13.0 profiles. I'm only seeing this bug now, but I think the easiest path would have been to edit profiles/hardened/linux/parent and set ../../releases/10.0, then upgrade portage and then switch over to 13.0. This would have worked then, I'm not sure now. (In reply to Rick Farina (Zero_Chaos) from comment #12) > (In reply to Jorge Manuel B. S. Vicetto from comment #11) > > Now that this bug has been sitting for over 6 months, I believe it should > > either be closed or that it should be assigned to another team. > > This issue wasn't caused nor is it related to release, so there's no point > > for us to stick in here. > > Unless someone is ready to do the work to support this request, I also > > suggest we close it as INVALID as this wasn't a "bug", but the result of a > > conscious decision of the hardened team. > > +1 This bug should have been assigned to hardened. I would have taken care of it immediately if I'd seen it. (In reply to Anthony Basile from comment #18) > I'm only seeing this bug now, but I think the easiest path would have been > to edit profiles/hardened/linux/parent and set ../../releases/10.0, then > upgrade portage and then switch over to 13.0. This would have worked then, > I'm not sure now. Thanks, I will put it on my list. I think I still have one or two systems sitting around with very old portage status (12+ months). (In reply to Anthony Basile from comment #19) > (In reply to Rick Farina (Zero_Chaos) from comment #12) > > (In reply to Jorge Manuel B. S. Vicetto from comment #11) > > > Now that this bug has been sitting for over 6 months, I believe it should > > > either be closed or that it should be assigned to another team. > > > This issue wasn't caused nor is it related to release, so there's no point > > > for us to stick in here. > > > Unless someone is ready to do the work to support this request, I also > > > suggest we close it as INVALID as this wasn't a "bug", but the result of a > > > conscious decision of the hardened team. > > > > +1 > > This bug should have been assigned to hardened. I would have taken care of > it immediately if I'd seen it. As this bug to me should likely be closed as OBSOLETE but you requested it, I'm going to reassign it and drop release from it (we have too many already). |
