It's come to my attention that there is a signficat problem with collisions between packages in $PKGDIR/All that has been agravated by the addition of new-style virtuals that have the same $P as their non-virtual counterparts. I propose that we deprecate the /All directory, put the real files in $CATEGORY/$P.tbz2, and put symlinks in /All for backward compatibility (if necessary). Is this bug severe enough that we need to fix it in 2.1? I imagine that the patch won't be disruptive in terms of 2.1 stablization. Plus, it's not really a new feature, but a bug fix. ;)
This would be a significant feature change and I don't think it should be done without lots of testing. Some of the cases off the top of my head that would need proper testing and probably help from the portage team would be tools such as eclean, qmerge, genpkgindex, emerge(-g|-G|-k|-K|-b|-B) to deal with all cases. As somebody that is currently maintaining ~7000 packages across 66 repos I know this change is going to cause alot of problems and am not in favor of it. I see the root of the problem being that virtual/foo.ebuilds should of never of been introduced into the tree and should be reverted asap.
(In reply to comment #1) > I see the root of the problem being that virtual/foo.ebuilds should of > never of been introduced into the tree and should be reverted asap. That's another solution that I've considered. We could simply make a rule that virtuals can't have the same $PN as their non-virtual counterparts. For example, virtual/qmail could be moved to virtual/qmail-virtual or something like that.
I realize the chances of reverting virtual/foo.ebuilds are probably slim to none. So to move fwd without having to make any changes to portage or any tools.. I'd also propose that we mandate all virtual/$PVR.ebuild be renamed to virtual/virtual-$PVR.ebuild
> It's come to my attention that there is a signficat problem with collisions > between packages in $PKGDIR/All that has been agravated by the addition of > new-style virtuals that have the same $P as their non-virtual counterparts. I > propose that we deprecate the /All directory, put the real files in > $CATEGORY/$P.tbz2, and put symlinks in /All for backward compatibility (if > necessary). Not exactly a new problem, just more apparent with all the new virtuals. > Is this bug severe enough that we need to fix it in 2.1? I imagine that the > patch won't be disruptive in terms of 2.1 stablization. Plus, it's not really > a new feature, but a bug fix. ;) I wouldn't like this in 2.1. It's not a regression. It's a change in behavior that might break some setups that rely on the current (mis)design. Proper fix IMO would be to add an alternate bindbapi implementation (also gives the chance to use a new container format), but I really suggest that only regressions and obvious bugfixes are going into release candidates. (In reply to comment #1) > I see the root of the problem being that virtual/foo.ebuilds should of > never of been introduced into the tree and should be reverted asap. I disagree that this is the root problem, it just has made the problem more apparent. Same discussion as with duplicate packages in general, see past -dev discussions. It's the "usual" namespace problem, renaming virtuals will just cover it up to some degree for a while.
(In reply to comment #4) > > It's come to my attention that there is a signficat problem with collisions > > between packages in $PKGDIR/All that has been agravated by the addition of > > new-style virtuals that have the same $P as their non-virtual counterparts. I > > propose that we deprecate the /All directory, put the real files in > > $CATEGORY/$P.tbz2, and put symlinks in /All for backward compatibility (if > > necessary). > > Not exactly a new problem, just more apparent with all the new virtuals. > > > Is this bug severe enough that we need to fix it in 2.1? I imagine that the > > patch won't be disruptive in terms of 2.1 stablization. Plus, it's not really > > a new feature, but a bug fix. ;) > > I wouldn't like this in 2.1. It's not a regression. It's a change in behavior > that might break some setups that rely on the current (mis)design. > Proper fix IMO would be to add an alternate bindbapi implementation (also gives > the chance to use a new container format), but I really suggest that only > regressions and obvious bugfixes are going into release candidates. That requires changes to all internal andexternal tools. (alot of work and testing) > (In reply to comment #1) > > I see the root of the problem being that virtual/foo.ebuilds should of > > never of been introduced into the tree and should be reverted asap. > > I disagree that this is the root problem, it just has made the problem more > apparent. Same discussion as with duplicate packages in general, see past -dev > discussions. Have we ever had a conflict where the version numbering were the same? > It's the "usual" namespace problem, renaming virtuals will just cover it up to > some degree for a while. Yes. Returning us to expected behavior causing the least ammount of breakage. Long run.. Yes it should be readdressed. But really while 2.0 exists I'd hate to see two unique methods of handling $PKGDIR/All
Another possible workaround would be a RESTRICT="buildpkg".
(In reply to comment #6) > Another possible workaround would be a RESTRICT="buildpkg". I've not tested but I would assume that restricting it would lead to broken dep graphs when using the -K/-G options and a given ebuild depends on virtual/foo.
(In reply to comment #5) > Have we ever had a conflict where the version numbering were the same? Yes. Half of the php stuff fails on this. The transition from dev-php/php to dev-lang/php was also affected by this. Besides, there are lots of packages that have exactly the same versions in dev-php{4,5}. > > It's the "usual" namespace problem, renaming virtuals will just cover it up to > > some degree for a while. I agree with genone here. It should be fixed, but not in 2.1. IMHO it's a pretty subtile bug we've been living with for years now, there's really no hurry to get it fixed right now.
(In reply to comment #7) > (In reply to comment #6) > > Another possible workaround would be a RESTRICT="buildpkg". > > I've not tested but I would assume that restricting it would lead to broken > dep graphs when using the -K/-G options and a given ebuild depends on > virtual/foo. ACK
(In reply to comment #3) > I'd also propose that we mandate all > virtual/$PVR.ebuild be renamed to virtual/virtual-$PVR.ebuild +1 on this, since proper fix for this flat namespace issue is not really trivial...
I would like to remind you guys that this will affect GRP building. We can work around it for the few places where the namespace conflicts, but I'm sure we're going to hit this sooner or later and it's going to be quite ugly. As much as I hate hiding problems, I think it might really be a good idea to go ahead with the virtual renames, until the changes can be made in portage and all of the tools.
(In reply to comment #10) > (In reply to comment #3) > > I'd also propose that we mandate all > > virtual/$PVR.ebuild be renamed to virtual/virtual-$PVR.ebuild > > +1 on this, since proper fix for this flat namespace issue is not really > trivial... > What about "new" style virtuals that took a similar but slightly different route? I'm of course speaking for vested intersts - the virtual/perl-* family :)
oops...
Created attachment 87708 [details, diff] All/ ${cat} symlink switch Just for kicks, I had a quick go at a patch. This works for local packages, but there's no support for updating an existing binary repository. The directory format of PORTAGE_BINHOST also does not allow for a quick fix. There are also a bunch of tools within portage (quickpkg, fixpackages, etc) that would need to be updated as well. In short, I think the moving from All/* to ${CATEGORY}/* is viable, keeping symlinks in All/ for backward compatibility, but I think it will need a fair bit of testing and so should wait until 2.2.
(In reply to comment #4) > Proper fix IMO would be to add an alternate bindbapi implementation (also gives > the chance to use a new container format) How about if we use a flag such as FEATURES="oldbinrepo" so that people that rely on the old $PKGDIR/All can delay migration until they're other tools have been updated for forward compatibility?
(In reply to comment #15) > (In reply to comment #4) > > Proper fix IMO would be to add an alternate bindbapi implementation (also gives > > the chance to use a new container format) > > How about if we use a flag such as FEATURES="oldbinrepo" so that people that > rely on the old $PKGDIR/All can delay migration until they're other tools have > been updated for forward compatibility? How about inverting that logic so that those wanting newstyle binrepo support can enable it? Then we hash out most of the bugs with the tools. That way we do it without forcing the new behavior on the larger majority right away. Eventually obsoleting the newstyle feature when we are content.
I was thinking FEATURES="oldbinrepo" would be in make.globals. Either way, it's essentially the same thing. How about if we have portage detect the repo format automatically. That way the user can use either type of repo transparently (could simply use $PKGDIR/All to detect the old format). The feature would control whether an empty $PKGDIR will default to the old or new format. Eventually, we can remove "oldbinrepo" from make.globals so that an empty $PKGDIR defaults to the new format. The transition will be transparent for the user (we'll provide a migration script anyhow, in case a user wants to convert the repo format from old to new or vice versa).
seem sane enough to allow for the testing and migration phases.
*** Bug 105308 has been marked as a duplicate of this bug. ***
Any news on this issue?
If we want to aim for maximum compatibility, we can keep the existing repository format (mostly), and only put tbz2 files directly in $PKGDIR/${CATEGORY}/${PF}.tbz2 when a collision occurs.
This is fixed in svn r4722. Collisions are in ${PKGDIR}/All/ are prevented by automatically bumping colliding packages to ${PKGDIR}/${CATEGORY}/ just before a collision would occur. Newly built packages are always stored in ${PKGDIR}/All/, so this should this be 100% compatible with previous behavior.
This has been released in 2.1.2_pre3-r3.
Is the following the intended behaviour? # emerge =app-emacs/gentoo-syntax-1.7 [...] # cd /usr/portage/packages; ls -l */gentoo-syntax* -rw-r--r-- 1 root root 39828 Sep 30 12:29 All/gentoo-syntax-1.7.tbz2 lrwxrwxrwx 1 root root 29 Sep 30 12:29 app-emacs/gentoo-syntax-1.7.tbz2 -> ../All/gentoo-syntax-1.7.tbz2 Emerging a package with the same name, but from a different category will move the existing binpkg from All to $CATEGORY: # emerge =app-xemacs/gentoo-syntax-1.7 [...] # cd /usr/portage/packages; ls -l */gentoo-syntax* -rw-r--r-- 1 root root 46428 Sep 30 12:30 All/gentoo-syntax-1.7.tbz2 -rw-r--r-- 1 root root 39828 Sep 30 12:29 app-emacs/gentoo-syntax-1.7.tbz2 lrwxrwxrwx 1 root root 29 Sep 30 12:30 app-xemacs/gentoo-syntax-1.7.tbz2 -> ../All/gentoo-syntax-1.7.tbz2 And emergeing the old one again will result in another move: # emerge =app-emacs/gentoo-syntax-1.7 [...] a1iulm2 packages # cd /usr/portage/packages; ls -l */gentoo-syntax* -rw-r--r-- 1 root root 39820 Sep 30 12:31 All/gentoo-syntax-1.7.tbz2 lrwxrwxrwx 1 root root 29 Sep 30 12:31 app-emacs/gentoo-syntax-1.7.tbz2 -> ../All/gentoo-syntax-1.7.tbz2 -rw-r--r-- 1 root root 46428 Sep 30 12:30 app-xemacs/gentoo-syntax-1.7.tbz2 IMHO, this is not very transparent. Wouldn't it make more sense to implement it as it was suggested in comment #17? And eventually drop $PKGDIR/All/ completely?
(In reply to comment #24) > IMHO, this is not very transparent. Wouldn't it make more sense to implement it > as it was suggested in comment #17? And eventually drop $PKGDIR/All/ > completely? That's the plan. However, current PORTAGE_BINHOST support relies on the All/ format. Of course, collisions are still a problem for the old PORTAGE_BINHOST protocol, but that's just how it is. We currently have experimental new PORTAGE_BINHOST support (in trunk) and the packages go into ${CATEGORY} directories by default. It just hasn't been released yet.
Is there a bug for the new support? Anything that makes the current situation better would be a major boon for Release Engineering.
I've just filed bug #194552 for the PORTAGE_BINHOST issue.
Created attachment 132740 [details, diff] script that moves packages from the "All" directory to separate ${CATEGORY} directories This script might be useful for people who are bothered by the behavior described in comment #24. Versions of portage since 2.1.2.x can use packages from inside ${CATEGORY} directories as well as the "All" directory. Newly built packages will always be placed in the "All" directory, so you'll have to run this script again each time that you build packages if you want to keep them all in ${CATEGORY} directories.
*** Bug 397599 has been marked as a duplicate of this bug. ***
