Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 133908 - collisions in $PKGDIR/All
Summary: collisions in $PKGDIR/All
Status: RESOLVED FIXED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: .tbz2 support (show other bugs)
Hardware: All Linux
: High normal (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords: InVCS
: 105308 397599 (view as bug list)
Depends on: 194552
Blocks: 147007
  Show dependency tree
 
Reported: 2006-05-20 16:19 UTC by Zac Medico
Modified: 2012-01-04 08:34 UTC (History)
12 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
All/ ${cat} symlink switch (binpkg-with-cats.patch,4.44 KB, patch)
2006-05-28 02:00 UTC, Jason Stubbs (RETIRED)
Details | Diff
script that moves packages from the "All" directory to separate ${CATEGORY} directories (move-from-all,472 bytes, patch)
2007-10-06 15:17 UTC, Zac Medico
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Zac Medico gentoo-dev 2006-05-20 16:19:00 UTC
It's come to my attention that there is a signficat problem with collisions between packages in $PKGDIR/All that has been agravated by the addition of new-style virtuals that have the same $P as their non-virtual counterparts.  I propose that we deprecate the /All directory, put the real files in $CATEGORY/$P.tbz2, and put symlinks in /All for backward compatibility (if necessary).

Is this bug severe enough that we need to fix it in 2.1?  I imagine that the patch won't be disruptive in terms of 2.1 stablization.  Plus, it's not really a new feature, but a bug fix.  ;)
Comment 1 solar (RETIRED) gentoo-dev 2006-05-20 16:56:38 UTC
This would be a significant feature change and I don't think it should 
be done without lots of testing. Some of the cases off the top of my
head that would need proper testing and probably help from the portage
team would be tools such as eclean, qmerge, genpkgindex, emerge(-g|-G|-k|-K|-b|-B) to deal with all cases.

As somebody that is currently maintaining ~7000 packages across 66 repos 
I know this change is going to cause alot of problems and am not in favor of it. 

I see the root of the problem being that virtual/foo.ebuilds should of 
never of been introduced into the tree and should be reverted asap.
Comment 2 Zac Medico gentoo-dev 2006-05-20 17:17:58 UTC
(In reply to comment #1)
> I see the root of the problem being that virtual/foo.ebuilds should of 
> never of been introduced into the tree and should be reverted asap.

That's another solution that I've considered.  We could simply make a rule that virtuals can't have the same $PN as their non-virtual counterparts.  For example, virtual/qmail could be moved to virtual/qmail-virtual or something like that.
Comment 3 solar (RETIRED) gentoo-dev 2006-05-20 17:23:06 UTC
I realize the chances of reverting virtual/foo.ebuilds are probably
slim to none. So to move fwd without having to make any changes to
portage or any tools.. I'd also propose that we mandate all
virtual/$PVR.ebuild be renamed to virtual/virtual-$PVR.ebuild
Comment 4 Marius Mauch (RETIRED) gentoo-dev 2006-05-20 17:33:02 UTC
> It's come to my attention that there is a signficat problem with collisions
> between packages in $PKGDIR/All that has been agravated by the addition of
> new-style virtuals that have the same $P as their non-virtual counterparts.  I
> propose that we deprecate the /All directory, put the real files in
> $CATEGORY/$P.tbz2, and put symlinks in /All for backward compatibility (if
> necessary).

Not exactly a new problem, just more apparent with all the new virtuals.

> Is this bug severe enough that we need to fix it in 2.1?  I imagine that the
> patch won't be disruptive in terms of 2.1 stablization.  Plus, it's not really
> a new feature, but a bug fix.  ;)

I wouldn't like this in 2.1. It's not a regression. It's a change in behavior that might break some setups that rely on the current (mis)design.
Proper fix IMO would be to add an alternate bindbapi implementation (also gives the chance to use a new container format), but I really suggest that only regressions and obvious bugfixes are going into release candidates. 

(In reply to comment #1)
> I see the root of the problem being that virtual/foo.ebuilds should of 
> never of been introduced into the tree and should be reverted asap.

I disagree that this is the root problem, it just has made the problem more apparent. Same discussion as with duplicate packages in general, see past -dev
discussions.
It's the "usual" namespace problem, renaming virtuals will just cover it up to some degree for a while.
Comment 5 solar (RETIRED) gentoo-dev 2006-05-20 17:46:42 UTC
(In reply to comment #4)
> > It's come to my attention that there is a signficat problem with collisions
> > between packages in $PKGDIR/All that has been agravated by the addition of
> > new-style virtuals that have the same $P as their non-virtual counterparts.  I
> > propose that we deprecate the /All directory, put the real files in
> > $CATEGORY/$P.tbz2, and put symlinks in /All for backward compatibility (if
> > necessary).
> 
> Not exactly a new problem, just more apparent with all the new virtuals.
> 
> > Is this bug severe enough that we need to fix it in 2.1?  I imagine that the
> > patch won't be disruptive in terms of 2.1 stablization.  Plus, it's not really
> > a new feature, but a bug fix.  ;)
> 
> I wouldn't like this in 2.1. It's not a regression. It's a change in behavior
> that might break some setups that rely on the current (mis)design.
> Proper fix IMO would be to add an alternate bindbapi implementation (also gives
> the chance to use a new container format), but I really suggest that only
> regressions and obvious bugfixes are going into release candidates. 

That requires changes to all internal andexternal tools. 
(alot of work and testing)
 
> (In reply to comment #1)
> > I see the root of the problem being that virtual/foo.ebuilds should of 
> > never of been introduced into the tree and should be reverted asap.
> 
> I disagree that this is the root problem, it just has made the problem more
> apparent. Same discussion as with duplicate packages in general, see past -dev
> discussions.

Have we ever had a conflict where the version numbering were the same?

> It's the "usual" namespace problem, renaming virtuals will just cover it up to
> some degree for a while.

Yes. Returning us to expected behavior causing the least ammount of breakage.
Long run.. Yes it should be readdressed. But really while 2.0 exists I'd hate 
to see two unique methods of handling $PKGDIR/All
Comment 6 Jason Stubbs (RETIRED) gentoo-dev 2006-05-20 19:12:31 UTC
Another possible workaround would be a RESTRICT="buildpkg".
Comment 7 solar (RETIRED) gentoo-dev 2006-05-20 21:24:17 UTC
(In reply to comment #6)
> Another possible workaround would be a RESTRICT="buildpkg".

I've not tested but I would assume that restricting it would lead to broken 
dep graphs when using the -K/-G options and a given ebuild depends on virtual/foo.

Comment 8 Simon Stelling (RETIRED) gentoo-dev 2006-05-21 02:00:04 UTC
(In reply to comment #5)
> Have we ever had a conflict where the version numbering were the same?

Yes. Half of the php stuff fails on this. The transition from dev-php/php to dev-lang/php was also affected by this. Besides, there are lots of packages that have exactly the same versions in dev-php{4,5}. 
 
> > It's the "usual" namespace problem, renaming virtuals will just cover it up to
> > some degree for a while.

I agree with genone here. It should be fixed, but not in 2.1. IMHO it's a pretty subtile bug we've been living with for years now, there's really no hurry to get it fixed right now.
Comment 9 Jason Stubbs (RETIRED) gentoo-dev 2006-05-21 02:27:37 UTC
(In reply to comment #7)
> (In reply to comment #6)
> > Another possible workaround would be a RESTRICT="buildpkg".
> 
> I've not tested but I would assume that restricting it would lead to broken 
> dep graphs when using the -K/-G options and a given ebuild depends on
> virtual/foo.

ACK
Comment 10 Jakub Moc (RETIRED) gentoo-dev 2006-05-23 05:18:08 UTC
(In reply to comment #3)
> I'd also propose that we mandate all
> virtual/$PVR.ebuild be renamed to virtual/virtual-$PVR.ebuild

+1 on this, since proper fix for this flat namespace issue is not really trivial... 
Comment 11 Chris Gianelloni (RETIRED) gentoo-dev 2006-05-23 06:31:02 UTC
I would like to remind you guys that this will affect GRP building.  We can work around it for the few places where the namespace conflicts, but I'm sure we're going to hit this sooner or later and it's going to be quite ugly.

As much as I hate hiding problems, I think it might really be a good idea to go ahead with the virtual renames, until the changes can be made in portage and all of the tools.
Comment 12 Michael Cummings (RETIRED) gentoo-dev 2006-05-23 06:31:38 UTC
(In reply to comment #10)
> (In reply to comment #3)
> > I'd also propose that we mandate all
> > virtual/$PVR.ebuild be renamed to virtual/virtual-$PVR.ebuild
> 
> +1 on this, since proper fix for this flat namespace issue is not really
> trivial... 
> 
What about "new" style virtuals that took a similar but slightly different route? I'm of course speaking for vested intersts - the virtual/perl-* family :)
Comment 13 Chris Gianelloni (RETIRED) gentoo-dev 2006-05-23 06:32:05 UTC
oops...
Comment 14 Jason Stubbs (RETIRED) gentoo-dev 2006-05-28 02:00:07 UTC
Created attachment 87708 [details, diff]
All/ ${cat} symlink switch

Just for kicks, I had a quick go at a patch. This works for local packages, but there's no support for updating an existing binary repository. The directory format of PORTAGE_BINHOST also does not allow for a quick fix. There are also a bunch of tools within portage (quickpkg, fixpackages, etc) that would need to be updated as well.

In short, I think the moving from All/* to ${CATEGORY}/* is viable, keeping symlinks in All/ for backward compatibility, but I think it will need a fair bit of testing and so should wait until 2.2.
Comment 15 Zac Medico gentoo-dev 2006-06-23 18:46:16 UTC
(In reply to comment #4)
> Proper fix IMO would be to add an alternate bindbapi implementation (also gives
> the chance to use a new container format)

How about if we use a flag such as FEATURES="oldbinrepo" so that people that rely on the old $PKGDIR/All can delay migration until they're other tools have been updated for forward compatibility?
Comment 16 solar (RETIRED) gentoo-dev 2006-06-24 22:55:26 UTC
(In reply to comment #15)
> (In reply to comment #4)
> > Proper fix IMO would be to add an alternate bindbapi implementation (also gives
> > the chance to use a new container format)
> 
> How about if we use a flag such as FEATURES="oldbinrepo" so that people that
> rely on the old $PKGDIR/All can delay migration until they're other tools have
> been updated for forward compatibility?

How about inverting that logic so that those wanting newstyle binrepo 
support can enable it? Then we hash out most of the bugs with the tools.
That way we do it without forcing the new behavior on the larger 
majority right away. Eventually obsoleting the newstyle feature when 
we are content.
Comment 17 Zac Medico gentoo-dev 2006-06-24 23:20:41 UTC
I was thinking FEATURES="oldbinrepo" would be in make.globals.  Either way, it's essentially the same thing.

How about if we have portage detect the repo format automatically.  That way the user can use either type of repo transparently (could simply use $PKGDIR/All to detect the old format).  The feature would control whether an empty $PKGDIR will default to the old or new format.  Eventually, we can remove "oldbinrepo" from make.globals so that an empty $PKGDIR defaults to the new format.  The transition will be transparent for the user (we'll provide a migration script anyhow, in case a user wants to convert the repo format from old to new or vice versa).
Comment 18 solar (RETIRED) gentoo-dev 2006-06-25 13:01:37 UTC
seem sane enough to allow for the testing and migration phases.
Comment 19 Zac Medico gentoo-dev 2006-07-25 23:18:13 UTC
*** Bug 105308 has been marked as a duplicate of this bug. ***
Comment 20 Wolfram Schlich (RETIRED) gentoo-dev 2006-10-12 02:59:58 UTC
Any news on this issue?
Comment 21 Zac Medico gentoo-dev 2006-10-14 01:52:50 UTC
If we want to aim for maximum compatibility, we can keep the existing repository format (mostly), and only put tbz2 files directly in $PKGDIR/${CATEGORY}/${PF}.tbz2 when a collision occurs.
Comment 22 Zac Medico gentoo-dev 2006-10-16 02:01:21 UTC
This is fixed in svn r4722.  Collisions are in ${PKGDIR}/All/ are prevented by automatically bumping colliding packages to ${PKGDIR}/${CATEGORY}/ just before a collision would occur.  Newly built packages are always stored in ${PKGDIR}/All/, so this should this be 100% compatible with previous behavior.
Comment 23 Zac Medico gentoo-dev 2006-10-16 19:40:31 UTC
This has been released in 2.1.2_pre3-r3.
Comment 24 Ulrich Müller gentoo-dev 2007-09-30 10:44:50 UTC
Is the following the intended behaviour?

# emerge =app-emacs/gentoo-syntax-1.7
[...]
# cd /usr/portage/packages; ls -l */gentoo-syntax*
-rw-r--r-- 1 root root 39828 Sep 30 12:29 All/gentoo-syntax-1.7.tbz2
lrwxrwxrwx 1 root root    29 Sep 30 12:29 app-emacs/gentoo-syntax-1.7.tbz2 -> ../All/gentoo-syntax-1.7.tbz2

Emerging a package with the same name, but from a different category will move the existing binpkg from All to $CATEGORY:

# emerge =app-xemacs/gentoo-syntax-1.7
[...]
# cd /usr/portage/packages; ls -l */gentoo-syntax*
-rw-r--r-- 1 root root 46428 Sep 30 12:30 All/gentoo-syntax-1.7.tbz2
-rw-r--r-- 1 root root 39828 Sep 30 12:29 app-emacs/gentoo-syntax-1.7.tbz2
lrwxrwxrwx 1 root root    29 Sep 30 12:30 app-xemacs/gentoo-syntax-1.7.tbz2 -> ../All/gentoo-syntax-1.7.tbz2

And emergeing the old one again will result in another move:

# emerge =app-emacs/gentoo-syntax-1.7
[...]
a1iulm2 packages # cd /usr/portage/packages; ls -l */gentoo-syntax*
-rw-r--r-- 1 root root 39820 Sep 30 12:31 All/gentoo-syntax-1.7.tbz2
lrwxrwxrwx 1 root root    29 Sep 30 12:31 app-emacs/gentoo-syntax-1.7.tbz2 -> ../All/gentoo-syntax-1.7.tbz2
-rw-r--r-- 1 root root 46428 Sep 30 12:30 app-xemacs/gentoo-syntax-1.7.tbz2

IMHO, this is not very transparent. Wouldn't it make more sense to implement it as it was suggested in comment #17? And eventually drop $PKGDIR/All/ completely?
Comment 25 Zac Medico gentoo-dev 2007-09-30 16:30:55 UTC
(In reply to comment #24)
> IMHO, this is not very transparent. Wouldn't it make more sense to implement it
> as it was suggested in comment #17? And eventually drop $PKGDIR/All/
> completely?

That's the plan. However, current PORTAGE_BINHOST support relies on the All/ format. Of course, collisions are still a problem for the old PORTAGE_BINHOST protocol, but that's just how it is. We currently have experimental new PORTAGE_BINHOST support (in trunk) and the packages go into ${CATEGORY} directories by default. It just hasn't been released yet.
Comment 26 Chris Gianelloni (RETIRED) gentoo-dev 2007-10-02 19:49:27 UTC
Is there a bug for the new support?  Anything that makes the current situation better would be a major boon for Release Engineering.
Comment 27 Zac Medico gentoo-dev 2007-10-02 20:42:10 UTC
I've just filed bug #194552 for the PORTAGE_BINHOST issue.
Comment 28 Zac Medico gentoo-dev 2007-10-06 15:17:10 UTC
Created attachment 132740 [details, diff]
script that moves packages from the "All" directory to separate ${CATEGORY} directories

This script might be useful for people who are bothered by the behavior described in comment #24. Versions of portage since 2.1.2.x can use packages from inside ${CATEGORY} directories as well as the "All" directory. Newly built packages will always be placed in the "All" directory, so you'll have to run this script again each time that you build packages if you want to keep them all in ${CATEGORY} directories.
Comment 29 Zac Medico gentoo-dev 2012-01-04 08:34:59 UTC
*** Bug 397599 has been marked as a duplicate of this bug. ***