Bugzilla Bug 133908

It's come to my attention that there is a signficat problem with collisions
between packages in $PKGDIR/All that has been agravated by the addition of
new-style virtuals that have the same $P as their non-virtual counterparts.  I
propose that we deprecate the /All directory, put the real files in
$CATEGORY/$P.tbz2, and put symlinks in /All for backward compatibility (if
necessary).

Is this bug severe enough that we need to fix it in 2.1?  I imagine that the
patch won't be disruptive in terms of 2.1 stablization.  Plus, it's not really
a new feature, but a bug fix.  ;)

This would be a significant feature change and I don't think it should 
be done without lots of testing. Some of the cases off the top of my
head that would need proper testing and probably help from the portage
team would be tools such as eclean, qmerge, genpkgindex,
emerge(-g|-G|-k|-K|-b|-B) to deal with all cases.

As somebody that is currently maintaining ~7000 packages across 66 repos 
I know this change is going to cause alot of problems and am not in favor of
it. 

I see the root of the problem being that virtual/foo.ebuilds should of 
never of been introduced into the tree and should be reverted asap.

(In reply to comment #1)
> I see the root of the problem being that virtual/foo.ebuilds should of 
> never of been introduced into the tree and should be reverted asap.

That's another solution that I've considered.  We could simply make a rule that
virtuals can't have the same $PN as their non-virtual counterparts.  For
example, virtual/qmail could be moved to virtual/qmail-virtual or something
like that.

I realize the chances of reverting virtual/foo.ebuilds are probably
slim to none. So to move fwd without having to make any changes to
portage or any tools.. I'd also propose that we mandate all
virtual/$PVR.ebuild be renamed to virtual/virtual-$PVR.ebuild

> It's come to my attention that there is a signficat problem with collisions
> between packages in $PKGDIR/All that has been agravated by the addition of
> new-style virtuals that have the same $P as their non-virtual counterparts.  I
> propose that we deprecate the /All directory, put the real files in
> $CATEGORY/$P.tbz2, and put symlinks in /All for backward compatibility (if
> necessary).

Not exactly a new problem, just more apparent with all the new virtuals.

> Is this bug severe enough that we need to fix it in 2.1?  I imagine that the
> patch won't be disruptive in terms of 2.1 stablization.  Plus, it's not really
> a new feature, but a bug fix.  ;)

I wouldn't like this in 2.1. It's not a regression. It's a change in behavior
that might break some setups that rely on the current (mis)design.
Proper fix IMO would be to add an alternate bindbapi implementation (also gives
the chance to use a new container format), but I really suggest that only
regressions and obvious bugfixes are going into release candidates. 

(In reply to comment #1)
> I see the root of the problem being that virtual/foo.ebuilds should of 
> never of been introduced into the tree and should be reverted asap.

I disagree that this is the root problem, it just has made the problem more
apparent. Same discussion as with duplicate packages in general, see past -dev
discussions.
It's the "usual" namespace problem, renaming virtuals will just cover it up to
some degree for a while.

(In reply to comment #4)
> > It's come to my attention that there is a signficat problem with collisions
> > between packages in $PKGDIR/All that has been agravated by the addition of
> > new-style virtuals that have the same $P as their non-virtual counterparts.  I
> > propose that we deprecate the /All directory, put the real files in
> > $CATEGORY/$P.tbz2, and put symlinks in /All for backward compatibility (if
> > necessary).
> 
> Not exactly a new problem, just more apparent with all the new virtuals.
> 
> > Is this bug severe enough that we need to fix it in 2.1?  I imagine that the
> > patch won't be disruptive in terms of 2.1 stablization.  Plus, it's not really
> > a new feature, but a bug fix.  ;)
> 
> I wouldn't like this in 2.1. It's not a regression. It's a change in behavior
> that might break some setups that rely on the current (mis)design.
> Proper fix IMO would be to add an alternate bindbapi implementation (also gives
> the chance to use a new container format), but I really suggest that only
> regressions and obvious bugfixes are going into release candidates. 

That requires changes to all internal andexternal tools. 
(alot of work and testing)

> (In reply to comment #1)
> > I see the root of the problem being that virtual/foo.ebuilds should of 
> > never of been introduced into the tree and should be reverted asap.
> 
> I disagree that this is the root problem, it just has made the problem more
> apparent. Same discussion as with duplicate packages in general, see past -dev
> discussions.

Have we ever had a conflict where the version numbering were the same?

> It's the "usual" namespace problem, renaming virtuals will just cover it up to
> some degree for a while.

Yes. Returning us to expected behavior causing the least ammount of breakage.
Long run.. Yes it should be readdressed. But really while 2.0 exists I'd hate 
to see two unique methods of handling $PKGDIR/All

Another possible workaround would be a RESTRICT="buildpkg".

(In reply to comment #6)
> Another possible workaround would be a RESTRICT="buildpkg".

I've not tested but I would assume that restricting it would lead to broken 
dep graphs when using the -K/-G options and a given ebuild depends on
virtual/foo.

(In reply to comment #5)
> Have we ever had a conflict where the version numbering were the same?

Yes. Half of the php stuff fails on this. The transition from dev-php/php to
dev-lang/php was also affected by this. Besides, there are lots of packages
that have exactly the same versions in dev-php{4,5}. 

> > It's the "usual" namespace problem, renaming virtuals will just cover it up to
> > some degree for a while.

I agree with genone here. It should be fixed, but not in 2.1. IMHO it's a
pretty subtile bug we've been living with for years now, there's really no
hurry to get it fixed right now.

(In reply to comment #7)
> (In reply to comment #6)
> > Another possible workaround would be a RESTRICT="buildpkg".
> 
> I've not tested but I would assume that restricting it would lead to broken 
> dep graphs when using the -K/-G options and a given ebuild depends on
> virtual/foo.

ACK

(In reply to comment #3)
> I'd also propose that we mandate all
> virtual/$PVR.ebuild be renamed to virtual/virtual-$PVR.ebuild

+1 on this, since proper fix for this flat namespace issue is not really
trivial... 

I would like to remind you guys that this will affect GRP building.  We can
work around it for the few places where the namespace conflicts, but I'm sure
we're going to hit this sooner or later and it's going to be quite ugly.

As much as I hate hiding problems, I think it might really be a good idea to go
ahead with the virtual renames, until the changes can be made in portage and
all of the tools.

(In reply to comment #10)
> (In reply to comment #3)
> > I'd also propose that we mandate all
> > virtual/$PVR.ebuild be renamed to virtual/virtual-$PVR.ebuild
> 
> +1 on this, since proper fix for this flat namespace issue is not really
> trivial... 
> 
What about "new" style virtuals that took a similar but slightly different
route? I'm of course speaking for vested intersts - the virtual/perl-* family
:)

oops...

Created an attachment (id=87708) [details]
All/ ${cat} symlink switch

Just for kicks, I had a quick go at a patch. This works for local packages, but
there's no support for updating an existing binary repository. The directory
format of PORTAGE_BINHOST also does not allow for a quick fix. There are also a
bunch of tools within portage (quickpkg, fixpackages, etc) that would need to
be updated as well.

In short, I think the moving from All/* to ${CATEGORY}/* is viable, keeping
symlinks in All/ for backward compatibility, but I think it will need a fair
bit of testing and so should wait until 2.2.

(In reply to comment #4)
> Proper fix IMO would be to add an alternate bindbapi implementation (also gives
> the chance to use a new container format)

How about if we use a flag such as FEATURES="oldbinrepo" so that people that
rely on the old $PKGDIR/All can delay migration until they're other tools have
been updated for forward compatibility?

(In reply to comment #15)
> (In reply to comment #4)
> > Proper fix IMO would be to add an alternate bindbapi implementation (also gives
> > the chance to use a new container format)
> 
> How about if we use a flag such as FEATURES="oldbinrepo" so that people that
> rely on the old $PKGDIR/All can delay migration until they're other tools have
> been updated for forward compatibility?

How about inverting that logic so that those wanting newstyle binrepo 
support can enable it? Then we hash out most of the bugs with the tools.
That way we do it without forcing the new behavior on the larger 
majority right away. Eventually obsoleting the newstyle feature when 
we are content.

I was thinking FEATURES="oldbinrepo" would be in make.globals.  Either way,
it's essentially the same thing.

How about if we have portage detect the repo format automatically.  That way
the user can use either type of repo transparently (could simply use
$PKGDIR/All to detect the old format).  The feature would control whether an
empty $PKGDIR will default to the old or new format.  Eventually, we can remove
"oldbinrepo" from make.globals so that an empty $PKGDIR defaults to the new
format.  The transition will be transparent for the user (we'll provide a
migration script anyhow, in case a user wants to convert the repo format from
old to new or vice versa).

seem sane enough to allow for the testing and migration phases.

*** Bug 105308 has been marked as a duplicate of this bug. ***

Any news on this issue?

If we want to aim for maximum compatibility, we can keep the existing
repository format (mostly), and only put tbz2 files directly in
$PKGDIR/${CATEGORY}/${PF}.tbz2 when a collision occurs.

This is fixed in svn r4722.  Collisions are in ${PKGDIR}/All/ are prevented by
automatically bumping colliding packages to ${PKGDIR}/${CATEGORY}/ just before
a collision would occur.  Newly built packages are always stored in
${PKGDIR}/All/, so this should this be 100% compatible with previous behavior.

This has been released in 2.1.2_pre3-r3.

Is the following the intended behaviour?

# emerge =app-emacs/gentoo-syntax-1.7
[...]
# cd /usr/portage/packages; ls -l */gentoo-syntax*
-rw-r--r-- 1 root root 39828 Sep 30 12:29 All/gentoo-syntax-1.7.tbz2
lrwxrwxrwx 1 root root    29 Sep 30 12:29 app-emacs/gentoo-syntax-1.7.tbz2 ->
../All/gentoo-syntax-1.7.tbz2

Emerging a package with the same name, but from a different category will move
the existing binpkg from All to $CATEGORY:

# emerge =app-xemacs/gentoo-syntax-1.7
[...]
# cd /usr/portage/packages; ls -l */gentoo-syntax*
-rw-r--r-- 1 root root 46428 Sep 30 12:30 All/gentoo-syntax-1.7.tbz2
-rw-r--r-- 1 root root 39828 Sep 30 12:29 app-emacs/gentoo-syntax-1.7.tbz2
lrwxrwxrwx 1 root root    29 Sep 30 12:30 app-xemacs/gentoo-syntax-1.7.tbz2 ->
../All/gentoo-syntax-1.7.tbz2

And emergeing the old one again will result in another move:

# emerge =app-emacs/gentoo-syntax-1.7
[...]
a1iulm2 packages # cd /usr/portage/packages; ls -l */gentoo-syntax*
-rw-r--r-- 1 root root 39820 Sep 30 12:31 All/gentoo-syntax-1.7.tbz2
lrwxrwxrwx 1 root root    29 Sep 30 12:31 app-emacs/gentoo-syntax-1.7.tbz2 ->
../All/gentoo-syntax-1.7.tbz2
-rw-r--r-- 1 root root 46428 Sep 30 12:30 app-xemacs/gentoo-syntax-1.7.tbz2

IMHO, this is not very transparent. Wouldn't it make more sense to implement it
as it was suggested in comment #17? And eventually drop $PKGDIR/All/
completely?

(In reply to comment #24)
> IMHO, this is not very transparent. Wouldn't it make more sense to implement it
> as it was suggested in comment #17? And eventually drop $PKGDIR/All/
> completely?

That's the plan. However, current PORTAGE_BINHOST support relies on the All/
format. Of course, collisions are still a problem for the old PORTAGE_BINHOST
protocol, but that's just how it is. We currently have experimental new
PORTAGE_BINHOST support (in trunk) and the packages go into ${CATEGORY}
directories by default. It just hasn't been released yet.

Is there a bug for the new support?  Anything that makes the current situation
better would be a major boon for Release Engineering.

I've just filed bug #194552 for the PORTAGE_BINHOST issue.

Created an attachment (id=132740) [details]
script that moves packages from the "All" directory to separate ${CATEGORY}
directories

This script might be useful for people who are bothered by the behavior
described in comment #24. Versions of portage since 2.1.2.x can use packages
from inside ${CATEGORY} directories as well as the "All" directory. Newly built
packages will always be placed in the "All" directory, so you'll have to run
this script again each time that you build packages if you want to keep them
all in ${CATEGORY} directories.