Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 524236 - sys-apps/portage-2.2.14_rc1 trouble emerging on case insensitive volumes
Summary: sys-apps/portage-2.2.14_rc1 trouble emerging on case insensitive volumes
Status: RESOLVED FIXED
Alias: None
Product: Gentoo/Alt
Classification: Unclassified
Component: Prefix Support (show other bugs)
Hardware: All OS X
: Normal major (vote)
Assignee: Gentoo Prefix
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2014-10-01 18:39 UTC by iloveosxzfs
Modified: 2014-11-19 21:41 UTC (History)
3 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
Initial detected file collisions. (conflicts.txt,50.50 KB, text/plain)
2014-10-01 18:39 UTC, iloveosxzfs
Details
Portage broken (broken.txt,675.71 KB, text/plain)
2014-10-01 18:40 UTC, iloveosxzfs
Details
Implement FEATURES=case-insensitive-fs (FEATURES-case-insensitive-fs-for-bug-524236.patch,4.01 KB, patch)
2014-10-02 18:34 UTC, Zac Medico
Details | Diff
Implement FEATURES=case-insensitive-fs (FEATURES-case-insensitive-fs-for-bug-524236.patch,4.47 KB, patch)
2014-10-02 18:48 UTC, Zac Medico
Details | Diff
Implement FEATURES=case-insensitive-fs (FEATURES-case-insensitive-fs-for-bug-524236.patch,13.16 KB, patch)
2014-10-10 22:16 UTC, Zac Medico
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description iloveosxzfs 2014-10-01 18:39:08 UTC
Created attachment 385908 [details]
Initial detected file collisions.

At first we get file collisions.

 * Detected file collision(s):
 *
 *      /Users/joe/gentoo/usr/lib/portage/pym/_emerge/UserQuery.py
 *      /Users/joe/gentoo/usr/lib/python2.7/site-packages/_emerge/UserQuery.py
 *      /Users/joe/gentoo/usr/lib/python3.3/site-packages/_emerge/UserQuery.py

Moving those files out of the way allows for a "successful" emerge, but when we actually then go try to use it:

joe@Josephs-MacBook-Pro ~ $ emerge
Traceback (most recent call last):
  File "/Users/joe/gentoo/usr/bin/emerge", line 50, in <module>
    retval = emerge_main()
  File "/Users/joe/gentoo/usr/lib/python2.7/site-packages/_emerge/main.py", line 1043, in emerge_main
    action=myaction, args=myfiles, opts=myopts)
  File "/Users/joe/gentoo/usr/lib/python2.7/site-packages/portage/proxy/objectproxy.py", line 30, in __call__
    result = object.__getattribute__(self, '_get_target')()
  File "/Users/joe/gentoo/usr/lib/python2.7/site-packages/portage/proxy/lazyimport.py", line 130, in _get_target
    __import__(name)
  File "/Users/joe/gentoo/usr/lib/python2.7/site-packages/_emerge/actions.py", line 71, in <module>
    from _emerge.depgraph import backtrack_depgraph, depgraph, resume_depgraph
  File "/Users/joe/gentoo/usr/lib/python2.7/site-packages/_emerge/depgraph.py", line 76, in <module>
    from _emerge.UserQuery import UserQuery
ImportError: No module named UserQuery
joe@Josephs-MacBook-Pro ~ $

So I reverted to the prior version of sys-apps/portage using the ebuild command since the portage command itself was now broken.

Later, based on the diagnosis of mgorny from #gentoo-portage, I created a case-sensitive ram disk, copied my entire prefix there, and tried again. The emerge succeeded and the new emerge command worked. So I rsync'd the changes from the ram disk back to my real prefix, ejected ram disk, and restored the real prefix to its original location. And the new emerge command is now installed fine.

amadio from #gentoo-prefix reports that copying only the UserQuery.py files from Linux into OS X prefix also was a successful workaround.
Comment 1 iloveosxzfs 2014-10-01 18:40:02 UTC
Created attachment 385910 [details]
Portage broken
Comment 2 Fabian Groffen gentoo-dev 2014-10-01 19:16:37 UTC
Can reproduce this.  Feels like the unmerge code doesn't take the collisions into account.  For reference, this is easily fixed by running:

  ebuild portage-2.2.14_rc1.ebuild merge

from $EPREFIX/usr/src/portage/sys-apps/portage.
Comment 3 Fabian Groffen gentoo-dev 2014-10-01 19:18:34 UTC
@portage: the respective files are not on the filesystem after merge (FEATURES=-collision-protect), any pointers on where to look are appreciated.
Comment 4 Zac Medico gentoo-dev 2014-10-01 20:29:08 UTC
(In reply to iloveosxzfs from comment #1)
> Created attachment 385910 [details]
> Portage broken

The log clearly shows the UserQuery.py* files being merged with camelcase names, and then unmerged with lowercase names.

In order to avoid this behavior, we would need case-insensitive support in the dblink.isowner method, since that's what's used to check if an unmerge candidate belongs to the new instance that was just merged.
Comment 5 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2014-10-01 21:59:05 UTC
(In reply to Zac Medico from comment #4)
> (In reply to iloveosxzfs from comment #1)
> > Created attachment 385910 [details]
> > Portage broken
> 
> The log clearly shows the UserQuery.py* files being merged with camelcase
> names, and then unmerged with lowercase names.
> 
> In order to avoid this behavior, we would need case-insensitive support in
> the dblink.isowner method, since that's what's used to check if an unmerge
> candidate belongs to the new instance that was just merged.

First, we need to agree that we won't allow installing two files with the same case-insensitive names. A QA check could suffice for a start but don't know if people really want that limitation.
Comment 6 Fabian Groffen gentoo-dev 2014-10-02 08:24:56 UTC
Not sure if we want to restrict case changes.  Collision protection already identifies there is a problem (and aborts).  The only real problem (IMO) at the moment is that unmerging removes the just installed file.  I'll see if I can do something with dblink.isowner as Zac suggests.
Comment 7 iloveosxzfs 2014-10-02 08:28:27 UTC
Case changes can still be accomplished on a case-insensitive file system by doing

mv foo bar
mv bar Foo

though I don't know whether that approach is feasible in this context.
Comment 8 Martin Väth 2014-10-02 16:06:09 UTC
(In reply to iloveosxzfs from comment #7)
> Case changes can still be accomplished on a case-insensitive file system

Not necessarily. For some filesystems there are options to report always
in upper or lower case.
Comment 9 iloveosxzfs 2014-10-02 16:13:59 UTC
Right. That is the distinction between the meaning of being case sensitive and the mean of being case preserving. You are right that I was assuming it would be case preserving, which is because I'm particularly thinking of HFS+J which is both case insensitive and case preserving.
Comment 10 Zac Medico gentoo-dev 2014-10-02 18:34:31 UTC
Created attachment 385946 [details, diff]
Implement FEATURES=case-insensitive-fs

When case-insensitive-fs is enabled in FEATURES, the dblink.isowner method and the _owners_db class will be case-insensitive. This causes the collision-protect and unmerge code to behave correctly for a case-insensitive file system. If the file system is case-insensitive but case-preserving, then case is preserved in the CONTENTS data of installed packages.

I'll be maintaining this patch in the following branch:

	https://github.com/zmedico/portage/tree/bug_524236
Comment 11 Zac Medico gentoo-dev 2014-10-02 18:48:39 UTC
Created attachment 385950 [details, diff]
Implement FEATURES=case-insensitive-fs

This fixes the previous patch so that dblink._match_contents correctly preserves the case of the CONTENTS key that it returns.
Comment 12 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2014-10-02 19:09:21 UTC
I'm against adding anything more to FEATURES. This variable is a horrible mistake, and features should be controlled via separate configuration variables.
Comment 13 Zac Medico gentoo-dev 2014-10-02 19:25:57 UTC
(In reply to Michał Górny from comment #12)
> I'm against adding anything more to FEATURES. This variable is a horrible
> mistake, and features should be controlled via separate configuration
> variables.

Are you sure that you want to add more environment variables instead? Maybe we should also have a way to configure things that doesn't involve environment variables. For example, when we added repos.conf, we replaced PORTDIR and PORTDIR_OVERLAY variables with an INI-style configuration file.
Comment 14 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2014-10-02 19:48:35 UTC
Yes, I would like to have proper configuration too. However, I don't have time myself to work on all that.
Comment 15 Zac Medico gentoo-dev 2014-10-02 20:14:11 UTC
(In reply to Michał Górny from comment #14)
> Yes, I would like to have proper configuration too. However, I don't have
> time myself to work on all that.

In this context, is adding a new environment variable really better than re-using the existing FEATURES variable? What's so bad about the FEATURES variable?
Comment 16 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2014-10-02 20:37:57 UTC
(In reply to Zac Medico from comment #15)
> (In reply to Michał Górny from comment #14)
> > Yes, I would like to have proper configuration too. However, I don't have
> > time myself to work on all that.
> 
> In this context, is adding a new environment variable really better than
> re-using the existing FEATURES variable? What's so bad about the FEATURES
> variable?

Yes, anything is better than adding up to that crap pile. What's so bad is that it is a collection of more than dozen totally unrelated options that sometimes collide one with another or force others. It's completely impossible to properly template it for CONFIG_PROTECT, and one basically has to diff manpage from time to time to know when you push yet another magical keywords into it.
Comment 17 Zac Medico gentoo-dev 2014-10-02 22:26:01 UTC
(In reply to Michał Górny from comment #16)
> Yes, anything is better than adding up to that crap pile. What's so bad is
> that it is a collection of more than dozen totally unrelated options that
> sometimes collide one with another or force others.

You seem biased against FEATURES. Anyway, we can discuss its pros/cons on a mailing list, since it's only tangentially related to this bug.  

> It's completely impossible to properly template it for CONFIG_PROTECT,

I'm not sure what you mean here. Are you talking about some interaction between case-insensive support and CONFIG_PROTECT? Anyway, it reminds me that the attached patch doesn't yet include case-insensitive support for CONFIG_PROTECT (though it should not be difficult to implement).
Comment 18 Zac Medico gentoo-dev 2014-10-10 22:16:09 UTC
Created attachment 386404 [details, diff]
Implement FEATURES=case-insensitive-fs

This fixes the previous patch to include case-insensitive CONFIG_PROTECT support (including etc-update and dispatch-conf).
Comment 19 Fabian Groffen gentoo-dev 2014-11-12 18:12:19 UTC
I've included this in the Prefix branch where it seems to work wonders
Comment 20 Fabian Groffen gentoo-dev 2014-11-12 18:22:46 UTC
2.2.10.1-r1 and 2.2.14 include the case-insensitive-fs patch and enable it through make.globals.

People must upgrade through 2.2.10.1-r1 because the patch must be active to emerge 2.2.14.  I couldn't find a way to express this in dependencies (Portage complained about circular deps and/or just ignored it) therefore I masked 2.2.14 for the moment.  To be unmasked after a week or so from now.
Comment 21 Zac Medico gentoo-dev 2014-11-13 01:24:36 UTC
For the master branch, I've posted this for review:

http://thread.gmane.org/gmane.linux.gentoo.portage.devel/4779
Comment 23 Fabian Groffen gentoo-dev 2014-11-18 10:21:29 UTC
I will backout the earlier one, and merge from master for the next prefix release
Comment 24 Ruud Koolen (RETIRED) archtester Gentoo Infrastructure gentoo-dev Security 2014-11-19 21:41:35 UTC
(In reply to Fabian Groffen from comment #20)
> I masked 2.2.14 for the moment.  To be unmasked after a week or so from now.

Unmasked.