Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 296716 - PMS patch for EAPI3/offset-prefix (EPREFIX, ED and EROOT)
Summary: PMS patch for EAPI3/offset-prefix (EPREFIX, ED and EROOT)
Status: RESOLVED FIXED
Alias: None
Product: Gentoo Hosted Projects
Classification: Unclassified
Component: PMS/EAPI (show other bugs)
Hardware: All All
: High normal (vote)
Assignee: PMS/EAPI
URL:
Whiteboard: in-eapi-3
Keywords:
Depends on:
Blocks: future-eapi
  Show dependency tree
 
Reported: 2009-12-13 11:57 UTC by Fabian Groffen
Modified: 2010-01-18 22:17 UTC (History)
1 user (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
PMS offset-prefix patch (r1) (prefix-r1.patch,14.49 KB, patch)
2009-12-13 11:58 UTC, Fabian Groffen
Details | Diff
PMS offset-prefix patch (r2) (prefix-r2.patch,14.48 KB, patch)
2009-12-13 12:30 UTC, Fabian Groffen
Details | Diff
PMS offset-prefix patch (r3) (prefix-r3,15.85 KB, patch)
2009-12-15 20:39 UTC, Fabian Groffen
Details | Diff
prefix-r4.patch (prefix-r4.patch,16.83 KB, patch)
2010-01-10 10:34 UTC, Fabian Groffen
Details | Diff
PMS offset-prefix patch (r5) (prefix-r5.patch,17.27 KB, patch)
2010-01-10 10:55 UTC, Fabian Groffen
Details | Diff
PMS offset-prefix patch (r5) rebased against current PMS head (0001-Offset-prefix-EPREFIX-ED-and-EROOT.patch,16.67 KB, patch)
2010-01-11 22:11 UTC, Ulrich Müller
Details | Diff
PMS offset-prefix patch (r6) (0001-Offset-prefix-EPREFIX-ED-and-EROOT.patch,17.91 KB, patch)
2010-01-16 19:59 UTC, Ulrich Müller
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Fabian Groffen gentoo-dev 2009-12-13 11:57:31 UTC
as attached, applies to current git at the moment of creation of this bug.
Comment 1 Fabian Groffen gentoo-dev 2009-12-13 11:58:28 UTC
Created attachment 212862 [details, diff]
PMS offset-prefix patch (r1)
Comment 2 Fabian Groffen gentoo-dev 2009-12-13 12:30:45 UTC
Created attachment 212873 [details, diff]
PMS offset-prefix patch (r2)

after feedback from ulm:

- fill in feature set for EAPI3 (offset-prefix support)
- add wording to make explicit that offset-prefix support is also available in EAPIs after 3
- fixed EPREFIX, ED and EROOT variable table to also include EAPI4
Comment 3 Ciaran McCreesh 2009-12-13 14:42:05 UTC
I think we need to get that todo sorted out.

Your "+files under the" is a very long line.

We've never used the wording "EAPI x and later" before, since EAPIs don't have an order. We've always instead pointed to the appropriate table. You might find it easier to do this if there's a table saying "EAPIs supporting offset prefix" or somesuch.

I think we need something in here mentioning whether ebuilds are required to handle offset prefix, or whether it's allowable for ebuilds to behave correctly only if EPREFIX is empty. The way it's worded now more or less implies that an ebuild that barfs if EPREFIX is set to non-empty (for example, if it uses $D where it should use $ED to do some tinkering) is broken.
Comment 4 David Leverton 2009-12-13 16:28:14 UTC
(In reply to comment #3)
> I think we need something in here mentioning whether ebuilds are required to
> handle offset prefix, or whether it's allowable for ebuilds to behave correctly
> only if EPREFIX is empty. The way it's worded now more or less implies that an
> ebuild that barfs if EPREFIX is set to non-empty (for example, if it uses $D
> where it should use $ED to do some tinkering) is broken.

My understanding from various mails is that the Prefix team expects it to be treated like any other arch, ie if the ebuild isn't keyworded for Prefix, there's no expectation that it'll work under Prefix.

grobian: is that correct?  Has it been formally stated anywhere, and if not, should it be?
Comment 5 Ciaran McCreesh 2009-12-14 18:53:01 UTC
Another thing: can we s/in the environment/in the environment by user configuration/, to make it clear that it's illegal for ebuilds to mess with those three vars?
Comment 6 Jeremy Olexa (darkside) (RETIRED) archtester Gentoo Infrastructure gentoo-dev Security 2009-12-15 15:33:20 UTC
(In reply to comment #4)
> (In reply to comment #3)
> > I think we need something in here mentioning whether ebuilds are required to
> > handle offset prefix, or whether it's allowable for ebuilds to behave correctly
> > only if EPREFIX is empty. The way it's worded now more or less implies that an
> > ebuild that barfs if EPREFIX is set to non-empty (for example, if it uses $D
> > where it should use $ED to do some tinkering) is broken.
> 
> My understanding from various mails is that the Prefix team expects it to be
> treated like any other arch, ie if the ebuild isn't keyworded for Prefix,
> there's no expectation that it'll work under Prefix.
> 
> grobian: is that correct?  Has it been formally stated anywhere, and if not,
> should it be?

Correct. I'm not sure it needs to be stated anywhere really. Is it *that* unclear? (We have separate profiles, etc)

(In reply to comment #5)
> Another thing: can we s/in the environment/in the environment by user
> configuration/, to make it clear that it's illegal for ebuilds to mess with
> those three vars?

To my understanding, the vars should be read-only. I also want to make one thing clear, 'the compiled package' (executable, docs, etc) should never rely on EPREFIX being set in the environment. An exception to this is when you are bootstrapping, but that is not relevant here.
Comment 7 Ciaran McCreesh 2009-12-15 15:35:30 UTC
(In reply to comment #6)
> Correct. I'm not sure it needs to be stated anywhere really. Is it *that*
> unclear? (We have separate profiles, etc)

Yeah, it needs to be stated. PMS is allegedly independent of profiles etc.
Comment 8 Fabian Groffen gentoo-dev 2009-12-15 16:21:54 UTC
Technically, if an ebuild has EAPI=3, then it should be Prefix aware, that's the result of doing Prefix as a new EAPI.  I personally think this is the cleanest way given the current situation.

The current state of gx86 however is that if it carries Prefix keywords, the ebuild works for Prefix people, because we use a Portage that always supports an offset-prefix installation, regardless of EAPI.
Comment 9 Ciaran McCreesh 2009-12-15 16:24:13 UTC
That's something we need to know: is it your position that any ebuild using EAPI 3 or 4 must work with arbitrary prefix values? Is it your intent to prevent people from using other new features in EAPIs 3 and 4 if they don't have time to prefixify their package?
Comment 10 Fabian Groffen gentoo-dev 2009-12-15 16:33:25 UTC
I have no strong preference to either options, since I think both suck to a certain degree.  I do feel that it is cleanest to require offset-compatability when an offset-enabled EAPI is being used.

I think it's worth to put the pros and cons next to each other before deciding which option we will officially put in place.
Comment 11 Ciaran McCreesh 2009-12-15 16:35:30 UTC
I think this should probably go to -dev@. Would you like to send an email asking for opinions?
Comment 12 Fabian Groffen gentoo-dev 2009-12-15 16:38:38 UTC
I plan to first address your comments posted in here, so we're all on the same
level, and then ask for a discussion on -dev.
Comment 13 Fabian Groffen gentoo-dev 2009-12-15 18:24:43 UTC
(In reply to comment #3)
> I think we need to get that todo sorted out.
> 
> Your "+files under the" is a very long line.

easily rewrapped, I tried not to introduce any diffs just for cosmetic reasons

> We've never used the wording "EAPI x and later" before, since EAPIs don't have
> an order. We've always instead pointed to the appropriate table. You might find
> it easier to do this if there's a table saying "EAPIs supporting offset prefix"
> or somesuch.

How about Table 12.3?  ulm asked me to add the x and later everywhere.  So I guess he's the best to defend that wording.

> I think we need something in here mentioning whether ebuilds are required to
> handle offset prefix, or whether it's allowable for ebuilds to behave correctly
> only if EPREFIX is empty. The way it's worded now more or less implies that an
> ebuild that barfs if EPREFIX is set to non-empty (for example, if it uses $D
> where it should use $ED to do some tinkering) is broken.

It would be interesting to see if this is possible at all.  Due to the brokenness of some build-systems removing D (and ROOT) never seemed to be a good idea to me, you need them too often such that ${ED%${EPREFIX}} would get tiresome.  I will bring the discussion regarding whether it is ok to have EAPI={3,4} without offset-awareness to -dev ML, later.
Comment 14 Fabian Groffen gentoo-dev 2009-12-15 18:27:45 UTC
(In reply to comment #5)
> Another thing: can we s/in the environment/in the environment by user
> configuration/, to make it clear that it's illegal for ebuilds to mess with
> those three vars?

how about just s/environment/calling environment/ there?
Comment 15 Ulrich Müller gentoo-dev 2009-12-15 19:05:58 UTC
(In reply to comment #13)
> How about Table 12.3?  ulm asked me to add the x and later everywhere.
> So I guess he's the best to defend that wording.

I think the "and later" is fine, as long as all our EAPIs are integers (which can be ordered). But I have no objections against a table. Probably this would be more in line with the rest of the document.
Comment 16 Ciaran McCreesh 2009-12-15 19:12:11 UTC
Well, we've got "The package manager must not attempt to perform any kind of comparison test other than equality upon EAPIs." in there... I'd rather we didn't restrict EAPIs to a hierarchy without very good reason.
Comment 17 Ulrich Müller gentoo-dev 2009-12-15 19:15:45 UTC
As I said, I have no objections against a table.
Comment 18 Fabian Groffen gentoo-dev 2009-12-15 20:39:32 UTC
Created attachment 213134 [details, diff]
PMS offset-prefix patch (r3)

This revision adds an explicit table specifying which EAPIs are offset-prefix aware, and drops the before and later wording again.
Comment 19 Jeremy Olexa (darkside) (RETIRED) archtester Gentoo Infrastructure gentoo-dev Security 2009-12-16 22:16:15 UTC
(In reply to comment #12)
> I plan to first address your comments posted in here, so we're all on the same
> level, and then ask for a discussion on -dev.
> 

For the record, the thread is here:
http://archives.gentoo.org/gentoo-dev/msg_bbadeffac202df43a276272dd39297c1.xml
Comment 20 Christian Faulhammer (RETIRED) gentoo-dev 2009-12-16 23:07:00 UTC
Some review:

+	value than the built in value is set in the calling environment, a
+	cross-prefix build is performed where using the existing utilities,
+	a package is build for the given \t{EPREFIX}, akin to \t{ROOT}.  See

* built-in?  And "built" on the last line?

* Do we need three columns and all variables separately in tab:offset-env-vars-table?

+The latter do have the variables \t{ED} and \t{EROOT} properly set,
+though

* Isn't this bad writing style?

* For the sake of consistence you should always write ${EPREFIX} instead of $EPREFIX in some cases.

* econf should be marked-up with \t always.

* The todos need to be sorted out.  Any proposals how to handle them?
Comment 21 Fabian Groffen gentoo-dev 2009-12-18 18:36:03 UTC
(In reply to comment #19)
> (In reply to comment #12)
> > I plan to first address your comments posted in here, so we're all on the same
> > level, and then ask for a discussion on -dev.
> > 
> 
> For the record, the thread is here:
> http://archives.gentoo.org/gentoo-dev/msg_bbadeffac202df43a276272dd39297c1.xml

Outcome seems to be that offset-prefix support should be optional. 
Comment 22 Ulrich Müller gentoo-dev 2009-12-18 21:43:19 UTC
The first hunk in eapi-differences.tex doesn't look right, since it adds the variables to the line containing the default phase functions:

-        \t{src\_compile}, \t{src\_install}, \t{src\_test}} \\
+        \t{src\_compile}, \t{src\_install}, \t{src\_test},
+        \t{EPREFIX}, \t{ED}, \t{EROOT}} \\

I suggest to add the following line instead (directly after REPLACED_BY_VERSION seems to be the best place):

+\t{EPREFIX}, \t{ED}, \t{EROOT} & \compactfeatureref{offset-prefix-vars} &
+    No & No & No & Yes & Yes \\

Comment 23 Fabian Groffen gentoo-dev 2009-12-18 21:50:40 UTC
Go ahead.  You know what the idea is.  All of you ask to either revert, or do it again in another different way, so please just fix it to whatever you're happy with.
Comment 24 Fabian Groffen gentoo-dev 2009-12-19 20:17:15 UTC
(In reply to comment #20)
> * The todos need to be sorted out.  Any proposals how to handle them?

% todo: Portage does not behave like this, and Prefix relies on that
%       root:root -> 0:0 (for systems where root name/group are different)
%       PORTAGE_INST_UID:PORTAGE_INST_GID <- Prefix sets that to user ids

options:
1) drop the clause from PMS
2) update the clause to say the PM will set ownership to the super user or its equivalent for the system/installation at hand
3) change the clause for offset-prefix aware EAPIs to say that the PM will set ownership to the configured user (PORTAGE_ROOT_USER)

% todo: Portage's implementation is different and required for Prefix
%       these files are unpacked with deb2targz if available, else it
%       falls back to "ar", which cannot be required to be GNU binutils

1) ignore it, PMS' definition stems from a GNU/Linux dominated environment like with the previous issue is the case, Portage just implements for more than just GNU/Linux
2) relax the clause to just mention that a deb-archive is being unpacked, just defining what's the outcome on disk, instead of restricting it to using GNU binutils to unpack
3) change the clause to read that for offset-aware EAPIs deb-file unpacking is implemented using deb2targz with a fallback to ar (if it's GNU).
Comment 25 Ulrich Müller gentoo-dev 2009-12-20 17:40:33 UTC
(In reply to comment #24)
> % todo: Portage does not behave like this, and Prefix relies on that
> %       root:root -> 0:0 (for systems where root name/group are different)
> %       PORTAGE_INST_UID:PORTAGE_INST_GID <- Prefix sets that to user ids

IMHO, 2) is best here:

> 2) update the clause to say the PM will set ownership to the super user or its
> equivalent for the system/installation at hand

> % todo: Portage's implementation is different and required for Prefix
> %       these files are unpacked with deb2targz if available, else it
> %       falls back to "ar", which cannot be required to be GNU binutils

Not sure about this one. Maybe split the item, as follows?

    \item ar archives (\t{*.a}). Ebuilds must ensure that the ar
    program is installed.
    \item deb archives (\t{*.deb}). Ebuilds must ensure that the
    deb2targz program or the ar program is installed.
Comment 26 Ciaran McCreesh 2009-12-20 17:49:11 UTC
The whole unpack thing's a frickin' mess. Non-prefix EAPIs will have to carry on using the existing wording, so the question's how to word it for prefix things.
Comment 27 Christian Faulhammer (RETIRED) gentoo-dev 2009-12-29 15:49:18 UTC
(In reply to comment #24)
> (In reply to comment #20)
> > * The todos need to be sorted out.  Any proposals how to handle them?
> 
> % todo: Portage does not behave like this, and Prefix relies on that
> %       root:root -> 0:0 (for systems where root name/group are different)
> %       PORTAGE_INST_UID:PORTAGE_INST_GID <- Prefix sets that to user ids
> 
> options:
> 2) update the clause to say the PM will set ownership to the super user or its
> equivalent for the system/installation at hand

 Totally my vote.

> % todo: Portage's implementation is different and required for Prefix
> %       these files are unpacked with deb2targz if available, else it
> %       falls back to "ar", which cannot be required to be GNU binutils
> 
> 1) ignore it, PMS' definition stems from a GNU/Linux dominated environment like
> with the previous issue is the case, Portage just implements for more than just
> GNU/Linux
> 2) relax the clause to just mention that a deb-archive is being unpacked, just
> defining what's the outcome on disk, instead of restricting it to using GNU
> binutils to unpack
> 3) change the clause to read that for offset-aware EAPIs deb-file unpacking is
> implemented using deb2targz with a fallback to ar (if it's GNU).

 2 should be fine, maybe saying that GNU binutils is preferred if available.
Comment 28 Fabian Groffen gentoo-dev 2010-01-10 10:34:45 UTC
Created attachment 215910 [details, diff]
prefix-r4.patch

This is the patch ulm gave me, but applied to the current head.
Comment 29 Fabian Groffen gentoo-dev 2010-01-10 10:47:30 UTC
(In reply to comment #26)
> The whole unpack thing's a frickin' mess. Non-prefix EAPIs will have to carry
> on using the existing wording, so the question's how to word it for prefix
> things.

Do you forsee any problems with changing the description (for all existing EAPIs) to allow deb2targz next to GNU binutils ar to be used to unpack a .deb file?  If so, which?  I ask because Portage is already implemented as such.
Comment 30 Fabian Groffen gentoo-dev 2010-01-10 10:55:52 UTC
Created attachment 215915 [details, diff]
PMS offset-prefix patch (r5)

This revision of the Prefix patch addresses both TODOs based on the input in this bug.
Comment 31 Ciaran McCreesh 2010-01-10 15:56:07 UTC
(In reply to comment #29)
> Do you forsee any problems with changing the description (for all existing
> EAPIs) to allow deb2targz next to GNU binutils ar to be used to unpack a .deb
> file?  If so, which?  I ask because Portage is already implemented as such.

Depends upon how you word it. Non-prefix EAPI ebuilds mustn't just DEPEND="deb2targz", for example. You probably also need to word it such that ebuilds *must* select ar where possible, except there's not really a nice way for ebuilds to do that.
Comment 32 Fabian Groffen gentoo-dev 2010-01-10 16:17:19 UTC
(In reply to comment #31)
> (In reply to comment #29)
> > Do you forsee any problems with changing the description (for all existing
> > EAPIs) to allow deb2targz next to GNU binutils ar to be used to unpack a .deb
> > file?  If so, which?  I ask because Portage is already implemented as such.
> 
> Depends upon how you word it. Non-prefix EAPI ebuilds mustn't just
> DEPEND="deb2targz", for example.

No, so it must perhaps be said even more clearly that the ebuild must guarantee in some what that the deb2targz program is available, if ...

> You probably also need to word it such that
> ebuilds *must* select ar where possible, except there's not really a nice way
> for ebuilds to do that.

Why is that a requirement?  It's not nice to force a deb2targz dependency on someone who doesn't need it, but what's the danger in accidentially doing so/having it?  Of course its awkward to just unconditionally depend on it while it is known that only a couple of arches need it, but that seems to be a detail that the ebuild writer/arch team member needs to deal with.
Comment 33 Ciaran McCreesh 2010-01-10 16:23:14 UTC
Ideally you'd keep it legal for package managers that only support basic unprefixed things to continue using exactly the same unpack code they've always used.
Comment 34 Fabian Groffen gentoo-dev 2010-01-10 16:28:18 UTC
A package manager that chooses not to fully implement offset installation support can do so without problems.  It will never face any platform that needs deb2targz to unpack a .deb file.
Comment 35 Ciaran McCreesh 2010-01-10 16:32:47 UTC
Depends upon how you word it... If you simply say that either binutils ar or dep2targz must be available, it strongly suggests that it's legal for an ebuild to DEPEND="dep2targz". What with the state of binary packages and all, it's pretty unlikely that anyone's going to be able to get away with not happening to  have binutils installed by coincidence anyway, but still, I'd rather not set a precedent of picking wording that makes that kind of situation legal since there's a remote possibility that dependencies might start being accurate and complete some day.
Comment 36 Fabian Groffen gentoo-dev 2010-01-10 16:48:22 UTC
Maybe we're not talking about the same thing.  This is taken from the patch:

    \item deb packages (\t{*.deb}).  Ebuilds must ensure that the
    deb2targz program is installed on those platforms where the GNU binutils
    ar program is not available and the installed ar program is
    incompatible with GNU archives.  Otherwise, ebuilds must ensure that
    GNU binutils is installed.
Comment 37 Ciaran McCreesh 2010-01-10 16:54:36 UTC
Yeah, but how does an ebuild specify that? What's the syntax for it?
Comment 38 Fabian Groffen gentoo-dev 2010-01-10 16:58:58 UTC
something like:
  ppc-aix? ( app-arch/deb2targz )
?
Comment 39 Ciaran McCreesh 2010-01-10 17:02:43 UTC
That's going to get really really messy, isn't it? Won't you need dozens of those all over the place?

*shrug* ah well. Guess we'll have to push hard for magic/unpack-blah in EAPI 5...
Comment 40 Fabian Groffen gentoo-dev 2010-01-10 17:05:12 UTC
Not really, it's just a single package, or maybe 2.  It's just because the package manager doesn't add the correct dependency based on the distfile that this awkwardness needs to take place.  Similar to xz-utils issue.
Comment 41 Ciaran McCreesh 2010-01-10 17:11:04 UTC
No, that's the point: the package manager *can't* add dependencies automatically, because not everything in A gets unpacked. That's why magic/unpack-blah deps are needed, and why they have to be specified explicitly.
Comment 42 Fabian Groffen gentoo-dev 2010-01-10 17:18:48 UTC
Great, that all sounds plausible, but shows that this issue needs to be resolved correctly at some point.  You can drop this entry from the current patch, as I just noted that Portage implements it differently, and that that allows a few Prefix arches to install one tool that we need.  It would be more reasonable to allow it per PMS though, IMO.
Comment 43 Ulrich Müller gentoo-dev 2010-01-11 22:11:53 UTC
Created attachment 216131 [details, diff]
PMS offset-prefix patch (r5) rebased against current PMS head

Find attached a new version of your patch that applies cleanly against the current head (where yesterday's removal of kdebuild conditionals introduced relatively large changes).

Except for resolution of conflicts with the kdebuild-removal patch, nothing should have changed.
Comment 44 Christian Faulhammer (RETIRED) gentoo-dev 2010-01-13 21:09:16 UTC
For the records: The patch is fine for me.
Comment 45 Ulrich Müller gentoo-dev 2010-01-16 19:59:57 UTC
Created attachment 216699 [details, diff]
PMS offset-prefix patch (r6)

I did another round of review, and here is yet another version. Only tiny changes, as compared to r5:
- In subsection "Installation commands": Added word "These" at the beginning
  of the third sentence (that had been mistakenly removed in r5).
- In subsubsection "Misc commands", item "unpack": Add word "the" in phrase:
  "... where _the_ GNU binutils ar program is not available ..."
- Some whitespace polishing.
Comment 46 Christian Faulhammer (RETIRED) gentoo-dev 2010-01-17 12:06:03 UTC
It has been pushed to the repository.
Comment 47 Ulrich Müller gentoo-dev 2010-01-18 22:13:51 UTC
EAPI 3 approved by council.