Specification being: profile-format = [pms | portage-1] If unspecified, defaults to pms; if it's a value other than those listed/supported, than it's up to the PM what to do (likely throw warnings at minimal if the profile is used). If the format is pms, than the rules of pms apply. If the format is portage-1, than it's pms rules, just allowing package.* and use.* to be directories rather than files (directory traversal/parsing being via alphanumeric sorting- ["1", "a1", "a2"] for example). Pretty much this format is portage's normal behaviour now, just codified so we can apply appropriate QA rules when PMS format is in use (gentoo-x86 tree). In terms of portage work, this is semi straightforward, although requires touching a couple of spots in config. The sole tricky part is that portage doesn't associate a make.profile to a specific repository, and realistically a given profile may cross multiple repositories. To handle that, just take the realpath of the profile, look for the rightmost instance of 'profiles' directory, than load up the repository configuration from ../metadata/layout.conf from that path chunk (preferably caching repoconfig instances so you're not parsing it every single time). Since portage currently allows portage-1 by default rather than pms-1, a compatibility phase where it throws warnings if a profile uses portage-1 behaviour but isn't marked as portage-1 is required obviously, although I'm hoping repository maintainers will move their asses on marking their repositories. End brain dump.
Adding PMS; as is, layout.conf isn't even part of PMS, but considering the manifest/cache/profile format controls that are in there, we should be adding it in. Realistically every manager knows of that file, and has for a while w/ the exception of pkgcore- which isn't a huge issue for pkgcore users, they're expected to be running latest version due to it's developmental state.
We already have the 'eapi' file for this. All this proposal does is reimplement the same concept in a different way, except that it doesn't interact properly with existing package managers.
(In reply to comment #2) > We already have the 'eapi' file for this. All this proposal does is reimplement > the same concept in a different way Different intentions here. The goal of profile-format is twofold; to allow new profile formats (literal on disk incompatible) to be developed/deployed, and to also once and for all fix the issue of package.mask being used as a directory while being incompliant with PMS. New formats should be positioned differently in the tree to avoid collisions, same thing that was done with cache-format; the sole reason portage-1 is exempted is because I'm adhering to reality on this, rather than letting a couple more years of incompatibility to continue via sticking strictly to the academic notion of how it should be. Sticking our heads in the sand and telling people "package.mask and friends have to be files, PMS says so" is reliant on the notion everything must be PMS; this marker is explicitly for enabling new development and addressing reality rather than allowing the existing stagnation to continue. Basically, EAPI is ebuild format versioning (atoms specifically, and realistically just that if one is designing sanely) w/in a format; trying to use it for versioning of the format as whole limits what can be done and slows development itself while ignoring reality. Doing as you propose, we'd have to wait for EAPI5; figure a year (again, pragmatic reality rules here, it'll be a year). Then every repository that uses directories, would have to jump to that EAPI. End result, compatibility is worse; specifically, for the vast majority of gentoo users who are using portage, it introduces a compatibility bottleneck just so that pkgcore/paludis can support real world profiles correctly. Reality is, those profiles explode now for PMS strict managers, and there is no way to detect if the upstream wants it parsed in PMS strict mode, or in portage-1 mode; profile-format adds that, while opening up a way to introduce new formats down the line. It's a pragmatic solution that also gives us a way out of this mess longer term. Or we do as you propose, abuse eapi markers and kick the can a few more feet down the road. > except that it doesn't interact properly with existing package managers. This is a bit of bullshit; it does. For paludis repositories configuration, you already spell out the repo format and the profile pathway within that repository. Via that, you can access layout.conf and swivel the profile format as needed. It's actually a bit simpler for paludis configuration. For portage style configuration (/etc/make.profile), I already laid out a way for this to work for existing profile formats that are out there; for any future/odder profile setups, the PM would have to be modified for awareness of it anyways so make.profile still would fly (not perfectly, but it's a flaw of the configuration format- orthogonal flaw specifically).
EAPI=funtoo-4. Define it as being identical to 4, except having one extra feature. Problem solved using existing wheel.
(In reply to comment #4) > EAPI=funtoo-4. Define it as being identical to 4, except having one extra > feature. Problem solved using existing wheel. Breaking compatibility for all existing portage instances that use that repo via requiring people to upgrade to a portage version w/ that EAPI just so that pkgcore/paludis can stop bitching about package.mask as a directory. Yeah, that's fucking sane and lovely for users. "Hey guys, we're going to put you through compatibility hell just so pkgcore/paludis can use your profiles. Also, when we develop a new profile format we'll put you through hell then too. Enjoy! ~PMS" Be a bit more realistic please. As is, these profiles aren't PMS compliant, and they do *not* need to be. Like it or not, we have to work with that fact rather than inducing pain on the user base trying to shoehorn PMS in.
It's about doing it right, which is a one time cost, as opposed to adding in yet another hack that will be with us forever.
You could hybridize both approaches by using layout.conf to set the default eapi for the profiles, so that they default to funtoo-4 instead of 0. Then you could bump the eapi via layout.conf without breaking compatibility for existing funtoo users.
@zac: the hybrid approach has some nasty interactions; specifically, it's retroactively modifying PMS rules for profiles. Additionally, it has the same compatibility issues doing per node EAPI has- specifically think through existing instances where a package.mask has a slotted atom; that requires EAPI=1 to be set. Now we're in a pickle; the default/unspecified for that repo is 5 (package.mask as a directory), but this one node specifies 2, which disallows as a directory. Upstreams options are to either: 1) pull the eapi marker, which means any PM that doesn't yet support your proposal, now views the profile as EAPI=0; breakage. 2) mark the node itself as eapi=5, which has the same flaws I outlined in #3. Basically it makes the transition worse- causes new pain just to enable existing functionality. Note your proposal still results in a PMS incompliant profile, unless PMS decides to suddenly retroactively require all PM's to support a controllable default (which isn't viable from a compatibility standpoint- either induces breakage or requires sitting on it for a year before using it). In contrast, profile-format doesn't add any new pain- it's already there. It addresses the existing pain points /now/, and gives us a way to swap in new profile formats down the line. To be clear, default profile eapi is useful, we just can't realistically slip it in (nor use it for this) due to PMS rules. Potentially a worthwhile addition to a new format... @ciaran: bluntly, if PMS had been done this right from the get go we wouldn't be in this mess. Further, define 'right'; academic right? Specifically breaking working bits for people just to adhere to rules that don't even actually /apply/ to the profiles in question? Or pragmatic right; marking those profiles as non pms so the manager can swivel it's support, having it work /now/, inducing no extra pain, while addressing the long term problem of swapping the profile format. What you're arguing is basically academic aesthetics; tweak the existing profile slightly and break existing profiles/trees just to try and mark them as PMS; the exact same issue will reoccur the next time someone wants to refactor profiles. It's just not sane.
(In reply to comment #8) > Now we're in a pickle; the default/unspecified for that repo is 5 (package.mask > as a directory), but this one node specifies 2, which disallows as a directory. pardon, meant "this one node specifies 1". Sorry for the typo.
PMS did get this right. It documented what Portage claimed it supported. Where things went wrong was when people started using something that Portage silently accepted that neither its documentation nor PMS said was supported. Repositories using package.mask as a directory are already broken.
(In reply to comment #3) > Basically, EAPI is ebuild format versioning (atoms specifically, and > realistically just that if one is designing sanely) w/in a format; trying to > use it for versioning of the format as whole limits what can be done and slows > development itself while ignoring reality. I'm not sure what you mean by this. Is there a potential change to the profile format that couldn't be handled by marking the specific directories that use it with a new EAPI? (In reply to comment #5) > Breaking compatibility for all existing portage instances that use that repo > via requiring people to upgrade to a portage version w/ that EAPI just so that > pkgcore/paludis can stop bitching about package.mask as a directory. > > Yeah, that's fucking sane and lovely for users. They wouldn't have to add the EAPI marker the moment Portage started supporting it, they could leave some time for users to upgrade. I'm not sure what Funtoo etc's policies are, but I wouldn't have thought they had such strict compatibility requirements as the main Gentoo tree, so they wouldn't have to wait too long. (In reply to comment #8) > Now we're in a pickle; the default/unspecified for that repo is 5 (package.mask > as a directory), but this one node specifies 1, which disallows as a directory. > Upstreams options are to either: > 1) pull the eapi marker, which means any PM that doesn't yet support your > proposal, now views the profile as EAPI=0; breakage. > 2) mark the node itself as eapi=5, which has the same flaws I outlined in #3. Profile EAPI only applies to the specific directory that it appears in, so as long as the directory with EAPI 1 in it doesn't use package.mask directories it's fine. It's perfectly OK to have other EAPI 5 profiles in the same repository, even if they have the EAPI 1 node as a parent. Incidentally, that's a significant advantage to using profile EAPI over specifying it tree-wide in layout.conf: with layout.conf, if the package manager doesn't support the new format then with layout.conf it can't use the repository at all, but with profile EAPI it can still use the older profiles that don't use the new features (unless it's used in profiles/ itself, but that can't be helped). > @ciaran: bluntly, if PMS had been done this right from the get go we wouldn't > be in this mess. Further, define 'right'; academic right? Specifically > breaking working bits for people just to adhere to rules that don't even > actually /apply/ to the profiles in question? Regardless of whether this particular feature should or shouldn't have been in PMS to start with, presumably there'll be other profile format changes in future. As above, with profile EAPI they can be introduced in new profiles without killing off compatibility with existing systems using older package managers and older profiles. > What you're arguing is basically academic aesthetics; tweak the existing > profile slightly and break existing profiles/trees just to try and mark them as > PMS; the exact same issue will reoccur the next time someone wants to refactor > profiles. It's just not sane. Next time someone wants to refactor profiles, existing users will be using Portage versions that don't support the new feature and therefore will break just as badly with layout.conf as with profile EAPI, if not worse. Your way only works this time because Portage already accepts the directories, and because until now it didn't look at layout.conf profile-format at all and therefore didn't check whether the value was something it recognised.
(In reply to comment #11) > (In reply to comment #3) > > Basically, EAPI is ebuild format versioning (atoms specifically, and > > realistically just that if one is designing sanely) w/in a format; trying to > > use it for versioning of the format as whole limits what can be done and slows > > development itself while ignoring reality. > > I'm not sure what you mean by this. Is there a potential change to the profile > format that couldn't be handled by marking the specific directories that use it > with a new EAPI? Re-read the 'slow' part; the problem w/ EAPI for things other than ebuild format itself is that it's a single axis of versioning, and that any changes to the profiles require sitting on them a fairly long while prior to deployment. One simple example of what cannot be handled by EAPI is default EAPI for the profile tree; literally shifting 0 to 2 as an example. That's not possible to roll out w/out a new format realistically. > (In reply to comment #5) > > Breaking compatibility for all existing portage instances that use that repo > > via requiring people to upgrade to a portage version w/ that EAPI just so that > > pkgcore/paludis can stop bitching about package.mask as a directory. > > > > Yeah, that's fucking sane and lovely for users. > > They wouldn't have to add the EAPI marker the moment Portage started supporting > it, they could leave some time for users to upgrade. In the interim, the PMS strict package managers remain broke. Further, the roll out phase requires getting that EAPI approved, deployed to stable for portage, than you start counting months till enough folk have upgraded that flipping the bit is viable. This is ignoring that jumping to an EAPI>1 for the profiles introduces it's own issues- namely that use atom's are now usable in profiles, but the semantics of that are completely ill defined (and likely varied in interpretation across the PMs). PMS views of profile/EAPI is murky (per ciaran's own statements I might add). Pretty much, if it's reliant on EAPI, the situation is going to continue for a potentially lengthy time while stepping into murky interpretations for EAPI>1. > I'm not sure what Funtoo > etc's policies are, but I wouldn't have thought they had such strict > compatibility requirements as the main Gentoo tree, so they wouldn't have to > wait too long. Bit more than just funtoo; there are normal gentoo overlays out there w/ it in addition. > (In reply to comment #8) > Incidentally, that's a significant advantage to using profile EAPI over > specifying it tree-wide in layout.conf: with layout.conf, if the package > manager doesn't support the new format then with layout.conf it can't use the > repository at all, but with profile EAPI it can still use the older profiles > that don't use the new features (unless it's used in profiles/ itself, but that > can't be helped). This is slightly disenguous; the same issue exists for both, and is reliant on carefully husbanding compatibility. Admittedly the fallout does vary slightly. One thing to note- the intention that all separate profile formats live at a different location (with the exception of portage-1 to address reality) means it would be possible for a repository to carry >1 profile formats; specifically for compatibility. The same can be done via some nastyness in PMS profile structuring; pointing out that there still is a compatibility option here. > > @ciaran: bluntly, if PMS had been done this right from the get go we wouldn't > > be in this mess. Further, define 'right'; academic right? Specifically > > breaking working bits for people just to adhere to rules that don't even > > actually /apply/ to the profiles in question? > > Regardless of whether this particular feature should or shouldn't have been in > PMS to start with, presumably there'll be other profile format changes in > future. As above, with profile EAPI they can be introduced in new profiles > without killing off compatibility with existing systems using older package > managers and older profiles. Additional note; there are certain cleanups we cannot deploy for example. PMS rules are that profiles cannot assume that they can override a repository level package.mask for example (yet the lion share of users have exactly that in use); realistically trying to resolve that via EAPI isn't sane- the solution would require stating "if EAPI5 is seen in the profile stack, than allow overriding of package.mask, else there is no gurantee" which makes things even more of a mess. > > What you're arguing is basically academic aesthetics; tweak the existing > > profile slightly and break existing profiles/trees just to try and mark them as > > PMS; the exact same issue will reoccur the next time someone wants to refactor > > profiles. It's just not sane. > > Next time someone wants to refactor profiles, existing users will be using > Portage versions that don't support the new feature and therefore will break > just as badly with layout.conf as with profile EAPI, if not worse. Your way > only works this time because Portage already accepts the directories, and > because until now it didn't look at layout.conf profile-format at all and > therefore didn't check whether the value was something it recognised. To be clear, I'm not denying I'm exploiting exactly what you described, but people are ignoring the longer term gains it allows. If we do this, *) we can get it sorted now without breaking anything that isn't already broken *) if desired, we can fold that profile format back into PMS via later EAPI extensions (year or two down the road), thus eliminating the sole instance where a profile format is allowed to live at the same spot as a seperate format. Cleanup of the hack complaints primarily. *) we can start deploying awareness of the profile formats that are bundled in a repository, specifically getting compatibility code in place for whenever someone creates a new format; note that the PM may not support the format, but it'll be able to give the user far more information as to what's going on rather than just puking badly. *) as mentioned above, with a minor tweak this can be extended to handle transitioning between profile formats. One thing I do not care to see out of this is a continuation of the previous head in the sand approach taken- specifically "PMS says no, so piss off"; we need to sort this not bitch and leave it broken for another couple of years.
The thing that is broken is not PMS, and it is not strict package managers. The thing that is broken is Funtoo and certain overlays. Nor is it "PMS says no, so piss off". It is "PMS says no, so either put it in a new Gentoo EAPI, or come up with an appropriate set of EAPI extensions for your distribution". We could have had a Funtoo EAPI years ago if they hadn't refused to do so.
(In reply to comment #13) > The thing that is broken is not PMS, and it is not strict package managers. The > thing that is broken is Funtoo and certain overlays. > > Nor is it "PMS says no, so piss off". It is "PMS says no, so either put it in a > new Gentoo EAPI, or come up with an appropriate set of EAPI extensions for your > distribution". We could have had a Funtoo EAPI years ago if they hadn't refused > to do so. While it's convenient to point and blame funtoo... this has been in portage (meaning >90% of our users have this support, quite likely >95% frankly) since at minimal, the start of '09; likely further back (I stopped searching at svn 12409). The issue has been known the whole time, including the incompatibility. The solution proffered thus far as has been "PMS says no"; I've yet to see /any/ attempt to solve this (eapi or otherwise) along those lines, let alone the funtoo eapi claim, and it's not like I'm new to EAPI after all. Regardless of interesting interpretations of history, here's why this line of discussion is pointless; it's going nowhere. The solution you offer is to break existing trees that work fine for majority of our users, in the name of supporting the pkgcore/paludis users. This isn't sane, it's being academic rather than pragmatic. What I'm proposing addresses it without any additional breakage; further it introduces the ability to switch formats down the line as needed. These are all positives, and bluntly, the discussion here reminds me of when I first added EAPI to portage- people trying to kick the can further down the road rather than architecting a solution that addresses the current need while opening up our options down the line. Like it or not, this does /not/ require PMS approval to go forward; as is, these profiles were already outside PMS anyways (just the same as the new md5-dict cache is, the manifest controls, etc); PMS says nothing about layout.conf despite basically every fucking overlay relying on it for eclass controls. That alone says a bit about PMS, but that's a seperate discussion. Bluntly; I've laid out a solution that fixes the problem now w/out additional pain. The counter proposal thus far has been to break shit further while addressing just that one point- the same issue can repeat down the line (and will, inevitably, nature of formats is they grow). Propose something sane and I'm interested; if the plan is to keep screaming "EAPI!" without addressing the issues it introduces than kindly be silent- you've already gotten your views across. No point in continuing the same debate w/out new arguments after all, unless the notion is to just illogically scream it down.
The EAPI solution is sane. What's not sane is the lengths people are going to to avoid doing anything EAPI related. Your solution introduces a massive amount of pain, by introducing a whole new incompatible mechanism that everyone will have to deal with. The "Make a Funtoo EAPI" offer has been open from the start. I even offered to help the Funtoo people do it when the issue first came up; the answer was no because at the time drobbins believed that PMS was something to do with Exherbo, not Gentoo. Giving Funtoo an EAPI with their proposed extensions is easy to do, solves the problem cleanly and allows everything everyone wants.
(In reply to comment #15) > The EAPI solution is sane. What's not sane is the lengths people are going to > to avoid doing anything EAPI related. Your solution introduces a massive amount > of pain, by introducing a whole new incompatible mechanism that everyone will > have to deal with. Spell out the "massive amount pain", since obviously I disagree. Preferably w/out the usual FUD mind you- I want exact faults, where it won't work (contrasting it against the current breakage), etc. In addition, I expect you to address how we can deal w/ profile changes PMS can't cover (repository package.mask negation, default eapi, etc) without swivelling the format. Finally, w/ the rules that that new formats cannot reuse the existing directory location, noting that there is nothing from stopping a repo from carrying multiple formats for compatibility, I expect a logical step by step breakdown of how this induces pain/complexity/hell. > The "Make a Funtoo EAPI" offer has been open from the start. I even offered to > help the Funtoo people do it when the issue first came up; the answer was no > because at the time drobbins believed that PMS was something to do with > Exherbo, not Gentoo. Giving Funtoo an EAPI with their proposed extensions is > easy to do, solves the problem cleanly and allows everything everyone wants. Again; how is this deployed w/out causing breakage for existing gentoo repos that are *not* funtoo? Further, as you said, the rules of EAPI>1 in profiles are murky- how are those rules going to be nailed down (and retroactively applied to existing EAPIs), or is your plan purely to double the number of EAPIs in use (going from EAPI={0,1,2,3,4} to EAPI={0,1,2,3,4}{,-profile-as-dir}) ? Work the solution out in full and address the issues you yourself have raised, and address the compatibility gap I laid out. Bluntly; I've been responding w/ technical points. You're thus far responding with vague FUD. I expect actual details, else troll elsewhere.</bluntly>
"Massive amount of pain" is that you're introducing a whole new mechanism that sort of behaves like EAPI, sort of doesn't and that has to interact with it in some unspecified way. Mechanisms are extremely expensive, and we already have one. It's not perfect, but it can be improved if it has to be. Default EAPIs, for example, if you really think you need them (which you don't, since GLEP 55 fixes that), are best done within the EAPI mechanism, not by introducing a whole new thing. There's no breakage problem -- people simply need to ensure that a sufficiently new package manager is being used. That's an issue when profiles are updated to use new things anyway, and all we're doing here is finally correcting a long-standing screwup. It has to be dealt with at some point, and delaying the inevitable only makes things worse. You don't need a whole bunch of new EAPIs either, since we can just go straight to 4 plus extensions for profiles -- or if you'd rather postpone figuring out the use deps in profiles issues, then give them 1 plus extensions. If those extensions turn out to be useful, then they can be merged into 5. If not, they can be quietly dropped. As for "FUD", the main FUD I see here is the usual "EAPIs are expensive" thing.
(In reply to comment #17) > "Massive amount of pain" is that you're introducing a whole new mechanism that > sort of behaves like EAPI, sort of doesn't and that has to interact with it in > some unspecified way. Mechanisms are extremely expensive, and we already have > one. It's not perfect, but it can be improved if it has to be. Default EAPIs, > for example, if you really think you need them (which you don't, since GLEP 55 > fixes that), are best done within the EAPI mechanism, not by introducing a > whole new thing. Crack is bad mm'kay; glep55 doesn't fix profile default eapi, it has zero relevance to it, it only applies (badly, but seperate discussion) to ebuilds, /not/ profiles. As for expensive new mechanism, that's FUD; the code for this detection/support was <40 lines at best, same for issuing warnings if the profile format isn't known/supported. Frankly it's more expensive long term to try and shoehorn everything into EAPI while locked to unchangable profile structure. > There's no breakage problem -- people simply need to ensure that a sufficiently > new package manager is being used. Breaking existing users who haven't yet upgraded. Wonder definition of compatibility you've got there. > That's an issue when profiles are updated to > use new things anyway, and all we're doing here is finally correcting a > long-standing screwup. It has to be dealt with at some point, and delaying the > inevitable only makes things worse. > > You don't need a whole bunch of new EAPIs either, since we can just go straight > to 4 plus extensions for profiles -- or if you'd rather postpone figuring out > the use deps in profiles issues, then give them 1 plus extensions. If those > extensions turn out to be useful, then they can be merged into 5. If not, they > can be quietly dropped. > > As for "FUD", the main FUD I see here is the usual "EAPIs are expensive" thing. EAPI's are expensive to create, document, and deploy; year turn around isn't unheard of. You've got a very selective definition. I'll note I asked for a logical breakdown (scenarios) of your arguments, where the current fails. Your resorting to vague handwavey shit, and bluntly, attempting to just scream down the support. PMS doesn't actually have a say in this (these profiles are already outside PMS view), so rather than getting this documented properly, I'm going to be pragmatic. Paludis can support it if it chooses; pkgcore already does, portage inherintly supports this solution already (and I'll be supplying a patch to make it fine grained in the next few days). If PMS becomes layout.conf aware, than it can deal with it. Till than it has zero say in the matter, so I'm removing the noise. If you wish to provide actual arguments rather than vague handwavey shit, go nuts, but I'm not going to play the trolling game with you; discuss it technically, or go away.
I've added profile-format support to our repos. I'm doing this for pragmatic reasons, not because I really care about this particular debate.
(In reply to comment #12) > Re-read the 'slow' part; the problem w/ EAPI for things other than ebuild > format itself is that it's a single axis of versioning, and that any changes to > the profiles require sitting on them a fairly long while prior to deployment. Maybe I'm being dumb, but I still don't see how profile-format is better than EAPI in this regard. > One simple example of what cannot be handled by EAPI is default EAPI for the > profile tree; literally shifting 0 to 2 as an example. That's not possible to > roll out w/out a new format realistically. What about changing the default EAPI for ebuilds? Is that considered less of a problem than the default EAPI for profiles? > In the interim, the PMS strict package managers remain broke. Further, the > roll out phase requires getting that EAPI approved, deployed to stable for > portage, than you start counting months till enough folk have upgraded that > flipping the bit is viable. Again, that's no worse than profile-format, except in this one instance. It doesn't seem sensible to permanently add a new mechanism that's an improvement over what we already have in exactly one case and is worse in others. > > I'm not sure what Funtoo > > etc's policies are, but I wouldn't have thought they had such strict > > compatibility requirements as the main Gentoo tree, so they wouldn't have to > > wait too long. > > Bit more than just funtoo; there are normal gentoo overlays out there w/ it in > addition. Hence "etc". It seems fairly common for overlays to require more recent Portage versions than the main tree. > One thing to note- the intention that all separate profile formats live at a > different location (with the exception of portage-1 to address reality) means > it would be possible for a repository to carry >1 profile formats; specifically > for compatibility. The same can be done via some nastyness in PMS profile > structuring; pointing out that there still is a compatibility option here. How would you write that in layout.conf? If a repository contains both PMS and Super-Whizbang-3000 profiles, then profile-format could be 1) "pms" to allow existing PMs to use the PMS profiles, in which case how does a new PM know to allow the new features in the new profiles? Reverse-engineer the profile path and see that it's under <repository>/profiles-w00t/ rather than <repository>/profiles/ ? 2) "super-whizbang-3000", in which case existing PMs will reject it outright. 3) "pms super-whizbang-3000", in which case your current code in pkgcore will still reject it, and even if it's updated to allow multiple formats and accept the repo if it supports at least one, you still have the issue in 1) where you need to figure out which profile uses which format. > Additional note; there are certain cleanups we cannot deploy for example. PMS > rules are that profiles cannot assume that they can override a repository level > package.mask for example (yet the lion share of users have exactly that in > use); realistically trying to resolve that via EAPI isn't sane- the solution > would require stating "if EAPI5 is seen in the profile stack, than allow > overriding of package.mask, else there is no gurantee" which makes things even > more of a mess. No, it would be "if the specific profile that's trying to override the repository package.mask is EAPI 5 then allow it".
(In reply to comment #18) > As for expensive new mechanism, that's FUD; the code for this detection/support > was <40 lines at best, same for issuing warnings if the profile format isn't > known/supported. Frankly it's more expensive long term to try and shoehorn > everything into EAPI while locked to unchangable profile structure. It looks like your current implementation is just a simple boolean "is it PMS or not?" Expanding it to support future features would require keeping track of which features are supported by which formats, possibly similar to what paludis already does for EAPIs.
(In reply to comment #20) > (In reply to comment #12) > > Re-read the 'slow' part; the problem w/ EAPI for things other than ebuild > > format itself is that it's a single axis of versioning, and that any changes to > > the profiles require sitting on them a fairly long while prior to deployment. > > Maybe I'm being dumb, but I still don't see how profile-format is better than > EAPI in this regard. > > > One simple example of what cannot be handled by EAPI is default EAPI for the > > profile tree; literally shifting 0 to 2 as an example. That's not possible to > > roll out w/out a new format realistically. > > What about changing the default EAPI for ebuilds? Is that considered less of a > problem than the default EAPI for profiles? Seperation of concerns here; default EAPI of ebuilds themselves is seperate from default EAPI of profile nodes; the two aren't really related, nor should they be. Looping back to my point, think through the current profile interpretation rules/implementations; there isn't a way to deploy default EAPI for a repositories profile nodes w/out either year+ compatibility sitting on the functionality and hoping people upgrade (which is what things were like pre-eapi), or marking it as a new format. > > In the interim, the PMS strict package managers remain broke. Further, the > > roll out phase requires getting that EAPI approved, deployed to stable for > > portage, than you start counting months till enough folk have upgraded that > > flipping the bit is viable. > > Again, that's no worse than profile-format, except in this one instance. It > doesn't seem sensible to permanently add a new mechanism that's an improvement > over what we already have in exactly one case and is worse in others. "worse in others" is inaccurate; profile-format is basically one extra configurable at the top level, w/ each format being allowed to have it's own rules; portage-1 rules are the same as PMS, sans the portage incompatibilities introduced (package.* and use.* as dirs primarily). Any "worse in others" applies to existing EAPI approach, so... as said, innacurate. > > > I'm not sure what Funtoo > > > etc's policies are, but I wouldn't have thought they had such strict > > > compatibility requirements as the main Gentoo tree, so they wouldn't have to > > > wait too long. > > > > Bit more than just funtoo; there are normal gentoo overlays out there w/ it in > > addition. > > Hence "etc". It seems fairly common for overlays to require more recent > Portage versions than the main tree. People really, really need to stop demanding EAPI than turn around and state "the solution is to require people to be running the most recent portage versions"; the latter statement directly contradicts the point of compatibility. > > One thing to note- the intention that all separate profile formats live at a > > different location (with the exception of portage-1 to address reality) means > > it would be possible for a repository to carry >1 profile formats; specifically > > for compatibility. The same can be done via some nastyness in PMS profile > > structuring; pointing out that there still is a compatibility option here. > > How would you write that in layout.conf? If a repository contains both PMS and > Super-Whizbang-3000 profiles, then profile-format could be > 1) "pms" to allow existing PMs to use the PMS profiles, in which case how does > a new PM know to allow the new features in the new profiles? Reverse-engineer > the profile path and see that it's under <repository>/profiles-w00t/ rather > than <repository>/profiles/ ? depends on the manager. For paludis configuration, the repository format is stated, including the profile path- knowing what the profile format is easy, you've got all of the necessary data provided right there to read layout.conf and swivel as needed. Pkgcore configuration allows the same. For make.profile symlinks, yep, inspection of the path. Perfect, nope, but neither is old style portage configuration. Addressing properly via the new repos.conf/etc portage configuration is potential- something that can be sorted as we go (we'd have this issue regardless of profile-format or not mind you). > 2) "super-whizbang-3000", in which case existing PMs will reject it outright. > 3) "pms super-whizbang-3000", in which case your current code in pkgcore will > still reject it, and even if it's updated to allow multiple formats and accept > the repo if it supports at least one, you still have the issue in 1) where you > need to figure out which profile uses which format. Figuring out which profile as said is a backtrack and loadup of the repo config. > > Additional note; there are certain cleanups we cannot deploy for example. PMS > > rules are that profiles cannot assume that they can override a repository level > > package.mask for example (yet the lion share of users have exactly that in > > use); realistically trying to resolve that via EAPI isn't sane- the solution > > would require stating "if EAPI5 is seen in the profile stack, than allow > > overriding of package.mask, else there is no gurantee" which makes things even > > more of a mess. > > No, it would be "if the specific profile that's trying to override the > repository package.mask is EAPI 5 then allow it". Seriously... that's a mess for people to actually interpret (which means they'll screw up); I'll note you've ignored default profile EAPI in addition. (In reply to comment #21) > (In reply to comment #18) > > As for expensive new mechanism, that's FUD; the code for this detection/support > > was <40 lines at best, same for issuing warnings if the profile format isn't > > known/supported. Frankly it's more expensive long term to try and shoehorn > > everything into EAPI while locked to unchangable profile structure. > > It looks like your current implementation is just a simple boolean "is it PMS > or not?" Expanding it to support future features would require keeping track > of which features are supported by which formats, possibly similar to what > paludis already does for EAPIs. Bit of an inaccurate statement; currently there are two formats, portage-1, and pms. Frankly the only real difference between the two is whether or not package.*/use.* is directory or file. As for "feature variations as paludis does for EAPI", same applies for as pkgcore does, and as portage does; what you're missing is that there isn't a /point/ to yet implementing that. When it's necessary, the bits will grow, till then your suggestion is overly complex without gain.
(In reply to comment #22) > Seperation of concerns here; default EAPI of ebuilds themselves is seperate > from default EAPI of profile nodes; the two aren't really related, nor should > they be. The point being that the way EAPI works for ebuilds doesn't allow changing the default away from 0 and people don't seem to be complaining, so why is it so important to allow that for profiles? > "worse in others" is inaccurate; profile-format is basically one extra > configurable at the top level, w/ each format being allowed to have it's own > rules; portage-1 rules are the same as PMS, sans the portage incompatibilities > introduced (package.* and use.* as dirs primarily). Any "worse in others" > applies to existing EAPI approach, so... as said, innacurate. "Worse in others" applies to mixing profile formats within a repository. > People really, really need to stop demanding EAPI than turn around and state > "the solution is to require people to be running the most recent portage > versions"; the latter statement directly contradicts the point of > compatibility. Just as a one-off in this case, to avoid introducing the redundant mechanism - define a funtoo-4 or whatever EAPI, wait for people to upgrade to a portage version that supports it, then start using it in the repositories and enforcing the rules more strictly for existing EAPIs. This only applies to the package.mask directories because portage currently allows them but will reject any funtoo-4 profile, so doing it this way minimises breakage. For every other new feature, portage will support the feature at the same time as it starts recognising the new EAPI, so there's no way of breaking existing working setups. > depends on the manager. For paludis configuration, the repository format is > stated, including the profile path- knowing what the profile format is easy, > you've got all of the necessary data provided right there to read layout.conf > and swivel as needed. Pkgcore configuration allows the same. The point is that layout.conf applies to the whole repository. There's no way to say "the 10.0 profiles in this repository are in PMS format, but the 20.0 profiles in the same repository are in ShinyNew format". The configuration format doesn't affect this unless you want to force users to configure manual overrides. > For make.profile symlinks, yep, inspection of the path. Perfect, nope, but > neither is old style portage configuration. Addressing properly via the new > repos.conf/etc portage configuration is potential- something that can be sorted > as we go (we'd have this issue regardless of profile-format or not mind you). So if the PM can't use profile-format to figure out the profile format, what's the point? > Seriously... that's a mess for people to actually interpret (which means > they'll screw up); I'll note you've ignored default profile EAPI in addition. How is it worse than "if the profile trying to override the repository package.mask is in ShinyNew format then allow it"? And default profile EAPI just controls which EAPI the PM assumes for profile nodes that don't specify it explicitly - after applying the default, all further processing works as if all the nodes do specify it. > Bit of an inaccurate statement; currently there are two formats, portage-1, and > pms. Frankly the only real difference between the two is whether or not > package.*/use.* is directory or file. As for "feature variations as paludis > does for EAPI", same applies for as pkgcore does, and as portage does; what > you're missing is that there isn't a /point/ to yet implementing that. > > When it's necessary, the bits will grow, till then your suggestion is overly > complex without gain. So it'll become complex in the future (and it will, since you're already talking about what sort of additional features could be introduced with this mechanism). How is that better than it being complex now, other than sweeping the complexity under the rug in the hope that the decision-makers won't notice until it's too late?
Round and round we go... (In reply to comment #23) > (In reply to comment #22) > > Seperation of concerns here; default EAPI of ebuilds themselves is seperate > > from default EAPI of profile nodes; the two aren't really related, nor should > > they be. > > The point being that the way EAPI works for ebuilds doesn't allow changing the > default away from 0 and people don't seem to be complaining, so why is it so > important to allow that for profiles? To phrase your argument accurately, "because there is no repository level default EAPI, why should we allow it for profiles." Aka, because the repository format issue isn't sorted, than we shouldn't allow profile EAPI defaults. They're seperate issues, solving one isn't blocked by solving the other. > > "worse in others" is inaccurate; profile-format is basically one extra > > configurable at the top level, w/ each format being allowed to have it's own > > rules; portage-1 rules are the same as PMS, sans the portage incompatibilities > > introduced (package.* and use.* as dirs primarily). Any "worse in others" > > applies to existing EAPI approach, so... as said, innacurate. > > "Worse in others" applies to mixing profile formats within a repository. Sigh. Said nothing about intermixing the formats; pms/portage-1 being a special case due to reality. > > People really, really need to stop demanding EAPI than turn around and state > > "the solution is to require people to be running the most recent portage > > versions"; the latter statement directly contradicts the point of > > compatibility. > > Just as a one-off in this case, to avoid introducing the redundant mechanism - > define a funtoo-4 or whatever EAPI, wait for people to upgrade to a portage > version that supports it, then start using it in the repositories and enforcing > the rules more strictly for existing EAPIs. This only applies to the > package.mask directories because portage currently allows them but will reject > any funtoo-4 profile, so doing it this way minimises breakage. For every other > new feature, portage will support the feature at the same time as it starts > recognising the new EAPI, so there's no way of breaking existing working > setups. Your one off is to keep breakage going till everyone has upgraded to the requisite portage version. Already pointed out the flaws with this, and pointed out profile-format doesn't carry that requirement (while opening up potentials down the line). Your proposal is basically the same as what I already addressed and discounted- I'll note however it's actually still broken (it's not just package.mask; it's package.* and use.* that can be directories). > > depends on the manager. For paludis configuration, the repository format is > > stated, including the profile path- knowing what the profile format is easy, > > you've got all of the necessary data provided right there to read layout.conf > > and swivel as needed. Pkgcore configuration allows the same. > > The point is that layout.conf applies to the whole repository. There's no way > to say "the 10.0 profiles in this repository are in PMS format, but the 20.0 > profiles in the same repository are in ShinyNew format". The configuration > format doesn't affect this unless you want to force users to configure manual > overrides. Please re-read the points outlined above, this is actually addressed already. Reiterating; w/ the sole exception of portage-1 vs pms, any new format has to live at a different offset; meaning profiles2 for all I care, point is, different on disk location. For non-make.profile user configuration, that offset combined w/ list of known profile-format for a repository is enough to detect when an unsupported profile is in use and complain appropriately. For make.profile only configuration, spotting unknown formats in profile-format (via repo configs being loaded up) can bitch appropriately. The logic behind that isn't particularly hard; if the pathway doesn't go into a repository, than it's the managers default user configuration interpretation (which is how it's been thus far for portage anyways). At some point, realistically make.profile should be replaced; regardless of this proposal, that's the truth; the sym approach isn't particularly grand, doing ${REPO_NAME}/profiles/blah or equivalent makes a bit more sense; realistically, single profile for all repo's used may not be wise anyways (paludis configuration allows per repo for example). Refactoring portage's make.profile user configuration basically is orthogonal to this; it's an issue that needs enhancement regardless of this proposal. > > For make.profile symlinks, yep, inspection of the path. Perfect, nope, but > > neither is old style portage configuration. Addressing properly via the new > > repos.conf/etc portage configuration is potential- something that can be sorted > > as we go (we'd have this issue regardless of profile-format or not mind you). > > So if the PM can't use profile-format to figure out the profile format, what's > the point? This is tiring; the PM can. Pathway intersects a repository, you know it's using that profiles bundled profiles; pkgcore may not implement that extra check yet, but that's for lazyness reasons. Manager realizes the pathway is into a repo, loads the repo configuration (layout.conf), recognizes that there is an unsupported profile-format in use (which because the formats are explicitly stated for that repo, means we know that pms/portage-1 profiles are guranteed at /profiles/, and have the same awareness of offset for any other support formats), bitches appropriately. Think it through; it will work. Optimal? No, but that's reality of dealing w/ make.profile; the mechanism is only needed for that user config, isn't needed for paludis/pkgcore configuration. > > Seriously... that's a mess for people to actually interpret (which means > > they'll screw up); I'll note you've ignored default profile EAPI in addition. > > How is it worse than "if the profile trying to override the repository > package.mask is in ShinyNew format then allow it"? And default profile EAPI > just controls which EAPI the PM assumes for profile nodes that don't specify it > explicitly - after applying the default, all further processing works as if all > the nodes do specify it. It's worse because the overall profile stacking behaviour can vary in ways people don't expect, versus w/ format, it's encoded as a rule of the format from the getgo- no variance. > > Bit of an inaccurate statement; currently there are two formats, portage-1, and > > pms. Frankly the only real difference between the two is whether or not > > package.*/use.* is directory or file. As for "feature variations as paludis > > does for EAPI", same applies for as pkgcore does, and as portage does; what > > you're missing is that there isn't a /point/ to yet implementing that. > > > > When it's necessary, the bits will grow, till then your suggestion is overly > > complex without gain. > > So it'll become complex in the future (and it will, since you're already > talking about what sort of additional features could be introduced with this > mechanism). How is that better than it being complex now, other than sweeping > the complexity under the rug in the hope that the decision-makers won't notice > until it's too late? This is a dumb criticism; it's not even cricitising profile-format, it's arguing that I should go mangle pkgcore implementation of profiles adding a half baked configuration object for what boils down to a single boolean swivel in format support. You want to do crazy/architectural-astronaut shit like that up front, do it in your own pkg manager. I'll manage my code how I please (and will expand that internal functionality when I need to, not prior). Stick to the proposal rather than trying to snipe at the source please, bit more productive.
(In reply to comment #24) > > "Worse in others" applies to mixing profile formats within a repository. > > Sigh. Said nothing about intermixing the formats; pms/portage-1 being a > special case due to reality. You didn't, but I did. If we're going to be developing new profile formats, surely we're going to want to have both old and new together, even if just for a transition period? > I'll note however it's actually still broken (it's not just package.mask; it's > package.* and use.* that can be directories). Nitpicking. > This is tiring; the PM can. Pathway intersects a repository, you know it's > using that profiles bundled profiles; pkgcore may not implement that extra > check yet, but that's for lazyness reasons. *Ahem*: (In reply to comment #18) > As for expensive new mechanism, that's FUD; the code for this detection/support > was <40 lines at best, same for issuing warnings if the profile format isn't > known/supported. You can't cite your implementation to prove that adding profile-format is a small change and then turn around and say there are all sorts of extra bits that you didn't bother to implement yet. > Think it through; it will work. Optimal? No, but that's reality of dealing w/ > make.profile; the mechanism is only needed for that user config, isn't needed > for paludis/pkgcore configuration. I'm not hugely familiar with the details of pkgcore's native configuration, but for paludis, the profile configuration is just a list of paths, which might not even be in the same repository that's being configured (overlays will almost always be using a profile from the main tree). So, to use layout.conf, it would have to check the profile path against the locations of /all/ configured repositories, and even then it'll only get a list of formats that it /might/ be. It would still need to check whether it's in <repo>/profiles/, <repo>/profiles2/, etc to know which format it actually is. That doesn't seem any better than what you'd have to do for make.profile. > This is a dumb criticism; it's not even cricitising profile-format, it's > arguing that I should go mangle pkgcore implementation of profiles adding a > half baked configuration object for what boils down to a single boolean swivel > in format support. > > You want to do crazy/architectural-astronaut shit like that up front, do it in > your own pkg manager. I'll manage my code how I please (and will expand that > internal functionality when I need to, not prior). Again, you brought up your implementation to claim that this is a simple change, it's entirely appropriate to evaluate how well it demonstrates what's needed to fully support this.
(In reply to comment #25) > (In reply to comment #24) > > > "Worse in others" applies to mixing profile formats within a repository. > > Sigh. Said nothing about intermixing the formats; pms/portage-1 being a > > special case due to reality. > > You didn't, but I did. If we're going to be developing new profile formats, > surely we're going to want to have both old and new together, even if just for > a transition period? If the new format wants to support inheriting from the old, they get to solve it. Being able to deploy both in parallel (as standalone) is a good enough basis. What you're doing here is trying to argue against profile-format via requiring all later formats to be able to inherit into the existing format. The point of this mechanism is to allow a clean break away from existing if we want. If you want any new format to be able to inherit old, fine, *design it that way*, I'm not going to require it up front (I will not constrain future designs since I do not know what they shall design). > > I'll note however it's actually still broken (it's not just package.mask; it's > > package.* and use.* that can be directories). > > Nitpicking. > > > This is tiring; the PM can. Pathway intersects a repository, you know it's > > using that profiles bundled profiles; pkgcore may not implement that extra > > check yet, but that's for lazyness reasons. > > *Ahem*: > > (In reply to comment #18) > > As for expensive new mechanism, that's FUD; the code for this detection/support > > was <40 lines at best, same for issuing warnings if the profile format isn't > > known/supported. > > You can't cite your implementation to prove that adding profile-format is a > small change and then turn around and say there are all sorts of extra bits > that you didn't bother to implement yet. Nitpicking bullshit. The only check that is lacking in pkgcore is if it intersects into a pathway; doing that in portage is simple, doing it in pkgcore is harder, but doable- it's just dependant on some cleanup I'm being lazy about. For portage and easily for paludis, it knows the repo's at the time of profile access, so it's not an issue. > > Think it through; it will work. Optimal? No, but that's reality of dealing w/ > > make.profile; the mechanism is only needed for that user config, isn't needed > > for paludis/pkgcore configuration. > > I'm not hugely familiar with the details of pkgcore's native configuration, but > for paludis, the profile configuration is just a list of paths, which might not > even be in the same repository that's being configured (overlays will almost > always be using a profile from the main tree). Specifically, paludis assumes that all profile pathways are PMS. I'm not seeing how that is pkgcore/portage/the-fucking-format's problem to solve. The nature of this change, it has to be phased in; I've laid out how that can occur, and mechanisms to actually do this. This can be done w/out breakage, and the approach solves existing breakage. I've laid out how make.profile can resolve it, meaning covering the actual users of the tree. Algo's not optimal, but that's a byproduct of the format (and orthogonally solved, since the user format needs work anyways). Your complaints pretty much are that it doesn't work out of the box for paludis, or that you don't like the algo, then point at EAPI as the solution (while ignoring the faults raised). Prety simple; this will work for portage, it can work for paludis if they deem to do it. If y'all don't want to support it, good for you. Funtoo's carrying it, and the few remainders I'll get converted; the portage patch will land shortly. Meaning at least for pkgcore 0.7.3, this is a solved issue. Being that this is outside of PMS, your criticisms are noted, disagreed with, and I'm moving forward. Feel free to push your own competing proposal, but this discussion ceased being productive for me, thus other things to do.
Checks/support are in git under profiles-formats: http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=e544ee13527a0568de2b556510280ed96fc1b3ff
This is fixed in 2.1.10.32 and 2.2.0_alpha72.
