Portage currently silently accepts package.mask being a directory in a repository, although PMS requires package.mask to be a file. Portage should instead warn or error when this happens to prevent people from accidentally abusing this.
I guess we can add something like FEATURES=pms-warnings and let people disable it if they want.
Just do it unconditionally. We're talking the tree here, not user configuration files, so enforcing QA can only be a good thing.
Disallowing it completely would hinder the use of portage in other distributions, and we shouldn't make their life harder just because we can.
I strongly disagree with disabling it. Althrought it is not able to be used in main tree, it can be used in overlays and other gentoo-derivated distributions. When/if the conversion to directory approach prove to be really benefitical, like it already is in kde overlay (we use it currently via script that merge it because we need paludis support) it can be added to pms and voted on council to became main-tree stuff.
Funtoo will continue to support this functionality regardless. This bug was created because I discussed this in -council, and the wrong people were there to hear it. My goal was to continue to support Paludis in Funtoo. The end result was that Ciaran has submitted this bug with the intent of forcing an incompatibility between Funtoo and Gentoo.
My goal is to ensure that PMS is followed, so that Gentoo and any other distribution that cares to can have a good solid specification-backed package format that can be used by any package manager or tool that cares to support it.
http://trac.pioto.org/paludis/ticket/761 is the associated paludis bug. Note that this is a rejected support request to follow this functionality. (from http://trac.pioto.org/paludis/ticket/761) The Package Manager Specification (PMS) aims to solve this by defining, independent of any package manager, what is and is not allowed in the tree, and what ebuilds may assume about their environment. I have taken the liberty of checking up on Ciaran's claims with regard to the council. I herein cite (http://www.gentoo.org/proj/en/council/meeting-logs/20090423-summary.txt) - Portage changing behaviour without EAPI bumps: David Leverton(dleverton) requested that the council mandate that portage is not allowed to change behaviour that is specified in PMS, as has occurred a few times in the past. Conclusion: The council decided that if PMS-conflicting changes occur in package managers then the council can mandate that versions that conflict will be masked. The council may take into account extenuating circumstances. This bug is attempting to force action reserved to the council, by the council, and should be closed. The council has not mandated that ALL versions conflicting with the PMS will be masked, but that they can, at the council's SOLE discretion, be masked.
I'm not asking for Portage to be masked or anything silly like that. I'm asking Portage to start providing errors for things that go against PMS and the documented EAPIs that Portage supports so that people can't screw things up accidentally. If you'd like to do something outside of Gentoo, you could try defining your own EAPI and asking for Portage support for it.
since portage is THE package manager of gentoo - why not just make portage behaviour the canonical one? it is not like other package managers have to be considered. So just change the pms to cover portage's behaviour and all is fine and dandy.
(In reply to comment #9) > since portage is THE package manager of gentoo - why not just make portage > behaviour the canonical one? > > it is not like other package managers have to be considered. > > So just change the pms to cover portage's behaviour and all is fine and dandy. > I agree, Ciaran, how quickly can the PMS be changed to reflect portage's ACTUAL behavior. Zac, can you help him with this?
The relevant code is the grablines function here: http://sources.gentoo.org/viewcvs.py/portage/main/trunk/pym/portage/util.py?view=markup It calls itself recursively on the sorted list of file names in a given directory.
(In reply to comment #10) > I agree, Ciaran, how quickly can the PMS be changed to reflect portage's ACTUAL > behavior. Zac, can you help him with this? PMS reflects Portage's documented behaviour and the behaviour described in the commit that introduced the feature. It does not describe an accidental, undocumented fluke caused by Portage not validating its input.
(In reply to comment #12) > (In reply to comment #10) > > I agree, Ciaran, how quickly can the PMS be changed to reflect portage's ACTUAL > > behavior. Zac, can you help him with this? > > PMS reflects Portage's documented behaviour and the behaviour described in the > commit that introduced the feature. It does not describe an accidental, > undocumented fluke caused by Portage not validating its input. Right, but I'm wondering how quickly it can be changed to document this behavior as it's being used in the wild. Rather than throwing warnings, lets simply extend the PMS.
The correct procedure for extending PMS is to EAPI control the change. The feature is already on the list of things to consider for EAPI 4, work on which will be started once EAPI 3's out of the door.
Ciaran, Again, I'm not Gentoo. I'm having significant trouble summoning the imagination which I would require to consture your attempt to use Gentoo bureaucracy to throw needless warnings at Funtoo users (and others as being anything other than petty. Your decision to do it on b.g.o is deeply concerning, especially as it is not a bug per GLEP. It's up to the council to mask packages or package managers which break PMS/EAPI, and I'm betting you knew that when you filed this Ciaran. The goal here should be to find a way to make this work, not throw warnings at users which will confuse them. You asked my several times what my thoughts are on EAPI-2 and EAPI-3, my answer is that I simply don't care, and this is why. Writing and enforcing standards takes time away from the council, from the package manager author and from third parties using that software who are attempting to innovate. "Portage should enforce PMS compliance..." Portage is a Package Manager not a QA tool (I think we can both agree that Portage isn't a QA tool). Repoman should probably enforce this on Gentoo's ports tree. Portage should be extensible, and should be expandable other than by council decree. I think this bug needs to be closed. (though I thought that 2 months ago) 1. Portage isn't a QA tool 2. the Paludis dev shouldn't be dictating policy to the portage devs (and vice-versa) 3. this functionality isn't in gentoo it's in funtoo and not germane to b.g.o 4. adult supervision is now required here.
Then why don't you propose a new EAPI with the extensions you require for Funtoo and get Portage to use that? There's no need to go through Gentoo at all that way, as you can deal with Zac directly. That way Gentoo's QA issues get fixed, which is what this is all about, whilst Funtoo gets to do its own thing. Let me repeat: I don't care what Funtoo does. I just don't want Funtoo's bad habits rubbing off on Gentoo.
(In reply to comment #16) > Then why don't you propose a new EAPI with the extensions you require for > Funtoo and get Portage to use that? There's no need to go through Gentoo at all > that way, as you can deal with Zac directly. That way Gentoo's QA issues get > fixed, which is what this is all about, whilst Funtoo gets to do its own thing. > > Let me repeat: I don't care what Funtoo does. I just don't want Funtoo's bad > habits rubbing off on Gentoo. > A few things, there was quite a discussion in userrel related to this. Ultimately a few things came up. 1) Please provide technical reasoning behind why having a package.* directory is a bug and not a feature. Please do not link to PMS and say its in there. This is not part of this question. 2) This behavior was existing prior to PMS being ratified 3) (Personal) A folder based system has potential for expanding the capabilities of the package.* functionality in new and interesting ways that might not exist in the current file "only" standard being pushed for. 4) As most people are here that would be involved in this: a) Jorge brought up that perhaps EAPI's are not the proper place for this kind of change as it doesn't directly affect how an ebuild is written. As such it was suggested that there's another mechanism outside of EAPI's that might be of value for such changes. Will you please discuss this possibility of such a system and let jmbsvicetto/myself/council know of the validity of such. As there will be more situations that don't directly affect the Ebuilds themselves. 5) A standard is only as good as its ability to be flexible and grow as the need arises. If people are rigid and inflexible to possibilities that don't exist now, then you will stagnate and cease to grow. PMS grew from the fact that new package managers started to appear, but what happens if PMS ends up hindering what it initially wanted to ensure (compatibility while allowing growth). Please work with each other to find a resolution that makes everyone as happy as possible, while sticking to technical reasoning. Please exclude the PMS standard definition and what it says currently as that ultimately is one of the issues that's causing this fight.
Additionally, on the 4a consideration. I expect and will be submitting to council an addition of the agenda to discuss this non-eapi based changes change on everyones behalf. I fully expect that by the 10th of september (3 weeks from now) that you will be able to schedule a meeting that will work for all parties and discuss this possibility and come to a satisfactory decision that everyone will agree to.
Submitted AOB to council for inclusion on the Sept 10th meeting...
(In reply to comment #17) > 1) Please provide technical reasoning behind why having a package.* directory > is a bug and not a feature. http://sources.gentoo.org/viewcvs.py/portage/main/trunk/man/portage.5?r1=4310&r2=4311 Explicitly documented as being only for /etc/portage/. There is no mention of this feature being permitted in /usr/portage/, which is also documented in that man page. Note also that Portage uses the same code for handling /etc/portage/ and /usr/portage/, which is why it works. This also means that other QA violations (e.g. parsing for dep specs for users is laxer than that required for ebuilds) will often pass through unnoticed. > 2) This behavior was existing prior to PMS being ratified That doesn't mean it's behaviour upon which people can rely. One of the important principles behind PMS that was agreed upon from the start is that there is Portage behaviour that cannot be relied upon -- this means the behaviour can change in subsequent Portage releases. It was undocumented unintentional behaviour before PMS was ratified, and it remains undocumented unintentional behaviour. This bug is about making it clear that such behaviour is undocumented and unintentional; turning it into intentional, documented behaviour (and possibly doing it in a different manner -- some developers have said they'd rather use package.mask.d than package.mask as a directory) is an issue for a future EAPI. As an example of why relying upon undocumented behaviour is bad: people who wrote their websites based upon what Internet Explorer 4 supported rather than what the standard said were in for a nasty shock, both with Internet Explorer 5 and other browsers. > 3) (Personal) A folder based system has potential for expanding the > capabilities of the package.* functionality in new and interesting ways that > might not exist in the current file "only" standard being pushed for. Yes, which is why it's an interesting feature to be discussed and introduced (possibly slightly differently than how Portage does it currently) in an EAPI controlled manner. > 4) As most people are here that would be involved in this: > a) Jorge brought up that perhaps EAPI's are not the proper place for this kind > of change as it doesn't directly affect how an ebuild is written. As such it > was suggested that there's another mechanism outside of EAPI's that might be of > value for such changes. Will you please discuss this possibility of such a > system and let jmbsvicetto/myself/council know of the validity of such. As > there will be more situations that don't directly affect the Ebuilds > themselves. The Council voted a while ago to introduce EAPI controls to profiles too, and Portage supports this. Profile contents are under EAPI control in the same way that ebuilds are. > 5) A standard is only as good as its ability to be flexible and grow as the > need arises. If people are rigid and inflexible to possibilities that don't > exist now, then you will stagnate and cease to grow. PMS grew from the fact > that new package managers started to appear, but what happens if PMS ends up > hindering what it initially wanted to ensure (compatibility while allowing > growth). Please work with each other to find a resolution that makes everyone > as happy as possible, while sticking to technical reasoning. Please exclude the > PMS standard definition and what it says currently as that ultimately is one of > the issues that's causing this fight. The way to grow is to put the feature in the next EAPI. I'm entirely in favour of seeing a discussion on this for EAPI 4, and I would be entirely happy for the Funtoo people to make their own EAPI funtoo-2 to allow this to happen. All the Funtoo people need to do to get this feature is: * Put together a very short document (probably five sentences) describing EAPI funtoo-2. Get this document approved by the Funtoo powers that be. * Ask Zac nicely to add in Portage support for it. Again, this one's trivial, since the code's already there. * Upgrade their users to a new Portage release. * Put a file under profiles/ named eapi containing just "funtoo-2". Unfortunately, Andrew's ruled out this based upon his earlier misconception that Portage doesn't use EAPIs and that EAPIs and PMS are purely a Paludis thing. That, combined with his stated aim to make Paludis and PMS fail in any way he can, means he won't even consider doing the above. For Gentoo, the process is similar, although the migration path is slightly trickier, since the top level profile has to remain EAPI 0. This is another reason some developers were favouring a package.mask.d solution plus minor infra voodoo for EAPI 4. In the mean time, since developers are mistakenly thinking that they're using documented, deliberate behaviour when making those files directories under EAPI 0 controlled profiles, Portage should be issuing warnings to correct those mistakes. (In reply to comment #18) > Additionally, on the 4a consideration. I expect and will be submitting to > council an addition of the agenda to discuss this non-eapi based changes change > on everyones behalf. The Council has already put profiles under EAPI control, and the mechanism for doing so is described in PMS. I suggest you retract your request until you've read up on that decision and its implications.
Portage documentation has been fixed in r14104.
That's not a fix and you know it. That isn't the proper mechanism for introducing changes. Please revert your change immediately and do this properly.
Please don't reopen this bug.
Why are you so strongly opposed to doing this properly? It's entirely clear from the initial commit what the purpose of the feature was, it's entirely clear that it's not behaviour that should have been in PMS from the get-go, and it's entirely clear that there is support for this feature to go in EAPI 4. What do you stand to gain from helping Andrew in his quest to make projects fail?
It's kind of bothering to see somebody who isn't even a member of the portage team do a commit related to such a topic which apparently doesn't make consensus at all. Zac, is there any way you can revert this until the council gets an opportunity to discuss it like Joshua requested it? Unless you guys reach an agreement before which would be the ideal solution. We can't really afford a conflict between portage and PMS on this or any other topic. By the way Joshua, the next council meeting will probably be on sept 21st not 10th. In the meantime I would advise everybody to take their discussions and conspiracy theories somewhere else. Denis.
(In reply to comment #25) > It's kind of bothering to see somebody who isn't even a member of the portage > team do a commit related to such a topic which apparently doesn't make > consensus at all. > > Zac, is there any way you can revert this until the council gets an opportunity > to discuss it like Joshua requested it? Unless you guys reach an agreement > before which would be the ideal solution. We can't really afford a conflict > between portage and PMS on this or any other topic. I think we can consider Arfrever a portage team member. Anyway, he asked me before he committed it and I approved because it documents the current behavior and I see that as an improvement. Therefore, I see not reason to revert it. We can always change things later as required by the community consensus.
As this is clearly a good feature to have, we should simply update PMS to document this behaviour.
Why does a person who had their Gentoo developer status revoked have commit access to the single document that allegedly defines _all_ Gentoo-compatible package managers? Having been using the directories "bug" in production for quite some time now, I think it's insane to have features removed and/or warnings that I cannot turn off introduced because the aforementioned non-dev can't figure out how to implement portage behavior in their own pet project that has next to no relevance for mainstream users and users of other meta-distributions based on Gentoo. The method we use to manage, via stacked profiles, any number of machine classes with similar or dissimilar hardware and feature sets is utilizing that "bug." Currently we're accomplishing class-based hardware and logical workload definitions and individual host configuration by utilizing package.* directories in stacked profiles. The result is a centralized, common and highly configurable system for managing what is typically kept under /etc/portage with significantly less administrative overhead and the additional benefit of being trivially kept under vcs. From my own production environment, an example would be our /etc/managed-portage/class/web/make.profile/package.mask/php-5.10 which logically describes and separates what is being masked. This mask applies to 14 machines spanning 3 different cpu types, and is only an example of what this particular feature of portage is capable of achieving. Additionally, in http://sources.gentoo.org/viewcvs.py/portage/main/trunk/pym/portage.py?rev=3495&view=markup (hint: look for "recursive=1") and http://bugs.gentoo.org/show_bug.cgi?id=133740, with both predating the initial RFC for PMS sent to gentoo-dev mailing list, this behavior is discussed and shown to be a design feature, not a flaw or lack of QA in portage. This proves with certainty that it is PMS and the views of the reporter of this bug that are flawed, and not the behavior of portage.
Can we please stop with the revert actions in this bug? Zac, do you agree with Arfrever resolution to this bug?
(In reply to comment #29) > Zac, > do you agree with Arfrever resolution to this bug? Well, typically we keep the bug open until the patch is included in a released version of portage. However, I do agree with the resolution. As said earlier, we can always change it in the future if the community consensus requires it.
(In reply to comment #28) Allen, you appear to be confused as to what we are discussing here. Note the "in a repository" in the initial comment. This has no impact upon user configuration or anything in bug 133740, all of which is beyond the scope of PMS. Anything you have under /etc will continue to work just as you expect. I also find it worrying that you chose to try to attack me on this. I didn't write the section of PMS in question, and I would have no difficulty implementing such a feature for Paludis -- Paludis already supports the feature in question for user configuration when emulating Portage's configuration format, so I've no idea where you're coming from on that angle. Please explain in detail why you think I can't figure out how to implement it, and why you chose to bring that up in this bug.
Re: r14104. While it documents the current behavior, it doesn't document the current PMS prohibition. I'd suggest at least noting that it's controversial ATM, pending council action or further compromise being worked out. Of course, unless there's a tagged release before council action (Sept 21st meeting, I read), it doesn't matter /that/ much, but IMO it would be the best compromise between the parties mean time, as it would document both the current behavior, and that said behavior may be reverted. That said, I'd like to see it adopted by council if not by previous compromise, for use in official overlays other than the Gentoo main tree repo with little delay (30-90 days), with Gentoo main tree conversion at 90-180 days, special-case, no wait for EAPI-4, because the behavior /was/ there, just not documented for in-tree use, since pre-PMS. The .d doesn't strike me as particularly workable, because of the nature of package.mask. Without further action (such as the infra combining scripts mentioned on -dev), therefore, people with extremely old clients will already be broken, as they'd be blind to anything in the .d dirs anyway, including critical package masks, thus making them broken by definition, regardless if they're broken due to inability to process package.* and use.* as a directory or because they don't see packages masked in package.mask.d as actually masked. But given the age of the code involved and the fact that people that let their PMs get that old are probably running portage, that's a /long/ way back, /well/ over the year that was the traditional pre-PMS wait-time for such changes. And they'd have yet another 90-180 days to upgrade after approval as well, before the main tree changed.
I've added a note about PMS: http://sources.gentoo.org/viewcvs.py/portage?view=rev&rev=14118
My personal opinion is to add it to EAPI 0 as the feature is handy and we can provide a generated package.mask for a while to make usage of all package managers possible without any breakage. EAPI 0 is defined as Portage behaviour prior to PMS introduction despite cases where Portage behaviour is obviously buggy. And all of us must admit that the undefined handling is quite old but handy.
Ok, I must correct myself...it is not that old (Portage 2.1.6.7). So the only thing left is it's usefulness. But breaking the specification process once makes having the whole process moot. So I have no strong opinion to it but tend to have it in a new EAPI...council decides here. (Great to contradict yourself in two posts in a row. :)
(In reply to comment #35) > Ok, I must correct myself...it is not that old (Portage 2.1.6.7). Actually, it's been supported in all package.mask files since portage-2.1, which is quite old. It was added for all package.* and use.* files in 2.1.6.7 (except for package.provided, which has support for it now in trunk).
(In reply to comment #36) > (In reply to comment #35) > > Ok, I must correct myself...it is not that old (Portage 2.1.6.7). > > Actually, it's been supported in all package.mask files since portage-2.1, > which is quite old. It was added for all package.* and use.* files in 2.1.6.7 > (except for package.provided, which has support for it now in trunk). As my opinion was not that clear to some people: If the package.* directory support was available (even accidentally) before setting EAPI 0 and PMS, it should be added to EAPI 0. If not, it should be available in EAPI 4. From the council meeting today I read that some people might argue that stating pre- or post-PMS is not possible for the 2.1 series (release date vs. fork date in repository), a council decision is possible by simple vote without having to discuss this topic over and over again. Especially the fact that it does not have any issues in Gentoo but in Funtoo and Paludis, this is something minor. Nonetheless, having that feature is handy and a council vote should end all discussions.
(In reply to comment #37) > As my opinion was not that clear to some people: > If the package.* directory support was available (even accidentally) before > setting EAPI 0 and PMS, it should be added to EAPI 0. If not, it should be > available in EAPI 4. The feature was undocumented in Portage until three weeks ago... The section of PMS in question was directly based off the Portage documentation. In any case, if you're taking the attitude that an undocumented and unused Portage feature existing is sufficient grounds for it being EAPI 0, I suggest you find someone who's prepared to find and document every single undocumented aspect of what Portage does, and then make sure Portage never changes any of those behaviours.
(In reply to comment #38) > (In reply to comment #37) > > As my opinion was not that clear to some people: > > If the package.* directory support was available (even accidentally) before > > setting EAPI 0 and PMS, it should be added to EAPI 0. If not, it should be > > available in EAPI 4. > > The feature was undocumented in Portage until three weeks ago... The section of > PMS in question was directly based off the Portage documentation. As you might know, Portage's implementation and documentation differed at some times during the course of the last decade. > In any case, if you're taking the attitude that an undocumented and unused > Portage feature existing is sufficient grounds for it being EAPI 0, I suggest > you find someone who's prepared to find and document every single undocumented > aspect of what Portage does, and then make sure Portage never changes any of > those behaviours. Can't we simply decide on a case-by-case basis? If I remember correctly, everyone agreed that this feature is really handy and introduction into EAPI 0 just backports current "features" of the official package manager of Gentoo. If it is unused Portage feature we could just make it official and use it...I am not willing to discuss this thing that does not even effect Gentoo (only Funtoo and Paludis have a problem here) for eternity. Either we have a decision by the body that is responsible (Council). This can be done by simple vote: 1) EAPI 0 or 2) Portage bug -> disable and let it go to EAPI 4. Every discussion until then is moot and I will no longer take part as all arguments have been exchanged.
(In reply to comment #39) > Can't we simply decide on a case-by-case basis? We shouldn't be deciding on retroactive changes to EAPI 0 without a *very* good reason, because doing so invalidates the entire process and screws things up for anyone attempting to implement the standard. The only times we should be retroactively changing EAPI 0 are if we can do it without breaking anything (which isn't the case here), if it's absolutely critical and can't wait for a new EAPI (which isn't the case here) or if there's a lot of existing code relying upon something that can't easily be altered (which isn't the case here). If we start arbitrarily invalidating the spec and every non-Portage implementation whenever we feel like it and when it's not absolutely necessary, there's no point to having a spec or third party implementations at all.
(In reply to comment #40) > We shouldn't be deciding on retroactive changes to EAPI 0 without a *very* > good reason, because doing so invalidates the entire process and screws > things up for anyone attempting to implement the standard. I tend to agree with this. And if we consider the latest (only?) PMS version approved by the council [1] to be the authoritative standard: 1. There is no mention that a package.* and use.* can be directories. 2. It doesn't say anything about profiles being controlled by EAPI. However, this was approved in the 2008-12-11 council meeting. [2] So I would say that the feature has to wait for EAPI 4, instead of retroactively changing the standard. Concerning Portage: IMHO the presence of some extended functionality hardly qualifies as a bug. After all, it doesn't break anything, and it is clearly documented [3] that it's non-standard behaviour. [1] <http://dev.gentoo.org/~tanderson/pms/eapi-2-approved/pms.html#x1-330003.4> [2] <http://www.gentoo.org/proj/en/council/meeting-logs/20081211-summary.txt> [3] <http://sources.gentoo.org/viewcvs.py/portage/main/trunk/man/portage.5?rev=14118&view=diff&r1=14118&r2=14117&p1=main/trunk/man/portage.5&p2=/main/trunk/man/portage.5>
I actually tend to agree with Ulrich. Were this to be a documented, intended feature of Portage in EAPI 0 pre-PMS, then PMS definitely would need to be changed(and we would in turn need to make sure PMS got changed). But it wasn't. I lean towards putting this feature in EAPI 3(quickly! so that portage doesn't finish up first) rather than wait for EAPI 4. That doesn't mean portage has to remove it for EAPI 0(Though an ignorable warning might be in order for those who are unaware of its implications), so funtoo(and trelane by extension) can be happy. ~Thomas~
I've talked to ciaranm, trelane, and zmedico, and try to summarise: - There is consensus that for PMS we should follow regular procedure, i.e. include the feature with the next EAPI. - There is no need to remove the feature from Portage. - About the question if Portage should emit a warning there is disagreement (see above comments by ciaranm and trelane). zmedico has suggested to output a warning by default, that can be turned off by some FEATURES setting. This looks like a good compromise to me. We could consider to make the warning unconditional, as soon as there is an EAPI that supports {package,use}.* directories in profiles.
(In reply to Ulrich Müller from comment #43) > zmedico has suggested to output a warning by default, that can be turned off > by some FEATURES setting. This looks like a good compromise to me. We could > consider to make the warning unconditional, as soon as there is an EAPI that > supports {package,use}.* directories in profiles. We've had a warning by default for some time now, that can be turned off by a profile-formats setting in metadata/layout.conf: http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=9d4459dfea72717c0565b0cfe39fa3a87a57ad9d
