Over the years I have found there is no technically feasible way to monitor what is being added to gentoo-x86/profiles/arch/* by whom, when, why, unless some policy is made explicit that tells developers how to do it. I've tried to somewhat control it with the header shown below, but as that very same example makes clear, developers simply ignore it in two ways: ---------------------------------------------- gentoo-x86/profiles/arch/hppa $ head package.mask # Copyright 1999-2010 Gentoo Foundation # Distributed under the terms of the GNU General Public License v2 # $Header: /var/cvsroot/gentoo-x86/profiles/arch/hppa/package.mask,v 1.20 2010/02/10 04 :24:41 abcd Exp $ # DON'T TOUCH THIS FILE. Instead, file a bug and assign it to <hppa@gentoo.org>. # Alexey Shvetsov <alexxy@gentoo.org> (04 Feb 2010) # Mask kde 4.4.0 =) # Note. Unmask libmsn at the same time, required by kopete. >=net-libs/libmsn-4.0 ---------------------------------------------- In this case, at least two kde@ developers not only touched the file, but entirely disregarded the request to instead to file a bug report as well. I am not filing this bug report to blame them[1], but to clarify what our documentation currently lacks. I think we should at least attempt to inform developers that touching arch profile files isn't trivial and possibly a minor sin, and that not informing the arch teams is a major sin, whereas requesting changes to those files in formal bug reports would be a +1 on their karma. [1] Especially if these developers made the changes with the general formal approval and foreknowledge of the team, in which case the entire KDE team should have known better group-wise and really actually *is* to blame.
(In reply to comment #0) > Over the years I have found there is no technically feasible way to monitor > what is being added to gentoo-x86/profiles/arch/* by whom, when, why, unless > some policy is made explicit that tells developers how to do it. I've tried to > somewhat control it with the header shown below, but as that very same example > makes clear, developers simply ignore it in two ways: > Is there a problem with gentoo-commits mailing list and procmail?
As I stated before I will say this again. ATs refused to do keyword requests in overlay and we add all new stuff with 4.4.0 final to main tree (this policy wont change, it works quite well). So only 2 options are left, mask it globaly or mask it in profiles and unmask on archs where we actually can do the keywording (amd64 x86). We usually add the bug at the time we properly make sure that amd64 and x86 build so we dont bother archies with 100% non-working enviroment. As rej was PITA on irc yesterday i will say it again. We were already creating that bug report when he started ranting about this (as now the bug is open). Usually arch team bug for keywording/stabilisation takes more than month (to get it completely done). So i don't get the reason why 24 hours delay on our side is so huge. I will gladly alter the policy for adding to main tree that we will open the bug first, but the list will follow LATER when we make sure it works. Rej is already known for keywording stuff that was not requested, nor desired to have keyworded [1]. We always open requests/bugs when at the time the thing is safe to keyword, not earlier, and that wont change. [1] - http://www.mail-archive.com/gentoo-dev@lists.gentoo.org/msg27939.html
I'm removing the alias as it doesn't work as people would expect it. As Jeroen stated, this bug is about a policy and not about people, so let's try to keep that in mind. I have to generally agree with him about touching individual arch profiles. It's mostly an arch team job and it should be coordinated with the arch teams. Maintainers and arch teams need to work together so what we need is to streamline the process and find compromises that work for all involved parties. As I've told Jeroen, I'll propose that in the future the KDE team tries to open a bug 1 month before adding a new minor release in the tree (around .8* or .9* releases) if there are new required dependencies, warning the arch teams about them and requesting their help. I think we should also consider adding the new dependencies to the tree as soon as they are released and reach a minimum level of stability. We'll still keep working in our overlay though and we'll likely have to ask arch teams to use it at times - if there are objections or concerns about using it, I'd like to hear them so we can address them and try to find alternatives or compromises. If arch teams are either unable or unwilling to help us in such a scenario, the only alternative will be to deal with masks in the base profile and or individual profiles - as was done here. I want to see that done as a last resort though and not as our standard method.
@Tomáš: Looks like you entirely missed the point of this bug report. It's not about kde@ policy, but about Gentoo policy. It certainly isn't about you or me, so please don't try to turn this discussion into something personal. Nobody needs another smear campaign, do we? Please remember that I am making an effort here to lightly brush aside your petty accusations. Now then, with regard to profiles/arch/ the developer documentation is pretty thinly spread[1][2][3][4]. At the very least, the Handbook should clarify that you inform arch teams (maybe "coordinate with" is better indeed) of the changes you'd like to see, if not how to use, change and not abuse profiles in the first place. In fact, most developer resources do mention keywording policy (you don't keyword for an arch if you can't test on that arch), and several arches expand on that by insisting that you have to be on the arch@ alias to be allowed to keyword ebuilds, probably because of all kinds of coordination mistakes in the tree's early days. Extending this one rule to arch sub-profiles looks like an easy route toward a definitive policy. Having the keywording rule apply to package.masking in sub-profiles would also simply mean that if you are not allowed to circumvent keywording issues through package.mask, your only alternative really is to not commit the ebuild or to drop the keyword. The arch teams should decide what goes into the arch sub-profiles, and no one else, exactly the same way as with keywords. [1] http://www.gentoo.org/proj/en/devrel/handbook/handbook.xml?part=3&chap=1 is short on detail, and could better connect these two statements: "There is a difference between using package.mask and ~arch for ebuilds. The use of ~arch denotes an ebuild requires testing. The use of package.mask denotes that the application or library itself is deemed unstable." "Important: Remember that arch teams alone are responsible for marking packages stable on their arch. Package maintainers should open stabilization bugs; they may not just mark packages stable." At least that's an interesting start to a bit of policy. [2] http://devmanual.gentoo.org/profiles/package.mask/index.html mentions subprofiles but not arch subprofiles or policy. [3] http://devmanual.gentoo.org/profiles/use.mask/index.html actually more verbose about arch subprofiles, but only mentions practical use, not policy. [4] The ebuild quiz fails to address (sub) profile policy.
Forgot this one of course, even though if have been hammering it into ebuild dev's for years: [5] http://devmanual.gentoo.org/keywording/index.html says: = Keywording on Upgrades = [...] Sometimes you may need to remove a keyword because of new unresolved dependencies. If you do this, you *must* file a bug notifying the relevant arch teams. This does actually say that if you can't keep keywords, because of unresolved deps, you drop the keywords and inform arch teams. All the documentation about package.mask already suggests that the file should be used for ebuilds with serious problems ("instability"), whereas it sadly states nowhere that conversely, package.mask is not to be used to conveniently hide keywording problems, whether it's the global package.mask or one in an arch sub-profile.
(In reply to comment #5) > conversely, package.mask is not to be used to conveniently hide keywording > problems, whether it's the global package.mask or one in an arch sub-profile. I agree in general, but I don't think you realize the gravity here. Example, - dev-libs/shared-desktop-ontologies is not keyworded for ~hppa. - kde-base/kdelibs depends on dev-libs/shared-desktop-ontologies - all other kde ebuilds depend on kdelibs You really want the keyword dropped for ~300 ebuilds, with every minor point release? I'm serious. If there should be a exception to the rule, this is it.
(In reply to comment #6) > If there should be a exception to the rule, this is it. And if this will be documented futher in devmanual, it should mentioned that different subprojects, such as KDE, might have differing policies over it.
(In reply to comment #6) > (In reply to comment #5) > > conversely, package.mask is not to be used to conveniently hide keywording > > problems, whether it's the global package.mask or one in an arch sub-profile. > > I agree in general, but I don't think you realize the gravity here. > > Example, > > - dev-libs/shared-desktop-ontologies is not keyworded for ~hppa. > - kde-base/kdelibs depends on dev-libs/shared-desktop-ontologies > - all other kde ebuilds depend on kdelibs > > You really want the keyword dropped for ~300 ebuilds, with every minor point > release? I'm serious. > > If there should be a exception to the rule, this is it. This is why I think it's so important to ensure cooperation with arch teams. Some might prefer to have their keywords simply dropped, while others might prefer to get the packages masked in their profile so all they need to do is remove the mask when they are able to test the ebuilds. Can / should we try to reach a consensus on one option about this or should we really insist on "cooperation" and "common sense" to address this type of issues? I'd prefer the latter.
(In reply to comment #6) > (In reply to comment #5) > > conversely, package.mask is not to be used to conveniently hide keywording > > problems, whether it's the global package.mask or one in an arch sub-profile. > > I agree in general, but I don't think you realize the gravity here. I think I do. > Example, > > - dev-libs/shared-desktop-ontologies is not keyworded for ~hppa. > - kde-base/kdelibs depends on dev-libs/shared-desktop-ontologies > - all other kde ebuilds depend on kdelibs Looks like a simple problem to solve. kde@ can see this one coming months ahead I should think, and it's not a kde-*/*-[MAJOR_VERSION] package so its keywords do not need to be dropped. The same isn't true for kde-*/*-[MAJOR_VERSION] of course, so if it had been a new kde-base/libkoscaf-4.4.0 (liboskaf - hehlol) dependency in kdelibs, that would have been a bit more of a problem. Still, kde@ doesn't need to decide for arch teams how to solve this problem - read on please. > You really want the keyword dropped for ~300 ebuilds, with every minor point > release? I'm serious. Yes, why not, if you have to. The arch team can rekeyword afterwards, can't they? As long as you notify the arch team, I don't see any problem with this Abusing profiles/arch/*/package.mask for that really is a stopgap measure, and requires coordination with the relevant arch team. Now if you have specific agreements with specific arch teams that you're allowed to p.mask whatever you see fit, then sure, go ahead. When there is no such agreement, you don't. There are other projects with hundreds of version restricted dependencies - GNOME and texlive spring to mind, and X doesn't have these versioning schemes but does require synced keywords in much the same manner - those projects solve the issues by warning arch teams ahead so that no one ends up dropping keywords at all, or by simply handling any needed arch specific package masking in coordination with those arch teams. There is really no need to go over anyone's head (and it pisses people off, you know). The other teams use overlays to develop new major versions as well but also manage to move new versions to the tree in a manner that everyone agrees with. > (In reply to comment #7) > > If there should be a exception to the rule, this is it. > > And if this will be documented futher in devmanual, it should mentioned that > different subprojects, such as KDE, might have differing policies over it. There's no need for an exception to the rule. There is no technical problem here. If the arch team decides that dropping the keywords is fine with them and they'll simply readd them later, there is no need to use package.mask. If package.masking for specific profiles cannot be avoided, there is still no need to shoot first and ask later - this could be handled much more subtlely. This is in fact how it has always been done, and I don't see why any project teams' "policy" should be allowed to overrule this fine tradition. Arch teams touch arch profiles and no one else - and this should be written down in our policies to reflect the status quo we've had since forever.
Can this be brought to the mailing list? No policy is going to be decided here. The idea that *only* arch teams can touch arch profiles is just plain silly. I also don't see why this is assigned to devrel and a locked bug...
Just to clarify my previous comment, when I say "while others might prefer to get the packages masked in their profile", I argue that the mask might be added by either the arch teams or the maintainers, but only *after* the issue is coordinated between the parties and the mask is added as agreed. I prefer we use "common sense" to solve disputes, but to be thorough, in case disputes cannot be solved before the maintainers want to add a new version to the tree and if arch teams are unable to react on a reasonable time frame, QA can always be asked to intervene.
I've talked with a few people on IRC and am thus opening this bug and reassigning it. The purpose is to leave the bug open while we start an ml discussion about policies. Although the bug has been unrestricted, I'd like to ask anyone wanting to add a comment to think carefully as this is about policy and not people.
(In reply to comment #11) > Just to clarify my previous comment, when I say "while others might > prefer to get the packages masked in their profile", I argue that the mask > might be added by either the arch teams or the maintainers, but only *after* > the issue is coordinated between the parties and the mask is added as agreed. That was my intention all along. Getting an advance warning really helps how you plan building and testing stuff. Also, yngwin remarked that the original notice I put in was perhaps too strongly worded, and we agreed that it could be reworded to imply you should inform the relevant arch team before making changes. Then again, informing the relevant arch team should be policy, not a repeated plea in some files in the tree. I put in those boilerplate warnings to try to stop people doing it without notification. > I prefer we use "common sense" to solve disputes, but to be thorough, in case > disputes cannot be solved before the maintainers want to add a new version to > the tree and if arch teams are unable to react on a reasonable time frame, QA > can always be asked to intervene. Handling profile changes is always a matter of cautious use of common sense. I happen to not have the hardware performance available to check whether removing a mask leaves nothing broken, whereas checking for dropped keywords is rather a lightweight check by comparison. Someone with the dep checking power to do a ~300 ebuild commit should also be able to check whether it is time yet to lift a package.mask on those ebuilds. Coordination between teams would greatly help here.
Ping. From the recruiter side of view, this is something that really confuses our recruits. The general approach is that developers are allowed to touch package specific arch files such as packages.use.mask but files that affect the general behaviour of an arch should be handled explicitely but arch members.
(In reply to comment #14) > From the recruiter side of view, this is something that really confuses > our recruits. I don't understand. This isn't policy (yet), and it hasn't been introduced in quizzes or in documentation. But generally I think it should come down to this. 1) Before you change an arch profile, notify the arch team in question. 2) Do not change an arch profile as a convenience. There are cases where the only alternative to arch profile changes would be to drop a lot of keywords, where (1) should apply equally since it is no different from dropping keywords, so arch teams should be notified anyway. An example of "convenient" arch profile changes(2), is a case where the newly introduced USE=foo would add an unkeyworded dependency, and where instead of rekeywording, foo is USE-masked to cover that up and not break the tree.
Please define "notify", and explain what should be done if an arch team * does not respond to IRC pings * does not react at all to a mail sent to the arch alias * and a keyword request is ignored What reaction times should be expected, in particular in the case of simple straightforward requests? You're very outspoken against pmasking a package in one arch to avoid broken dependencies. This is indeed a workaround, but it is a workaround intended to help busy and understaffed arch teams. Compared to testing for a keyword request, it is far less work to _copy_ a few mask lines from the global package mask to an arch package mask. Also, this will not break anything, since for users of that arch nothing changes. IMHO, every developer should be allowed to _copy_ a mask entry from a global package.mask to an arch package.mask after a keyword request has been filed for some time.
(In reply to comment #15) > (In reply to comment #14) > > From the recruiter side of view, this is something that really confuses > > our recruits. > > I don't understand. This isn't policy (yet), and it hasn't been introduced in > quizzes or in documentation. So? It is not a policy yet but you have to teach them *something*. Poking arch teams for every single thing wont work. Developers should be allowed to do minor alternations ( masking packages, package specific use flags ) if needed without waiting for an arch member to do it for them
(In reply to comment #16) > Please define "notify", File a bug report. > and explain what should be done if an arch team > * does not respond to IRC pings > * does not react at all to a mail sent to the arch alias > * and a keyword request is ignored Why do you put it like that? That's a worst case scenario (an arch is abandoned entirely) we haven't yet seen. Degrading an arch to ~arch ("dev" or "experimental") across the board is usually the long term solution. Dropping keywords is, too. If you want to stop supporting an arch team support your package, then drop their keywords, and tell them you did so everyone, users included, can see what happened and why. (Arches die slowly and painfully, which is why dropping keywords isn't always such a bad solution - it elegantly lessens the workload of the arch team, and if no users of that arch cry out, no damage has been done when ultimately the last keyworded versions are removed.) Putting tons of atoms in profile/arch/foo/package.mask is just as damaging to ~arch users (think typical embedded arches like mips and powerpc) as dropping keywords is, only it says nowhere in our documentation that you should inform the arch team. But let's not digress - dying architectures are not the subject of this bug report. Informing arch teams when you mask packages, in whichever way, is. > What reaction times should be expected, in particular in the case of simple > straightforward requests? As I said. It doesn't matter here. > You're very outspoken against pmasking a package in one arch to avoid broken > dependencies. This is indeed a workaround, but it is a workaround intended to > help busy and understaffed arch teams. Compared to testing for a keyword > request, it is far less work to _copy_ a few mask lines from the global package > mask to an arch package mask. Also, this will not break anything, since for > users of that arch nothing changes. > > IMHO, every developer should be allowed to _copy_ a mask entry from a global > package.mask to an arch package.mask after a keyword request has been filed for > some time. Now YOU are saying that the arch team SHOULD be notified before you do anything like that (touching the profile or dropping keywords). So we agree...
One more thought: if you maintain dozens or hundreds of packages and want to remove a huge profiles entry to unmask some version, how do you proceed? What tools do you use to check reverse dependencies? How could we turn all this into information we could enter into our documentation?
Hi Jeroen, > > File a bug report. > [...] > Now YOU are saying that the arch team SHOULD be notified before you do > anything like that (touching the profile or dropping keywords). So we > agree... Yes sure, I dont mind filing a bug. I'll happily document everything that I do that way. That is reasonable indeed. Just to make it clear, I learned during my recruitment that I should "get the ok from an arch team before I do anything". That's subtly different, since it is sometimes hard to get a response. So, could we agree to the following: if for some valid reason a (temporary?) mask entry makes sense, * notify the arch about it in a bug report ("we need these and these keywords on these and these packages, as long as they are not there yet, we'd like to temporarily mask x and y on your arch") * if there is no opposition within a few days, the arch package mask may be modified (with an appropriate comment) > (Arches die slowly and painfully, which is why dropping keywords isn't always > such a bad solution - it elegantly lessens the workload of the arch team, and > if no users of that arch cry out, no damage has been done when ultimately the > last keyworded versions are removed.) Sure... we have to find a compromise between * reducing the amount of keyworded packages to a manageable size * and not dropping needlessly keywords if it's just about a reasonable delay > One more thought: if you maintain dozens or hundreds of packages and want > to remove a huge profiles entry to unmask some version, how do you > proceed? What tools do you use to check reverse dependencies? So far I personally use repoman, but then I do have a rather fast machine. I have been told on #g-dev that there are faster tools available, but have not looked into it yet...
> One more thought: if you maintain dozens or hundreds of packages and want to > remove a huge profiles entry to unmask some version, how do you proceed? What > tools do you use to check reverse dependencies? As an idea, since the metadata is machine independent, maybe infra could set up some service where changes like this can be tested? Just brute-force computing power going over a modifiable tree... That could be helpful also for others (arm, ppp?)...
Ping- opinions?
(In reply to comment #22) > Ping- opinions? > I don't think that this would work. As I already said, recruiters are teaching recruits that they can touch package.use.mask on their own but when they need to touch anything else they should always ask the arch team. If the arch team is dead^W unresponsive then they can always consult QA for that.
Time to end this tragedy. Below is a patch which I would like to suggest for the devmanual. In effect, it codifies the current practices and common sense. Opinions? diff --git a/archs/text.xml b/archs/text.xml index 4eb9750..a9f1059 100644 --- a/archs/text.xml +++ b/archs/text.xml @@ -17,6 +17,17 @@ attempts to provide a standard set of packages at specific versions and install it onto every platform <d/> rather, we simply attempt to provide whatever happens to work best in any situation. </note> + +<p> +In general, all arch-specific files may only be modified by the respective architecture +team members. This is particularly the case for the arch-specific profile subtrees. +However, there an important exception exists: A package maintainer can add a new package +to the arch specific package.mask if this only delays visibility for that arch but effects +nothing else. This can be used e.g. in the case when dependencies introduced in a version +bump have not been keyworded yet for that arch. Similarly, the maintainer can remove that +mask again if all that does is to end the previously introduced delay in ebuild visibility. +</p> + </body> <section>
(In reply to comment #24) > Time to end this tragedy. Below is a patch which I would like to suggest for > the devmanual. In effect, it codifies the current practices and common sense. > Opinions? > --- a/archs/text.xml > +++ b/archs/text.xml > @@ -17,6 +17,17 @@ attempts to provide a standard set of packages at specific > versions and install > it onto every platform <d/> rather, we simply attempt to provide whatever > happens to work best in any situation. > </note> > + > +<p> > +In general, all arch-specific files may only be modified by the respective > architecture > +team members. This is particularly the case for the arch-specific profile > subtrees. > +However, there an important exception exists: A package maintainer can add a > new package > +to the arch specific package.mask if this only delays visibility for that arch > but effects Effects as in bring about, or affects, as in have an effect upon? > +nothing else. This can be used e.g. in the case when dependencies introduced > in a version > +bump have not been keyworded yet for that arch. Similarly, the maintainer can How about dropping the keyword, if that doesn't subsequently affect a multitude of packages? Most devs previously seemed to agree that the /least/ changes to the tree should be preferred over any dev/team's convenience over another dev/team's. So do you mask a USE flag for everyone and then unmask it in two arch profiles, making for a total of changes to three files, or do you mask it in all arch profiles but the ones you have tested, making for the total of arch profiles (say, 15) minus two, in this example affecting 13 (arch pro-)files instead of 3 files. > remove that > +mask again if all that does is to end the previously introduced delay in > ebuild visibility. I don't think that the maintenance of entries in arch profiles being the responsibility of the committer has been debated much at all, but I like it. :)
Andreas, First of all thank you for the patch. Could you please attach a git formatted patch so you can get a kudos for your contribution? Secondly, I believe (even though I can't find written proofs) that individual developers can touch arch specific files when the arch members are unresponsive. This, of course, implies that the developer tried to contact the arch team and waited for a $X amount of time before touching the files on his own.
Was this ever on the mailing lists and if yes, was there a discussion result?
@Markos: I'll prepare something, also taking into account Jer's comments. @Thomas: I am pretty sure it has been debated to death in ancient past. I will come up with some links. I am not so sure if is productive to raise it on the mailing list again, however, if you think so, I can certainly send my patch there for review.
I can't recall a recent thread in either -dev or -project MLs. However, I believe the proposed patch covers what the majority of developers do anyway.
Ping.
Speaking as a recruiter, what we teach new developers is to never touch arch profiles without asking the arch team first. Like I said in my previous comment, there should be a note that it is ok to touch the profile is arch team is unresponsive. Should we send this to the mailing list and formalize a policy? Because I don't think there is one I also believe all the devrel handbook related sections should just go away and document the respective policy in the devmanual once and for all.
(In reply to comment #31) > Should we send this to the mailing list and formalize a policy? Because I > don't think there is one Since nobody has found links to an existing discussion, I suppose we must. > I also believe all the devrel handbook related sections should just go away > and document the respective policy in the devmanual once and for all. Sounds good to me.
(In reply to comment #32) > (In reply to comment #31) > > Should we send this to the mailing list and formalize a policy? Because I > > don't think there is one > Since nobody has found links to an existing discussion, I suppose we must. > > > I also believe all the devrel handbook related sections should just go away > > and document the respective policy in the devmanual once and for all. > Sounds good to me. Is it me or did you just volunteer to send the email to gentoo-dev ML? :)
This should probably be covered in devmanual instead
Ping
No progress here. Also, after reading all 35 comments, I am not even sure what exactly this is about. Closing for now, feel free to reopen with a patch attached.
