See the tracker. Please really serious drop that thing. You're using it for 2 (*two*) ebuilds, and the support code is almost as large as the eblits themselves... If you really believe you need a central place for ~350 lines of code that is used by one package having two versions, please commit an eclass for this. Per-package eclasses have been discussed in PMS team today, and they won't happen.
I used to have multiple ebuilds (5+), that's why eblits made sense back then. I would still like to centralize the eblit code in some form, but always felt that an eclass was inappropriate for a single package. Hence, I used the glibc eblit logic. If this is being prohibited by PMS now, well, let me see what toolchain does for glibc and I'll look at copying that instead. That said, I still think the concept of per-package eclasses isn't in itself a terrible idea, but it would need to be more robust and better thought out than what eblits are. Perhaps for the future...
(In reply to Joshua Kinard from comment #1) > That said, I still think the concept of per-package eclasses isn't in itself > a terrible idea, but it would need to be more robust and better thought out > than what eblits are. Perhaps for the future... It has been considered for the future, and the effort necessary was found pointless for 3-10 packages that are going to use them.
(In reply to Michał Górny from comment #2) > (In reply to Joshua Kinard from comment #1) > > That said, I still think the concept of per-package eclasses isn't in itself > > a terrible idea, but it would need to be more robust and better thought out > > than what eblits are. Perhaps for the future... > > It has been considered for the future, and the effort necessary was found > pointless for 3-10 packages that are going to use them. Never say never :) It's not concrete to say what packages might do in the future if a hypothetical capability might be added. The only way to know for sure would to be to add it (possibly as an eclass of sorts) and then see what happens over a window of time. But, this bug isn't a suitable place to discuss such hypotheticals. For now, I'll go along with PMS, and look at doing the eblit stuff differently after I see what others do (because I don't have any good ideas at the moment and have other priorities on my plate right now). Also... (In reply to Michał Górny from comment #0) > See the tracker. Please really serious drop that thing. You're using it for > 2 (*two*) ebuilds, and the support code is almost as large as the eblits > themselves... I have to dispute this blind assertion. If you actually bothered to count the lines, excluding comments and whitespace, the four base eblit handing functions occupy 33 lines of code in each ebuild. If you look in ${FILESDIR}/eblits/, you'll see that across all of the existing eblits, there's 203 lines of code (excluding whitespace/comments). I think 33 LOC to load up to 203 LOC, broken out as modules and used on a per-machine basis, is a good tradeoff. If eblits were ever sanctioned into an official eclass, it'd be those 33 LOC, plus comments and other minutiae that would be centralized and further reduce the size of the individual ebuilds. Only the code needed to load specific eblits and the control variables at the top would need to be ebuild-specific. I started such an eclass, but haven't worked out how to sanely implement the guard variable that's required to make sure that the pkg_* eblit code can still be executed if installing from binary packages and ${FILESDIR} is not available. Probably really easy, just my bash-fu is quite rusty.
Also, per the tracker bug: > Per PMS 11.1 [1, note: wording improved in git master] the directory pointed > by FILESDIR must not be accessed from global scope of an ebuild and in pkg_* > functions. The value of this variable is only provided for setting variables > such as PATCHES that are used later in src_* phases. > > [1]:https://projects.gentoo.org/pms/latest/pms.html#x1-11800011.1 I believe the eblit use I have only violates the loading from pkg_* bit, namely, pkg_setup. The global scope access bug was fixed a few years ago by implementing the load_eblit_funcs' function that does the actual sourcing of eblit files in ${FILESDIR} once, then sets ${MIPS_SOURCES_EBLITS_LOADED} to '1' (the guard variable I mentioned) to prevent the eblits from being sourced again if the ebuild is being processed for metadata generation or being installed from binary packages. Is the one call of 'load_eblit_funcs' from pkg_setup the only thing that would really need to be done differently to solve this?
I would really appreciate if you just removed the ugly repetitive thing completely. We probably should also ban sourcing anything external inside ebuilds, that's just asking for UB.
Joshua, if you'd like to discuss per-package eclasses/eblits, please open a separate ([Future EAPI]) bug for that with some outline. Otherwise, soon enough the bug will be closed and nobody will be able to figure out where the discussion took place.
(In reply to Michał Górny from comment #6) > Joshua, if you'd like to discuss per-package eclasses/eblits, please open a > separate ([Future EAPI]) bug for that with some outline. Otherwise, soon > enough the bug will be closed and nobody will be able to figure out where > the discussion took place. Okay, I'll try to get to that sometime this week. My time is limited during the weekdays. Thanks!
<QA hat> We will assume maintainer timeout in 30 days and inline the eblits into ebuilds if the issue is not solved until then. </QA hat>
(In reply to Michał Górny from comment #8) > <QA hat> > > We will assume maintainer timeout in 30 days and inline the eblits into > ebuilds if the issue is not solved until then. > > </QA hat> Is there a set way to test/debug the underlying issue so I can see if there is additional logic that can go into the ebuild to square off the FILESDIR access issues? As stated in Comment #4, the only issue seems to be the calling of 'load_eblit_funcs' from within pkg_setup. Outside of that, the eblits are not hurting anything since the metadata issue was resolved several years ago.
No. Eblits are a no-go. In any form. Don't try to work around this, just finally accept that whole Gentoo will not be making exceptions for two developers. Thank you.
(In reply to Michał Górny from comment #10) > No. Eblits are a no-go. In any form. Don't try to work around this, just > finally accept that whole Gentoo will not be making exceptions for two > developers. Thank you. I can't concur with this assessment. You've offered no technical basis, other than the main eblit loading code is "ugly" and "repetitive", which is the exact same argument that I could make about the inlining of the existing eblit code. I favour modularity over correctness, and would prefer a repetition of the smaller loading code than repetition of the larger code that is currently in the eblit files. As far as I can tell, with the metadata issue being resolved years ago, and there's no sourcing of eblits from global scope like in the autoconf package, the only violation, if any, here, is the sourcing something in $FILESDIR. I'd like to find a technical solution to this, and if that means fleshing out a proper eblits eclass, fine (and maybe eblits move to something like $EBLITSDIR, or ./eblit in CWD). I have a partial eclass worked up, except I got hung up implementing the global guard variable that fixed the metadata issue. I have one, maybe two days a week right now to do anything Gentoo-wise, and this is incredibly low on my priority list. Rather than being ham-fisted in this rather minor QA issue, some help in doing eblits the right way would be more beneficial. I am attaching the unfinished eblit.eclass I started back in 2016. The only remaining function to be implemented is 'load_eblit_funcs' from mips-sources, which is where the guard variable comes into play to avoid the metadata/pkgdir issue. That's where I got stuck. Finishing that would allow the other packages using eblits to have a common, unambiguous set of functions for loading eblits and go a long way towards solving this issue.
Created attachment 464762 [details] Unfinished eblit eclass implementation
CC'ing Mike because glibc and autoconf packages could benefit from a finished eblit.eclass.
I'm strongly opposed to any effort like this. Figuring out toolchain ebuilds is hard enough as it is, without 7 more files containing random code that is randomly loaded based on heuristics and moon phase logic. Furthermore, that effort should have been done years ago. It is really tiresome that you just push hacks, then every time someone complains about them you tell them it's gonna be official. It's not. Deal with it and move on. We have enough bad code lying around. No point in making it worse for three packages whose two authors believe they made a great invention and can't accept that it's not as great as it sounds.
(In reply to Michał Górny from comment #14) > I'm strongly opposed to any effort like this. Figuring out toolchain ebuilds > is hard enough as it is, without 7 more files containing random code that is > randomly loaded based on heuristics and moon phase logic. The mips-sources eblits are not the same as the toolchain or glibc eblits and do not depend on heuristics, moon phases, or the daily odor of Cthulhu's breath -- all of them are sourced/loaded, and for the show_ip** ones, a USE flag determines which info message is actually shown. Simply put: - err_disabled_mach-v*.eblit: Contains one function to handle if a particular mips-sources machine type is "disabled", usually when I had a problem updating the patch for a new kernel version. Not really an issue any more, so this might be able to go away eventually. - err_only_one_mach_allowed-v*.eblit: Contains one function to print an error if a user tries enabling more than one machine patch at a time (this is slated to go away, as I've reset my patch hierarchy to make it easier to send stuff upstream) - pkg_postinst-v*.eblit: Implements pkg_postinst() for all mips-sources ebuilds. - pkg_setup-v*.eblit: Implements pkg_setup() for all mips-sources ebuilds. - show_ip**_info-v*.eblit: Implements a single function to display a MIPS machine-specific information message. Can probably go away when I get IP27 and IP30 sent upstream....but I have no ETA for that. - src_unpack-v*.eblit: Implements src_unpack() for all mips-sources ebuilds. That's a pretty clear break down of what each eblit is doing, and if you'd like, I have no problem further documenting them someplace. > Furthermore, that effort should have been done years ago. It is really > tiresome that you just push hacks, then every time someone complains about > them you tell them it's gonna be official. It's not. Deal with it and move > on. No one *ever* raised the notion that the concept of centralized bits of logic on a per-package basis was a problem. The one major issue where the eblit loading code was screwing up the rsync metadata generation because of assuming that FILESDIR was always present was fixed several years ago. Certainly, there have been spurious attempts to implement these things as "elibs" or such names that never really got off of the ground, but that was back in the era where the Portage repo was a quarter of the size it is today and the perceived benefits just not as wide-spread. There hasn't been any real organizing effort to try and make the idea better and actually implement it. I think if that were done, more developers would utilize the concept. > We have enough bad code lying around. No point in making it worse for three > packages whose two authors believe they made a great invention and can't > accept that it's not as great as it sounds. IMHO, I don't think it's bad code. But that an entirely subjective argument and has no basis on technical grounds. Furthermore, I don't regard it as a "great invention" -- I copied the code from an earlier version in the glibc ebuilds and tweaked it to work for my case (and later on, the metadata fix was added). To me, it's useful in centralizing commonly-used code for the way mips-sources is setup, which has been working fine for years. The ground that I do acknowledge is a policy one, per PMS. My understanding again is the core issue there is for binary packages only, the sourcing of the individual eblit code in FILESDIR is implicitly assuming that FILESDIR is always available, which PMS clearly states is not the case. My understanding is the guard variable and loading of eblits in a local function solved this problem. Is that not correct? How can I test for this? Another idea, looking at Section 4.3 of PMS, it says this at the very bottom: http://dev.gentoo.org/~ulm/pms/head/pms.html#x1-360004.3 """ A package directory may contain other files or directories, whose purpose is not covered by this specification. """ So would there still be a QA violation if I moved the eblits out of FILESDIR and into their own subdirectory? This could serve as a workaround until something more concrete can be put together (I will need help on this as my time is limited). I know it seems like a fruitless argument at the moment because I am only maintaining two active mips-sources ebuilds. That's largely a side-effect of how I was organizing the patches in the past. Each machine type needed its own copy of various support patches, and it was becoming a nightmare to manage. I've fixed that in my local workdir and reset things so that the machine support is more hierarchical (e.g., IP30 support now requires IP27 support) to make sending patches upstream easier as time permits. As such, keeping additional ebuilds (e.g., LTS) is a lot easier now.
Then please use the well-established path of using an eclass. Do you see any problem with that?
[After mid-air collision] Why not move all the code that is currently in eblits into an mips-sources.eclass? At first glance this seems to be straight forward. A single line of inherit is much cleaner than some 50 lines of eblit loading code.
(In reply to Michał Górny from comment #16) > Then please use the well-established path of using an eclass. Do you see any > problem with that? (In reply to Ulrich Müller from comment #17) > [After mid-air collision] > > Why not move all the code that is currently in eblits into an > mips-sources.eclass? At first glance this seems to be straight forward. > > A single line of inherit is much cleaner than some 50 lines of eblit loading > code. eclasses are (or were) intended for global scope and for use by multiple packages. E.g., kernel-2 is used by gentoo-sources, hardened-sources, mips-sources, and linux-headers, to name a few. Having an eclass for just a single package runs counter to that notion. That's where the idea of eblits originally derived from. You put common code used by multiple ebuilds from different packages into an eclass, and you common code used by multiple ebuilds from the same package into an eblit/elib/whatever. The idea is/was to reduce code duplication and contribute towards keeping the repo somewhat sane. Additionally, eclasses don't have a very sane mechanism for versioning. git might help solve that now with rename tracking, but it was a serious issue in the CVS days. Even with git, to maintain the change history on that eclass, you could only have one version of that eclass active at a time. I can't rule out that a later mips-sources ebuild might need adjusted code in an eclass that could break older ebuilds. I would need multiple copies of the single-package eclass at that point then, or I'd have to duplicate the functions inside the eclass so that different ebuilds would use the appropriate function. This seems inappropriate for a global-scope eclass, where preserving change history is somewhat important. That's why making eblits into an eclass by moving the loading code, which shouldn't change very often, into an eclass is the better solution. Multiple ebuilds from different packages could use it for their own local eblits files. The issue is that the notion of "per-package eclasses" isn't sanctified in an EAPI right now, and given past comments on this bug and elsewhere, seems to be opposed purely on cosmetic grounds rather than functional or technical grounds. I am not adamantly opposed to just inlining the code and being done with it. But I want to codify the opposition to the idea somewhere on something a bit more substantive than it simply being "ugly" and/or "repetitive". We have the valid PMS violation, and I acknowledge that. I've also proposed solutions to that, everything from moving the eblit code outside of FILESDIR into a package-specific directory that isn't covered by PMS §4.3 to doing a proper eclass and I guess a GLEP. Some questions I have: 1. From Comment #1, why does the PMS team oppose the notion of per-package eclass-like capability? Is that documented some place that I can go read up on? 2. It's been years since the idea of elibs was last proposed. Would it be worth putting the idea to the wider developer community at large again to see if there really might be any tangible benefits to having such a capability as "per-package eclasses"?
Three points regarding eclasses vs eblits: 1. I don't like eclasses used by one or a few packages either. However, that is an established thing and I can live with it. It's better to have one necessary evil than two, especially that eblits don't solve all use cases for eclasses. 2. Eclasses are less scattered. Eblits will make figuring out what exactly happens in an ebuild even harder, and we already have problems with multiple inheritance and function exports. 3. Versioning in eclasses may not be perfect but it's simple. I'd rather not have to figure out where eblits override stuff, where they are common, etc.
Created attachment 464902 [details] Discussion in #gentoo-pms from 2016-06-19 (In reply to Joshua Kinard from comment #18) > eclasses are (or were) intended for global scope and for use by multiple > packages. None of our documentation seems to say that. The only requirement is that an eclass is used by more than one ebuild: "An eclass is a collection of code which can be used by more than one ebuild." (Devmanual) "Eclasses serve to store common code that is used by more than one ebuild, [...]" (PMS) Then again, most packages won't have more than a couple of ebuilds at any given time, so presumably moving parts of their code into an eclass doesn't offer any significant advantage. Kernel packages may be somewhat special there with respect to the number of versions. > Some questions I have: > > 1. From Comment #1, why does the PMS team oppose the notion of per-package > eclass-like capability? Is that documented some place that I can go read up > on? > > 2. It's been years since the idea of elibs was last proposed. Would it be > worth putting the idea to the wider developer community at large again to > see if there really might be any tangible benefits to having such a > capability as "per-package eclasses"? I am attaching the log of a discussion in #gentoo-pms (with some lines unrelated to the topic removed for better readability). The GLEP draft we are referring to is this: https://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo/users/antarus/projects/gleps/glep-XX.txt?revision=1.2&view=markup Related discussion in the gentoo-dev mailing list: https://archives.gentoo.org/gentoo-dev/message/32b3dacf4c2b0f0c35b9352c93fcce41 In summary, we don't think that the rationale given for per-package eclasses is compelling. The number of eclasses won't be significantly reduced by that proposal, and our opinion is that it doesn't outweigh the necessary effort and the increased complexity. Packages having a need for such a feature can simply use regular eclasses. (If at some later time the number of such eclasses should become overwhelming, we can still reiterate the issue.)
(In reply to Joshua Kinard from comment #18) > 2. It's been years since the idea of elibs was last proposed. Would it be > worth putting the idea to the wider developer community at large again to > see if there really might be any tangible benefits to having such a > capability as "per-package eclasses"? Just for the record: yocto/oe (and probably the bsd's too) have the 'include' directive; they seem happy with it, moreover they do not seem to keep a lot of versions of the same package (unlike us), and still it allows them to share common code per-package. I think 'include' (aka 'source' which, unlike eclasses has no variable mangling, no export_function, nothing) would cover simply and efficiently the eblits/elibs case. I think something like that would definitely benefit code sharing (I can think of a bunch of my ebuilds where there is a lot of long and repetitive code copied over at each version). I think that if eblits are not widely used at the moment it's not because only 2 developers need them but rather because it's a complete pain to setup properly (if possible at all).
(In reply to Alexis Ballier from comment #21) > Just for the record: yocto/oe (and probably the bsd's too) have the > 'include' directive; they seem happy with it, moreover they do not seem to > keep a lot of versions of the same package (unlike us), and still it allows > them to share common code per-package. You're reading it the other way around. Not having multiple versions make this sane. The more versions you have, the more hassle you get to maintain that 'common code' correctly, and the more confusing it is for others when they just are trying to read your ebuild. > I think 'include' (aka 'source' > which, unlike eclasses has no variable mangling, no export_function, > nothing) would cover simply and efficiently the eblits/elibs case. > > I think something like that would definitely benefit code sharing (I can > think of a bunch of my ebuilds where there is a lot of long and repetitive > code copied over at each version). I think that if eblits are not widely > used at the moment it's not because only 2 developers need them but rather > because it's a complete pain to setup properly (if possible at all). Think not of yourself but of others. Sustainability often requires us to sacrifice some convenience/space savings in order to make it easier for others to change stuff. Don't presume you'll be the only one touching your package forever.
(In reply to Michał Górny from comment #22) > (In reply to Alexis Ballier from comment #21) > > Just for the record: yocto/oe (and probably the bsd's too) have the > > 'include' directive; they seem happy with it, moreover they do not seem to > > keep a lot of versions of the same package (unlike us), and still it allows > > them to share common code per-package. > > You're reading it the other way around. Not having multiple versions make > this sane. The more versions you have, the more hassle you get to maintain > that 'common code' correctly, Hmm, what about just 'no' ? You know, you can version those includes by major, major.minor or ebuild... or whatever you like. > and the more confusing it is for others when > they just are trying to read your ebuild. confusing is very subjective and more often than not just a synonym for "not being used to"... > > I think 'include' (aka 'source' > > which, unlike eclasses has no variable mangling, no export_function, > > nothing) would cover simply and efficiently the eblits/elibs case. > > > > I think something like that would definitely benefit code sharing (I can > > think of a bunch of my ebuilds where there is a lot of long and repetitive > > code copied over at each version). I think that if eblits are not widely > > used at the moment it's not because only 2 developers need them but rather > > because it's a complete pain to setup properly (if possible at all). > > Think not of yourself but of others. Sustainability often requires us to > sacrifice some convenience/space savings in order to make it easier for > others to change stuff. Don't presume you'll be the only one touching your > package forever. I'm thinking of others: I too often see changes in ebuilds that are not properly forward ported to the latest and authoritative version (because $other didn't understand what version this is); I too often see patches that apply the same diff to 10 ebuilds for the same package, etc. How this is organized depends on the package: For texlive-core I would definitely use an unversioned include of utility functions & SRC_URI building code. For ffmpeg I'd use major.minor versioned includes since no change that affect ebuild code are made upstream between micro releases. The naming would be self-explanatory: texlive-core-utils.inc and ffmpeg-3.2.inc. Anyone should be able to grasp what the include is about... PS: I take this reply as your offer to do the code backports/forwardports when needed, I'll be sure to poke you when I'll have to :)
How are you going to deal with stabilization workflow? Are you going to fork the includes anytime you do a new change that could affect the stable version? Or do you prefer just letting everything land straight in stable?
(In reply to Michał Górny from comment #24) > How are you going to deal with stabilization workflow? KEYWORDS dont go into eclasses ? err. eblits sorry :) > Are you going to fork > the includes anytime you do a new change that could affect the stable > version? Or do you prefer just letting everything land straight in stable? Just like with eclasses, it depends on the kind of change at hand. If we're talking about, e.g., a new configure option, a new useflag, a patch etc. that changes very often then this should not have been defined in the eblit in the first place... As the above KEYWORDS thing, it's more about common sense than a PMS thing. For the ffmpeg example above, ffmpeg-x.y.inc would be almost set in stone when some ffmpeg-x.y.z is stable. This is somewhat the implicit rule I've been applying there since ffmpeg-x.y.z+1 is usually fast tracked to stable for sec. vulns.
Joshua, would you prefer having mips-sources eblits inlined or replaced by an eclass? Would you like to do it yourself, or do you prefer if I wrote it and submitted for review?
(In reply to Michał Górny from comment #26) > Joshua, would you prefer having mips-sources eblits inlined or replaced by > an eclass? Would you like to do it yourself, or do you prefer if I wrote it > and submitted for review? I was just working on this yesterday. I doubt you're going to like the solution I've gone with, but it will solve the specific issue and eliminate the PMS violation. It's temporary until I have some free time in the next few months to try and put together an eblits eclass and propose it to -dev or as a feature in an upcoming EAPI. PMS 4.3 says the following at the bottom: "A package directory may contain other files or directories, whose purpose is not covered by this specification." As such, I've moved the eblits out of $FILESDIR and into an "eblits" folder, removed $FILESDIR, and modified the ebuilds to access the eblits from the new directory. A quick look through the rest of PMS does not seem to indicate this is wrong in any way. Basic 'ebuild' and 'emerge' commands from my local repo seem fine with this change, and 'repoman full' finds no issues. The only side-effects noted are that the eblit files are no longer included in the Manifest file, and I haven't yet figured out how to test metadata cache regeneration on just that specific package directory. If there's a cheat sheet to various QA tests I can run, I'll be more than happy to do so to see if this change exposes any new issues. I probably need to do a binpkg test w/o /usr/portage being available as well. None of this is committed yet. I've got updated kernel ebuilds/patches that are all going along with this change that I am still testing out.
(In reply to Joshua Kinard from comment #27) This is not a contest for finding loopholes in PMS. Yes, the spec says that there can be other files or dirs. For example, there used to be a CVS directory. Also we cannot completely exclude that a future EAPI will require such files, and then older package managers should not error out but simply ignore them. That does not at all imply that ebuilds would be allowed to access such files. How do you even obtain a path to your eblits directory? PMS doesn't provide any variable pointing to the package dir.
(In reply to Ulrich Müller from comment #28) > (In reply to Joshua Kinard from comment #27) > > This is not a contest for finding loopholes in PMS. Yes, the spec says that > there can be other files or dirs. For example, there used to be a CVS > directory. Also we cannot completely exclude that a future EAPI will require > such files, and then older package managers should not error out but simply > ignore them. I'm not trying to make it into one, either. A legitimate issue was raised about these ebuilds I maintain, but less-than-optimal solutions were proposed. Inlining means my workflow will take longer as I have to keep duplicated common code in sync more often, and an eclass means I can't maintain versioned sub-components anymore. So, I looked for, and found, a third solution that solves the root issue of this specific bug ($FILESDIR access) and which also lets me maintain my existing workflow with minimal disruption. > That does not at all imply that ebuilds would be allowed to access such > files. How do you even obtain a path to your eblits directory? PMS doesn't > provide any variable pointing to the package dir. That's why I'm asking if this solution works for you guys, and why I did not immediately commit my changes. I had a snow day yesterday, hence the opportunity to take a crack at this. I've ran several tests to make sure nothing else illegal stands out, and I did a quick read over PMS to see if there were any other prohibitions against accessing files in the package directory other than FILESDIR, and I can't immediately find anything. If I missed something, please let me know. As for a variable that points back to the package directory, yes, I noticed that myself. We should add such a variable to the next version of PMS and Portage, in my opinion. What I did instead was to parse the ${EBUILD} variable with parameter expansion that removes a matching suffix: # Eblits directory EBLITSDIR="${EBUILD%/*.ebuild}/eblits"
(In reply to Joshua Kinard from comment #29) > What I did instead was to parse the ${EBUILD} variable with parameter > expansion that removes a matching suffix: https://projects.gentoo.org/pms/6/pms.html#x1-14500011.3.3.16 11.3.3.16 Reserved Commands and Variables Except where documented otherwise, all functions and variables that contain any of the following strings (ignoring case) are reserved for package manager use and may not be used or relied upon by ebuilds: [...] ebuild [...]
...and before you start playing with ${BASH_SOURCE}, take the following into consideration: 1) you will break binary packages. 2) QA will not let you commit any non-standard files to the repository. 3) any kind of external code sourcing outside scope of PMS will most likely be banned in a few weeks. 4) we could have fixed this half a year ago if you just used existing tools instead of constantly trying to reinvent the wheel. You are blocking a Portage patch for over half a year. So please stop trying to find workarounds and loopholes, wasting your and everyone else's time and decide: eclass or inline?
(In reply to Michał Górny from comment #31) > ...and before you start playing with ${BASH_SOURCE}, take the following into > consideration: > > 1) you will break binary packages. Can you (or someone) provide me a command I can use to test for this? I've asked several times now for test cases that I can use to check things, and no one has yet to provide any. > 2) QA will not let you commit any non-standard files to the repository. I'd suggest making sure repoman checks for this then, because as I highlighted above, "repoman full" threw no errors. > 3) any kind of external code sourcing outside scope of PMS will most likely > be banned in a few weeks. But it is not banned now. So this point is invalid as it applies to this discussion at this point in time. > 4) we could have fixed this half a year ago if you just used existing tools > instead of constantly trying to reinvent the wheel. You are blocking a > Portage patch for over half a year. And *this* is the kind of important point you could have made a LONG time ago. I pointed out in Comment #11 that you, nor anyone else, have offered a technical argument on why eblits are frowned upon. The primary arguments put forth have either been programmer's opinion ("ugly", "repetitive") or PMS violations that are either cosmetic in origin or already addressed. I looked back on this bug and on the parent tracking bug, and I see NO reference to this Portage patch that's supposedly being blocked. What is this patch and why are eblits blocking it? What's the bug number? That's somewhat important information to know and might have changed my thinking had it been presented a bit earlier. It seems like something you should add to this bug and to the other eblit bugs as a blocker or at least to the parent tracking bug. Otherwise, people who stumble across this bug in the future are going to assume it's purely an argument between programmers over coding tricks. > So please stop trying to find workarounds and loopholes, wasting your and > everyone else's time and decide: eclass or inline? I would argue the time is not wasted. I've asked for help in doing this properly in an eblits eclass, and you've demurred. I've asked for technical-based arguments against, and was given subjective opinion instead. Even the log file from #gentoo-pms contains little on technical grounds, only that there is a perception that adding support for per-package eclasses would not be worth the time invested. So blame isn't on me alone. That said, I'll switch to an eclass, but it's not going to happen until this weekend or next. You caught me at a bad month (frankly, a bad year) where I've had too many other things on my plate and being this is a completely cosmetic issue, has the lowest of low priorities. Furthermore, the tone you've taken in this issue has been less-than encouraging. It would be helpful if you'd be more willing to work with other developers on solving these issues in the future, rather than simply dictating policy.
(In reply to Joshua Kinard from comment #32) > (In reply to Michał Górny from comment #31) > > ...and before you start playing with ${BASH_SOURCE}, take the following into > > consideration: > > > > 1) you will break binary packages. > > Can you (or someone) provide me a command I can use to test for this? I've > asked several times now for test cases that I can use to check things, and > no one has yet to provide any. Since binary packages are not really specified, it's hard to figure out proper test cases. For a start, ensure that: a. emerge -1vB foo [i.e. building binary package only] works, b. emerge -1vK foo [i.e. installing binary package] works afterwards. You should also test the equivalent functions of paludis. However, I don't know the commands there. > > 2) QA will not let you commit any non-standard files to the repository. > > I'd suggest making sure repoman checks for this then, because as I > highlighted above, "repoman full" threw no errors. repoman is not an ultimate answer whether something is legitimate, and repoman certainly doesn't cover corner cases of doing things completely outside the policy. > > 3) any kind of external code sourcing outside scope of PMS will most likely > > be banned in a few weeks. > > But it is not banned now. So this point is invalid as it applies to this > discussion at this point in time. Do you seriously believe it's worthwhile to deploy an ugly hack to work around policy just to have it banned in one week from now? Because as I see it, this sounds like abuse of procedure -- i.e. forcing something in knowingly before the policy prohibits that. > > 4) we could have fixed this half a year ago if you just used existing tools > > instead of constantly trying to reinvent the wheel. You are blocking a > > Portage patch for over half a year. > > And *this* is the kind of important point you could have made a LONG time > ago. I pointed out in Comment #11 that you, nor anyone else, have offered a > technical argument on why eblits are frowned upon. The primary arguments > put forth have either been programmer's opinion ("ugly", "repetitive") or > PMS violations that are either cosmetic in origin or already addressed. > > I looked back on this bug and on the parent tracking bug, and I see NO > reference to this Portage patch that's supposedly being blocked. What is > this patch and why are eblits blocking it? What's the bug number? That's > somewhat important information to know and might have changed my thinking > had it been presented a bit earlier. It seems like something you should add > to this bug and to the other eblit bugs as a blocker or at least to the > parent tracking bug. Otherwise, people who stumble across this bug in the > future are going to assume it's purely an argument between programmers over > coding tricks. The Portage patch is the one that catches invalid references to FILESDIR in global scope. It is no different from the points already raised and disregarded as you as 'cosmetic'. > > So please stop trying to find workarounds and loopholes, wasting your and > > everyone else's time and decide: eclass or inline? > > I would argue the time is not wasted. I've asked for help in doing this > properly in an eblits eclass, and you've demurred. I've asked for > technical-based arguments against, and was given subjective opinion instead. > Even the log file from #gentoo-pms contains little on technical grounds, > only that there is a perception that adding support for per-package eclasses > would not be worth the time invested. So blame isn't on me alone. > > That said, I'll switch to an eclass, but it's not going to happen until this > weekend or next. You caught me at a bad month (frankly, a bad year) where > I've had too many other things on my plate and being this is a completely > cosmetic issue, has the lowest of low priorities. Furthermore, the tone > you've taken in this issue has been less-than encouraging. It would be > helpful if you'd be more willing to work with other developers on solving > these issues in the future, rather than simply dictating policy. It would be helpful if I had all the time in the world and lots of money to work with every single developer that believes he is special and deserves to have complex solutions especially for a single package he maintains that sums up to two ebuilds and to which he commits two times a year.
Furthermore, I should point out that new eclass are subject to gentoo-dev@ review. Considering that the deadline passes in 8 days, you should submit it for review tomorrow.
(In reply to Michal Górny from comment #33) > (In reply to Joshua Kinard from comment #32) > > (In reply to Michal Górny from comment #31) > > > ...and before you start playing with ${BASH_SOURCE}, take the following into > > > consideration: > > > > > > 1) you will break binary packages. > > > > Can you (or someone) provide me a command I can use to test for this? I've > > asked several times now for test cases that I can use to check things, and > > no one has yet to provide any. > > Since binary packages are not really specified, it's hard to figure out > proper test cases. For a start, ensure that: > > a. emerge -1vB foo [i.e. building binary package only] works, > > b. emerge -1vK foo [i.e. installing binary package] works afterwards. > > You should also test the equivalent functions of paludis. However, I don't > know the commands there. I only have the time to verify that Portage works on these systems. It's bad enough I've got new kernel issues to debug and am getting no help from other kernel developers either. > > > 3) any kind of external code sourcing outside scope of PMS will most likely > > > be banned in a few weeks. > > > > But it is not banned now. So this point is invalid as it applies to this > > discussion at this point in time. > > Do you seriously believe it's worthwhile to deploy an ugly hack to work > around policy just to have it banned in one week from now? Because as I see > it, this sounds like abuse of procedure -- i.e. forcing something in > knowingly before the policy prohibits that. And when are you going to knock it off with subjective/personal opinions about someone else's code? If the only basis for your argument against is that it is "ugly", then how can I honestly hope to ever convince you otherwise? Assume for a moment I do end up putting together an eblits eclass one day. Going by your attitude right now, it seems like it would be an utterly pointless exercise because you consider the very idea "ugly" and I assume would likely move to block the adoption outright. If you don't like the code, then help make it better. Right now, you're throwing around ultimatums and using your position on QA to enforce what is an *extremely* pedantic corner case of PMS violation in the tree. I've tried asking for options, but the only two options it seems you'll remotely accept is to inline the bloody code or move it into an eclass, both options that I've already stated are unacceptable at the present time. Right now, I think I am going to look into setting up a developer overlay, moving mips-sources there, and removing it from the tree until I have time to actually work on the ebuilds themselves and maybe refactor the code before adding it back. As it currently stands, I cannot meet your self-imposed deadline, and, frankly, your attitude is not at all encouraging and makes me not even want to try to work with you. > > > 4) we could have fixed this half a year ago if you just used existing tools > > > instead of constantly trying to reinvent the wheel. You are blocking a > > > Portage patch for over half a year. > > > > And *this* is the kind of important point you could have made a LONG time > > ago. I pointed out in Comment #11 that you, nor anyone else, have offered a > > technical argument on why eblits are frowned upon. The primary arguments > > put forth have either been programmer's opinion ("ugly", "repetitive") or > > PMS violations that are either cosmetic in origin or already addressed. > > > > I looked back on this bug and on the parent tracking bug, and I see NO > > reference to this Portage patch that's supposedly being blocked. What is > > this patch and why are eblits blocking it? What's the bug number? That's > > somewhat important information to know and might have changed my thinking > > had it been presented a bit earlier. It seems like something you should add > > to this bug and to the other eblit bugs as a blocker or at least to the > > parent tracking bug. Otherwise, people who stumble across this bug in the > > future are going to assume it's purely an argument between programmers over > > coding tricks. > > The Portage patch is the one that catches invalid references to FILESDIR in > global scope. It is no different from the points already raised and > disregarded as you as 'cosmetic'. How long ago did you submit this patch? I did a quick search of my portage-dev folder and the web archives back to Jan 2016 for any reference to "FILESDIR", and the patch you proposed a few hours ago seems to be the only match. Was there an earlier version under a different title or name? You said this bug has been blocking a patch for 6+ months. Yet the FILESDIR patch appears to only be a few hours old, and written exclusively to block eblits-consuming packages. If this is correct, this is behaviour that borders on vindictive more than actually solving a real-world problem. > > > So please stop trying to find workarounds and loopholes, wasting your and > > > everyone else's time and decide: eclass or inline? > > > > I would argue the time is not wasted. I've asked for help in doing this > > properly in an eblits eclass, and you've demurred. I've asked for > > technical-based arguments against, and was given subjective opinion instead. > > Even the log file from #gentoo-pms contains little on technical grounds, > > only that there is a perception that adding support for per-package eclasses > > would not be worth the time invested. So blame isn't on me alone. > > > > That said, I'll switch to an eclass, but it's not going to happen until this > > weekend or next. You caught me at a bad month (frankly, a bad year) where > > I've had too many other things on my plate and being this is a completely > > cosmetic issue, has the lowest of low priorities. Furthermore, the tone > > you've taken in this issue has been less-than encouraging. It would be > > helpful if you'd be more willing to work with other developers on solving > > these issues in the future, rather than simply dictating policy. > > It would be helpful if I had all the time in the world and lots of money to > work with every single developer that believes he is special and deserves to > have complex solutions especially for a single package he maintains that > sums up to two ebuilds and to which he commits two times a year. Again, I never made any claim that I am somehow special, so I'd HIGHLY recommend that you drop that assertion. You have yet to offer a *single* technical argument against the specific eblit code in mips-sources, and I already stated that the specific case of the binpkgs should be addressed with the guard variable added a few years ago. The only real issue here is a violation on paper, to which I offered up a workaround, but you've batted that away with yet another paper violation. So as far as I can tell, your only concern here is that I cave to your demands to remove what is in your eyes "ugly code" and nothing else. This is toxic and not how development works in Gentoo. As for my commit frequency, that's due in part to re-writing/refactoring whole chunks of platform code in arch/mips/sgi-ip27 over the last year in the kernel, as well as maintaining the IP30 patches. I am trying to get things organized to send stuff upstream, but now I have to double back and redo the PCI code for these systems before that can happen. So you'll forgive me if I have a lack of motivation on fixing a purely pedantic issue right now just to make you happy.
I can still write the patch for you if you answer the simple binary question.
Created attachment 467384 [details] Completed eblit.eclass implementation I have finished the eblit.eclass implementation, which moves all of the eblit loading/sourcing functions into itself and removes their duplication in each ebuild. The guard variable and "load_eblit_funcs" implementation needs to remain in the ebuild, however, since it is in this function where the different eblit versions are called from. I have tested this implementation via binpkgs: - emerge -1vB =sys-kernel/mips-sources-4.10.3 - emerge -1vK =sys-kernel/mips-sources-4.10.3 And can observe no ill effects. Even when defining specific USE flags to enable the external machine patches I maintain, the binpkg remembers that USE change and displays the correct machine-specific messages. I tried renaming $PORTDIR to something else on my filesystem to mimic the effect of the portage repo not being present, but emerge won't work with that, so a more robust testcase will need to be devised later to make sure that binpkgs made from eblit-consuming ebuilds are fully autonomous in the absence of $EBLITSDIR. I also ran egencache and can observe no ill effects: # egencache --repo=local --update sys-kernel/mips-sources # echo $? 0 The only glaring issue is that we still need to mangle a variable to point at the eblits directory in the package directory. Since PMS 11.3.3.16 states that we cannot mangle the ${EBUILD} variable, because it contains the string "ebuild", I have instead chosen to mangle FILESDIR, which remains defined even if the package directory does not contain an actual "files" subdirectory. This is a TEMPORARY workaround until the appropriate patches can be submitted to, or bugs filed for, PMS, Portage, and any other external package managers. This mangling is defined as: EBLITSDIR="${FILESDIR%/files}/eblits" I will attach a revised mips-sources ebuild that highlights the changes needed to use this proposed new eclass.
Created attachment 467386 [details] Preliminary mips-sources-4.10.3 ebuild highlighting use of eblit.eclass The attached ebuild is a preliminary version for the 4.10.3 kernel that applies a diff from linux-mips.org, followed by a series of external patches for various SGI machines. It is incomplete because I still have potential spinlocking bugs to address on IP27/IP30 before I officially commit anything, assuming no one finds any overt issues with the eblit.eclass implementation and usage. The following distfiles are needed for anyone wanting to test this ebuild out for themselves: linux-mips git diff: http://dev.gentoo.org/~kumba/mips/misc/mipsgit-4.10.0-20170314.diff.xz External machine patches: http://dev.gentoo.org/~kumba/mips/misc/mips-sources-4.10.0-patches-v1.tar.xz Place both tarballs into ${DISTDIR}. Last, the following tarball of eblits should be unpacked into the same package directory as the attached mips-sources-4.10.3 ebuild. It will expand into an "eblits" subdirectory. Then run "repoman manifest" to update the Manifest file (this will, for now, not manifest the eblit files). http://dev.gentoo.org/~kumba/mips/misc/mips-sources-eblits-20170317.tar.xz If I missed anything that should allow an objective observer to test things for themselves to make sure I did not miss something, please let me know.
Since review of the proposed eblit.eclass and corresponding use in the ebuild might take longer than the current deadline on this bug, I'd like to request that the deadline either be removed entirely or extended for at least one month. If no one finds any issues with things in the next few days, then I'll submit the eclass to gentoo-dev for consideration sometime next week, time permitting. Follow-on patches to update PMS to support an "EBLITSDIR" will happen afterwards, unless QA has any advice on other ways to proceed. If such a change requires a new EAPI, then I can queue that for whatever the next proposed EAPI is. It also shouldn't be too hard to add the appropriate implementation into Portage, likely by re-using code that implements FILESDIR (but this is an uneducated guess at the moment). Other package managers I am uncertain about. They might have to implement EBLITSDIR on their own if it is accepted into PMS. Implementation as an eclass seems to be a fairly straightforward way of implementing eblit functionality with minimal disruption. Previous attempts involved GLEPs that were never finished and discussions on the topic that stalled. This way at least gets things off the ground and keeps the door open for the capability to expand into something more if there is significant interest.
Created attachment 467388 [details] Completed eblit.eclass implementation v2 eblit.eclass v2 that fixes minor typos in the comment documentation that I made.
(In reply to Joshua Kinard from comment #37) > The only glaring issue is that we still need to mangle a variable to point > at the eblits directory in the package directory. Since PMS 11.3.3.16 > states that we cannot mangle the ${EBUILD} variable, because it contains the > string "ebuild", I have instead chosen to mangle FILESDIR, which remains ...which is yet another proof that you are trying hard to work around policies. > defined even if the package directory does not contain an actual "files" > subdirectory. This is a TEMPORARY workaround until the appropriate patches > can be submitted to, or bugs filed for, PMS, Portage, and any other external > package managers. This mangling is defined as: Package managers won't be defining absurd variable to account for non-PMS features. > EBLITSDIR="${FILESDIR%/files}/eblits" This evaluates to /var/tmp/portage/foo/bar-1/eblits on my system. Mostly because you are violating the first rule about all paths defined by PMS and are accessing the tree above tha provided path. Here's a blog post specifically written to help you understand PMS: https://blogs.gentoo.org/mgorny/2017/03/17/why-you-cant-rely-on-repository-format-pms/ (In reply to Joshua Kinard from comment #39) > Since review of the proposed eblit.eclass and corresponding use in the > ebuild might take longer than the current deadline on this bug, I'd like to > request that the deadline either be removed entirely or extended for at > least one month. If no one finds any issues with things in the next few > days, then I'll submit the eclass to gentoo-dev for consideration sometime > next week, time permitting. No. You are attempting to stall QA work by resubmitting the same thing over and over again, and just changing it a little in an attempt to work around policies. > Follow-on patches to update PMS to support an "EBLITSDIR" will happen > afterwards, unless QA has any advice on other ways to proceed. If such a > change requires a new EAPI, then I can queue that for whatever the next > proposed EAPI is. It also shouldn't be too hard to add the appropriate > implementation into Portage, likely by re-using code that implements > FILESDIR (but this is an uneducated guess at the moment). Other package > managers I am uncertain about. They might have to implement EBLITSDIR on > their own if it is accepted into PMS. Yes, this requires a new EAPI. Which means the existing ebuilds need to be fixed for all current EAPIs. This pretty much reminds me the case of dynamic SLOTs where explicit breakage was left in Gentoo for a few years because of 'planned' new EAPI features which never reached even a patch.
Furthermore, please test your work with the following Portage patch applied: https://github.com/gentoo/portage/pull/140
(In reply to Michal Górny from comment #41) > (In reply to Joshua Kinard from comment #37) > > The only glaring issue is that we still need to mangle a variable to point > > at the eblits directory in the package directory. Since PMS 11.3.3.16 > > states that we cannot mangle the ${EBUILD} variable, because it contains the > > string "ebuild", I have instead chosen to mangle FILESDIR, which remains > > ...which is yet another proof that you are trying hard to work around > policies. No, it's proof that I am trying to work with you to implement this the right way. Yes, that means CURRENTLY, a hack has to be used. If this is unacceptable, then that's fine, but for the love of Cthulhu, knock it off with the assertion that I am "trying to work around policies". Because if I seriously wanted to work around policies, I could flat ignore ignore you and this bug, but that obviously wouldn't be very conducive to getting anything done. So either please help me out here or turn this over someone else on QA that will. > > defined even if the package directory does not contain an actual "files" > > subdirectory. This is a TEMPORARY workaround until the appropriate patches > > can be submitted to, or bugs filed for, PMS, Portage, and any other external > > package managers. This mangling is defined as: > > Package managers won't be defining absurd variable to account for non-PMS > features. > > > EBLITSDIR="${FILESDIR%/files}/eblits" > > This evaluates to /var/tmp/portage/foo/bar-1/eblits on my system. Mostly > because you are violating the first rule about all paths defined by PMS and > are accessing the tree above tha provided path. This is the kind of testing I need to know about. I can't test every possible combination on my system. Feedback like this is good, but I reiterate again, help me do the right way. If I have to submit a patch for PMS first or request a new EAPI, then point me to the right resources so I can do it right. > Here's a blog post specifically written to help you understand PMS: > https://blogs.gentoo.org/mgorny/2017/03/17/why-you-cant-rely-on-repository- > format-pms/ > > (In reply to Joshua Kinard from comment #39) > > Since review of the proposed eblit.eclass and corresponding use in the > > ebuild might take longer than the current deadline on this bug, I'd like to > > request that the deadline either be removed entirely or extended for at > > least one month. If no one finds any issues with things in the next few > > days, then I'll submit the eclass to gentoo-dev for consideration sometime > > next week, time permitting. > > No. You are attempting to stall QA work by resubmitting the same thing over > and over again, and just changing it a little in an attempt to work around > policies. Wrong. I'm trying to work with you here by proposing alternate solutions, but because you've got a blatantly obvious bias against the idea entirely, it seems like this will end up going no where and the ONLY solution you'll remotely accept is one of the two defined by yourself. If this is an incorrect assumption, then please explain your position and give me an idea of whether this idea is utterly dead for all time or not. If you actually are open to implementing it somehow, then outline for me the next steps you would like to see me take to work towards it, or point me towards any existing documentation that does so. > > Follow-on patches to update PMS to support an "EBLITSDIR" will happen > > afterwards, unless QA has any advice on other ways to proceed. If such a > > change requires a new EAPI, then I can queue that for whatever the next > > proposed EAPI is. It also shouldn't be too hard to add the appropriate > > implementation into Portage, likely by re-using code that implements > > FILESDIR (but this is an uneducated guess at the moment). Other package > > managers I am uncertain about. They might have to implement EBLITSDIR on > > their own if it is accepted into PMS. > > Yes, this requires a new EAPI. Which means the existing ebuilds need to be > fixed for all current EAPIs. This pretty much reminds me the case of dynamic > SLOTs where explicit breakage was left in Gentoo for a few years because of > 'planned' new EAPI features which never reached even a patch. Then what is the procedure to request a new EAPI for PMS?
(In reply to Michał Górny from comment #42) > Furthermore, please test your work with the following Portage patch applied: > > https://github.com/gentoo/portage/pull/140 github isn't letting me fetch a raw patch file of this pull. Is this available somewhere? The current implementation of eblit.eclass doesn't touch FILESDIR anymore, so I have my doubts that this patch will actually have an effect.
(In reply to Joshua Kinard from comment #44) > (In reply to Michał Górny from comment #42) > > Furthermore, please test your work with the following Portage patch applied: > > > > https://github.com/gentoo/portage/pull/140 > > github isn't letting me fetch a raw patch file of this pull. Is this > available somewhere? The current implementation of eblit.eclass doesn't > touch FILESDIR anymore, so I have my doubts that this patch will actually > have an effect. https://github.com/gentoo/portage/pull/140.patch (In reply to Joshua Kinard from comment #43) > Then what is the procedure to request a new EAPI for PMS? I'm planning to write a draft for wiki soon, but long story short: a. open future EAPI bug (if there isn't one already), b. do an RFC on gentoo-dev@, c. write a patch for portage, d. write a patch for PMS. in any order. Once you have all that done (and patches approved), you can add the item to the agenda when Council handles new EAPI. And ofc you don't have to do everything yourself, if you can convince someone to do it for you. That said, the first step towards that would be to stop trying to find workarounds and use accepted practices for current ebuilds. Then start over.
(In reply to Joshua Kinard from comment #39) > Follow-on patches to update PMS to support an "EBLITSDIR" will happen > afterwards, unless QA has any advice on other ways to proceed. I think it is very unlikely that support for any variant of eblits will be added to PMS. If there is demand for shared code for the ebuilds of one package, then IMHO the previously proposed per-package eclasses are the way to go (but see my comment #20). <QA hat> Until then, you should either move the eblits code to a dedicated eclass, or inline it into the ebuilds. </QA hat>
(In reply to Ulrich Müller from comment #46) > <QA hat> > Until then, you should either move the eblits code to a dedicated eclass, or > inline it into the ebuilds. > </QA hat> To clarify, by "eblits code" I mean the code of the eblits themselves, not any support code for their loading.
(In reply to Michał Górny from comment #45) > (In reply to Joshua Kinard from comment #44) > > (In reply to Michał Górny from comment #42) > > > Furthermore, please test your work with the following Portage patch applied: > > > > > > https://github.com/gentoo/portage/pull/140 > > > > github isn't letting me fetch a raw patch file of this pull. Is this > > available somewhere? The current implementation of eblit.eclass doesn't > > touch FILESDIR anymore, so I have my doubts that this patch will actually > > have an effect. > > https://github.com/gentoo/portage/pull/140.patch Thanks. This does break the eclass I proposed because it appears your change copies the contents of $FILESDIR to $PORTAGE_TMPDIR and changes the value of $FILESDIR to point there. That explains your observation in your earlier comment. What is the aim of this change? Is it done specifically to break eblit-using packages or are there other motivations? I see no updates on the portage-dev ML thread, so what's the impetus behind this change? What's the history? I can make things work again if I go back to frobbing the $EBUILD variable, FYI. So if the end goal is to outright deny any kind of non-ebuild access to PKGDIR other than copying FILESDIR, it needs more work (but that almost seems like something that sandbox would be better suited for, IMHO). > (In reply to Joshua Kinard from comment #43) > > Then what is the procedure to request a new EAPI for PMS? > > I'm planning to write a draft for wiki soon, but long story short: > > a. open future EAPI bug (if there isn't one already), > > b. do an RFC on gentoo-dev@, > > c. write a patch for portage, > > d. write a patch for PMS. > > in any order. Once you have all that done (and patches approved), you can > add the item to the agenda when Council handles new EAPI. And ofc you don't > have to do everything yourself, if you can convince someone to do it for you. This is probably why no one's bothered to do an eblits eclass. I was honestly half-expecting to be chastised for not searching the wiki for a guide that has been around for several years, but to find out this specific process isn't fully documented? > That said, the first step towards that would be to stop trying to find > workarounds and use accepted practices for current ebuilds. Then start over. I don't know the exact start date of when Mike added eblits to the tree, but odds are incredibly likely they were done so at a time when policy either was not defined on this point, or was not specific enough that eblits were not considered a violation then. That's why I don't consider them as such. I need you to understand this. It's not like eblits were added, as far as I know, after PMS limited the scope of $FILESDIR or just so several of us could stick it to QA or anything. On top of that, the reason no one's tried to change anything is because the implementation works fine as-is, at least as far as mips-sources is concerned. It looks like the boilerplate code in each consuming package for eblits was done a little differently, though I think mips-sources' version is the most robust due to it not apparently suffering from the binpkg issue. Going by the #gentoo-pms log file, it really seems like QA's effort to purge eblits started as a curious discussion on a lazy afternoon. I understand you guys are just doing your jobs and working to make sure that PMS is followed, but it would honestly help a lot if you'd pause for a moment and look at it from my point of view for a few moments. Here's an ebuild that's been doing something that, functionally-speaking, works to get the job done, and doing so for a number of years. Because nothing was functionally broken, there was no drive to change the implementation. Yes, things elsewhere in the tree changed and the ebuilds just never got updated to adhere to those changes. None of our QA tools were ever updated to flag the implementation as wrong, either. So we kept using the implementation. Some of us really do isolate ourselves to a small subset of the Project and just don't pay attention to what's happening elsewhere. I am not alone in this regard. Hopefully that helps you see my point of view on things. (In reply to Ulrich Müller from comment #46) > (In reply to Joshua Kinard from comment #39) > > Follow-on patches to update PMS to support an "EBLITSDIR" will happen > > afterwards, unless QA has any advice on other ways to proceed. > > I think it is very unlikely that support for any variant of eblits will be > added to PMS. If there is demand for shared code for the ebuilds of one > package, then IMHO the previously proposed per-package eclasses are the way > to go (but see my comment #20). > > <QA hat> > Until then, you should either move the eblits code to a dedicated eclass, or > inline it into the ebuilds. > </QA hat> As far as I am concerned, the words "eblit" and "per-package eclass" mean the same thing to me. "eblit" is just quicker to type out. I am not wedded to the current eblit implementation at all, and honestly would have no problem helping with a completely different implementation under a different name. The thing that I care about most is not duplicating chunks of code per-ebuild that I have to then spend more effort maintaining, outside of the multitude of kernel patches I already maintain. Call it my OCD if you will, but I really, really detest duplicating code or having to work with duplicated code. I am only proposing the current eblit implementation as an eclass because it accomplishes the following objectives: 1. Minimizes changes required to package managers and PMS. 2. Encapsulates the base functionality in an eclass that can be used by multiple different packages (thus fulfilling the concept of an eclass). 3. Can be implemented somewhat quickly versus writing a new implementation from scratch. The only change I see right now that requires that PMS be modified for eblits is to add a definition for $EBLITSDIR and have the package managers support it. I changed to doing it this way because of the prohibition on using $FILESDIR for anything other than package support files. I have a sense that a proposition to change that part of PMS would meet stiff resistance. Equally, I can understand that changing a broad-level package manager specification to support a tiny new variable seems like a lot of churn, but the eblit files have to be stored //somewhere// if they're to be used. So my question is, are there any better ways to store common/shared pieces of ebuild code that doesn't seem appropriate for a full eclass? Maybe the definition of the "eclass" directory could be extended to support subdirectories that follow CATEGORY/PKG format and store things there? You'd lose the ability to manifest the files and have to commit with vanilla git versus repoman, but that's a larger issue common to the other top-level, non-pkg dirs. I think PMS would still need to be modified, however, because I don't think the current definition of the "eclass" directory supports subdirectories.
(In reply to Joshua Kinard from comment #48) > (In reply to Michał Górny from comment #45) > > (In reply to Joshua Kinard from comment #44) > > > (In reply to Michał Górny from comment #42) > > > > Furthermore, please test your work with the following Portage patch applied: > > > > > > > > https://github.com/gentoo/portage/pull/140 > > > > > > github isn't letting me fetch a raw patch file of this pull. Is this > > > available somewhere? The current implementation of eblit.eclass doesn't > > > touch FILESDIR anymore, so I have my doubts that this patch will actually > > > have an effect. > > > > https://github.com/gentoo/portage/pull/140.patch > > Thanks. This does break the eclass I proposed because it appears your > change copies the contents of $FILESDIR to $PORTAGE_TMPDIR and changes the > value of $FILESDIR to point there. That explains your observation in your > earlier comment. > > What is the aim of this change? Is it done specifically to break > eblit-using packages or are there other motivations? I see no updates on > the portage-dev ML thread, so what's the impetus behind this change? What's > the history? The aim is to catch ebuilds accessing FILESDIR outside src_*, i.e. the number of violations involving reading files there in pkg_* (mostly READMEs) and globs in global PATCHES array. In other words, things that are going to break binary packages horribly. > I can make things work again if I go back to frobbing the $EBUILD variable, > FYI. So if the end goal is to outright deny any kind of non-ebuild access > to PKGDIR other than copying FILESDIR, it needs more work (but that almost > seems like something that sandbox would be better suited for, IMHO). The goal is to catch common mistakes, not prevent people from intentionally breaking stuff. For the latter, we have the ban hammer. > > (In reply to Joshua Kinard from comment #43) > > > Then what is the procedure to request a new EAPI for PMS? > > > > I'm planning to write a draft for wiki soon, but long story short: > > > > a. open future EAPI bug (if there isn't one already), > > > > b. do an RFC on gentoo-dev@, > > > > c. write a patch for portage, > > > > d. write a patch for PMS. > > > > in any order. Once you have all that done (and patches approved), you can > > add the item to the agenda when Council handles new EAPI. And ofc you don't > > have to do everything yourself, if you can convince someone to do it for you. > > This is probably why no one's bothered to do an eblits eclass. I was > honestly half-expecting to be chastised for not searching the wiki for a > guide that has been around for several years, but to find out this specific > process isn't fully documented? It's not like anybody has been really interested in helping PMS team out. > (In reply to Ulrich Müller from comment #46) > I am only proposing the current eblit implementation as an eclass because it > accomplishes the following objectives: > > 1. Minimizes changes required to package managers and PMS. > 2. Encapsulates the base functionality in an eclass that can be used by > multiple different packages (thus fulfilling the concept of an eclass). > 3. Can be implemented somewhat quickly versus writing a new implementation > from scratch. So to summarize, you believe that adding a regular eclass for one package is bad. But you don't mind adding a complex hacking eclass + ugly PMS hack + modifying three packages to support eblits that will be used in one package that consists of two ebuilds. Makes lots of sense. Everything just not to change the status quo from 10 years ago. This whole discussion is outside the scope of this bug. Please fix the problem immediately, in a legitimate, non-hacky way or tell us how you prefer it done. Any further eblit discussion can occur when you have some proposal. It won't happen overnight, so there's really no point in making me even more angry.
(In reply to Joshua Kinard from comment #48) > So my question is, are there any better ways to store common/shared pieces > of ebuild code that doesn't seem appropriate for a full eclass? Maybe the > definition of the "eclass" directory could be extended to support > subdirectories that follow CATEGORY/PKG format and store things there? Why do you think that this isn't appropriate for a regular eclass? From a functional point of view it makes no difference if that eclass (or whatever it is to be called) lives in the top-level eclass dir, in eclass/${CATEGORY}/${PN}, or in the package directory. I wouldn't even mind if there was more than one version of such an eclass in the regular eclass directory. > You'd lose the ability to manifest the files and have to commit with vanilla > git versus repoman, but that's a larger issue common to the other top-level, > non-pkg dirs. I think PMS would still need to be modified, however, because > I don't think the current definition of the "eclass" directory supports > subdirectories. Extending PMS for per-package eclasses is a different discussion, and also on a longer time scale than the issue at hand. Until then, ebuilds must follow the existing specification. Maybe I should also note that already in the first council-approved PMS version (March 2009) FILESDIR was only legal in src_* phases, and not guaranteed to be consistent between phases: https://projects.gentoo.org/pms/2/pms.pdf (page 42 there)
(In reply to Michal Górny from comment #49) > (In reply to Joshua Kinard from comment #48) > > (In reply to Michal Górny from comment #45) > > > (In reply to Joshua Kinard from comment #44) > > > > (In reply to Michal Górny from comment #42) > > > > > Furthermore, please test your work with the following Portage patch applied: > > > > > > > > > > https://github.com/gentoo/portage/pull/140 > > > > > > > > github isn't letting me fetch a raw patch file of this pull. Is this > > > > available somewhere? The current implementation of eblit.eclass doesn't > > > > touch FILESDIR anymore, so I have my doubts that this patch will actually > > > > have an effect. > > > > > > https://github.com/gentoo/portage/pull/140.patch > > > > Thanks. This does break the eclass I proposed because it appears your > > change copies the contents of $FILESDIR to $PORTAGE_TMPDIR and changes the > > value of $FILESDIR to point there. That explains your observation in your > > earlier comment. > > > > What is the aim of this change? Is it done specifically to break > > eblit-using packages or are there other motivations? I see no updates on > > the portage-dev ML thread, so what's the impetus behind this change? What's > > the history? > > The aim is to catch ebuilds accessing FILESDIR outside src_*, i.e. the > number of violations involving reading files there in pkg_* (mostly READMEs) > and globs in global PATCHES array. In other words, things that are going to > break binary packages horribly. Okay, that sounds reasonable. But I will point out that the eblit use in mips-sources, based on current tests, does not break binary packages. I want that acknowledged on the record, unless there's a more concise test that proves otherwise. You keep ignoring this fact, and it's really irritating that you continue to insist that the mips-sources implementation of eblits is broken code, when it clearly is not. > > I can make things work again if I go back to frobbing the $EBUILD variable, > > FYI. So if the end goal is to outright deny any kind of non-ebuild access > > to PKGDIR other than copying FILESDIR, it needs more work (but that almost > > seems like something that sandbox would be better suited for, IMHO). > > The goal is to catch common mistakes, not prevent people from intentionally > breaking stuff. For the latter, we have the ban hammer. I am going to assume the statement "not prevent people from intentionally breaking stuff" is a shot at me again, despite the fact I've asked you repeatedly to stop. I stated the fact that changing back to frobbing the $EBUILD variable defeated your patch so that if you wanted to modify the patch to catch that, you could. I was not indicating any intent to "intentionally break stuff". So I will ask one last time, back off of the personal accusations and attacks. > > (In reply to Ulrich Müller from comment #46) > > I am only proposing the current eblit implementation as an eclass because it > > accomplishes the following objectives: > > > > 1. Minimizes changes required to package managers and PMS. > > 2. Encapsulates the base functionality in an eclass that can be used by > > multiple different packages (thus fulfilling the concept of an eclass). > > 3. Can be implemented somewhat quickly versus writing a new implementation > > from scratch. > > So to summarize, you believe that adding a regular eclass for one package is > bad. But you don't mind adding a complex hacking eclass + ugly PMS hack + > modifying three packages to support eblits that will be used in one package > that consists of two ebuilds. Makes lots of sense. Everything just not to > change the status quo from 10 years ago. Michal, the point of my proposals was an attempt to turn this bug from the bikeshedding event it's become into a fruitful discussion on development of a feature. It really seems like you are so stuck in your opinion that the eblit code is a hack that you can't see it in any other way. This is underscored by the fact you continue, despite my repeated requests to stop, to refer to either the existing eblit implementation or the proposals I've tried to make as either "ugly" or a "hack". Furthermore, what does this say about your work in other areas of Gentoo? It really seems like you have a tendency to go off and declare some random bit of code as "hacky" or "broken", and then you propose broad-level changes to fix it in a way that you think is correct. Some of the work you've done is indeed pretty awesome (multilib especially), but it really seems to me that when you form an opinion of a specific piece of code or work, you will absolutely not accept any other solution except your own. Development in Gentoo is a team effort. No one person can ever get it all right, and this includes both me and you. Yes, I like my ugly-as-Cthulhu eblit code, but if someone came along and showed me a better way of doing it, I'd drop it in a heartbeat. Merging all of the code I have in the individual eblit files is, to me, ugly and hack'ish, which is why I am so vehemently set against doing it, and it really gets under my skin that you steadfastly refuse to work with me on this in any way other than the two options you and you alone have set forth. I threw out a couple of ideas, including, yes, "a complex hacking eclass" and "ugly PMS hack". I was hoping to get constructive feedback back. What do I get instead? More insults. Furthermore, you've gone as far as to create a patch to Portage that *specifically* attacks this feature. I don't know what your issue is with the eblit code, but it certainly goes far beyond a mere PMS violation. You seem to be full of ideas to improve Gentoo, so if you were the one implementing something akin to an eblit-like function, how would you do it? Seriously, non-sarcastically, please enlighten me. > This whole discussion is outside the scope of this bug. Please fix the > problem immediately, in a legitimate, non-hacky way or tell us how you > prefer it done. Any further eblit discussion can occur when you have some > proposal. It won't happen overnight, so there's really no point in making me > even more angry. I'll fix the ebuilds this weekend by inlining the code, as much as I want to spit on such an abomination. But I am also going to initiate a last-rite of mips-sources for 30 days, setup a developer overlay, and move my work there and run things how I want to run them until I get these stupid machines to work right and maybe get something like eblits or per-package eclasses or whatever to work before I move these ebuilds back into the tree. But that depends entirely on whether or not you or the other members of QA are open to working with me on whatever, if any, modifications to PMS to support per-package eclasses/eblits/whatever. As it stands right now, I really feel like any effort to develop the idea would be ignored or shot down by yourself, especially once you've gotten your way and these eblit bugs are closed. I really hope I get proven wrong on this last assertion. Furthermore, I am going to contact comrel to review the comments on this bug. I have asked you several times to cease with opinionated statements about my work, and you have continued to throw around words like "ugly" or "hacky" or related. I'm not claiming to have been a saint myself on this issue, but I feel that, in light of the past several days, third-party review and advice on how to handle this better is needed for both of our sakes.
(In reply to Ulrich Müller from comment #50) > (In reply to Joshua Kinard from comment #48) > > So my question is, are there any better ways to store common/shared pieces > > of ebuild code that doesn't seem appropriate for a full eclass? Maybe the > > definition of the "eclass" directory could be extended to support > > subdirectories that follow CATEGORY/PKG format and store things there? > > Why do you think that this isn't appropriate for a regular eclass? From a > functional point of view it makes no difference if that eclass (or whatever > it is to be called) lives in the top-level eclass dir, in > eclass/${CATEGORY}/${PN}, or in the package directory. Maybe it's a balance thing, but mips-sources is probably used by all of five people in the world, and I probably pass for two of those five. The way I've got things structured, I've got per-machine informational notes in their own eblit file, src_unpack, pkg_setup, pkg_postinst, etc, all divided up into concise modules. If I were to copy this over wholesale into specific eclasses, then when the casual observer looks into eclass/ and see all of these mips-sources-specific eclasses, it really just doesn't look fair. It also means that, with a flat namespace, I'd have to rename the files to something that would indicate their relationship to mips-sources and that really does border one "ugly", to borrow one of Michal's favourite terms. One of my chief design goals with mips-sources was to minimize its footprint in the tree as much as possible. These ebuilds, before eblits, used to be ~18-20k each in size, and I believe there's a really old bug where I was yelled at for having them that big and them taking up too much space in the tree. That is partly why I switched to eblits, as well as moving the pacthes into a tarball. Both allowed me to decrease the footprint to just a few K per ebuild so that, perceptually, this package would not be unfair in its consumption of treespace. Maybe it's just a sign of things changing and me not keeping up, but when I became a developer in 2003, eclasses were strictly controlled. I even think, but can't be sure, that common code had to be repeated in a certain number of ebuilds before it could even be considered for an eclass. The goal back then was to prevent people from moving arbitrary small snippets of whatever random code they wanted into a top-level folder. That notion has just stuck with me, and so I strongly feel that mips-sources really does not need an eclass solely dedicated to itself. This is why I am rather passionate about the idea of eblits (or any feature that would be package-specific, but allow for code-sharing). Honestly, I probably need to refactor the code in mips-sources anyways to be smaller. I wrote most of it years ago back when my bash-fu was at a beginners stage. If I can make it smaller, I might get over whatever it is with myself other inlining a bunch of it. Problem is that I've been staring at C and Python for so long now, my bash-fu now is probably no better than it was when I wrote mips-sources. > I wouldn't even mind if there was more than one version of such an eclass in > the regular eclass directory. > > > You'd lose the ability to manifest the files and have to commit with vanilla > > git versus repoman, but that's a larger issue common to the other top-level, > > non-pkg dirs. I think PMS would still need to be modified, however, because > > I don't think the current definition of the "eclass" directory supports > > subdirectories. > > Extending PMS for per-package eclasses is a different discussion, and also > on a longer time scale than the issue at hand. Until then, ebuilds must > follow the existing specification. > > Maybe I should also note that already in the first council-approved PMS > version (March 2009) FILESDIR was only legal in src_* phases, and not > guaranteed to be consistent between phases: > https://projects.gentoo.org/pms/2/pms.pdf (page 42 there) As far as I know, and I can only speak for the eblit-loading code in mips-sources, not glibc or autoconf or elsewhere, the eblit functions are only ever sourced during src_* operations. Previous iterations of eblits loaded them in global scope, and that turned up in some of infra's metadata operations, which is why a bug was filed against mips-sources back in ~2011 (I can't find the bug number at the moment), and the issue was deemed resolved by implementing "load_eblit_funcs" and using the guard variable. Michal suggested using -B and -K to verify no errors popped up, and indeed, on both an amd64 machine and my SGI Octane, binary packages appear to be unaffected, so this tells me that eblits are really only sourced during src_* phases. In any other phase, they're simply residing in $FILESDIR and should not actually be accessed. I even ran egencache on that specific ebuild and it didn't throw any kind of error or return a non-zero value. Neither are thorough tests, but I rarely use binpkgs, so I don't know of all of the possible ways to test them to see if the mips-sources eblit code really does have a bug in it or not. It's a bit of a moot point now, since I am forced to abandon the concept entirely for the time being, but I do appreciate your cordiality and feedback on them.
https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=9b62c3f713bdb9ce280e520279818d3298b1717e commit 32dfd81aaf6c26d5564cb0665451b4307290c64e Author: Joshua Kinard <kumba@gentoo.org> Date: Sat Mar 18 14:16:42 2017 -0400 sys-kernel/mips-sources: Update 4.4.43 to 4.4.55 Package-Manager: Portage-2.3.5, Repoman-2.3.2
Thank you.
