As of now, as far as I can see, the only way to not strip a file is to use RESTRICT=strip in an ebuild. However, portage supports STRIP_MASK to explicitly disallow certain files from being stripped. This is quite handy in some ways, especially with some ocaml packages. [more or less copy/paste from bug #203887 for justification] ocamlc -custom uses a nasty trick to have runnable executables withtout needing an interpreter, some packages use it (see ocamlc manpage for details). It embeds an interpreter and the bytecode in one file. Stripping them removes the bytecode... See [1] for a probably better reference. [1] http://caml.inria.fr/pub/ml-archives/caml-list/2002/08/492887df149fb58b1c21ffa89e6fee66.en.html [/end of copy paste] I tend to consider that restricting stripping on the whole package only for this is an overkill, especially with bug #203886, where I'd like to provide native code executables or (interpreter + bytecode) depending on an useflag. With portage just doing something like: use ocamlopt || export STRIP_MASK="*/bin/*" does what is needed and nothing more (assuming the regexp is correct). However it doesn't seem well documented, and I'm not sure if it's allowed to do that or not (esp. since paludis still strips the file if I use this ebuild, thus the package becomes unuseable). If I have a voice, I vote for this to be allowed.
Do we classify STRIP_MASK as a Portage internal config option, and thus beyond the scope of PMS and ebuilds? I'm inclined to say yes...
(In reply to comment #1) > Do we classify STRIP_MASK as a Portage internal config option, and thus beyond > the scope of PMS and ebuilds? I'm inclined to say yes... Imho that's what it currently is. But then you can see this bug as a feature request to allow this in a more "sane" manner if this one isn't (even if I can't see any problem with it). grepping the tree shows that glibc ebuilds have been doing this for a while, so imho something has to be done in one way or the other. For the reasons I mentionned, I think such a finegrained way to do it should be allowed, and because portage seems to have supported STRIP_MASK for a while, it could just be allowed without backward compatibilities issues. [thinking again it seems I misunderstood your comment, but well...]
Should probably be an EAPI 2 request then.
But do ebuilds need to know about STRIP_MASK?
of course they do ... it's been around since EAPI-0 and their are things relying on it already
*Should* ebuilds be relying upon it? It looks suspiciously like it's a user config variable like INSTALL_MASK, and thus not something ebuilds should be touching or using.
yes, ebuilds need to set STRIP_MASK so it can execute prepallstrip properly
Last time I looked, prep* was considered a package manager internal too and not suitable for ebuild use.
your refusal to acknowledge that some prep functions are exported to ebuilds and expected to be used is irrelevant they are exported, ebuilds are using them, and are part of EAPI-0+
Are they exported, or do they just happen to be around because of how Portage sets the env up? The fact that people are using them does not imply that people *should* be using them... I mean, do you consider what's being done in bug #167029 to be legitimate?
notice i said "some". i'm not claiming every prep function deserves should be used by ebuilds, but certainly some are. what needs to be done by the ruby.eclass is there because Bug 179800 is not implemented yet.
As there does not seem to be any non political reason to not use it, I think I'll use it for ocaml packages. That will greatly help me improving the quality of those packages.
It's not a political reason, it's a technical one -- the prep* functions are considered to be Portage internals by the Portage people, and are thus subject to arbitrary removal or change.
Considering the summary has just been changed to postpone it to a hypothetical future eapi, let me make a summary for ocaml stuff: - lots of ebuilds are now using it to not strip files that shouldnt be - this is needed for bytecode, bytecode is needed for arches that do not have (good) native code support - portage handles this very well. I still think postponing this to a future eapi is a political reason; this has nothing to do with prep* functions and isn't more a user defined variable than cflags where afaik its sane to append stuff.
I would like to ask the council to vote on retroactively adding STRIP_MASK to EAPI-0+ or close this bug and let non-portage PMs suffer from it.
Where can I find documentation how STRIP_MASK works? It's neither in Portage man pages nor in the devmanual.
Here's the code located in /usr/lib/portage/bin/ebuild-helpers/prepstrip: # The noglob funk is to support STRIP_MASK="/*/booga" and to keep # the for loop from expanding the globs. # The eval echo is to support STRIP_MASK="/*/{booga,bar}" sex. set -o noglob strip_this=true for m in $(eval echo ${STRIP_MASK}) ; do [[ /${x#${ED}} == ${m} ]] && strip_this=false && break done set +o noglob
Isn't OCaml what should be fixed instead? I doubt it's anywhere close to good design when applications requires to be non-stripped to work properly.
STRIP_MASK was introduced at 20060113: <http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;f=bin/prepstrip;h=2ef74a4c8dd1ef5dab77264c7a9f2d85a3d5fb19> This was included with portage-2.1 which went stable (on last arch) at 20060611. So IF we add it to the PMS, it would certainly qualify for EAPI 0. On the other hand, it's an obscure and undocumented feature that is used by only 22 packages in the tree. @aballier: I suggest that you post it to the gentoo-dev mailing list, and depending on the outcome of the discussion there the issue can be brought forward to the council.
Here's the list of packages that set STRIP_MASK: app-emulation/qemu app-misc/geneweb app-misc/ledit app-misc/wyrd app-text/active-dvi app-text/htmlc dev-lang/ghc dev-lang/go dev-lisp/clisp dev-ml/findlib dev-ml/oasis dev-ml/ocamlnet dev-scheme/schoca dev-util/coccinelle dev-util/omake net-misc/unison net-p2p/mldonkey sys-boot/grub sys-cluster/ceph sys-devel/dev86 sys-freebsd/freebsd-lib sys-libs/glibc
(In reply to Michał Górny from comment #18) > Isn't OCaml what should be fixed instead? I doubt it's anywhere close to > good design when applications requires to be non-stripped to work properly. It isn't good design, some apps are avoiding to use that "broken" feature and it is becoming less needed these days, but that's the classical first reaction of an outsider. If you think you can fix it, then please do. Others have tried. Anyway, this isn't all about ocaml. .o files (think *crt*.o) have no need for being stripped too. (In reply to Ulrich Müller from comment #19) > @aballier: I suggest that you post it to the gentoo-dev mailing list, and > depending on the outcome of the discussion there the issue can be brought > forward to the council. If you feel like doing it, then please do. I'm sorry but fixing PMS 6 years later is at the bottom of my priority list :/
Considering the prep* functions aren't stable or EAPI controlled, it's a bit of a stretch to consider this "fixing PMS"...
(In reply to Ciaran McCreesh from comment #22) > Considering the prep* functions aren't stable or EAPI controlled, it's a bit > of a stretch to consider this "fixing PMS"... This isn't about prep* functions.
(In reply to Alexis Ballier from comment #23) > This isn't about prep* functions. Comment 7 thinks it is.
(In reply to Ciaran McCreesh from comment #24) > (In reply to Alexis Ballier from comment #23) > > This isn't about prep* functions. > > Comment 7 thinks it is. Read the code. glibc does it that way, others do not as far as I've seen. You should also do some research as of why glibc does it that way.
(In reply to Ciaran McCreesh from comment #24) > (In reply to Alexis Ballier from comment #23) > > This isn't about prep* functions. > > Comment 7 thinks it is. It is about the package manager stripping binaries. The internal Portage function doing this is called prepallstrip, but this doesn't mean that it needs to be called from (or even exposed to) ebuilds. Doesn't Paludis also strip binaries after src_install and before merging? (In reply to Alexis Ballier from comment #21) > If you feel like doing it, then please do. I'm sorry but fixing PMS 6 years > later is at the bottom of my priority list :/ Well, then RESO WONT. Obscure and undocumented Portage-only feature.
(In reply to Ulrich Müller from comment #26) > Doesn't Paludis also strip binaries after src_install and before merging? It does, just not using a utility named prep*. > (In reply to Alexis Ballier from comment #21) > > If you feel like doing it, then please do. I'm sorry but fixing PMS 6 years > > later is at the bottom of my priority list :/ > > Well, then RESO WONT. Obscure and undocumented Portage-only feature. Then any uses of it in the tree that are required for correct operation need to be replaced with RESTRICT=strip.
(In reply to Ciaran McCreesh from comment #27) > Then any uses of it in the tree that are required for correct operation need > to be replaced with RESTRICT=strip. Yes, only those uses that are essential for correct operation. Cases where STRIP_MASK is just "nice to have" can stay, IMHO.
(In reply to Ulrich Müller from comment #28) > Then any uses of it in the tree that are required for correct operation need > to be replaced with RESTRICT=strip. start by fixing glibc then; it's either this, an entirely unstripped glibc or bug #46186
(In reply to Alexis Ballier from comment #29) > start by fixing glibc then; it's either this, an entirely unstripped glibc > or bug #46186 So to summarise, you want the feature but it's so low on your priority list that you're not willing to champion it in the -dev ML or present it to the council. But others shall do the work and clean up the breakage.
(In reply to Ulrich Müller from comment #28) > (In reply to Ciaran McCreesh from comment #27) > > Then any uses of it in the tree that are required for correct operation need > > to be replaced with RESTRICT=strip. > > Yes, only those uses that are essential for correct operation. Cases where > STRIP_MASK is just "nice to have" can stay, IMHO. you know what this implies, right? either a serious regression by restricting stripping on an entire package or changing those STRIP_MASK uses to RESTRICT=strip + STRIP_MASK + prepallstrip so that there is no regeression on the portage side and other PMs also have working packages, yet totally unstripped.
(In reply to Alexis Ballier from comment #31) > you know what this implies, right? > either a serious regression by restricting stripping on an entire package or > changing those STRIP_MASK uses to RESTRICT=strip + STRIP_MASK + prepallstrip > so that there is no regeression on the portage side and other PMs also have > working packages, yet totally unstripped. As I said, discuss it in the -dev ML, and/or propose it for the next council meeting's agenda (preferably with a PMS patch). And feel free to reopen this bug then. Otherwise, also the members of the PMS team are volunteers. I can't speak for others, but I'm not willing to do the grunt work for you if the feature is "at the bottom of your priority list". (In reply to Alexis Ballier from comment #29) > (In reply to Ulrich Müller from comment #28) > > Then any uses of it in the tree that are required for correct operation need > > to be replaced with RESTRICT=strip. BTW, I haven't written the above sentence that you're attributing to me.
(In reply to Ulrich Müller from comment #30) > (In reply to Alexis Ballier from comment #29) > > start by fixing glibc then; it's either this, an entirely unstripped glibc > > or bug #46186 > > So to summarise, you want the feature but it's so low on your priority list > that you're not willing to champion it in the -dev ML or present it to the > council. But others shall do the work and clean up the breakage. Sort of, but I would phrase it differently. See comment #15 for the council part. I do not _want_ the feature in the usual sense, I claim it is _needed_ and is a PMS deficiency not to document it. I am not involved with PMS. If PMS does not reflect reality and is broken, I consider it fair to ignore it. If PMS people want to impose not well thought rules by forbidding stuff that has always been working fine (for reasons I, moreover, consider doubtful) then I indeed expect those people to provide solutions for the cases where it's needed. And correct solutions, not the RESTRICT=strip bloat. (In reply to Ulrich Müller from comment #32) > Otherwise, also the members of the PMS team are volunteers. I can't speak > for others, but I'm not willing to do the grunt work for you if the feature > is "at the bottom of your priority list". Fair enough, the case is closed :) This bug has been ignored for 6 years, bug #46186 was closed in 2004: I understand that as nobody cares and the current status quo is good enough. No need to spend my time on it :) Note: We have a problem of vocabulary here, what you call a "feature" I consider a "PMS deficiency".
The phrase "reflect reality" isn't a helpful one to use. By Council decree, "reality" isn't what's allowed in the tree; what PMS specifies is what's allowed in the tree.
(In reply to Ciaran McCreesh from comment #34) > The phrase "reflect reality" isn't a helpful one to use. By Council decree, > "reality" isn't what's allowed in the tree; what PMS specifies is what's > allowed in the tree. yes, and by this very definition, you have been acting as the council itself in this bug.
(In reply to Alexis Ballier from comment #35) > (In reply to Ciaran McCreesh from comment #34) > > The phrase "reflect reality" isn't a helpful one to use. By Council decree, > > "reality" isn't what's allowed in the tree; what PMS specifies is what's > > allowed in the tree. > > yes, and by this very definition, you have been acting as the council itself > in this bug. I don't follow. The Council approved the most recent edition of PMS, not me.
OK guys. We're not on the mailing list here. We got your points. All of them. Now may I respectfully ask you to shut up and/or carry your discussion elsewhere. @ulm: it still is a bug. probably one in reality, which slowly gets distorted here.
I've nothing against including this in PMS. However, STRIP_MASK was proposed previously and the council hasn't accepted it for EAPI 2: http://archives.gentoo.org/gentoo-dev/msg_6dcca100f593cce5d25870f357bbe318.xml So please follow regular procedure for EAPI 6, starting with discussion in gentoo-dev.
This should probably be done in some kind of 'dostrip' (in consistency with 'docompress').
(In reply to Michał Górny from comment #39) > This should probably be done in some kind of 'dostrip' (in consistency with > 'docompress'). docompress accepts one or more plain paths without globbing, whereas STRIP_MASK accepts a single filename pattern. So I wonder if we could have a dostrip with an interface similar to docompress?
(In reply to Ulrich Müller from comment #40) > (In reply to Michał Górny from comment #39) > > This should probably be done in some kind of 'dostrip' (in consistency with > > 'docompress'). > > docompress accepts one or more plain paths without globbing, whereas > STRIP_MASK accepts a single filename pattern. That is not correct; STRIP_MASK accepts multiple patterns separated by whitespace. For simple cases, a plain path will suffice (like usr/lib/grub). There may be other cases where more fine-grained pattern matching is necessary. Are we still looking for a gentoo-dev discussion to move this bug forward? If so, I am happy to start it.
(In reply to Mike Gilbert from comment #41) > > Are we still looking for a gentoo-dev discussion to move this bug forward? > If so, I am happy to start it. Please do. I don't care how the semantics precisely look, but I do need a way to exempt specific files from stripping, that is guaranteed to work independent of package manager. I'm fine with adding it to a future EAPI as well, just that *unless* we get this feature I won't be able to remove the remaining calls to prepstrip (see bug 259422).
PS. For obvious reasons this is a feature that 99% of the packages should not and will not need. The rest (boot loaders, glibc, ...) fulfills some critical functionality though.
(In reply to Andreas K. Hüttel from comment #42) > I don't care how the semantics precisely look, but I do need a way to exempt > specific files from stripping, that is guaranteed to work independent of > package manager. If the answer to the question asked in comment #40 is positive, we can use docompress as a blueprint for a dostrip command in the next EAPI. > I'm fine with adding it to a future EAPI as well, just that *unless* we get > this feature I won't be able to remove the remaining calls to prepstrip (see > bug 259422). I disagree. The very first council-approved PMS version already said (in 2009) that prep* are internal functions of the PM and that ebuilds may neither use nor rely upon them. So any calls to them are a QA violation and should be removed.
> > > I'm fine with adding it to a future EAPI as well, just that *unless* we get > > this feature I won't be able to remove the remaining calls to prepstrip (see > > bug 259422). > > I disagree. The very first council-approved PMS version already said (in > 2009) that prep* are internal functions of the PM and that ebuilds may > neither use nor rely upon them. So any calls to them are a QA violation and > should be removed. Well I give you three options: * a QA violation that should be removed * a massive code duplication between portage and e.g. the sys-libs/glibc ebuild (re-implementing the entire stripping procedure) * or a non-functional gdb in all multithreaded applications There's a reason why noone fixed this QA violation for 8 years... it's not just "toolchain can't be bothered" here.
(In reply to Ulrich Müller from comment #40) > (In reply to Michał Górny from comment #39) > > This should probably be done in some kind of 'dostrip' (in consistency with > > 'docompress'). > > docompress accepts one or more plain paths without globbing, whereas > STRIP_MASK accepts a single filename pattern. So I wonder if we could have a > dostrip with an interface similar to docompress? I need to check in detail still, but on a quick glance the semantics of docompress look useful. I think I can work with that.
(In reply to Andreas K. Hüttel from comment #45) > Well I give you three options: > * a massive code duplication between portage and e.g. the sys-libs/glibc > ebuild (re-implementing the entire stripping procedure) Why not? This could be in a eclass. > * or a non-functional gdb in all multithreaded applications That is what we have now, because prep* is unspecified and we cannot rely doing anything useful.
(In reply to Ulrich Müller from comment #47) > (In reply to Andreas K. Hüttel from comment #45) > > Well I give you three options: > > * a massive code duplication between portage and e.g. the sys-libs/glibc > > ebuild (re-implementing the entire stripping procedure) > > Why not? This could be in a eclass. This is not accounting that it bites its own tail by requiring to look at "$FEATURES" to respect FEATURES=nostrip, splitdebug, installsource, etc. I like the idea of moving every bash code out of PM(S) to eclasses, but we're far from it on this side. > > * or a non-functional gdb in all multithreaded applications > > That is what we have now, because prep* is unspecified and we cannot rely > doing anything useful. That is what we have now on the very small minority (I don't have numbers but I would say that's <1%) of users not using portage, yes. Defining STRIP_MASK and prep* would have made this minority happy since almost 10 years now. Really, if you look at the implications, either the QA violation is OK or those functions need to be added to current EAPIs: EAPI7 will probably come in 2018; I would expect glibc ebuilds with those changes and this EAPI to be stabilized on all arches in 2019 or 2020 if we follow current procedures of not bumping @system packages EAPIs too quickly in order to preserve the upgrade path and give proper testing to our libc. That's a more than 2 years plan.
So, apparently Portage is considered to define the standard, and PMS isn't needed here? Then I suppose this bug can be closed again.
Created attachment 499966 [details, diff] First draft of a specification. Attached is a first draft of a specification. For the final version this should presumably be merged with the previous subsection (controllable compression).
(In reply to Ulrich Müller from comment #50) > Created attachment 499966 [details, diff] [details, diff] > First draft of a specification. > > Attached is a first draft of a specification. For the final version this > should presumably be merged with the previous subsection (controllable > compression). Thanks. Some nits: +stripping of debug symbols, the package manager may compress a subset of the files under the \t{ED} s/compress/strip/ The current API seems very good to me. I think I saw toolchain using globs (e.g. *crt*.o), hope that works for them.
(In reply to Alexis Ballier from comment #51) > The current API seems very good to me. I think I saw toolchain using globs > (e.g. *crt*.o), hope that works for them. I think that should be fine, with some small ebuild adjustments. If the dostrip call is done after the files are copied to ${D}, the globs can be evaluated by the ebuild rather than the package manager. For example: src_install() { emake DESTDIR="${D}" install cd "${ED}" dostrip -x foo/bar/*baz*.o }
(In reply to Ulrich Müller from comment #50) > Created attachment 499966 [details, diff] [details, diff] > First draft of a specification. > > Attached is a first draft of a specification. For the final version this > should presumably be merged with the previous subsection (controllable > compression). Looks good to me (all bugs already found above :).
(In reply to Alexis Ballier from comment #51) > +stripping of debug symbols, the package manager may compress a subset of > the files under the \t{ED} > > s/compress/strip/ Corrected. > The current API seems very good to me. I think I saw toolchain using globs > (e.g. *crt*.o), hope that works for them. I'd rather not have the command itself handle wildcards, as none of the other do* commands (especially docompress) does that.
(In reply to Mike Gilbert from comment #52) > (In reply to Alexis Ballier from comment #51) > > The current API seems very good to me. I think I saw toolchain using globs > > (e.g. *crt*.o), hope that works for them. > > I think that should be fine, with some small ebuild adjustments. If the > dostrip call is done after the files are copied to ${D}, the globs can be > evaluated by the ebuild rather than the package manager. > > For example: > > src_install() { > emake DESTDIR="${D}" install > cd "${ED}" > dostrip -x foo/bar/*baz*.o > } Unless I'm missing something, this will not work or at least is a bit confusing: We use "/" in the include list to strip everything, this wildcard would expand to e.g. "foo/bar/baz.o", without the leading "/". (In reply to Ulrich Müller from comment #54) > > The current API seems very good to me. I think I saw toolchain using globs > > (e.g. *crt*.o), hope that works for them. > > I'd rather not have the command itself handle wildcards, as none of the > other do* commands (especially docompress) does that. I fully agree with that. If the above is a real issue and not a simple misunderstanding on my side, I would rather see dostrip behave exactly the same as docompress for consistency and avoiding any confusion. Moreover, it is easy to write an eutils function expanding a wildcard to match files in $D.
(In reply to Alexis Ballier from comment #55) > Unless I'm missing something, this will not work or at least is a bit > confusing: > We use "/" in the include list to strip everything, this wildcard would > expand to e.g. "foo/bar/baz.o", without the leading "/". "Whether an item is to be excluded is determined as follows: For each item in the exclusion list, pretend it has the value of the \t{ED} variable prepended, then: ..." So, this becomes "${ED}foo/bar/baz.o". ${ED} is guaranteed to always end with a slash. ${ED} generally expands to something like "/tmp/portage/cat/pkg/image/". So, we end up excluding "/tmp/portage/cat/pkg/image/foo/bar/baz.o". I don't see why any leading slash would be necessary in the exclusion list.
(In reply to Mike Gilbert from comment #56) > "Whether an item is to be excluded is determined as follows: For each item > in the exclusion list, pretend it has the value of the \t{ED} variable > prepended, then: ..." > > So, this becomes "${ED}foo/bar/baz.o". Right, the intention is that do{compress,strip} work for arguments beginning with or without a slash. > ${ED} is guaranteed to always end with a slash. ${ED} generally expands to > something like "/tmp/portage/cat/pkg/image/". According to our roadmap, that will no longer be the case in EAPI 7 (see bug 465772). So we will need some adjustment of wording there. The portage implementation adds any missing / in the docompress call already: https://gitweb.gentoo.org/proj/portage.git/tree/bin/phase-helpers.sh?id=7d2c4fb609454b76d30c42fc7a0bb720decc39a3#n120 (see f=${f%/} in line 127).
(In reply to Ulrich Müller from comment #57) > (see f=${f%/} in line 127). Sorry, I meant line 128 of course: [[ ${f:0:1} = / ]] || f="/${f}"
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=9aaa652c86b21b925bc56fafd57b561b86ace0f8 commit 9aaa652c86b21b925bc56fafd57b561b86ace0f8 Author: Zac Medico <zmedico@gentoo.org> AuthorDate: 2018-03-26 19:12:51 +0000 Commit: Zac Medico <zmedico@gentoo.org> CommitDate: 2018-03-28 05:09:15 +0000 Add dostrip for EAPI 7 This patch includes the essential parts of the dostrip implementation from portage-mgorny. All of the prepstrip code has moved to bin/estrip, without any changes except the addition of argument parsing for estrip --ignore, --queue, and --deque modes which are equivalent to the corresponding ecompressdir modes. Due to overlap, also ban the non-standard prepstrip and prepallstrip helpers in EAPI 7, with a die message suggesting to use 'dostrip' instead. Also ignore the non-standard STRIP_MASK variable for EAPI 7. Bug: https://bugs.gentoo.org/203891 bin/eapi.sh | 4 +++ bin/ebuild-helpers/prepall | 2 +- bin/ebuild-helpers/prepallstrip | 4 +++ bin/ebuild-helpers/prepstrip | 4 +++ bin/ebuild.sh | 2 +- bin/estrip | 74 +++++++++++++++++++++++++++++++++++++++-- bin/misc-functions.sh | 6 ++++ bin/phase-helpers.sh | 29 ++++++++++++++++ bin/save-ebuild-env.sh | 4 +-- 9 files changed, 122 insertions(+), 7 deletions(-)}
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/pms.git/commit/?id=373ec493c322530b72f7bafb7db846a522704184 commit 373ec493c322530b72f7bafb7db846a522704184 Author: Ulrich Müller <ulm@gentoo.org> AuthorDate: 2017-10-29 09:30:43 +0000 Commit: Ulrich Müller <ulm@gentoo.org> CommitDate: 2018-03-31 15:30:43 +0000 EAPI 7 has controllable stripping and dostrip. Bug: https://bugs.gentoo.org/203891 dependencies.tex | 4 +++- eapi-differences.tex | 7 ++++++ pkg-mgr-commands.tex | 62 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 72 insertions(+), 1 deletion(-)}
