There's precisely one place where we use compgen in Portage: >bin/save-ebuild-env.sh:86: unset -f $(compgen -A function ___) While not wrong, it's unusual for scripts to employ compgen & it takes people by surprise on occasion (see e.g. https://forums.gentoo.org/viewtopic-t-1163902.html). bin/save-ebuild-env.sh first start using it with: ``` commit ab46499322311c1faa710c63d0a5339e49a9061a Author: Arfrever Frehtes Taifersar Arahesis <Arfrever@Apache.Org> Date: Wed Sep 26 22:31:20 2012 +0200 Add eapi.sh with ___eapi_*() functions and use these functions in other files. ``` then later on, vapier extended it: ``` commit 29f3837382babcac38d4c0e6498620b0dbc6df67 Author: Mike Frysinger <vapier@gentoo.org> Date: Thu Nov 12 20:53:20 2015 -0500 ebuild: unset all funcs/vars that start with ___ Since the __* (two) namespace is reserved, and ___* (three) has rarely (if ever) been used in ebuilds, we can nuke all funcs/vars that start with that. It makes clean up easier for us. ```
pkgcore also has a history with compgen: - https://github.com/pkgcore/pkgcore/commit/2e31a6843166c6fa23bf02dfe846ef161db29f65 - https://github.com/pkgcore/pkgcore/commit/4cd7ecf0224a4529fa780af72816d813d721620e - https://github.com/pkgcore/pkgcore/commit/8abd796a02f2d5a57a0e495f1f6d9aa1c7ec4174 - https://github.com/pkgcore/pkgcore/blob/83dd99d490fa4fd0785053b99ec617fd0851d409/src/pkgcore/ebuild/eapi.py#L257
FYI in prefix branch we made this one conditional # BEGIN PREFIX LOCAL: compgen is not compiled in during bootstrap if type compgen >& /dev/null ; then # Clear out the triple underscore namespace as it is reserved by the PM. unset -f $(compgen -A function ___) unset ${!___*} fi # END PREFIX LOCAL
The other alternative is to do some fairly minor grepping of `declare -F` (as pkgcore in fact once did) which should achieve the same effect for a fairly stable one-time cost of a second fork. The rationale for why this is cleared out isn't really documented, but if there's a good reason then a good solution should be used, rather than just sometimes disabling it.
(In reply to Fabian Groffen from comment #2) > # BEGIN PREFIX LOCAL: compgen is not compiled in during bootstrap What is the reason for this?
As I was copied in, I suppose I should also explain my stake in this. I have rewritten some portions of portage (for the better, I hope) in such a way that the use of compgen is naturally extended. In particular, the filter-bash-environment script has been entirely overhauled. The present instance of the script is written in python and is a regex-employing aberration. My rewrite is written in pure bash and relies heavily on compgen -A variable in particular. I also use compgen to filter out the names of the special variables in bash, rather than have portage rely on an ill-maintained, hard-coded list to do so. Below is an example of how that technique works. # Filter out shell variables that are defined by bash when it starts. while read -r; do __filter[$REPLY]= done < <(env -i "$BASH" -c 'compgen -A variable') There's more, but you get the idea. My stance is that the compgen builtin is a feature that is materially useful to portage, more so than the current code may suggest. I would not welcome being unable to use it. Therefore, I am wondering whether there is a compelling reason for compgen ever being disabled in the first place.
While parsing declare -F isn't a problem, here is why I am less then eager to parse variable names from declare -p. { FOO=$'\nBAR BAZ QUUX='; declare -p; } | while IFS=' =' read -r _ _ name _; do printf '%s\n' "$name"; done | grep QUUX This prints QUUX, despite no such variable existing. Perhaps the interface could be improved in the future. In the meantime, it is what it is.
I agree that parsing declare -p is impractical. One possible alternative is `${!a*}` but I'm not sure you can do that without doing it dozens of times, one for every possible variable prefix. The problem with compgen is that it is only available for use when bash is configured with --enable-progcomp / --enable-readline, which feels like a powerful argument that script authors are not allowed to assume that it will exist, regardless of how useful it may be in non-programmable-completion contexts. Maybe the answer is to ask that it always be available as a builtin, even when the programmable completion system isn't enabled.
(In reply to Eli Schwartz from comment #7) > I agree that parsing declare -p is impractical. One possible alternative is > `${!a*}` but I'm not sure you can do that without doing it dozens of times, > one for every possible variable prefix. Indeed, that is the problem with ${!prefix*}. It is frustrating. > > The problem with compgen is that it is only available for use when bash is > configured with --enable-progcomp / --enable-readline, which feels like a What a nuisance. > powerful argument that script authors are not allowed to assume that it will > exist, regardless of how useful it may be in non-programmable-completion > contexts. > > Maybe the answer is to ask that it always be available as a builtin, even > when the programmable completion system isn't enabled. Yes, it's probably worth bringing it up, irrespective of how this particular bug pans out. 5.2 improves upon the quoting strategy employed for declare -p but it could easily change again.
I posted a query to the bug-bash list (the archives haven't indexed it yet).
Having mulled it over, it occurs to me that I could do this instead. eval "printf %s\\\n $(printf '${!%s*} ' {A..Z} {a..z} _)" It's not a pleasant thing to behold but it gets the job done. I would have to make further concessions elsewhere but those are of a lesser concern.
Created attachment 864600 [details, diff] Patch for master branch Refrain from using compgen.
Created attachment 864601 [details, diff] Patch for prefix branch Refrain from using compgen. Brings the code back into line with the master branch.
https://lists.gnu.org/archive/html/bug-bash/2023-06/msg00094.html
(In reply to Eli Schwartz from comment #13) > https://lists.gnu.org/archive/html/bug-bash/2023-06/msg00094.html Thanks.
At this juncture, I do not care in the slightest whether Gentoo promises to make compgen available or not. This is not merely on account of the antipathy demonstrated by upstream in Eli's thread, but also by another discussion in which I learned that compgen has a history of odd bugs. As such, I have no intention of writing any new code that uses compgen. Parsing declare -F is perfectly acceptable.
Anothing point to consider: the current approach taken by portage isn't robust. It can be undone by any prior meddling with the value of IFS. $ bash -c 'foo() { :; }; bar() { :; }; IFS=a; printf "<%s> " $(compgen -A function)' <b> <r foo> To address that would require either resetting the value of IFS or reading the output line by line, at which point one might as well as be using declare -F in the first place.
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=f24dd0d9808559571509add2c8c69c1bcb2bfec6 commit f24dd0d9808559571509add2c8c69c1bcb2bfec6 Author: Kerin Millar <kfm@plushkava.net> AuthorDate: 2023-06-25 20:03:04 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2023-07-03 20:18:37 +0000 bin/save-ebuild-env.sh: refrain from using the compgen builtin For the compgen builtin to be available requires that bash be compiled with --enable-readline. Rather than rely on compgen to generate a list of function names, parse their names out of declare -F instead. Specify -v for the following unset command, so that bash is coerced into unsetting only variables, as is intended. Expand the applicable variable names with "${!___@}". Owing to the constraints placed on identifiers, it's not particularly important that it be done this way, but I think it better expresses the intent. Signed-off-by: Kerin Millar <kfm@plushkava.net> Bug: https://bugs.gentoo.org/909148 Signed-off-by: Sam James <sam@gentoo.org> bin/save-ebuild-env.sh | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=27ed51fc126465ee5a4499bf6379e5c2b4da37cc commit 27ed51fc126465ee5a4499bf6379e5c2b4da37cc Author: Sam James <sam@gentoo.org> AuthorDate: 2023-07-03 20:21:32 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2023-07-03 20:21:32 +0000 sys-apps/portage: master no longer needs bash[readline] Bug: https://bugs.gentoo.org/909148 Signed-off-by: Sam James <sam@gentoo.org> sys-apps/portage/portage-9999.ebuild | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
The bug has been closed via the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=858dfd771ac4c6c9315ac5851f4aeeb233fc21d5 commit 858dfd771ac4c6c9315ac5851f4aeeb233fc21d5 Author: Sam James <sam@gentoo.org> AuthorDate: 2023-08-09 02:54:12 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2023-08-09 02:54:12 +0000 sys-apps/portage: add 3.0.50 Closes: https://bugs.gentoo.org/908971 Closes: https://bugs.gentoo.org/640658 Closes: https://bugs.gentoo.org/894398 Closes: https://bugs.gentoo.org/895908 Closes: https://bugs.gentoo.org/909067 Closes: https://bugs.gentoo.org/909148 Closes: https://bugs.gentoo.org/909853 Closes: https://bugs.gentoo.org/910035 Closes: https://bugs.gentoo.org/910376 Closes: https://bugs.gentoo.org/911594 Closes: https://bugs.gentoo.org/911574 Signed-off-by: Sam James <sam@gentoo.org> sys-apps/portage/Manifest | 1 + sys-apps/portage/portage-3.0.50.ebuild | 229 +++++++++++++++++++++++++++++++++ 2 files changed, 230 insertions(+)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/proj/portage.git/commit/?id=4f4adbfb1b60ca04c8c04cd858d2cdebdf201b8c commit 4f4adbfb1b60ca04c8c04cd858d2cdebdf201b8c Author: Sam James <sam@gentoo.org> AuthorDate: 2023-08-10 01:43:14 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2023-08-10 01:43:14 +0000 NEWS: add missing entry for compgen removal Bug: https://bugs.gentoo.org/909148 Fixes: f24dd0d9808559571509add2c8c69c1bcb2bfec6 Signed-off-by: Sam James <sam@gentoo.org> NEWS | 3 +++ 1 file changed, 3 insertions(+)
