What about saying "emerge <atom>" or "ebuild xx.ebuild install" and getting a shell prompt having exactly that environment where the ebuild-functions (pkg_setup, src_build, src_install, etc.) get called normally, to be able to debug an ebuild step by step, or just execute the ebuild-function directly ? Attached patch is for portage-2.1.1-r1, use it like this (using crossdev as sample here, as it is a small package): $ FEATURES=ebuildshell emerge crossdev Calculating dependencies... done! >>> Emerging (1 of 1) sys-devel/crossdev-0.9.17-r2 to / * checking ebuild checksums ;-) ... [ ok ] * checking auxfile checksums ;-) ... [ ok ] * checking miscfile checksums ;-) ... [ ok ] EBUILD crossdev pkg_setup $ type pkg_setup pkg_setup is a function pkg_setup () { return } EBUILD crossdev pkg_setup $ pkg_setup EBUILD crossdev pkg_setup $ exit exit >>> Unpacking source... EBUILD crossdev src_unpack $ exit 1 exit !!! ERROR: sys-devel/crossdev-0.9.17-r2 failed. Call stack: ebuild.sh, line 1578: Called dyn_unpack ebuild.sh, line 708: Called call-ebuildshell 'src_unpack' ebuild.sh, line 1068: Called die !!! (no error message) !!! If you need support, post the topmost build error, and the call stack if relevant. If you enter "exit 1" (as seen above), portage stops operation to be able to abort the current build.
Created attachment 101944 [details, diff] add 'call-ebuildshell()' to bin/ebuild.sh
rather then overloading each phase invocation, would suggest fixing ebuild.sh implementation so it's a lib, and having a consumer of that lib. Fair bit cleaner.
hmm, have no idea on how you mean that, because in the end I want the prompt exactly where the phase would be called normally.
Created attachment 102194 [details, diff] one fix, one improvement, based on original patch Independent of basics-discussion, the improvement is: 'type' the to-be-called phase-function before the prompt.
Created attachment 113966 [details, diff] patch against portage-trunk/bin/ebuild.sh at revision 6261 Since having qa_call(), call-ebuildshell() only needs to be called within qa_call() now. Another bugfix: using 'chmod g+w' instead of 'chown g+w'. There is one drawback: this feature should not be recorded into vdb, as it causes confusion when unmerging a package while having forgot that this feature was set during merge.
(In reply to comment #0) > What about saying "emerge <atom>" or "ebuild xx.ebuild install" and getting a > shell prompt having exactly that environment where the ebuild-functions > (pkg_setup, src_build, src_install, etc.) get called normally, to be able to > debug an ebuild step by step, or just execute the ebuild-function directly ? > Like app-shells/sandboxshell?
(In reply to comment #6) > > Like app-shells/sandboxshell? > No - or does sandboxshell accept emerge-atoms now ? I'd like to be able to do this kind of commands in that shell fex: EBUILD$ src_unpack EBUILD$ cd $S EBUILD$ cp somefile{,.orig} EBUILD$ vi somefile EBUILD$ diff -u somefile{.orig,} > ${FILESDIR}/some-new.patch EBUILD$ vi $EBUILD add a line "epatch ${FILESDIR}/some-new.patch"
Perhaps we should implement this so that you simply run `ebuild foo.ebuild shell` and it puts you into an interactive shell inside the ebuild environment. Wouldn't that be a nice way to do it?
(In reply to comment #8) > Perhaps we should implement this so that you simply run `ebuild foo.ebuild > shell` and it puts you into an interactive shell inside the ebuild environment. > Wouldn't that be a nice way to do it? Good idea, I can live with that, although it might be less comfortable IMO...
(In reply to comment #9) > Good idea, I can live with that, although it might be less comfortable IMO... Having the die automatically drop into the ebuild shell seems like a nice FEATURE. I think we should have both.
(In reply to comment #9) > (In reply to comment #8) > > Perhaps we should implement this so that you simply run `ebuild foo.ebuild > > shell` and it puts you into an interactive shell inside the ebuild environment. > > Wouldn't that be a nice way to do it? > > Good idea, I can live with that, although it might be less comfortable IMO... > hmm, not so good: 'ebuild foo.ebuild shell' lacks <command> - "shell" is not a <command>, just indicates how to execute a <command>. But this works: FEATURES=ebuildshell ebuild foo.ebuild <command>
There's one drawback with my ebuildshell-implementation: FEATURE 'ebuildshell' should not be recorded in metadata, as this can confuse users (did me before noticing that): when upgrading/remerging a package, having forgotten that it was merged _with_ 'ebuildshell', the old package still uses 'ebuildshell' for unmerge. Now when the new emerge-output is redirected to somewhere, the ebuildshell exits without doing anything. So this FEATURE should be enable-able only by environment, not by vdb-metadata. Or it might not be a FEATURE, but something different - cmdline-switch fex...
(In reply to comment #12) > So this FEATURE should be enable-able only by environment, not by vdb-metadata. Generally, the build time FEATURE list should not be interpreted as being valid at any time other than build time. Current versions of portage (>2.1.2.2) explicitly ignore the FEATURES list from installed and binary packages. > Or it might not be a FEATURE, but something different - cmdline-switch fex... The way that you've hooked into qa_call(), it behaves something like stepping through breakpoints in a debugger. Perhaps we should create a command line interface similar to gdb for this type of functionality. At any point it would allow you to drop into a real bash shell or simply continue to some other point. Currently, the ebuildshell bash instance does not have job control because of the way that output is piped through tee for logging. We can create a pseudo-terminal like the 'script' command does and use it to enable full job control inside the ebuildshell. It can be done using python's tty, pty and termios modules.
(In reply to comment #13) > Generally, the build time FEATURE list should not be interpreted as being valid > at any time other than build time. Current versions of portage (>2.1.2.2) > explicitly ignore the FEATURES list from installed and binary packages. Ah - and I was wondering why I didn't encounter this problem recently. > The way that you've hooked into qa_call(), it behaves something like stepping > through breakpoints in a debugger. Perhaps we should create a command line > interface similar to gdb for this type of functionality. At any point it would > allow you to drop into a real bash shell or simply continue to some other > point. Maybe http://bashdb.sourceforge.net/ could help here ? I didn't use bashdb myself (yet), but it looks very cool. The python-debugger (also available along bashdb) works well. > > Currently, the ebuildshell bash instance does not have job control because of > the way that output is piped through tee for logging. We can create a > pseudo-terminal like the 'script' command does and use it to enable full job > control inside the ebuildshell. It can be done using python's tty, pty and > termios modules. > Would be really cool, thanks!
hows the status of all this? i would have needed an "ebuild debugger" many times (i used the ebuildshell patch successfully, it's just _way_ easier to try out things, and find problems).
Created attachment 157409 [details, diff] patch to work with >= bash-3.2p20, in use with Portage 2.2.00.10708-prefix
note (probably mainly to self): while the patch is on the prefix code, there are no prefix specifics involved. In fact the patch is not prefix-compliant (uses portage user and group, instead of $PORTAGE_USER and $PORTAGE_GROUP).
Yes, I'm applying attachment#157409 [details, diff] unmodified to installed usr/lib/portage/bin/ebuild.sh of both main gentoo portage-2.1.6.4 as well as prefix portage-2.2.*. When one is aware of (both missing) job-control and Ctrl-C handling, it definitively is useful already. Just type in commands for the current phase or copy&paste from what is shown before the first prompt (by [type <phasecommand>]).
Created attachment 198919 [details, diff] updated patch for prefix-portage-2.2, as well as main portage-2.1.6.13 Have added this patch to the Prefix tree now (r48916). Try FEATURES=ebuildshell there, it really helps (me) tracking down build errors.
http://overlays.gentoo.org/proj/alt/browser/trunk/prefix-overlay/sys-apps/portage/files/portage-2.2.01.20020-ebuildshell.patch?rev=60287
Created attachment 325990 [details, diff] respin ebuildshell, with some fixes As of 2.2.01.21165-prefix, ebuildshell is broken. I'm enclosing a respun patch which includes the following changes: o update for __-prefixed filter_readonly_variables functions o fix nasty interaction when bashdb is installed to $EPREFIX (by turning off the extdebug shopt in the ebuildshell subshell) o add flag to SUPPORTED_FEATURES in pym/portage/const.py I can't obsolete since it's not my patch; Michael, it's up to you to obsolete your patch if you're happy with my changes. NB: only tested in prefix 2.2.01.21165!!!
Here's two small problems I wasn't sure how to fix correctly: First, In gen_usr_ldscript, we set ED to "what it should be" if it's not already set. But, apparently, it's always in $PORTAGE_READONLY_VARS. So in EAPI's without ED, gen_usr_ldscript crashes when invoked by the ebuild-shell. We might suspect that similar problems will exist elsewhere... perhaps the ebuildshell needs access to these variables, and we should be doing something programmatically to prevent them from being read from ebuildshell-environment-outcome? Second, my fix for the bashdb interaction breaks portage backtraces by disabling extdebug -- I think? I found the whole bashdb problem to be exceedingly confusing. I'd say it really sucks to loose backtraces given that ebuildshell users are obviously debugging. No clue how to fix it better than I did, though, as I didn't even want to start looking into what bash does with bashdb and #141906... ugh, what a mess!
Created attachment 326048 [details, diff] Respin ebuildshell, with some more fixes This version also adds passing along EAPI-specific variables to the subshell (otherwise i.e.: dodir breaks). Hopefully this obviates the concern about readonly variables.
Created attachment 326182 [details, diff] Yet another ebuildshell spin adjusted for the __ namespace In addition to the above changes, this version adjusts the call-ebuildshell api itself for the "__" namespace.
(In reply to Greg Turner from comment #24) > Created attachment 326182 [details, diff] [details, diff] > Yet another ebuildshell spin adjusted for the __ namespace Confirmed to work with portage-2.2.7, both w/ and w/o prefix, thank you Greg!
Created attachment 374152 [details, diff] portage-2.2.8-ebuildshell-update
Created attachment 374154 [details, diff] patch for ebuild to reenable ebuildshell reenable ebuildshell confirmed to work.
Patch (attachment #374152 [details, diff]) still works fine with sys-apps/portage-2.2.14.
The patch seems pretty reasonable to me. Here are some things I would change: 1) Replace PORTAGE_USER/PORTAGE_GROUP variables with PORTAGE_USERNAME/PORTAGE_GRPNAME variables that are already supported by portage. Note that the corresponding chown/chmod calls which reference these variables would not be needed if we simply removed the $T/ebuildshell-environment-* files when finished with them. However, they could trigger permission issues later if they failed to be removed for some reason (due to the user killing the the ebuild processes). 2) Unset __call-ebuildshell inside save-ebuild-env.sh, in order to filter it from the environment when appropriate. 3) Document the new __filter_readonly_variables options. 4) Maybe add a --ebuildshell option to the ebuild(1) command. In fact, FEATURES=ebuildshell should be prohibited from being used with emerge --jobs/--quiet-build, since stdin has to be /dev/null in that case. Also, I have some questions about some code in the __call-ebuildshell fuction: 1) Isn't the 'shopt -u extdebug' call unnecessary, since 'shopt -p | grep -v extdebug$' should already filter that out? 2) Isn't 'unset BASH_ENV' unnecessary, since bash is called with 'env -i'?
Created attachment 432114 [details, diff] ebuildshell.patch: updated, preserves exported variables across phases now Updated patch, now can preserve exported variables across multiple phases. (In reply to Zac Medico from comment #29) > The patch seems pretty reasonable to me. Here are some things I would change: > > 1) Replace PORTAGE_USER/PORTAGE_GROUP variables with > PORTAGE_USERNAME/PORTAGE_GRPNAME Don't remember why I've initially added user/group handling at all, removed now. Instead, all the temporary files are removed both before and after use now. > 2) Unset __call-ebuildshell inside save-ebuild-env.sh, in order to filter it > from the environment when appropriate. done. > 3) Document the new __filter_readonly_variables options. Have realized that __filter_readonly_variables() does not match too well here, using filter-bash-environment.py with extra options instead. Do you still want me to use __filter_readonly_variables() with the necessary extensions? > 4) Maybe add a --ebuildshell option to the ebuild(1) command. In fact, > FEATURES=ebuildshell should be prohibited from being used with emerge > --jobs/--quiet-build, since stdin has to be /dev/null in that case. Agreed - but haven't looked for where to implement these things yet. Anyone? Also, I've not used the ebuild(1) command for ages, but 'emerge --nodeps' instead. IIRC, the ebuild(1) command is deprecated, no? > Also, I have some questions about some code in the __call-ebuildshell > fuction: > > 1) Isn't the 'shopt -u extdebug' call unnecessary, since 'shopt -p | grep -v > extdebug$' should already filter that out? The 'shopt -p | grep -v extdebug$' is for the returning env, while the 'shopt -u extdebug' is for the prompting shell. The extdebug handling was added by Greg Turner - I've never used bash-debugger yet... > 2) Isn't 'unset BASH_ENV' unnecessary, since bash is called with 'env -i'? This is a relict from before using 'env -i', removed.
Created attachment 432256 [details, diff] updated ebuildshell.patch Updates: * Do not use user's real history file, but the one in temporary HOME. * Disable history expansion with !, often used in function bodies, to allow for copy&paste. * Pre-fill history with "$@", to allow for running normal command with [CursorUp+Enter]. * Avoid potential variable name conflict with $status. * Add some comments (and reordering) to the created bash env file.
Created attachment 432966 [details, diff] updated ebuildshell.patch Updates: * Preserve even non-exported variables across phases - actually that is everything visible to the EXIT trap. * With one exception: Variables declared using 'local' at the shell-promt (copy&paste of function body lines) are not preserved across phases.
Comment on attachment 432966 [details, diff] updated ebuildshell.patch still has some issues
Created attachment 433098 [details, diff] updated ebuildshell.patch Updates: * Filter PS{1..4} from return-env. * Use "trace" attribute rather than some shell function for "local" vars declared at ebuildshell prompt (using copy&paste), to identify for filtering from return-env. * For returning variables only, use 'declare -g' as alias. * Improved reentrant support.
(In reply to Michael Haubenwallner from comment #34) > * For returning variables only, use 'declare -g' as alias. Support for -g option was introduced in BASH 4.2. Maybe use this option only when sufficiently new version of BASH is detected. If ebuildshell feature cannot work without -g option, then disable ebuildshell feature (with a warning) when old version of BASH is detected.
Created attachment 434176 [details, diff] updated ebuildshell.patch Updates: * Need the local='declare -t' alias before defining functions, or recursive ebuildshell's return-env will return local variables. * Warn when preserving variables across phases does not work due to <bash-4.2 (In reply to Arfrever Frehtes Taifersar Arahesis from comment #35)
should have an `ebuild xxx.ebuild shell` step too so you can do: ebuild foo.ebuild clean unpack prepare shell
(In reply to SpanKY from comment #37) > should have an `ebuild xxx.ebuild shell` step too so you can do: > ebuild foo.ebuild clean unpack prepare shell /me fails to see the point of a 'shell' step: the idea is to get a shell prompt for each step (that uses phase-functions defined in the ebuild), as in $ ebuild xxx.ebuild --shell unpack prepare ...
(In reply to Michael Haubenwallner from comment #36) > Created attachment 434176 [details, diff] [details, diff] > updated ebuildshell.patch > > Updates: > > * Need the local='declare -t' alias before defining functions, or recursive > ebuildshell's return-env will return local variables. > > * Warn when preserving variables across phases does not work due to <bash-4.2 > (In reply to Arfrever Frehtes Taifersar Arahesis from comment #35) Bumped patch in Prefix: https://gitweb.gentoo.org/repo/proj/prefix.git/commit/?id=be550519a4d29b437b70be2d5c72fc431edc11e0 Added pull request: https://github.com/gentoo/portage/pull/92
Created attachment 496448 [details, diff] updated ebuildshell.patch Using with FEATURES=userpriv in main Gentoo Linux leads to permission errors with .bash_history when there is a pkg_setup phase - needs explicit chown.
Created attachment 534066 [details, diff] updated ebuildshell patch Rebased to 2.3.40 +8 commits, plus these changes: Update to byte regex (br'<regex>'). Preserve HOME across env -i, as bash -i seems to access ~/.bash_history before HISTFILE is set in --rcfile passed.
