Hi, Here is a patch to make emerge _really_ quiet when used with the "--quiet" option. There are two goals to this changes: - improve performances when emerge is used on a slow terminal (remotely, or in X term emulators like gnome-terminal) - only show to the user what matters to the user: progress, infos and errors. Summary: ======== - "--quiet" changes portage.noiselimit. This can also be done using by default using the "quiet" FEATURE flag, and is disabled by "--verbose". - it also adds a PORTAGE_QUIET variable to settings, which is taken into account by ebuild.sh. - "noiselimit" prevents many emerge/portage messages to be displayed, mainly the merge/unmerge files lists. - "PORTAGE_QUIET" put ebuild.sh in quiet mode (but if the action is "config"). In this mode, stdout and stderr are hiden to the user. But then there is always a log file, even if PORT_LOGDIR is not set. It is then in the $T directory. The only printed messages are those that go to file descriptor 3. The "eecho" function can be used to display messages to both stdout for the log and FD 3 for display. The "einfo", "ewarn", etc. functions are overloaded to also duplicate their output. - Optionnaly, by defining PORT_SHORTLOGDIR, user can choose to keep a quiet version of the logs, similar to the quiet display. - Optionnaly (enabled by default but disable with "--nospinner"), the compilation, installation, merge and unmerge progress are displayed using spinners, that should run slow enough to not stress the terminal. Patchs details: =============== The patch was big, so I've split it in several parts... I will attach them as an archive. They should apply in filenames order on portage-2.0.50_pre10. 01--portage.py--log_counter.patch: This one is a small improvement of the LOG_COUNTER logic. The goal is to have a single log for a whole installation/uninstallation life cycle, whereas there are currently three (one for build, one for preinst/postinst, one for prerm/postrm). This one can be applied and tested separatly from the others, I only post it here because the next one breaks if it is not there, but I will also report it as a separate bug. 02--portage.py--base.patch: This are the basic changes made to portage.py. It is mainly some function definition for printing messages depending on "noiselimit". 03--portage.py--messages.patch: This one replaces some "write" and "print" by their conditionnal version. It mainly removes the dump of merged/unmerged files. 04--ebuild.sh--base.patch: This are the basic changes made to ebuild.sh. It changes the per-ebuild logging, depending on PORTAGE_QUIET. It also define "eecho", and overloads "einfo", "ewarn", etc. 05--ebuild.sh--messages.patch: This one replaces some "echo" with "eecho", and add a few messages so that quiet output is coherent. 06--emerge--base.patch: This is the code that set settings depending on "--quiet". 07--emerge--messages.patch: This one replaces some "write" and "print" by their conditionnal version. You can stop at this patch, quiet output should work. 08--emergehelp.py--doc.patch: Updates the "--quiet" documentation; 09--make.conf--doc.patch: Documents PORT_SHORTLOGDIR and the "quiet" FEATURE flag. 10--portage.py--spinner.patch: Adds spinners to merge and unmerge if in quiet mode. 11--ebuild.sh--spinner.patch: Adds spinners for compilation, installation and packaging, if in quiet mode. 12--emerge--spinner.patch: Makes spinners depend on "--nospinner" not being in the options. Misc. remarks: ============== - What about stderr? I've choosed to hide it in quiet mode, just like stdout, because it contains many messages (configure warnings for instances) that are imho useless for users. Another reason was that I've not found a way to duplicate it without introducing some delays that results in confused logs. - What about non-{einfo,ewarn,...} important messages from ebuilds? Well, they won't be displayed in quiet mode (they are still in logs though). I don't think it it such a big issue though: o they were already missed by users because of the general emerge verbosity o they may be replaced by "einfo", and will then be displayed o they may also be replaced by "eecho", which acts just like "echo", but with duplication. Sure, this wouldn't work with versions of portage that don't define "eecho" yet, but this can be workaround by providing an "eecho.eclass", until all supported portage version provide it natively. o since most of this messages appear in postinst and postrm, we may disable quiet mode for this actions. I don't really like this solution though, some are really verbose (scrollkeeper updates for instance), so I would prefer to keep it in quiet mode. - What about performances? There is no difference in a linux console (without fb), but a clear improvement in my X term emulator (gnome-terminal). I will attach a few benchs. - What about readibility? Well, it's a matter of taste, but I like this new output. I will attach a "console screenshot" of a few emerges. - What about the original "--quiet" behavior? I've thought it was useless, too close to the default output. But I may be wrong, tell me if you want me to restore it and use a different option name. TODO: ===== - There are probably other messages that should be removed in quiet mode, I will try to update the patchs when I see some. There are also some line jump to add or remove, some other messages that could be modified, etc. - Short logs may be completly useless for some packages which don't print anything. Maybe I can remove them in that case. - There are some external command output that could be filtered in quiet mode. For "wget" for instance, the progress bar could be enough. And for "rsync", maybe a simple spinner, or an "updated files counter", something like this. I will make some experiments on this later. And that's all... Sorry for my verbosity, I should try to bugreport with --quiet :) Reproducible: Always Steps to Reproduce:
Created attachment 23317 [details] quietmode.tar.gz Here are the patchs.
Created attachment 23318 [details] benchmarks.txt Some benchmarks I've made to see the quiet vs. verbose difference.
Created attachment 23319 [details] console-screenshot.txt A copy of the console for after emerging two packages.
Created attachment 23663 [details] quietmode.tar.gz Here is an updated version. It should work with 2.0.50_pre12. Changes: - a small ebuild.sh fix for the "recurse | tee ...log" call, that I will report also in a separate bug - emerge and portage.py now share their spinner code, defined in output.py. Also, all the printing functions that depend on a noiselimit have been moved to output.py (including "writemsg", but I've put some update patches for other modules that were calling portage.writemsg) - maybe a few more messages marked conditional, but there is probably still work to do here. The patch splitting is different, but most of my previous report is still relevant, even if files names changed. The spinner part is no more in separate patches. New files list: 01--portage.py--log_counter.patch 02--output.py--code.patch 03--portage.py--code.patch 04--portage.py--messages.patch 05--emerge.py--code.patch 06--emerge.py--messages.patch 07--ebuild.sh--code.patch 08--ebuild.sh--messages.patch 09--emergehelp.py--doc.patch 10--make.conf--doc.patch 11--md5check.py--writemsg.patch 12--repoman--writemsg.patch (this two last ones are just for compatibility)
Created attachment 23788 [details] quietmode.tar.gz Updated for 2.0.50_pre15.
Created attachment 23793 [details, diff] 13--post_release_fixes.patch Doh!... Please add this one line patch to the ones from the previously attached archive if you want to test it. (btw, this _pre15 patches are still okay with _pre16)
Created attachment 24281 [details] quietmode.tar.gz Updated for 2.0.50_pre19. No change but some re-diff I think.
Created attachment 24592 [details] quietmode.tar.gz Updated for _pre21. Changes: - modified the unmerge output in emerge so that it is now one line per package in quiet mode. A side effect is that it would fix bug #28261. - in ebuild.sh, the original versions of "einfo" is now renamed "veinfo" ("v" stands for "verbose"). This function could be used in several eclasses as a replacement for einfo when the message to display does not deserve to be printed in quiet mode (for instance "Applying something.patch" or "Updating gnome 2 schemas"). The same apply to all the display functions, einfo was just an example. - added a bash wrapper for wget so that the fetching output fits in one line if you use it as FETCH_COMMAND.
Created attachment 25035 [details] quietmode.tar.gz rediffed for 2.0.50_pre22 + very minor updates
Created attachment 25203 [details] quietmode.tar.gz Updated for 2.0.50. (partial rediff + a few more messages cleanups)
Created attachment 25332 [details, diff] 01--portage.py--log_counter.patch This is the only patch that i had to change to get the patchset apply on 2.0.50-r1.
Created attachment 38870 [details] quietmode.tar.gz Still the same old patch, synced with portage 2.0.51_pre20. I think the only notable change is that quiet mode can be disabled when needed from ebuilds, using some restrict flags. That may be useful in some rare case: for instance, if a game ebuild uses 'check_license' in its 'pkg_setup', the 'interactive_setup' restrict flag will make it inconditionally verbose during this step of the build process.
Created attachment 38872 [details] quiet_screenshot.txt Up-to-date console screenshot, that shows some details that were not in the firsts versions (single line output for wget, single line output for each package to unmerge, die message pointing to verbose log, etc.)
Created attachment 39579 [details] quietmode_2.0.51_pre23.tar.gz Here is the patchset updated for .51_pre23. There are several important changes in this version: - "bin/output.sh" completly replaces /etc/init.d/functions.sh. - the short logs (logs for einfo messages, etc.) are now stored in there own format, without ansi escape sequences or things like that. - "bin/elogreader" is a new python command-line tool to read this short logs. - following the changes that were made to portage_exec.spawn() for ebuilds logging, there is now also more python code to manage the quiet output, and less bash hacks.
Created attachment 39582 [details] quietmode_2.0.51_pre23.tar.gz oops, wrong one.
Created attachment 39608 [details] quietmode_2.0.51_pre24.tar.gz minor rediff for _pre24 + minor bugfix for "elogreader"
Created attachment 39882 [details] quietmode_2.0.51_rc1.tar.gz Upadted for .51_rc1. Changes: - moved the compilation/installation spinner from bin/output.sh to pym/portage_exec.py. It is still some bash code, but now spawned by spawn(). It's a bit cleaner than the background "tail logfile | <spinner shell code>" i was using before imho. - some minor improvements and bugfixes on bin/elogreader - minor rediffs and messages cleanups
Here is an updated presentation of the content of this patches: ======================================================= == Overview of some important variables: Bash: * PORT_SHORTLOGDIR: the directory where short logs must be stored. It should be defined in make.conf. * PORT_SHORTLOGFILE: internal to ebuild.sh/output.sh. It is the location of the short log file. It is set only if PORT_SHORTLOGDIR is set. It value is ${PORT_SHORTLOGDIR}/${LOG_COUNTER}-${PF}.log. * PORT_LOGFILE: exported by portage_exec.spawn(), used in ebuild.sh (simply to be displayed in die message) * PORTAGE_QUIETFD: indicates the file descriptor number of the quiet output. It is exported by portage_exec.spawn() and is used by output.sh functions. * PORTAGE_QUIET: indicates that portage and spawned process should be quiet. In fact, it is no more used by ebuild.sh, but is still usefull for two things: - with tools like "ebuild" (or any other tool that uses portage.py), defining PORTAGE_QUIET on the command line allows to force the quiet mode. - spawned process other than ebuild.sh, like the fetch command, can use this variable as an indication that they should reduce their verbosiy. That's what "wget-quiet" does. * PORTAGE_SPINNERS: just like PORTAGE_QUIET, this var can be used to force portage using spinner for tools other than emerge. That's now its only use: it used to be the way to enable spinners in output.sh, but this is now done in portage_exec.spawn(). Python: * portage.settings.quiet: indicates that we are in quiet mode. This boolean is set if PORTAGE_QUIET is in the environment, and is forced by emerge when using --quiet. * portage.settings.spinners: indicates that we should use spinners if in quiet mode. This boolean is set if PORTAGE_SPINNERS is in the environment, and is forced by emerge when not using --no-spinner. * output.noiselimit: integer that indicates what kind of portage messages should be displayed. It defaults to 0, which gives a behavior similar to the current one. If set to 1, users will also see debug messages, and if set to -1, he will have a much more quiet output. This variable is used by all the print*/write* functions of output.py. * output.spinner: this is a global instance of the Spinner class. It is used by both emerge and portage.py, and also by the output.print_out() function when used with the 'spin' parameter. ======================================================= == Patch / new files details: 01--portage.py--logcounter_fix.patch: This small patch makes portage use a single counter value per package instance, for its whole life, from installation to unmerging. This way it produces less logs files (exactly one per pkg installation). It has been reported separately on bug #37498. 02--python_output_code.patch: - moves "writemsg()" from portage_utils.py to output.py - adds there a Spinner class, used by both emerge and portage.py - adds there other display functions (print_out, print_err, etc.) - fixes many imports in other python files to reflect the writemsg move - introduces two evil globals ('noiselevel' and 'spinner') 03--portage.py--shortlog+quietmode--code.patch: - adds support for PORT_SHORTLOGDIR into the portage.config class and in portage.doebuild(). - adds a "quiet" boolean field to portage.config. - adds support for a "quiet" boolean parameter to spawnebuild(). It's up to doebuild() to decide when quiet mode should be enabled. This decision is based on settings.quiet, on the ebuild action, and on RESTRICT flags. - adds a few sanity checks: mainly, PORT_LOGDIR should be different from PORT_SHORTLOGDIR - finally, in doebuild(), it forces PORT_LOGDIR to $T if not set when in quiet mode (because we dont want users to have some ebuilds that fail with no verbose message to report) 04--portage.py--messages.patch: This is a long and boring patch that adds some noiselimit to many portage messages. It also adds support for output.spinner instead of the merge/unmerge dumps if in quiet mode (by using print_out(...,spinner=1)). 05--emerge--parse_options.patch Very small patch, that forces portage.settings to reflect the "quiet" FEATURES flag and/or the -v and -q options. Also takes care of the --nospinner option. 06--emerge--messages.patch: Same as patch #04, but for emerge messages. 07--portage_exec.py--quietmode+spinner.patch: Adds support for a quiet output and spinner to spawn(): - About stdout/stderr * if logfile=None, quiet=0 and spinner_period=0, then there is no magic * if logfile is set, quiet=0 and spinner_period=0, then stdout/stderr go to a `tee logfile` process * if logfile is set, quiet=1 and spinner_period!=1, then stdout/stderr go to a `tee logfile | <some bash code for the spinner>` process * if logfile is not set and quiet=1, they go to /dev/null (used for instance by the 'depend' action in quiet mode) * if spinner_period is set but not quiet or not logfile, then it is ignored (there was no use for that setup, but can be changed if needed) - About quiet output: When quiet=1, a new file descriptor is created on standard output. Its number is declared in the spawned process environment (PORTAGE_QUIETFD). This will be used by output.sh functions. 08--ebuild.sh--shortlog+quietmode--code.patch: - adds creation of the shortlog file - replaces functions.sh sourcing by output.sh sourcing - makes sure that PORTAGE_QUIETFD is not saved saved in ${T}/environment (because it can change depending on the ebuild action) 09--ebuild.sh--messages.patch: Replaces many "echo" by the appropriate output.sh function. 10--portage.py--spinner.patch In spawnebuild(), set the spinner_period parameter, depending on the ebuild action. 20--doc+conf.patch: - Adds some documentation in cnf/make.conf and cnf/make.conf.x86 about PORT_SHORTLOGDIR, wget-quiet (for FETCHCOMMAND), and the 'quiet' feature flag. - Adds --quiet doc in pym/emergehelp.py and man/emerge.1 There are other docs that still lacks: - a manpage for elogreader - doc for the different output.sh functions in man/ebuild.5 - doc in cnf/make.conf.* (but x86) bin/output.sh: Replaces and complement /etc/init.d/functions.sh. Here are the functions that are defined: - v*() (vinfo, verror, etc.) are very similar to the e* from functions.sh, and only output to stdout. They are based on the e*() that were in functions.sh. - elog() writes a message to the $PORT_SHORTLOGFILE if defined - q*() output messages to the $PORTAGE_QUIETFD (the quiet output file descriptor) if defined. - p*() output messages to both standard and quiet outputs (call q*() and v*()) - e*() output messages to both standard and quiet outputs, and also call elog() to write them in the log file. - techo() is an echo equivalent that is redirected to the current terminal (either stdout or quiet output) - tbell() send an "\a" char to the user terminal, using techo() bin/elogreader: This is a command line tool that filters and prettyprints short log files. It mainly lacks a manpage, and some user feedback (I personaly like it the way it is, but I'm opened to any suggestion.) bin/wget-quiet: This is a small wrapper for wget. The goal is to have a single-line output when emerge is in quiet mode (PORTAGE_QUIET in environment). It can safely replace wget in FETCHCOMMAND.
Created attachment 39950 [details, diff] portage_exec.py--dev_null_bugfix.patch Doh! Shame on me, here is a quick fix for an obvious bug of the previous patches tarball.
For crying out loud, why havn't this patch been added to portage yet? It's been lingering here for almost 1 year for apparently no reason at all even though it fixes a IMO huge design flaw in the very core of what is Gentoo, namely Portage & it's crippeling verbosity. Please someone with the apropriate permissions, add this to portage to save us all from the compleatly meaningless compilespam.
The patches would need some rework to be integrated tho, because there were quite some changes since my last sync (portage_exec.py, the fancy spinner, etc.). Plus it adds a few globals that i could probably kill, things like that. Nothing hard i think, but should be done, and i will wait to have some 2.0.52_pre snapshots to do it, since this is for sure to big for a 2.0.51 revision.
Putting a hold on feature requests for portage as they are drowning out the bugs. Most of these features should be available in the next major version of portage. But for the time being, they are just drowning out the major bugs and delaying the next version's progress. Any bugs that contain patches and any bugs for etc-update or dispatch-conf can be reopened. Sorry, I'm just not good enough with bugzilla. ;)
*** Bug 117043 has been marked as a duplicate of this bug. ***
I know solar posted a patch to the ML...Can you attach it here ( along with the two ugly ones you didn't post to the ML? :) )
Most of this functionality exists in trunk, so InSVN
Released in 2.1_pre2. Please reopen if it's not quiet enough.
