Hey devs, the work is finally done - and it works flawlessy (at least on my box at work). Here is how i added it: 1) A new variable has been added to /etc/make.globals: ELOGFILE. This variable holds the current logfile. It defaults to "/var/log/emerge.inst.log". 2) I added a new class to "portage.py": instlog. This class handles the maintenance of EINSTLOG. It has two relevant functions: dump() and flush(). dump() dumps ELOGFILE to stdout. flush() wipes all content from the ELOGFILE. 3) Modification to ebuild.sh: I had to add a var called ELOGGING. This defaults to "no". If we run into a pkg_preinst or pkg_postinst, ELOGGING is set to "yes" 4) Modification to functions.sh: einfo() and ewarn() are modfified to write to ELOGFILE when ELOGGING is set to "yes" 5) Added flush() and dump() to depgraph.merge() In a nutshell: - user types "emerge foobar". - ELOGFILE is set to "/var/log/emerge.inst.log (make.globals). - /usr/bin/emerge runs into merge(): the ELOGFILE gets flushed with flush(). - /usr/bin/emerge happily merges around and calls /usr/sbin/ebuild.sh - if we run into a pkg_postinst() or pkg_preinst(), ebuild.sh sets ELOGGING to "yes" or "no". - /usr/sbin/ebuild.sh calls /sbin/functions.sh. - /sbin/functions.sh drops the output into ELOGFILE when ELOGGING is set to "yes" (that means, it drops the output when we run into ewarn/einfo in pkg_postinst/pkg_preinst). - /usr/bin/emerge finishes because mylist[] is done. - /usr/bin/emerge uses dump() to dump the ELOGFILE to stdout. Well thats it. Give it a try - i will attach the 5 files (ebuild.sh,emerge,functions.sh,make.globals,portage.py) tar bzipped. Thats the way to install it: $ tar jxvf foo.tar $ cp ebuild.sh /usr/sbin/ebuild.sh $ cp emerge /usr/bin/emerge $ cp functions.sh /sbin/functions.sh $ cp make.globals /etc/make.globals $ cp portage.py /usr/lib/python2.2/site-packages/portage.py Use it and you should see something like that right after "clean": ------- | >>> These messages have been logged to /var/log/emerge.inst.log : | * $PN: message | * $PN: message ---- (BTW, "emerge gentoo-stats colortail" is nice for testing - doesn't take long to download/install it and produces quite some output) So what do you say? -phoen][x-
Created attachment 6039 [details] portage-modified.tar.bz2
*** Bug 4138 has been marked as a duplicate of this bug. ***
Created attachment 6046 [details] patches.tar.bz2 Okay i'm evil. I forgot the patches. :( Here they are. -phoen][x-
Wouldn't it be handy to also save them permanently (in a second file)?? That way you can see all of them after you have done several emerges in a row or for future reference? It shouldn't be too hard to implement I guess.
Heyas Sam, the file would fill up pretty fast. And it would need user interaction (eg "rm /var/log/emerge.inst.log") to ~clean~ it up again. Sorry, but I dont think that this is a good idea. This very information we log at the moment, isn't really interesting for more like 10 minutes IMHO. You read the info and follow the instructions. The next merge places new instuctions in the file. That's how i wanted to implement it, and that's how i implemented it. :) However, if one of you (/me looks at the Cc: list) likes Sam's idea, try to explain me why we need it and i'll add it. -phoen][x-
I haven't tested it yet, although the basic scheme looks sound. What happens if the user hits <ctrl>-c during emerge? Will the already-generated log messages be displayed?
*** Bug 12202 has been marked as a duplicate of this bug. ***
*** Bug 13577 has been marked as a duplicate of this bug. ***
Great. I would find it nice too, if a permant log could be enabled by option (by command line or config). This would be nice for automated emerge's. One reason: It's not always possible to emerge all needed things on line. E.g. I can't do 'emerge postfix fcron' on one line, because fcron needs an mta and therefor ssmtp is choosen, but that conflicts with postfix. There are more such problems (e.g. global use-flag sasl, but I have to emerge squid with USE="-sasl" because squid wants saslv1. But collecting einfo and ewarn messages are great feature I allways missed. Thanks for that.
Sorry for that long line, I don't know how I've done this. :(
*** Bug 14302 has been marked as a duplicate of this bug. ***
*** Bug 15206 has been marked as a duplicate of this bug. ***
I agree that a more permanent collection would be nice. Personally I often get into trouble because I do emerge -u world. If several packages are being emerged I never get to see the ewarn or einfo messages belonging to the packages in "the middle". Perhaps a feature to buffer up the messages and less them at the end of the multiple emerge?
I vote that this would be a very nice feature as well.
*** Bug 21525 has been marked as a duplicate of this bug. ***
This enhancement seems pretty popular, is any work currently been done on it or being planned. Design, programming, or otherwise?
*** Bug 22471 has been marked as a duplicate of this bug. ***
I didn't know this bug.. I've made a patch (#22471) which is very simple. Can you guys look at the bug. But this patch looks good as well :) I think if we make a logfile, it should be separated files. example 030609_01_info.log yymmdd nn nn=number of emrging of day. Then after emerge it should say "You got messages during emerge. Please look at /var/log/somewhere/030609_01_info.log" Sorry, my poor English :)
*** Bug 24913 has been marked as a duplicate of this bug. ***
*** Bug 25256 has been marked as a duplicate of this bug. ***
*** Bug 18079 has been marked as a duplicate of this bug. ***
+ echo -e "${WARN}* ${HILITE}${P}${NORMAL}: ${*}" >> ${ELOGFILE} This is the first thing that caught my eye: The variables ${WARN} , ${HILITE} and ${NORMAL} all specify bash escape codes for color and should not be written to a text log file.
+ def dump (self): + """ + Dumps the logfile to stdout. + """ + print ">>> These messages have been logged to",self.elogfile,":" + self.open(mode="r") + for line in self.mylogfile.readlines(): + print line, + self.close I'm assuming this is a typo, it should probably be: self.close() (this isn't really critical as python will take care of closing it, due to reference counting *I think*, but anyway...) Also, when will the flush function be used?
I would like to see a similar feature: emailing of important emerge messages to root. I've written a small patch to do just that, which I'll attach directly. The major flaw with my patch is that you get an email for every invocation of einfo, ewarn, or eerror. Perhaps my approach is the wrong way, and instead we should log everything then at the end of each emerge simply mail the log to root. What are the chances of getting this functionality worked into this whole mess as well?
Created attachment 16300 [details, diff] patch for functions.sh implementing emergemail feature to try it out, set FEATURES="emergemail" and try an emerge that has some einfo or ewarn messages.
the mail thingy is used in debian too, i like the idea
Created attachment 16341 [details, diff] updated functions.sh patch With some lovely advice from Karsten Schulz on gentoo-dev, I managed to remove the previously noted nasty limitation of my implementation. Now you get all the einfo messages in one email (if there were any), all the ewarns in one, etc. I've used it on a couple ebuilds and it works quite nicely. Please try it out and comment. To try it out, you might do.. % cd /sbin % patch < /home/me/functions.sh.patch % emerge stuffit (stuffit chosen because it has no deps and it fails immediately with an einfo)
I forgot to mention that you have to add "emergemail" to FEATURES in /etc/make.conf. So that's: % cd /sbin % patch < /home/me/functions.sh.patch % $EDITOR /etc/make.conf # add emergemail to FEATURES % emerge stuffit (stuffit chosen because it has no deps and it fails immediately with an einfo)
I testet the new functions.sh with the following ebuilds: app-benchmarks/dbench (einfos) app-emulation/vmware-workstation (einfo, ewarn) app-emulation/point2play/point2play-1.0.1.ebuild (eerror) After that, I emerged -U --deep world: [ebuild U ] sys-libs/db-3.2.9-r7 [3.2.9-r2] [ebuild U ] app-editors/vim-6.2-r2 [6.2] with some einfos from db and vim without any problems. It works for me. Karsten
With exim i had to patch as follows below. I wonder if there is any clean solution for this that works with all mta's ? Or maybe it's only exim that is problematic... --- patch lookalike begins here: (Excerpt starts at line 1136). [ -z "${CCACHE_SIZE}" ] && export CCACHE_SIZE="2G" ccache -M ${CCACHE_SIZE} &> /dev/null fi + if has emergemail ${FEATURES} &>/dev/null; then + addwrite "/var/spool/exim/input/" + addwrite "/var/spool/exim/msglog/" + fi fi # "$*"!="depend" && "$*"!="clean"
I see, this could be a problem with other MTAs. If people with courier, postfix, and sendmail could post patches as well we'd have decent coverage. I'll piece the patches together and post the results. Can't do it until I get back from a trip this weekend. Mondayish.
You could stick to SMTP mailing. While you are at it, you could also add EMERGEMAIL_SERVER and EMERGEMAIL_MAILTO (would make sense to use default values like 127.0.0.1/root in make.globals). This would provide a clean interface to all MTA's, since SMTP is standarized in an RFC. Furthermore, it would not require a working local MTA (since you can use EMERGEMAIL_SERVER in make.conf to override the SMTP server). Only an idea. -phoen][x-
emergemail works well with postfix MTA. My tests (see comment #29) were done with postfix installed on my system. But i like to encourage you to consider phoenix' suggestion to use smtp (at least as an additional option) have fun! Karsten
I have found two more issues. Issue 1) _Occasionally_ i get a mail looking like this: information from emerge of gcc-3.2 From: abuse@isp.com <-- not my real adress ... To: CFLAGS="" CXXFLAGS="" ----------------- This happens for gcc 3.2,2.95.3 and xfree 4.2.1 It seems to have something do do with doing emerge --pretend --update --deep world after having done rsync. But I'm not sure yet. Maybe it's the rsync. Issue 2) epatch uses info so all packages that uses it to apply patches sends me a mail telling me this. I feel this information is unecessary and that maybe the eclass which epatch is in should be changed or that the messages should be filtered.
see also bug #2834
Okay.. I'm struggling a bit switching over to SMTP mailing. I can do it with telnet, but telnet is not default install in gentoo. Is there another way to do SMTP mailing from the command line, without necessarily having an MTA? Otherwise I can submit a patch with telnet SMTP mailing, based on EMERGEMAIL_SERVER and EMERGEMAIL_MAILTO as was suggested.
*** Bug 28397 has been marked as a duplicate of this bug. ***
*** Bug 29751 has been marked as a duplicate of this bug. ***
please add this feature asap, maybe including mailx feature to mail root
Please consider people who are installing the system from scratch. Sending email to root during a base install is somewhat pointless. Of particular concern to me is that early emerges can pull in dozens of dependancies, some of which may generate their own messages. I had gone quite a way down an initial install before I noticed any of these messages (they aren't mentioned anywhere in the docs that I have seen). Since some of them seemed quite important, I decided to re-install everything from scratch, redirecting all emerge commands (including 2>&1) to individual log files (disk space is cheap). This way I could browse the files afterwards looking for these potentially helpful messages. Note that when logging command output that you must append if a log file is already there. See bug 31177 for an example where that wasn't done inside genkernel.
I got an idea, how about making portage write the info to a file(one for each ebuild that writes info) in /var/log/portage (or some other place), and then point the user to read the info from those files after the emerges have completed?
Failing more sophisticated fixes, a possible quick and dirty interim fix is to mark all important messages with a characteristic text string, such as an initial #####. The the "Find in History" option can be used in the screen build log, or the entire log can be tee'd to a file which can be searced trivially by most editors. I'd be really happy to see a fix of some sort - I'm *sure* I'm missing important stuff on "emerge -u world".
*** Bug 32358 has been marked as a duplicate of this bug. ***
Have anyone considered ncurses for displaying messages during install? You could create a window with a confirmation dialog for each message.
confirmation requires interactivity which breaks the general non-interactive rule for emerge, so it's not a viable option.
This improvement is in the air. Publish early, publish often. Please put it in production tree, and let's see what happens.
*** Bug 34856 has been marked as a duplicate of this bug. ***
(assigned from Bug 34856) Sorry for asking, but I just don't get the technical point at the moment: Is this intended to show messages from pgk_setup _before_ emerge takes place? (see mozilla-gtk2 vs. galeon-gtk1 problem) Or if not, would it be possible to use it for that like: 'display pkg_setup' 'countdown 10 seconds (chance to hit ctrl+c)' 'normal emerge' TIA
Is it not possible to use ncurses to stack dialogs? Just keep them there and continue merging.
*** Bug 35575 has been marked as a duplicate of this bug. ***
Just had a question about the portage-modified.tar.bz2 ball when following your instructions about where to copy things: Thats the way to install it: $ tar jxvf foo.tar $ cp ebuild.sh /usr/sbin/ebuild.sh $ cp emerge /usr/bin/emerge $ cp functions.sh /sbin/functions.sh $ cp make.globals /etc/make.globals $ cp portage.py /usr/lib/python2.2/site-packages/portage.py I found that inside of ebuild.sh it looks at: if [ -e /etc/init.d/functions.sh ] then source /etc/init.d/functions.sh > /dev/null 2>&1 elif [ -e /etc/rc.d/config/functions ] then source /etc/rc.d/config/functions > /dev/null 2>&1 fi and so it is looking for the new ebuild inside of /etc/init.d/ when you said to place it in /sbin/ Did I miss something here?? I fixed this by cp /tmp/functions.sh (new file from tar ball) /sbin/functions.sh.new and changed the lines about /etc/init.d/functions.sh to point to the /sbin/functions.sh.new (.new as to keep my *working* function.sh working. Well just thought I would give some user feedback ;-) Jeremy G.
*** Bug 38397 has been marked as a duplicate of this bug. ***
*** Bug 36467 has been marked as a duplicate of this bug. ***
I'm attaching a revised functions.sh.patch that alters the sendmail stuff to stop mail clients showing "To: undisclosed-recipients: ;" (I use kmail).
Created attachment 25855 [details] updated functions.sh.patch
*** Bug 43191 has been marked as a duplicate of this bug. ***
I'd vote for this feature as well. Any plans on when this will be implemented officially?
Why not use the python smtplib to send e-mail through any SMTP server? I'm attaching a simple mailsend script that will send the specified file to the recipient from the sender using the specified host. USE: mailsend filename sender recipient host "subject" (subject isn't being sent for some reason. Not enough time to look at it right now) Maybe the scripts could be modified to use something like this mailsend instead of a local MTA.
Created attachment 27224 [details] Python script to send file by e-mail (uses smtplib)
My concern about this would be if something like this is put in the scripts, there still has to be a way to send to a local user with an mda or something.
Should be simple enough; a variable in make.conf (oh no! Another one!) detailing which local user or e-mail address to send mail to.
mailsend accepts arguments: filename sender recipient host subject So if you need to send the contents of a file to a local user you just need to pass the local user as recipient and localhost or 127.0.0.1 as host and the mail will be sent using the local MTA. The advantages of doing this through SMTP are 2: 1) it will work no matter which MTA is installed on the host. 2) if no MTA is installed, the user should be able to set a different mail host. Regarding Stewart's comment, two variables should be set: mail recipient (local or remote) and mail host (which could default to localhost). Sender and subject should be automatically generated.
Created attachment 27382 [details] Python script to send file by e-mail (uses smtplib) The subject is now sent correctly. It was a stupid mistake.
Sending mail assumes that someone is going to read it. Most of the time, they aren't. It's junk. Most machines probably have no MUA anyway. It would be far better (simpler, easier to write, easier to read, easier to grep) to write a numbered logfile in /var/log somewhere.
Granted, but with every option comes the notion that they are "optional" :P I do like the idea of a log file regardless of the e-mail functionality - I just like the notion that a daily / per-merge log could be mailed to me. With the user / mailhost variables, it could be sent anywhere, even somewhere like Hotmail or Yahoo.
*** Bug 45664 has been marked as a duplicate of this bug. ***
*** Bug 47910 has been marked as a duplicate of this bug. ***
Please incorporate Owen Gunden's excellent "emergemail" (Comment #25) patch into Portage - I have to keep repatching functions.sh.
*** Bug 51823 has been marked as a duplicate of this bug. ***
The script in comment #55 works great. I wish I had this one installed on my sun u10 before silo got updated... The system coudn't boot any more, because silo has not been executed, but it was mentioned in an ewarn (which scrolled by during the update of more than 30 packages). Who is currently in charge of maintaining this bug-report? It would be great, when it becomes included in Gentoo 2004.2
Hey, 57 persons are in CC. The bug is there since 11-2002. 21 duplicates have been filled The proposed patch will soon blow a candel Machines keep crashing for their maintener don't see the ewarns Don't you think it's time to do something about it (and I don't talk about WONTFIX) ?
the feature kind of already exists with PORT_LOGDIR but no one has mentioned that fact
PORT_LOGDIR logs the entire build process. This processed a email of the ewarn itself.. There have been patches submitted but they won't include it in the source tree.. Thats the problem.
A log dir of all commits is not the same like an email about the einfos/ewarns. Administrators should receive a (short) note on their desk that something has changed instead of pulling it manually out of the directory. So, if it is already implemented, why is the email-script missing?
Well, for one thing, the default Gentoo install does not include an mta or an mda. (Similarly, methods that rely on syslog are also problematic, since a syslog daemon isn't installed until one of the last steps of the install process.) Given those facts, I don't think that anybody has managed to sell the portage team that the current patches are the "right" solutions to this problem. My personal opinion is that the "right" solution would be to: (a) create a new function (or extend einfo / ewarn) that would be called in ebuilds to log important text to a log file for just this purpose (b) Modify portage to collect all of the information that has not yet officially been presented to the user and either spit it all out at then end of the emerge, or e-mail it to a particular user, or just leave it as is in the log file (with the user choosing between those options by using appropriate FEATURES settings). (c) Fix all ebuilds in portage to use the new system appropriately (d) Ensure that the new portage code is robust to <ctrl>-C's, power failures, etcetera. This "solution" requires a lot of work, though, and it still lacks the important step of selling the portage team on the idea that it is more important for them to work on this problem than it is for them to work on the myriad other significant bugs in portage.
I beg to differ. They don't have to work on anything. If you read threw this laundry list, there were patches made for this. it was just a matter of them adding it. Not writing it. And this was added as part of the features line, with variables in make.conf for how to send the email.. This has already been thought off. my question is why it didn't get added.
which patch? functions.sh belongs to baselayout, not to portage. I haven't looked at the tarballs, but I'm sure they won't apply to current portage code anyway. Also, do you really want one mail per einfo/ewarn statement (it's possible for them to be called in any shellscript) and all mails sent to root with no option to change that (I'm sure when we add this the request will come)? The patch only focuses on mail notifications, but there are several other solutions (delayed display, message queue, syslog, ...) What I want to say is that this issue needs a generic solution, not a specialized one.
I don't believe this. This is a very real problem which could be so easily solved. To say it can't be done because it's not a perfect solution is just daft and only ends up with Debian-like delayed development. Yes it is specialized... there is a specific need! Is Gentoo not about choice? What actual harm would be caused by adding this as a FEATURE, with a comment warning about it's requirements and the fact that all einfo/ewarns would be mailed? functions.sh is indeed in baselayout and appears to log all ewarns to syslog as the facility "rc-scripts". Since when was portage an rc-script? I for one would find an email far more useful than a syslog entry (push rather than pull). If the patch in #55 causes portage to break, say so and something can be done about it. Otherwise, please include it.
I attempted to apply the "updated functions.sh.patch" to functions.sh in the ~x86 baselayout so I could see how this works before I comment and got the following message: "malformed patch at line 57" What did I do wrong to get this message?
Re: #79... That's hardly surprising - look at the first line: --- functions-old.sh 2003-08-18 21:49:35.000000000 -0400 There have been a lot of changes to functions.sh since then. You can still manually patch it in an editor and it works. Just paste the first lot of additions at about line 325, before this line: # void esyslog(char* priority, char* tag, char* message) ...and the rest will be obvious. I don't want to keep updating the patch because the negative response to this bug would mean I would be forever doing that.
To fix the patch, change 53 in the third line to 54. As for the rest of it, at least for me email is a perfect solution, since procmail allows to do whatever I want with it (log to a file, send to someone else, etc.) I was looking for something like this for some time.
I agree with the previous comment in this bug that functions.sh is not part of portage. Because of this I'm not sure that we should be patching functions.sh for a portage specific feature. What I would like to do is look into reworking the patch that is provided in this bug and patching ebuild.sh instead. Also, I like the idea of using the mailsend script that is included in this bug to send mail, that way we aren't depending on the existence of a sendmail program. My thought is to look into patching ebuild.sh and using the mailsend script instead of calling sendmail. I will propose a patch once I have this done. What do you think?
Do not post to this bug again if you aren't fully aware of the following comments. The best way to get this bug ignored or closed is to make it unreadable/unfollowable. Comment #72 -- PORT_LOGDIR Comment #75 -- Design and Reasons Comment #77 -- Options and issues I will _never_ accept odd patches. If someone reports that it's flawed, it will be ignored. If you really want this feature, work it out into something that is a very clean, very small patch that affects nothing beyond it's target and doesn't do things that are insane like mailing 500 messages to the admin. If there is a flaw in your design, I will pick at it. The cleaner you make the additions, the less time you'll have to spend explaining why your solution is legitimate and reasonable. http://www.gentoo.org/proj/en/portage/index.xml Read sections 8 and 9. They mention most of this. Potential help for a better design: ${T}/something can be used to store entries. It should remain until the merge completes. Binaries will never see anything prior to pkg_preinst. One screen of e{info,warn,error} is up to 24 distinct messages. Some systems will not have an MTA. Some systems will not have an MUA. Some systems will not have a logger. If you have tried all possibilities to break your patch, and have found nothing, you may posting it here with descriptions of what you have done to test and the description of the patch and design. =========== Continued flaming in this bug will get it closed, and it will then be completed whenever the portage team gets around to it. We do not have time to deal with complaining of this magnitude. Help out in a useful way, or be quiet. Trying to help is great. Even if your patch or solution isn't accepted, you or someone else may become inspired to make something better. ===========
*** Bug 54623 has been marked as a duplicate of this bug. ***
*** Bug 55081 has been marked as a duplicate of this bug. ***
*** Bug 57661 has been marked as a duplicate of this bug. ***
I just returned from two hours of hunting an error after -u world which I finally solved by manually checking the ebuilds for einfos manually. I don't want to be impolite, but why hasn't this been included yet? Users are continiously requesting it and I don't want to know if an invisible einfo already caused massive system damage.
Although I also would love to see such a feature in Portage, there are at least some tools that can provide similar things, take a look at http://forums.gentoo.org/viewtopic.php?t=67849 .
*** Bug 21392 has been marked as a duplicate of this bug. ***
Created attachment 36713 [details, diff] Ebuild einfo/ewarn/eerror sent to syslog Patch against: sys-apps/baselayout-1.9.4-r3
Just uploaded a new patch regarding this functionality. This patch simply logs the einfo/ewarn/eerror output. This should alleviate many of the concerns expressed with previous patches, including: log file size, output log persistence, sending emails, enabling/disabling output. By sending the output to the system logger, you can configure your system logger to handle all of these issues, just like for any other system log events. Facility: user.info, user.warning, or user.err Program: ebuild Output format designed to allow simple regexp matching for further log redirection or searching.
I am working on a patch to do this also. However, I believe this should be done in portage itself, and not in baselayout since it is a portage specific function. The approach I am working on is to patch ebuild.sh with custom versions of ewarn, einfo and eerror which display the messages and log them to a file, then I will need to figure out where in portage to write the code to actually send the email.
so many duplicates... we should probably change this bug summary name to catch those search terms... (bug within a bug!)
Please do not use email or syslog, or else make them configurable options along with a normal file log option. Any solution to this problem must deal with bootstrapping the initial install as well as systems that do not have an MTA or logger loaded. The current system using PORT_LOGDIR almost works, but it would be real nice to split out the einfo/ewarn/eerror output from the compile and install messages. Another deficiency of PORT_LOGDIR is that not all of the information sent to the terminal is logged. Please re-read comment #83.
Can I just offer my $0.02. Logging to file should be the default. Email should be configurable. But even if emails are configured, always log to file (emails can go missing, after all) IMHO, full einfo/ewarn should NOT go to syslog, but at most a one-line entry in syslog saying "new ewarn info added to file ... for package ..." should be made. One line for each package. I'm not sure whether einfo should be added too. Perhaps instead of syslog, logging to /var/log/emerge.log? Rather than saying log to syslog|email|nowhere, make each option configurable. So you can hav all turned on if you like. So, configuration options would be: email address (optional; maybe default of root@localhost) level of email reporting (none|ewarn|einfo) (default none) level of syslogging (none|ewarn|einfo) (default none) level of emerge.log logging (maybe we need a better name for this?) (none|ewarn|einfo) (default none) One last thing: (it may not be my place to say this, but that won't stop me :-)The status of this bug is still NEW, but it seems that a lot of stuff has been written. Assigning it to someone might make things move along a little more quickly.
Best would be absolutely no email support. Adding it would require a dep from sys-apps/portage on virtual/mta, which is not an option. If the einfo messages are logged to a file, anybody who needs them emailed would be able to write a script to get it to happen automatically - especially if a notification is syslogged. As for ease of use for users, such mail support would go into a different tool rather than portage itself. At least that is the case until portage is modularized enough that it can support a use flag for the dep.
Also, forgot to mention that creating a /var/log/portage directory will make portage log all output of each emerge to individual files. Using /var/log/emerge.log to know which files to check in /var/log/portage (or even some find -mtime magic) allows you to get at this information fairly easily. Addendum to the previous comment: mail support in portage is fine as long as it uses python to open an smtp connection a configurable server to send the mail.
As I am one of the people, if not the only person, that requested email support, then I am happy for initial support to just include logging. As long as any important output is available after the emerge has completed, then I will be happy enough. I can always knock up a script to tell me when the file has changed, but I'll probably look at it 1) from time to time and/or 2) after an emerge and/or 3) (more likely) just when things go wrong.
Why would it *require* a dependency? I don't see portage depending on distcc or cvs, and they're both used when FEATURES is set accordingly. As I see it, the "proper" way to do this, would be to add a FEATURES flag, such as "elogging", along with an ELOGGING_TYPE flag in /etc/make.conf that can be set to one or more of mail, syslog or file. In the case of mail, ELOGGING_ADDRESS could specify the recipient. In the case of file, ELOGGING_FILEDIR could specify a directory to log to, or specifying 'file' could simply indicate that you want to log einfo/ewarn to PORT_LOGDIR. Personally, I think this stirs things up too much to happen, so a simpler alternative: Could PORT_LOGDIR be used? It could be extended to support more than one target, comma or space delimited. If a target starts with a /, it's a directory to log to. If it starts with mailto:, it's an address to send mail to via the system MTA. We still need to be able to specify the "log level", such as full emerge logging (as PORT_LOGDIR does now), or just einfo/ewarn logging (most useful to 99% of us). Whatever is best, I would do something like: PORT_LOGDIR="/var/log/portage,full mailto:mal@komcept.com,einfo" With full as the default if none specified (allowing current make.conf's to continue unhindered). If comma is a problem in paths, another character can be used or it can be backslash escaped. Whether or not an mta/logger are present are largely irrelevant. A simple comment in make.conf should stop newbies from trying to log bootstrap einfo/ewarn to syslog or mail. A simple check for the existance of /usr/bin/sendmail, or indeed by using python's own SMTP API is enough to avoid total failure. It would also not be hard to fall back to file logging, if PORT_LOGDIR is set, but no MTA is found. Again, it's all about choice. You don't have to log to mail, and those admins who do, will do knowing full well that mailing may cause emerge delays, or that the mail might not get there. However, I would bet that 99% of the time it will be mailing a local domain anyway. I apologise for the length, it would be good to find some common accepted method. Please comment.
To me, email all is the best answer. I continue to use emergemail (Comment #55), repatching functions.sh as necessary. It has saved my bacon numerous times.
Let's not invent the wheel when there's a fully configured SUV already parked in the driveway. Regarding using a logger: IMHO, this is the right way to address the problem. True, there may not always be a logger installed, such as in a bootstrap installation situation, but the need for logging during this phase is arguably minimal: Logging is much more useful for maintaining an existing system and knowing when your world updates internally contain recommendations for a revdep-rebuild or advise of conflicts or changes. And it's easy for the patch to just bypass logging if there is no logger present; my posted patch already does this. Regarding using an MTA: This is where the logger is useful. 95% of all system loggers are configurable to do many things based on both the source and a regexp of the contents of the log message, including sending email (and even that can be done externally, so if you want to use Python+SMTP, feel free to configure your logging that way). The email generation can be based on a regexp, so it can limit the messages it sends based on the regexp. Alternatively, the mail program invoked by the logger can do its own filtering and history maintenance, or you could have a cron job that did something like emailing whenever the ebuild logfile was updated (obviously only for systems with cron installed :). Regarding using a PORT_LOGDIR: This is where the logger is useful. You don't want everything, just certain output (like ewarn/eeror). Just setup your logging to log to the directory/ies and file/s you'd like it to. The logger will automatically rotate log files based on many factors, including size and age. Log files will have appropriate persistence, and will be located in customary locations (unless you redirect them otherwise). [Yes, I have read comment #83] Regarding modifying portage rather than baselayout: While I tend to agree that the einfo/ewarn/eerror functions should probably have been part of the portage package because that's the only place those seem to be used, I don't think it's appropriate to make significant cross-package changes to add a small enhancement. The new generation of portage is being designed even now, so I'm assuming that the designers are watching portage bugs and will take these issues into account where appropriate for the new portage; minimizing churn in the current generation will actually help those folks. Most loggers can invoke an arbitrary external command, so you get very significant amounts of flexibility based on the filters applied by the logger. See my wheel/SUV comment above, and apply the maxim that one of Unix's strengths is the ability to utilize many different small utilities in concert rather than building huge multifunction applications (epitomized by MSWord). Yes, I'm making the argument/assumption that logging/emailing/RSSing/paging/text-messaging/whatevering during system installation/bootstrap is of low value. I know this can be argued, but it should be juxtaposed against the effort required to develop a custom portage-internal solution to all the problems that are already handled by system utilities designed for just such purposes.
Okay.. forget about mail/syslog/etc for the time being. Just create a patch that has the following 2 problems solved and add that stuff later. 1. Lines are never intermingled even if there are 100 concurrent emerges. 2. Lines from pkg_setup to post_install remain in one block.
Created attachment 36784 [details, diff] patch to ebuild.sh to collect einfo/eerror/ewarn messages This patch to the cvs version of ebuild.sh will collect einfo/eerror/ewarn messages in ${T}/messages so that they can be emailed/logged/handled another way later.
*** Bug 60706 has been marked as a duplicate of this bug. ***
*** Bug 60912 has been marked as a duplicate of this bug. ***
Lovely :) When do you think we'll have this working?
A modification similar to William's from above I intend to slip in after .51 is out the door. Right now, under a bit of a moratorium on new features for 2.0.51* . I intend to just nail down info/warn output, summarizing at the end... the extra stuff people are after is another thing.
How about this patch to ebuild.sh: http://dev.gentoo.org/~eldad/ebuild.sh-enotice.patch Other than overriding einfo/ewarn/eerror, it also goes a step further and put the notice file in a user-defined directory (PORT_ENOTICE_DIR env in /etc/make.conf)
I changed the patch abit, and also written a nice (python) tool to read/delete the notices: http://dev.gentoo.org/~eldad/enotice
Ok, I've tried your patch eldad but I can't get it to work. Here's what I did: # vi /etc/make.conf (set PORT_ENOTICE_DIR to /var/log/portage-enotices) # mkdir /var/log/portage-enotices # cd /usr/sbin/ # ls -l ebuild.sh (note that it's a symlink) # cd /usr/lib/portage/bin # wget "http://dev.gentoo.org/~eldad/ebuild.sh-enotice.patch" # patch < ebuild.sh-enotice.patch (patch succeeds) # cd /usr/local/sbin # wget "http://dev.gentoo.org/~eldad/enotice" # chmod u+x enotice # emerge gentoolkit (there's an einfo at the bottom) # enotice No notices found. # ls -a /var/log/portage-enotices/ ./ ../ what am I doing wrong?
Probably a permission problem on PORT_ENOTICE_DIR. I've changed the patch abit to create and set the right premissions to PORT_ENOTICE_DIR. If you (or anyone) else have a problem with it, mail me directly first - this bug is long enough as it is...
*** Bug 65564 has been marked as a duplicate of this bug. ***
*** Bug 66385 has been marked as a duplicate of this bug. ***
So when is this going to make it into the production portage system? I need this feature too, and have patched emerge and functions.sh so as to write einfo and ewarn output to file descriptor 3, if (and only if) fd3 is available. It's pretty simple, but it works like a champ. # emerge blah woof tweet 3> /tmp/emessages There are only two patches, and they're quite simple. If fd 3 is directed to a file, then this file gets the output. If you want to, you can subsequently use 3>> to append to the same file. If there's no redirection, emerge works as before. There are no settings in make.conf, no environment vars to set, no muss, no fuss, no dirty dishes .... :-) If the devs (or anyone else) want my patches I'll be happy to post them, or email them, but if these mods are about to go to production then perhaps it's not worth the effort.
Another portage update, time to re-apply eldad's patches! (hint, hint)
so, why is this still not in portage? i mean, is there any doubt as to the usefulness of this feature? any difficulty in the implementation?
remade the patch against 2.0.51 rc9, on http://dev.gentoo.org/~eldad/.
This feature request has been opened for almost TWO YEARS and work on this really basic feature apparently still hasn't been mainstreamed into Gentoo, nor is the feature mentioned in the online Gentoo Portage docs. Why the delay?
It would be useful to show "all" einfo messages at the end of emerge process, instead in the middle as so being redirected to /dev/brain/null.
*** Bug 69380 has been marked as a duplicate of this bug. ***
Can someone tell me what's the status of this feature? Is it documented (concisely!) anywhere?
No, Lindsay. You have stumbled upon a game of bug-tennis. There's one hidden somewhere in every RT and Bugzilla. The bug report just goes back and forth until the umpire calls "out". "New Bugs Please!"
:sigh: another new revision of portage. Time to bitch about having to manually patch again!
Echoing Owen's sigh. Please put emergemail into portage.
actually i think that currently eldad's enotice feature is the best solution and the only one that has a patch which works on current portage. but this bug is out of control. maybe we need to start a new one..
The bug is out of control because this feature is something a _lot_ of people _really_ want, and if it were mainlined into portage, the bug would go away and we'd all be happy :-) This feature is pretty important for good system administration. Because of the way ebuild.sh is now spawned, my fd3 solution no longer works. Oh well....
Eldad's patch works well. It's _one_ patch, and one very convenient utility to pretty-up and manage portage einfo/ewarn output from various ebuilds, although the output is perfectly legible without it. I'll shut up now (as long as eldad's patch stays current with portage version updates). :-)
As stated above, .51 had no further features stuck into it, and I intend to add equivalent functionality into post .51 (so complaining about a release w/out it is pointless, .51 *needed* to go out, not sit collecting 101 features). Either way, here's how you do it without patching portage. stick this in your /etc/portage/profile/bashrc if [ "$EBUILD_PHASE" != "depend" ]; then func=$(type einfo | tail -n +3) eval "einfo() { $func }" fi haven't tested it, but that should be a way to do it via bashrc w/out having to patch portage.
I've done a bit of shell programming but I don't quite grok this. I have no /etc/portage/profile/bashrc. Do you mean /etc/portage/bashrc? I don't mind patching portage. This is, after all, Open Source :-) Having complained as much as anyone about the lack of this feature, I'll put in 2 cents worth and say that eldad's patch and enotice management utility are extremely good from a human interface POV. einfo/ewarn output is saved w. indexing so the messages emitted from any past ebuild can be easily found and reviewed, or deleted from the collection. The ability to save and easily retrieve and manage these messages has already saved me a bunch of time on troubleshooting several ebuilds! I'm a happy camper. I'd suggest that eldad's patch and utility be the basis, at least from a UI perspective, of any such system put into portage.
Brian, I've played with your code snippet in /etc/portage/bashrc and I don't see how it's supposed to work, or what its advantage would be if it did. Since ${PORTAGE_BASHRC} is sourced _before_ functions.sh, any definition of einfo() in the bashrc is overridden, and type(P) will return an error if einfo() hasn't already been defined.
Created attachment 43541 [details] eldad's ewarn/einfo logging incorporated into /etc/portage/profile/profile.bashrc Looking at the ebuild.sh code, I believe Brian intended to put his logging code into /etc/portage/profile/profile.bashrc (rather than /etc/portage/profile/bashrc), which is sourced _after_ functions.sh. Eldad's patches can be fully incorporated into this file and will work. See the attached profile.bashrc. I had to dispense with the temporary directory and write directly to the target file in PORT_ENOTICE_DIR since the order of execution doesn't permit a move of the temp file to this directory without losing output from ebuild's postinst phase. This accomplishes the goal of enabling this feature _without_ patching portage, but it may not conform to policy. The intended purpose of profile.bashrc isn't very well documented. It's not in the portage man page, and I can google it only in a few relatively recent patches on the gentoo.org website.
First, I must say that I didn't read all the comments very carefully so maybe someone has already said what I'm saying here. Anyway, it seems that the discussion here is too low-level i.e. implementation specific: "Emailing is stupid", "No, logging is stupid" etc bla bla. A little "code by intention" and modularity would do good here :-) Why not add an _optional_ setting EINFO_HANDLER or something? It would contain the name of the program (script) that processes einfos/ewarns. The only modification needed to portage would be that in addition to showing the messages to tty, it would run the given program and pass the data to it as parameters or environment variables or whatever. Then we (the community) would be free to add different handlers for different purposes. Please, I think this feature is VERY IMPORTANT. Currently the oldest of my gentoo systems is slightly broken (see the still open bug #71160!) and I suspect the reason is that I have missed some ewarn that told me to do something manually. (If this strategy is used, someone might eventually create a handler that handles the messages similarly to etc-update so it could hand-guide the user through all the tasks that need to be done.) Furthermore, I would add a new function (= log level) for messages that tell the user to do something manually, because these messages are really important.
Ville, we pretty much have this already. See Eldad Zack's Gentoo page at http://dev.gentoo.org/~eldad/ for both a non-patch fix for the problem and a very convenient tool for viewing enotices in an indexed way. Two things need to happen to get this solution adopted. The portage devs need to nail down the current functionality of /etc/portage/profile/profile.bashrc which is undocumented (and hence subject to going away) and the required profile.bashrc script and the enotice reading tool need to be published as part of the portage ebuild, or at least made available as accessories referenced in the main gentoo documentation.
btw, sometimes I get messages like "QA notice: /bin/passwd is dynamically linked and built with LDFLAGS=...", "QA notice: eclas <classname> inherited illegaly from ... in ...",... They shows up in the middle of build. It would be fine if they could be logged, too.
Please change it to critical, since missed einfo/ewarns can easily cause broken systems. This should be fixed ASAP.
I strongly approve suggestion #135
Posting noise here won't help anything, it'll just annoy us and therefore rather *delay* this feature. This has been evaluated as a portage-2.1 feature and will show up in that version, changing the criticality won't change that.
*** Bug 78130 has been marked as a duplicate of this bug. ***
*** Bug 79245 has been marked as a duplicate of this bug. ***
*** Bug 81240 has been marked as a duplicate of this bug. ***
I messed up a PHP install this week becuase the message about not using mssql and sybase together scrolls off VERY quickly on a dual Xeon box... This is definitely the worst problem in an otherwise excellent system. Pretty please, with sugar on top, sort it out!!! [Disclaimer: Yeah it's a bit rude, no I'm not a developer, yes I donate money to open source, and NO, common sense definitely has not prevailed with this bug].
Lars, we pretty much have a non-patch fix for this available which works OK and addresses the problem. See comment #133 and Eldad's page referenced in the post. An 'official' fix is promised, but until then you can use these tools to save off the information output by ebuilds and browse it selectively at your convenience. It's simple, but pretty slick.
*** Bug 87717 has been marked as a duplicate of this bug. ***
*** Bug 90607 has been marked as a duplicate of this bug. ***
*noise*, *noise* Sorry, can't resist, but for a bug opened 2002-11-29 05:08 PDT, comment #137 is just too delicious. ;)
*** Bug 45711 has been marked as a duplicate of this bug. ***
*** Bug 96239 has been marked as a duplicate of this bug. ***
*** Bug 98118 has been marked as a duplicate of this bug. ***
Fixed on or before 2.0.51.22-r1
Marked incorrectly and got batched.
The latest Gentoo notice logging/reading tool (a la Eldad Zack's page at http://www.gentoo.org/~eldad) is now available at http://www.fmp.com/enotice
*** Bug 100380 has been marked as a duplicate of this bug. ***
Putting a hold on feature requests for portage as they are drowning out the bugs. Most of these features should be available in the next major version of portage. But for the time being, they are just drowning out the major bugs and delaying the next version's progress. Any bugs that contain patches and any bugs for etc-update or dispatch-conf can be reopened. Sorry, I'm just not good enough with bugzilla. ;)
This bug had the "InCVS" keyword a while back, now it does not. Does this mean that the functionality has been dropped out of CVS ?
It is InCVS and working with the HEAD line of development; ie. that which will become the next major version. (Comment #156 will be unneccessary)
Jason, Comment #156 ?? _This_ is comment #156. You apparently mistyped. What comment did you mean?
Oops!! Sorry :-( I'm short of sleep, my spouse is sick and I'm not hitting on all cylinders. Please excuse. I'll shut up now.... LH
Good news. The HEAD stuff that has been referenced several times is very modular and can be backported to 2.0 without causing great upheaval.
Just for clarification: This is still in line with comment #137, the only reason for backporting it is the goals for 2.1 have been shifted largely (from my perspective). My assumption in comment #137 was for 2.1 (with much less changes than currently planned) to be released early 2005.
Created attachment 65206 [details] enotice-0.2.9. Ebuild notice log reader/manager Eldad wrote this and I did a bunch of work on it to make it more useful. See http://www.fmp.com/enotice or http://www.gentoo.org/~eldad for more detaiils. The tarball contains a man page, a changelog and an install utility which will doubtless become an ebild depending on how the whole issue gets resolved in portage and/or gentoolkit.
I've looked in CVS at the make.conf.example for portage 2.1. I can probably port enotice to work with portage elog files with no problem, but one issue needs to be clarified. The enotice reader as it's now written requires that all lines in a notice/elog file file begin with one of three tags: "info:", "warn:" or "error:" with no spaces between these tags and the subsequent text. I assume some similar system will be used in portage 2.1 to identify lines marked with colored asterisks in ebuild output, yes? Does the "log" value for PORTAGE_LOG_CLASSES cause _all_ output to be logged? If portage elog logging is to include tags as described above, will these still be included if the only value for PORTAGE_LOG_CLASSES is "log"? I could probably grub around in gentoo CVS and find this in code, but if someone could give me more details it would be helpful.
http://www.gentoo.org/proj/en/glep/glep-0042.html I don't think this addresses the problems that enotice is here to work around. The small notices are still important (the ones that won't get released as "news" items). The fact still remains... When you are emerging more than one package, einfo output to the terminal is worthless.
(In reply to comment #163) > http://www.gentoo.org/proj/en/glep/glep-0042.html > > I don't think this addresses the problems that enotice is here to work around. > The small notices are still important (the ones that won't get released as > "news" items). > > The fact still remains... When you are emerging more than one package, einfo > output to the terminal is worthless. GLEP 42 is not designed to solve this problem. GLEP 42 is designed to push news to users PRIOR to things being install ( although, I see more and more things being output during pkg_setup which I'd prefer over post_inst ). I personally thinkg GLEP 42 has a bit more work ahead of it before it accomplishes it's goal ( pre-emptive over just output ). For actual logging I think enotice and elog and quite helpful, but they are different beasts entirely.
Sorry for being too vague then. My point is that official work is being done on something ebuild-info related, that isn't einfo logging, which IMHO is more important.
(In reply to comment #165) > Sorry for being too vague then. My point is that official work is being done on > something ebuild-info related, that isn't einfo logging, which IMHO is more > important. And already implemented ... so your point is pointless ;)
Would you mind giving more info? Is this in the next major version of portage? Is the closure of this bug pending it's testing? ;)
Just committed the first two pacthes of elog in trunk, so it should show up in 2.0.54_pre1 which should be out shortly. Still missing example configs and docs.
*** Bug 113481 has been marked as a duplicate of this bug. ***
Released in 2.1_pre1.
It would be cool to have the hostname of the computer which is sending the mail somewhere... atm it is not possible to distinguish mails from different computers. Or is there a way to customize the subject?
Is the information in the from field not enough?
I'll probably add another var for the subject. Further feature requests/bugs about the elog subsystem should be stated in new bugs, this one really is big enough already ;)
Sorry to add to this already long bug, but it pertains to the fix. Is there documentation on how to use these features in any location other than this bug (which is now pretty gargantuan)? I've tried looking through it and through portage a little, but can't find out exactly what's been done or how to use it. If anyone could shed a little light I'd be most greatful. Thanks!
atm all docs regarding it are located in /etc/make.conf.example, those will (hopefully ;) be included in make.conf(5) soon, also in coming versions there will be a list of changes in /usr/share/doc/portage-<version>/NEWS.
*** Bug 121301 has been marked as a duplicate of this bug. ***
Jakub, why did you kick me off the CC list??
Jakub, please explain why you removed a lot of people from the CC!
(In reply to comment #178) > Jakub, please explain why you removed a lot of people from the CC! > So that I wouldn't pointlessly spam tens of people with duplicates on a bug that has been fixed.
(In reply to comment #179) > (In reply to comment #178) > > Jakub, please explain why you removed a lot of people from the CC! > > > > So that I wouldn't pointlessly spam tens of people with duplicates on a bug > that has been fixed. > please don't do that again, people are able to determine for themself if and what email they receive from bugzilla, thanks.
Stop adding me to CC, thanks!
*** Bug 122889 has been marked as a duplicate of this bug. ***
*** Bug 130403 has been marked as a duplicate of this bug. ***
*** Bug 132896 has been marked as a duplicate of this bug. ***
Considering the fact that I've made some software contributions to a workaround this problem (and received a lot of thanks from a bunch of Gentoo users for my efforts), I'd like to ask a small favor of the people on the CC list for this bug, especially anyone on the gentoo dev team dealing with ebuild issues. Please take a look at Bug #136203 (it's relatively short!) and tell me, in private correspondence: 1. If the enhancement I suggest there has sufficient merit to warrant considertation by anyone other than Jacub Moc. 2. If it does, how should I bring it to the attention of people who can give it some thoughtful consideration and criticism?
(In reply to comment #185) > Considering the fact that I've made some software contributions to a workaround > this problem (and received a lot of thanks from a bunch of Gentoo users for my > efforts), I'd like to ask a small favor of the people on the CC list for this > bug, especially anyone on the gentoo dev team dealing with ebuild issues. > > Please take a look at Bug #136203 (it's relatively short!) and tell me, in > private correspondence: > > 1. If the enhancement I suggest there has sufficient merit to warrant > considertation by anyone other than Jacub Moc. > > 2. If it does, how should I bring it to the attention of people who can give > it some thoughtful consideration and criticism? > 1) Feel free to re-open the other bug. 2) That is not an issue for bugs, mail gentoo-dev, preferably with a well thought out proposal. 3) Getting developers to provide that information is a big step, moreso that the technical angle of making the information available; ie someone has to provide it in the first place. Often volunteers are glad their package works and they are glad that they can share their work with others, but documenting every little aspect is not their concern. I would encourage developers to take and integrate patches for such things, but I don't see it becoming a requirement anytime soon.
*** Bug 147152 has been marked as a duplicate of this bug. ***
*** Bug 149386 has been marked as a duplicate of this bug. ***
*** Bug 155746 has been marked as a duplicate of this bug. ***
*** Bug 169263 has been marked as a duplicate of this bug. ***
How do you get the elog stuff to be displayed last after the "emerge -u world"? I'm not interested of an e-mail nor a log to search to the right location.
(In reply to comment #191) > How do you get the elog stuff to be displayed last after the "emerge -u world"? > > I'm not interested of an e-mail nor a log to search to the right location. I'd write a small wrapper around emerge that prints the files elog generated, then deletes them.
(In reply to comment #192) > > How do you get the elog stuff to be displayed last after the > > "emerge -u world"? > I'd write a small wrapper around emerge that prints the files elog generated, > then deletes them. Could you attache your wrapper to this bug?
> Could you attache your wrapper to this bug? Uhm, i don't have one, but something like this should do the job: First, set PORTAGE_ELOG_SYSTEM=custom and PORTAGE_ELOG_COMMAND="/usr/local/bin/savemergelog ${PACKAGE} ${LOGFILE}" in make.conf. Don't forget PORTAGE_ELOG_CLASSES="warn error log" /usr/local/bin/savemergelog would be something like this: #!/bin/bash echo "logged messages for package $1:" cat $2 Note that I didn't test it. Also, this doesn't really belong on bugzilla. If you need more help, feel free to contact me by mail instead.
