Please mark these packages as stable.
*** Bug 176340 has been marked as a duplicate of this bug. ***
sparc stable.
mail-filter/dspam-3.8.0-r1 USE="clamav ldap -daemon -debug -large-domain -logrotate -mysql -postgres -sqlite -sqlite3 -user-homedirs -virtual-users" 1. emerges on x86 2. passes collision test www-apps/dspam-web-3.8.0 USE="-vhosts" 1. emerges on x86 2. passes collision test (don't have a setup to test further) Portage 2.1.2.7 (default-linux/x86/2007.0/desktop, gcc-4.1.2, glibc-2.5-r2, 2.6.20.12 i686) ================================================================= System uname: 2.6.20.12 i686 Genuine Intel(R) CPU T2300 @ 1.66GHz Gentoo Base System release 1.12.9 Timestamp of tree: Sun, 27 May 2007 17:30:01 +0000 dev-java/java-config: 1.3.7, 2.0.32 dev-lang/python: 2.3.5-r3, 2.4.4-r4 dev-python/pycrypto: 2.0.1-r5 sys-apps/sandbox: 1.2.17 sys-devel/autoconf: 2.13, 2.61 sys-devel/automake: 1.4_p6, 1.5, 1.6.3, 1.7.9-r1, 1.8.5-r3, 1.9.6-r2, 1.10 sys-devel/binutils: 2.16.1-r3 sys-devel/gcc-config: 1.3.16 sys-devel/libtool: 1.5.22 virtual/os-headers: 2.6.17-r2 ACCEPT_KEYWORDS="x86" AUTOCLEAN="yes" CBUILD="i686-pc-linux-gnu" CFLAGS="-O2 -march=prescott -pipe -fomit-frame-pointer" CHOST="i686-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/kde/3.5/env /usr/kde/3.5/share/config /usr/kde/3.5/shutdown /usr/share/X11/xkb /usr/share/config" CONFIG_PROTECT_MASK="/etc/env.d /etc/env.d/java/ /etc/gconf /etc/php/apache2-php5/ext-active/ /etc/php/cgi-php5/ext-active/ /etc/php/cli-php5/ext-active/ /etc/revdep-rebuild /etc/terminfo /etc/texmf/web2c" CXXFLAGS="-O2 -march=prescott -pipe -fomit-frame-pointer" DISTDIR="/usr/portage/distfiles" EMERGE_DEFAULT_OPTS="--nospinner" FEATURES="collision-protect distlocks metadata-transfer parallel-fetch sandbox sfperms strict test userfetch userpriv usersandbox" GENTOO_MIRRORS="http://mirror.switch.ch/mirror/gentoo/ http://gentoo.inode.at/" LINGUAS="en de en_GB de_CH" MAKEOPTS="-j3" PKGDIR="/usr/portage/packages" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --delete-after --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --filter=H_**/files/digest-*" PORTAGE_TMPDIR="/var/tmp" PORTDIR="/usr/portage" SYNC="rsync://rsync.gentoo.org/gentoo-portage" USE="X a52 aac acl acpi alsa apache2 asf avahi berkdb bitmap-fonts cairo cdr cdrom cli cracklib crypt cups dbus divx dri dts dvd dvdr dvdread eds emboss encode evo fam ffmpeg firefox flac fortran gdbm gif gnome gpm gstreamer gtk hal iconv ipv6 isdnlog java jpeg kde kdeenablefinal kerberos ldap libg++ mad midi mikmod mmx mono mp3 mpeg mudflap ncurses nls nptl nptlonly ogg opengl openmp oss pam pcre pdf perl png pppd python qt3 qt3support qt4 quicktime readline reflection rtsp ruby samba sdl session smp spell spl sse sse2 sse3 ssl svg tcpd test tetex theora threads tiff truetype truetype-fonts type1-fonts unicode vcd vorbis wifi win32codecs wxwindows x264 x86 xine xml xorg xprint xv xvid zlib" ELIBC="glibc" INPUT_DEVICES="keyboard mouse" KERNEL="linux" LINGUAS="en de en_GB de_CH" USERLAND="GNU" VIDEO_CARDS="i810 fbdev vesa" Unset: CTARGET, INSTALL_MASK, LANG, LC_ALL, LDFLAGS, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, PORTDIR_OVERLAY
Works fine on x86, too - I'm running it on a production server.
This version doesn't appear to work at all for me unless I set the execute bit and run it as root. It just doesn't think ANYTHING is spam. And unfortunately their configuration guides don't provide any insight into what might be the issue. I believe Uberlord is having the same issue as me as well.
Doug, you first have to train it. The FAQ might help: http://dspam.nuclearelephant.com/faq.shtml
(In reply to comment #5) Maybe dspam user doesn't have the necessary rights. Install it with debug USE flag enabled and study the logs stored in /var/log/dspam.
It is trained. It has like 20,000 e-mails through it. It worked fine with the previous 3.6.4 version. I'll check out the logging through and see if I can't figure out what I messed up.
(In reply to comment #8) > I'll check out the logging through and see if I can't figure out what I messed > up. Doug? Have you worked it out?
Philantrop looked at my configs and gave them the green light of approval. However nothing affected this issue in anyway positive. For example, this morning when I checked my e-mail I had: 60 new e-mails 59 in my Inbox 3 real e-mails, 56 spams in my Inbox 1 in my spam folder I guess the only positive thing I can say about this version is that at least it doesn't get false positives.
You have 2 options: a) keep training dspam; eventually it will catch 99% of your spam. b) cleanup dspam database and retrain it from scratch; I used it for a user who didn't took training job seriously enough.
I'll admit I've had this issue and started off from scratch again. Its started to catch stuff (from X-Spam headers) but its better then it was doing before, and now I've told it to ignore that so we'll see how it goes, as I need to do quite a bit more training. I had the same issue as Cardoe in otherwords.
x86 stable as it does actually work..just needs some proper training...
The issue appears to be specific to user-homedirs, which just doesn't seem to be working properly. dspam-web contains no support for user-homedirs.. as can be seen by this function which pulls the data.. sub GetPath { my($USER) = @_; my($PATH); # Domain-scale if ($CONFIG{'DOMAIN_SCALE'} == 1) { my($VPOPUSERNAME, $VPOPDOMAIN); $VPOPUSERNAME = (split(/@/, $USER))[0]; $VPOPDOMAIN = (split(/@/, $USER))[1]; $VPOPDOMAIN = "local" if ($VPOPDOMAIN eq ""); $PATH = "$CONFIG{'DSPAM_HOME'}/data/$VPOPDOMAIN/$VPOPUSERNAME/" . "$VPOPUSERNAME"; return $PATH; # Normal scale } elsif ($CONFIG{'LARGE_SCALE'} == 0) { $PATH = "$CONFIG{'DSPAM_HOME'}/data/$USER/$USER"; return $PATH; # Large-scale } else { if (length($USER)>1) { $PATH = "$CONFIG{'DSPAM_HOME'}/data/" . substr($USER, 0, 1) . "/". substr($USER, 1, 1) . "/$USER/$USER"; } else { $PATH = "$CONFIG{'DSPAM_HOME'}/data/$USER/$USER"; } return $PATH; } &error("Unable to determine filesystem scale"); } Because with user-homedirs the data is in /home/$USER/.dspam With this config, it picked the "normal scale" An Error Has Occured The following error occured while trying to process your request: No historical data is available in /var/spool/dspam/data/local/cardoe/cardoe
yeah confirmed. USE=user-homedirs is busted. I turned that off and now dspam is chugging along.
I've added a pkg_setup function to ebuild. Now dspam-web will refuse to install while dspam is installed with user-homedirs.
Alin: Great. I think there's more problems with dspam + user_homedirs though. But now that I'm not using them things seem better. TP True Positives: 357 TN True Negatives: 634 FP False Positives: 3 FN False Negatives: 105 SC Spam Corpusfed: 0 NC Nonspam Corpusfed: 2 TL Training Left: 1861 SHR Spam Hit Rate 77.27% HSR Ham Strike Rate: 0.47% OCA Overall Accuracy: 90.17% The only issue I'm having now is the performance screen on dspam messes up constantly... Since last reset 0 missed 0 missed 0 caught -1 delivered 100% caught 0% missed Total processed by filter missed missed caught delivered From corpus fed fed Yet those are my stats above...
Actually... un 18 14:47:21 [kernel] dspam[13076]: segfault at 00002b36bcea3b30 rip 00002b36bbf0b222 rsp 00007fffeecc43c8 error 4 Jun 18 14:47:21 [maildrop] Unable to filter message. Getting a whole lot of dspam segfaults... Any suggestions how to debug this.. I have fetchmail -> maildrop -> dspam
Debugging a segfault is usually like this: a) Get a piece of input that crashes dspam. b) Install dspam (or at least compile it from source without installing it) with -g added to your CFLAGS/CXXFLAGS. Also you should check that no one strips the executable (FEATURES=nostrip). c) Use a debugger (e.g. kdbg) to discover which part of the code is to blame. I didn't noticed any segfaults on my system, but then again my system is x86 while yours seems to have 64-bit architecture (amd64?).
(gdb) bt #0 _hash_drv_seek (map=0x5163a0, offset=<value optimized out>, hashcode=11903221412579597457, flags=0) at hash_drv.c:1094 #1 0x00002b7f0f6702d8 in _hash_drv_get_spamrecord (map=0x5163a0, wrec=0x7fff9b5605d0) at hash_drv.c:1182 #2 0x00002b7f0f670370 in _ds_get_spamrecord (CTX=<value optimized out>, token=<value optimized out>, stat=0x45aa0) at hash_drv.c:691 #3 0x00002b7f0f670cc1 in _ds_getall_spamrecords (CTX=0x515510, diction=0x517e10) at hash_drv.c:612 #4 0x00002b7f0f667814 in _ds_operate (CTX=0x515510, headers=0x519c70 "Return-Path: <ivghqqisqf@net.co>", body=<value optimized out>) at libdspam.c:879 #5 0x00002b7f0f66863e in dspam_process (CTX=0x515510, message=0x514c70 "Return-Path: <ivghqqisqf@net.co>\nX-Original-To: cardoe@cardoe.com\nDelivered-To: cardoe@spunkymail-mx7.g.dreamhost.com\nReceived: from corporativos24235-116.etb.net.co (unknown [190.24.235.116])\n "...) at libdspam.c:578 #6 0x0000000000409794 in process_message (ATX=0x7fff9b564ed0, message=0x512830, username=0x5126a0 "cardoe", result_string=0x7fff9b564e68) at dspam.c:514 #7 0x000000000040a0aa in process_users (ATX=0x7fff9b564ed0, message=0x5126e0) at dspam.c:1797 #8 0x000000000040a88c in main (argc=<value optimized out>, argv=0x7fff9b565a58) at dspam.c:258
* QA Notice: Package has poor programming practices which may compile * fine but exhibit random runtime failures. * cssclean.c:143: warning: implicit declaration of function 'dirname' It needs to #include <libgen.h>
Here's a better backtrace.. Program received signal SIGSEGV, Segmentation fault. 0x00002b76be596772 in _hash_drv_seek (map=0x5173a0, offset=4837344, hashcode=11903221412579597457, flags=0) at hash_drv.c:1094 1094 hash_drv.c: No such file or directory. in hash_drv.c (gdb) bt #0 0x00002b76be596772 in _hash_drv_seek (map=0x5173a0, offset=4837344, hashcode=11903221412579597457, flags=0) at hash_drv.c:1094 #1 0x00002b76be596a1f in _hash_drv_get_spamrecord (map=0x5173a0, wrec=0x7fffec63d700) at hash_drv.c:1182 #2 0x00002b76be5954b3 in _ds_get_spamrecord (CTX=0x516510, token=11903221412579597457, stat=0x7fffec63d750) at hash_drv.c:691 #3 0x00002b76be59522a in _ds_getall_spamrecords (CTX=0x516510, diction=0x518e10) at hash_drv.c:612 #4 0x00002b76be587786 in _ds_operate (CTX=0x516510, headers=0x51ac70 "Return-Path: <ivghqqisqf@net.co>", body=0x51b080 "\n\n<META") at libdspam.c:879 #5 0x00002b76be586d57 in dspam_process (CTX=0x516510, message=0x515c70 "Return-Path: <ivghqqisqf@net.co>\nX-Original-To: cardoe@cardoe.com\nDelivered-To: cardoe@spunkymail-mx7.g.dreamhost.com\nReceived: from corporativos24235-116.etb.net.co (unknown [190.24.235.116])\n "...) at libdspam.c:578 #6 0x0000000000403c9e in process_message (ATX=0x7fffec641f90, message=0x513830, username=0x5136a0 "cardoe", result_string=0x7fffec641db8) at dspam.c:514 #7 0x0000000000406c17 in process_users (ATX=0x7fffec641f90, message=0x5136e0) at dspam.c:1797 #8 0x0000000000403558 in main (argc=6, argv=0x7fffec642b38) at dspam.c:258
I've added a patch to dspam-3.8.0-r1 to fix the QA notice. Also I've added a warning in pkg_setup about the hash backend - user is advised to select a database backend. Doug, does sqlite backend works for ya?
revbump (see bug 184194)
(In reply to comment #23) > I've added a patch to dspam-3.8.0-r1 to fix the QA notice. Also I've added a > warning in pkg_setup about the hash backend - user is advised to select a > database backend. > > Doug, does sqlite backend works for ya? > @Alin: The hash driver only works for the tokenizer SBPH and pvalue makrov. All other tokenizers require another driver. This issue is described in the DSPAM documentation markov.txt
revbump (see bug 185718)
revbump (see bug 189033) For the record, all issues exposed by previous comments have been worked out.
dspam-3.8.0-r4 has totally broken my previous working setup that everyone worked so hard on. I know Philantrop is having problems with it as well and he's masked it on his machine.
Indeed. On the first attempt, it wouldn't even compile on an x86 server where -r3 works just fine. Now it compiles but I'm too tired to install and test it (especially since it's a production machine :-) ) today. Will try to see if it works for me tomorrow.
(In reply to comment #28) > dspam-3.8.0-r4 has totally broken my previous working setup that everyone > worked so hard on. I know Philantrop is having problems with it as well and > he's masked it on his machine. > Can you explain what is broken? Do you have any log entries? Any errors? Any more info?
(In reply to comment #29) > Indeed. On the first attempt, it wouldn't even compile on an x86 server where > -r3 works just fine. > How can this be? The only difference between -r3 and -r4 is the $Header line in the ebuild: --- dspam-3.8.0-r4.ebuild 2007-08-17 20:19:27.000000000 +0200 +++ dspam-3.8.0-r3.ebuild 2007-07-19 11:05:46.000000000 +0200 @@ -1,6 +1,6 @@ # Copyright 1999-2007 Gentoo Foundation # Distributed under the terms of the GNU General Public License v2 -# $Header: /var/cvsroot/gentoo-x86/mail-filter/dspam/dspam-3.8.0-r4.ebuild,v 1.1 2007/08/17 18:19:27 mrness Exp $ +# $Header: /var/cvsroot/gentoo-x86/mail-filter/dspam/dspam-3.8.0-r3.ebuild,v 1.2 2007/07/19 08:39:01 mrness Exp $ WANT_AUTOCONF="latest" WANT_AUTOMAKE="latest" > Now it compiles but I'm too tired to install and test it (especially since it's > a production machine :-) ) today. Will try to see if it works for me tomorrow. > Are you talking about -r4? What have you done to get it compiling on your system?
The ebuild code is unchanged. The only difference between -r3 and -r4 is the cron script, so...
(In reply to comment #32) > The ebuild code is unchanged. The only difference between -r3 and -r4 is the > cron script, so... > How can then the cron job influence the compilation of DSPAM?
Created attachment 128937 [details] mail-filter:dspam-3.8.0-r4:20070819-094433.log (In reply to comment #32) > The ebuild code is unchanged. The only difference between -r3 and -r4 is the > cron script, so... ... you don't believe a word. Nor would I if I were you. :-) Here's the build log, though. Meanwhile, I've updated and everything seems to work fine.
(In reply to comment #34) > Created an attachment (id=128937) [edit] > mail-filter:dspam-3.8.0-r4:20070819-094433.log > > (In reply to comment #32) > > The ebuild code is unchanged. The only difference between -r3 and -r4 is the > > cron script, so... > > ... you don't believe a word. Nor would I if I were you. :-) > > Here's the build log, though. Meanwhile, I've updated and everything seems to > work fine. > It is not about beliving. It is about trying to find out what went wrong in order to get better quality in the ebuild. That's all. Anyway... do I understand you right, that -r4 is now working in your setup? If so, is it possible for you to tell or guess what went wrong when you tried the first time the -r4 edition of the ebuild?
(In reply to comment #35) > It is not about beliving. It is about trying to find out what went wrong in > order to get better quality in the ebuild. That's all. I know, Steve. Look at my email. :-) > Anyway... do I understand you right, that -r4 is now working in your setup? Seems so. Dspam happily catches all the spam I get since the update. The improved cron script hasn't run yet but I've compared it to the previous one, of course, and it looks good to me. > so, is it possible for you to tell or guess what went wrong when you tried the > first time the -r4 edition of the ebuild? Look at the build log: mv -f .deps/config_shared.Tpo .deps/config_shared.Po mv: cannot stat `.deps/config_shared.Tpo': No such file or directory make[3]: *** [config_shared.o] Error 1 make[3]: *** Waiting for unfinished jobs.... Seems like a parallel make issue to me. I'm using -j10 and distcc on that server so it might very well be that. The few other updates between the failure and yesterday's successful build are extremely unlikely to have had any influence on the outcome.
(In reply to comment #36) > (In reply to comment #35) > > It is not about beliving. It is about trying to find out what went wrong in > > order to get better quality in the ebuild. That's all. > > I know, Steve. Look at my email. :-) > Aha. You are a Gentoo dev :) > > Anyway... do I understand you right, that -r4 is now working in your setup? > > Seems so. Dspam happily catches all the spam I get since the update. The > improved cron script hasn't run yet but I've compared it to the previous one, > of course, and it looks good to me. > > > so, is it possible for you to tell or guess what went wrong when you tried the > > first time the -r4 edition of the ebuild? > > Look at the build log: > > mv -f .deps/config_shared.Tpo .deps/config_shared.Po > mv: cannot stat `.deps/config_shared.Tpo': No such file or directory > make[3]: *** [config_shared.o] Error 1 > make[3]: *** Waiting for unfinished jobs.... > > Seems like a parallel make issue to me. I'm using -j10 and distcc on that > server so it might very well be that. > Hmm... I guess it is distcc. Maybe we should disable parallel builds? > The few other updates between the failure and yesterday's successful build are > extremely unlikely to have had any influence on the outcome. >
(In reply to comment #30) > (In reply to comment #28) > > dspam-3.8.0-r4 has totally broken my previous working setup that everyone > > worked so hard on. I know Philantrop is having problems with it as well and > > he's masked it on his machine. > > > Can you explain what is broken? Do you have any log entries? Any errors? Any > more info? > I can't give any more info then that since dspam never logs squat. Running it from the command line, it doesn't spit out any errors or anything.. it just returns immediately. cardoe@meyer ~ $ /usr/bin/dspam --user cardoe --stdout --deliver=innocent,spam cardoe@meyer ~ $ echo $? 1 cardoe@meyer ~ $ /usr/bin/dspam --user cardoe --stdout --debug --deliver=innocent,spam cardoe@meyer ~ $ echo $? 1 from /etc/dspam/dspam.conf: # # Logging: Disabling logging for users will make usage graphs unavailable to # them. Disabling system logging will make admin graphs unavailable. # SystemLog on UserLog on yet: cardoe@meyer ~ $ ls -l /var/log/dspam/ total 0
I think I discovered the issue. The ownership of /etc/dspam/dspam.conf changed to root:root from root:dspam. I've made it dspam:dspam now.
(In reply to comment #38) > (In reply to comment #30) > > (In reply to comment #28) > > > dspam-3.8.0-r4 has totally broken my previous working setup that everyone > > > worked so hard on. I know Philantrop is having problems with it as well and > > > he's masked it on his machine. > > > > > Can you explain what is broken? Do you have any log entries? Any errors? Any > > more info? > > > > I can't give any more info then that since dspam never logs squat. Running it > from the command line, it doesn't spit out any errors or anything.. it just > returns immediately. > > cardoe@meyer ~ $ /usr/bin/dspam --user cardoe --stdout --deliver=innocent,spam > cardoe@meyer ~ $ echo $? > 1 > This is normal. Could you try: /usr/bin/dspam --user cardoe --classify --stdout --deliver=innocent,spam < /path/to/your/testmessage Or try this: echo "some stupid text"|/usr/bin/dspam --user cardoe --classify --stdout --deliver=innocent,spam Or try this: echo "some stupid text"|/usr/bin/dspam --user cardoe --stdout --deliver=innocent,spam > cardoe@meyer ~ $ /usr/bin/dspam --user cardoe --stdout --debug > --deliver=innocent,spam > cardoe@meyer ~ $ echo $? > 1 > See above... > from /etc/dspam/dspam.conf: > > # > # Logging: Disabling logging for users will make usage graphs unavailable to > # them. Disabling system logging will make admin graphs unavailable. > # > SystemLog on > UserLog on > > > yet: > cardoe@meyer ~ $ ls -l /var/log/dspam/ > total 0 > Can you post the output of: ls -lahd /var/log/dspam // SteveB
(In reply to comment #39) > I think I discovered the issue. The ownership of /etc/dspam/dspam.conf changed > to root:root from root:dspam. I've made it dspam:dspam now. > The ownership is not a problem as long as DSPAM can read the file. No writing is needed.
(In reply to comment #39) > I think I discovered the issue. The ownership of /etc/dspam/dspam.conf changed > to root:root from root:dspam. I've made it dspam:dspam now. I don't understand why. The ebuild install dspam.conf with owner:group=dspam:dspam. People, I'm still trying to convince amd64 peeps to mark this stable. Please don't clutter this bug with even more comments. amd64 team, please do your magic.
revbump (see bug 191271)
As a consequence of bug 193081, I've commited -r7 which has a far better polished pkg_config. Apparently I cannot convince amd64 team to mark dspam stable, so I will close this bug with FIXED (wrt sparc and x86 stabilization). If there is some amd64 user who want to stabilize dspam and dspam-web, better open his/her own bug.
