DSPAM is crashing when run in daemon mode as the content filter of postfix. First it starts refusing connections on its LMTP socket and then within a few minutes it segfaults. Tested dspam-3.8.0-r7/-r8 with postfix-2.4.5/2.4.6. Reproducible: Always Actual Results: Nov 27 05:16:36 rocks dspam[19374]: Daemon process starting Nov 27 05:17:14 rocks postfix/lmtp[19297]: C72D399C8D: to=<rumi@xxx.hu>, orig_to=<rumi@xxx.hu>, relay=xxx.hu[/var/run/dspam/dspam.sock], delay=0.1, delays=0.09/0/0/0, dsn=4.4.2, status=deferred (lost connection with xxx.hu[/var/run/dspam/dspam.sock] while sending end of data -- message may be sent more than once) Nov 27 05:17:15 rocks postfix/lmtp[19297]: 9CE8F99C8C: to=<rumi@xxx.hu>, orig_to=<rumi_ml@xxx.hu>, relay=none, delay=0.44, delays=0.44/0/0/0, dsn=4.4.1, status=deferred (connect to xxx.hu[/var/run/dspam/dspam.sock]: Connection refused) Nov 27 05:18:42 rocks postfix/lmtp[19297]: 7A95799C8F: to=<rumi@xxx.hu>, orig_to=<rumi_ml@xxx.hu>, relay=none, delay=0.63, delays=0.63/0/0/0, dsn=4.4.1, status=deferred (connect to xxx.hu[/var/run/dspam/dspam.sock]: Connection refused) It refuses all further connections all the way until it segfaults. Relevant configuration details: main.cf: content_filter = dspam-lmtp:unix:/var/run/dspam/dspam.sock master.cf: dspam-lmtp unix - - n - 1 lmtp -o lmtp_data_done_timeout=1200 -o lmtp_send_xforward_command=yes 127.0.0.1:10025 inet n - n - - smtpd -o content_filter= -o local_recipient_maps= -o relay_recipient_maps= -o smtpd_restriction_classes= -o smtpd_client_restrictions= -o smtpd_helo_restrictions= -o smtpd_sender_restrictions= -o smtpd_recipient_restrictions=permit_mynetworks,reject -o mynetworks=127.0.0.0/8 -o strict_rfc821_envelopes=yes -o smtpd_error_sleep_time=0 -o smtpd_soft_error_limit=1001 -o smtpd_hard_error_limit=1000 dspam.conf: #TrustedDeliveryAgent "/usr/lib/cyrus/deliver" #UntrustedDeliveryAgent "/usr/lib/cyrus/deliver %u" #QuarantineAgent "/usr/lib/cyrus/deliver %u" DeliveryHost 127.0.0.1 DeliveryPort 10025 DeliveryIdent localhost DeliveryProto SMTP #ServerPort 10024 #ServerQueueSize 32 ServerPID /var/run/dspam/dspam.pid ServerMode auto ServerParameters "--deliver=innocent,spam -d %u" ServerIdent "rocks.dynaweb.hu" ServerDomainSocketPath "/var/run/dspam/dspam.sock" #ClientHost "/var/run/dspam/dspam.sock"
Backtrace with CFLAGS="-O0 -ggdb" FEATURES="nostrip": (gdb) run Starting program: /usr/bin/dspam --daemon [New LWP 22046] Program received signal SIGSEGV, Segmentation fault. [Switching to LWP 22046] 0xb7f18fc7 in _hash_drv_seek (map=0x8067db8, offset=100663544, hashcode=8805600035631203381, flags=0) at hash_drv.c:1094 1094 hash_drv.c: No such file or directory. in hash_drv.c (gdb) bt #0 0xb7f18fc7 in _hash_drv_seek (map=0x8067db8, offset=100663544, hashcode=8805600035631203381, flags=0) at hash_drv.c:1094 #1 0xb7f1926e in _hash_drv_get_spamrecord (map=0x8067db8, wrec=0xb7a435dc) at hash_drv.c:1182 #2 0xb7f17dfb in _ds_get_spamrecord (CTX=0x80666d8, token=8805600035631203381, stat=0xb7a43620) at hash_drv.c:691 #3 0xb7f17b38 in _ds_getall_spamrecords (CTX=0x80666d8, diction=0x806a4e8) at hash_drv.c:612 #4 0xb7f09b8b in _ds_operate (CTX=0x80666d8, headers=0x806aae0 "Received: from balu.sch.bme.hu (balu.sch.bme.hu [152.66.208.40])", body=0x806a6d8 "All") at libdspam.c:879 #5 0xb7f0919e in dspam_process (CTX=0x80666d8, message=0x8067698 "Received: from balu.sch.bme.hu (balu.sch.bme.hu [152.66.208.40])\n\tby rocks.xxx.hu (Postfix) with ESMTP id ADF088B81D\n\tfor <rumi@xxx.hu>; Tue, 27 Nov 2007 06:28:14 +0100 (CET)\nReceived: from pool-"...) at libdspam.c:578 #6 0x0804c22b in process_message (ATX=0x8065550, message=0x80663d0, username=0x8066478 "rumi@xxx.hu", result_string=0xb7a48a78) at dspam.c:514 #7 0x0804f860 in process_users (ATX=0x8065550, message=0x8066518) at dspam.c:1797 #8 0x0805748e in process_connection (ptr=0x8064f18) at daemon.c:712 #9 0xb7c8e18b in ?? () from /lib/libpthread.so.0 #10 0x08064f18 in ?? () #11 0xb7a49b90 in ?? () #12 0xb7a49b90 in ?? () #13 0xb7a49b90 in ?? () #14 0xb7a49480 in ?? () #15 0x00000000 in ?? ()
The Connection refused problem occurs a lot earlier than the segfault so I'd assume these may be two separate issues. I've tried to reconfigure DSPAM and postfix to use a TCP socket instead of a UNIX domain socket for the LMTP transfer towards DSPAM but the results are the same. Here is a forum topic about another user struggling with DSPAM and postfix: http://forums.gentoo.org/viewtopic-t-619112-postdays-0-postorder-asc-start-0.html If you look at his configuration you may notice that he is using the postgresql backend and not the hashdb backend I am using of which the functions seem to be called around the crash point. Still he does get the Connection refused problem but apparently he does not get the crash.
1) daemon mode requires a thread-safe storage driver. Apparently postgres driver is not (see also bug 199700). 2) hash driver has pretty narrow possibilities when it comes to PValue and Tokenizer. pkg_postinst issue messages for both restrictions, so please apply them to your case. Steve, can you help me here please?
Could you post the output of: dspam --version Could you as well post the whole dspam.conf?
Is this issue still open? Please post back. // SteveB
Created attachment 137609 [details] dspam.conf
# dspam --version DSPAM Anti-Spam Suite 3.8.0 (agent/library) Copyright (c) 2002-2006 Jonathan A. Zdziarski http://dspam.nuclearelephant.com DSPAM may be copied only under the terms of the GNU General Public License, a copy of which can be found with the DSPAM distribution kit. Configuration parameters: '--prefix=/usr' '--host=i686-pc-linux-gnu' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--datadir=/usr/share' '--sysconfdir=/etc' '--localstatedir=/var/lib' '--with-storage-driver=hash_drv' '--with-dspam-home=/var/spool/dspam' '--sysconfdir=/etc/mail/dspam' '--enable-daemon' '--enable-ldap' '--enable-clamav' '--disable-large-scale' '--enable-domain-scale' '--enable-syslog' '--enable-debug' '--enable-bnr-debug' '--enable-long-usernames' '--with-dspam-group=dspam' '--with-dspam-home-group=dspam' '--with-dspam-mode=2511' '--with-logdir=/var/log/dspam' '--disable-virtual-users' '--disable-preferences-extension' '--enable-homedir' '--build=i686-pc-linux-gnu' 'build_alias=i686-pc-linux-gnu' 'host_alias=i686-pc-linux-gnu' 'CFLAGS=-march=i686 -O3 -pipe -Wl,-z,now' 'CXXFLAGS=-march=i686 -O3 -pipe -Wl,-z,now'
(In reply to comment #5) > Is this issue still open? Please post back. Yes it is open. I've posted what you asked for but sorry I cannot further assist in this one because that was a production system that I had to get back to work within a short time so when I realized that DSPAM is a buggy PoS I reverted to SA instantly...
(In reply to comment #8) > (In reply to comment #5) > > Is this issue still open? Please post back. > > Yes it is open. I've posted what you asked for but > sorry I cannot further assist in this one because that > was a production system that I had to get back to work > within a short time so when I realized that DSPAM is a > buggy PoS I reverted to SA instantly... > DSPAM is not a buggy PoS! :) Just look at the statistics for my merged group in DSPAM (read the readme if you want to know what merged groups are in DSPAM): mail ~ # dspam_stats -H globaluser globaluser: TP True Positives: 666679 TN True Negatives: 494405 FP False Positives: 8497 FN False Negatives: 2564 SC Spam Corpusfed: 46448 NC Nonspam Corpusfed: 44362 TL Training Left: 0 SHR Spam Hit Rate 99.62% HSR Ham Strike Rate: 1.69% PPV Positive predictive value: 98.74% OCA Overall Accuracy: 99.06% mail ~ # This is around 1.2 million messages processed by DSPAM and no error with DSPAM. And this is only my merged group. DSPAM processes many mails per day on that instance without even coming close to a 10MB memory usage. Anyway... you use the hash_drv and it was introduced not long ago into DSPAM. I don't know how well it is integrated but it would not surprise me if the driver has issues. Are there any options on your side to use another (more mature) storage driver? Like MySQL, PostgreSQL, SQLite, etc?
I did a quick check about dspam_signature_data and found a pretty big bug in mysql_drv.c! According to my current dspam database, the maximum size of the signature field is 32767 (this is the result of "select max(length) from dspam_signature_data" query, I would have to check if this is the real maximal value). Despite that, mysql_drv.c use only a buffer of 1024 bytes for the SQL insertion phrase, when it should be 32767*2 + 1 + whatever_is_needed_for_the_rest_of_the_phrase! Are we the first who discover this bug?
Scratch my previous comment. Although it isn't exactly optimal (see buffer management and its unnecessary strlen calls), the code doesn't seem to lead to buffer overflow. On the other hand, I don't understand the content and 32767 length limit of the data field.
(In reply to comment #10) > I did a quick check about dspam_signature_data and found a pretty big bug in > mysql_drv.c! > > According to my current dspam database, the maximum size of the signature field > is 32767 (this is the result of "select max(length) from dspam_signature_data" > query, I would have to check if this is the real maximal value). > The reason for the 32767 size is that the field for the data is a normal blob. The blob is according to the documentation found here http://dev.mysql.com/doc/refman/5.0/en/blob.html 65KB in size. And since the data is saved in dual byte the 32767 is normal. Here how to calculate the size: BLOB = L + 2 bytes (max size is 2^16 - 1 or 65,535 bytes, 65KB) MEDIUMBLOB = L + 3 bytes (max size is 2^24 - 1 or 16,777,215 bytes, 16MB) LONGBLOB = L + 4 bytes (max size is 2^32 - 1 or 4,294,967,295 bytes, 4GB) > Despite that, > mysql_drv.c use only a buffer of 1024 bytes for the SQL insertion phrase, when > it should be 32767*2 + 1 + whatever_is_needed_for_the_rest_of_the_phrase! > No. mysql_drv.c uses a dynamically allocated buffer and a char of 1024 for adding to buffer: buffer *query; char scratch[1024]; The buffer gets later initialized: query = buffer_create (NULL); if (query == NULL) { LOG (LOG_CRIT, ERR_MEM_ALLOC); return EUNKNOWN; } And then DSPAM adds the stuff into scratch and then appends or writes the stuff into the buffer called query. > Are we the first who discover this bug? > Yes. We are the first one to complain about it. On my installation I have switched away from normal BLOB some time ago. I think it is time to open a new bug report and start adding the patches to work around this issue. Or we could hijack this bug report and start posting the patches. I have so far fixed the whole MySQL driver. Exchanged around 8% of the source code inside mysql_drv.c. I am still testing the code, but I could start posting patches here. But I would not like the patches to be included until I am not 100% finished with coding on it and until you say that the patch is working as expected. What do you think?
(In reply to comment #11) > Scratch my previous comment. Although it isn't exactly optimal (see buffer > management and its unnecessary strlen calls), the code doesn't seem to lead to > buffer overflow. > No buffer overflows but problems with inserting to big data into MySQL happen pretty often if you have huge data. > On the other hand, I don't understand the content and 32767 length limit of the > data field. > It is coming from MySQL BLOB size. Do this and you will get bigger numbers: alter table dspam_signature_data change data data mediumblob not null; And then increase the DSPAM MaxMessageSize and increase max_allowed_packet in MySQL and do some training with huge mails (take this here http://www.cs.virginia.edu/~cs101/hws/hw6/markov/textfiles/bible.txt to get a huge text file. Copy it twice to double the size) and you will see a huger number.
(In reply to comment #13) > It is coming from MySQL BLOB size. I see. In that case the 'length' field type is wrong; it should be changed to unsigned smallint. I'm still puzzled by the 'data' content. It doesn't seem to contain parts of message body or header. (In reply to comment #12) > I think it is time to open a new bug report and start adding the patches to > work around this issue. Or we could hijack this bug report and start posting > the patches. I have so far fixed the whole MySQL driver. Exchanged around 8% of > the source code inside mysql_drv.c. I am still testing the code, but I could > start posting patches here. But I would not like the patches to be included > until I am not 100% finished with coding on it and until you say that the patch > is working as expected. > > What do you think? Sure, just add them here. I will not add them until we are happy with the results.
(In reply to comment #14) > (In reply to comment #13) > > It is coming from MySQL BLOB size. > > I see. In that case the 'length' field type is wrong; it should be changed to > unsigned smallint. > I'm still puzzled by the 'data' content. It doesn't seem to contain parts of > message body or header. > It contains data in binary format. The limit on length is because of this: /* * struct _ds_spam_signature - A historical classification signature * * A binary representation of the original training instance. The spam * signature contains all the metadata used in the original decision * about the message, so that a 1:1 retraining can take place if the * message is submitted for retraining (e.g. was misclassified). The * signature contains a series of _ds_signature_token structures, which * house the original set of tokens used and their frequency counts in * the message. A spam signature is a temporary piece of data that is * usually purged from disk after a short period of time. */ struct _ds_spam_signature { void *data; long length; }; The data then is added with this into MySQL: snprintf (scratch, sizeof (scratch), "insert into dspam_signature_data(uid, signature, length, created_on, data) values(%d, \"%s\", %ld, current_date(), \"", (int) p->pw_uid, signature, SIG->length); buffer_cat (query, scratch); buffer_cat (query, mem); buffer_cat (query, "\")"); SIG is nothing more then a pointer to _ds_spam_signature. And since SIG->length is only a signed long it can have the value from −32'768 to +32'767 and that is the reason why the length field in MySQL is never longer then 32767. The data filed it self is 65KB if you use BLOB. The query below should on your installation result in 65535: select max(octet_length(data)) from dspam_signature_data > (In reply to comment #12) > > I think it is time to open a new bug report and start adding the patches to > > work around this issue. Or we could hijack this bug report and start posting > > the patches. I have so far fixed the whole MySQL driver. Exchanged around 8% of > > the source code inside mysql_drv.c. I am still testing the code, but I could > > start posting patches here. But I would not like the patches to be included > > until I am not 100% finished with coding on it and until you say that the patch > > is working as expected. > > > > What do you think? > > Sure, just add them here. I will not add them until we are happy with the > results. > Currently I am working externally in the office of a customer where I can not access my environment with SSH. After work I will post the patches. Is this okay with you?
@Alin: Fixing this whole issue with signed/unsigned stuff the proper way would require us to change as well the way the data is saved in the storage engine. We would sure need to set all the INT fields to INT UNSIGNED. Then we would as well need to change some structs in libdspam_objects.h and for sure the storage drivers (mysql, pgsql and sqlite. I am not sure about the hash driver). And we would need to offer a migration path/way for the existing users. The migration path could be as simple as one SQL script (for example MySQL): ALTER TABLE `dspam_signature_data` CHANGE `data` `data` LONGBLOB NOT NULL, CHANGE `length` `length` INT( 11 ) UNSIGNED NOT NULL DEFAULT '0' ALTER TABLE `dspam_stats` CHANGE `spam_learned` `spam_learned` INT( 11 ) UNSIGNED NOT NULL DEFAULT '0', CHANGE `innocent_learned` `innocent_learned` INT( 11 ) UNSIGNED NOT NULL DEFAULT '0', CHANGE `spam_misclassified` `spam_misclassified` INT( 11 ) UNSIGNED NOT NULL DEFAULT '0', CHANGE `innocent_misclassified` `innocent_misclassified` INT( 11 ) UNSIGNED NOT NULL DEFAULT '0', CHANGE `spam_corpusfed` `spam_corpusfed` INT( 11 ) UNSIGNED NOT NULL DEFAULT '0', CHANGE `innocent_corpusfed` `innocent_corpusfed` INT( 11 ) UNSIGNED NOT NULL DEFAULT '0', CHANGE `spam_classified` `spam_classified` INT( 11 ) UNSIGNED NOT NULL DEFAULT '0', CHANGE `innocent_classified` `innocent_classified` INT( 11 ) UNSIGNED NOT NULL DEFAULT '0' This all would require a quite big time investment from us. Are you prepared for something like that? If we go that path, then I could contact Sensery Network and talk with them. They anyway contacted me last week asking for our Gentoo patches (quite a lot of people outside Gentoo use our patch set for DSPAM. Many of them consider the Gentoo DSPAM package as THE DSPAM package). However... I have since one week running DSPAM with fixed signed/unsigned values for most stuff, but only for the MySQL driver so far. As mentioned before: I have not finished all my tests and don't know how good/bad the patches are. I don't think that they are ultra bad. But I can't be objective on this one, since I coded them.
Ach! I forgot to post some proof: Download http://www.cs.virginia.edu/~cs101/hws/hw6/markov/textfiles/bible.txt and try to train it with stock DSPAM. -> will not work 100% With the patched DSPAM for signed/unsigned stuff I get this: mail / # dspam --user globaluser --process --deliver=summary --stdout < /tmp/bible.txt X-DSPAM-Result: globaluser; result="Innocent"; class="Innocent"; probability=0.0000; confidence=1.00; signature=1,4755c2c538801608415579 mail / # Checking for signature data I get this: mail / # mysql --user=$(sed -n "3,1p" /etc/mail/dspam/mysql.data) --password=$(sed -n "4,1p" /etc/mail/dspam/mysql.data) --socket=$(sed -n "1,1p" /etc/mail/dspam/mysql.data) -e "select uid,signature,octet_length(data),length,created_on from dspam_signature_data where signature='1,4755c2c538801608415579'" $(sed -n "5,1p" /etc/mail/dspam/mysql.data) +-----+--------------------------+--------------------+---------+------------+ | uid | signature | octet_length(data) | length | created_on | +-----+--------------------------+--------------------+---------+------------+ | 1 | 1,4755c2c538801608415579 | 8804556 | 8804556 | 2007-12-04 | +-----+--------------------------+--------------------+---------+------------+ mail / # As you see the length is way above 65KB and it works :) I know that no one is so crazy to train 766'111 words with an anti spam filter. Especially not with anything more complex then unigram (I used noise with osb with burton graham naive and bcr). But to know that it is possible with DSPAM makes me more confident to use DSPAM.
I'm not sure is worth changing to LONGBLOB because: a) most spam messages (99.99% ?) are below 100 Kb (MaxMessageSize will be set by default to it in the next version) b) mysql's max_allowed_packet would have to be dramatically increased c) the price we would have to pay performance wise is probably high I agree with your changes in dspam_stats table, but when it comes to dspam_signature_data, I think this is more appropriate: ALTER TABLE `dspam_signature_data` CHANGE `length` `length` SMALLINT UNSIGNED NOT NULL DEFAULT '0' Anyway, we need to fix this dspam_signature_data issue. The question is not whether we do it or not, but when we do it.
(In reply to comment #18) > I'm not sure is worth changing to LONGBLOB because: > a) most spam messages (99.99% ?) are below 100 Kb (MaxMessageSize will be > set by default to it in the next version) > That is not 100% true but I see your point. > b) mysql's max_allowed_packet would have to be dramatically increased > Yes. That is sure. > c) the price we would have to pay performance wise is probably high > I am honest with you: Believe it or not. The patched DSPAM is faster then the old one. > I agree with your changes in dspam_stats table, but when it comes to > dspam_signature_data, I think this is more appropriate: > ALTER TABLE `dspam_signature_data` > CHANGE `length` `length` SMALLINT UNSIGNED NOT NULL DEFAULT '0' > This would basically limit us to have 65kb data. I personally would change to LONGBLOB and fix the signed/unsigned issue because: 1) Changing from BLOB to LONGBLOB costs us 2 byte more storage for field data per entry in dspam_signature_data. This is less then 2mb for 1 million entries. I don't find this much. 2) Changing length in dspam_signature_data from smallint to int unsigned is costing us as well 2 bytes more storage per entry in dspam_signature_data. This is again less then 2mb for 1 million entries. Which is not much. I am not saying that every one should now bump up his DSPAM to have 20MB in MaxMessageSize. This would be insane. But I am 100% for having a DSPAM being able to store more data while still not forcing any one to use that functionality and still being stable. Stability is one of the main reasons I started doing the patches. It is not about having huge data in DSPAM. It is about stability and predictability of the anti spam filter. I am all about having DSPAM recognize and honor max_allowed_packet and adapt when a user selects to have a low max_allowed_packet. In fact I have rewritten some stuff in mysql_drv.c to just do that. Assume the whole query would be 4mb. Assume the user has max_allowed_packet of 1mb. I rewrote some stuff to split the query around 1mb and still deliver the proper result. So I am coding for the worst but still thinking about the insanity others could have. If MySQL would allow me to concatenate BLOB's to work around max_allowed_packet, then I would do it. But unfortunately MySQL does not allow this. Having max_allowed_packet of 1mb and trying to concatenate a field having already 800kb and just doing "update dspam_signature_data set data=concat(data,'<300kb of data>') where uid=1" will trigger max_allowed_packet restrictions. Even if the total query send is 320kb. MySQL will not allow me to execute that if max_allowed_packet is 1mb. But I captured that condition in the new code and I do print out warnings suggesting to increase the max_allowed_packet to another size. I am basically keeping up the service (DSPAM) operational without crashing it but still try my best to do what is requested from DSPAM. > Anyway, we need to fix this dspam_signature_data issue. The question is not > whether we do it or not, but when we do it. > For MySQL I have this fixed. If you only want to fix this signed/unsigned stuff, then I think that doing the fixes for the other drivers then MySQL should not be a problem for me. I could do it. But the user must alter the table structure. This is for sure.
@Alin: I am too much ITIL (http://en.wikipedia.org/wiki/ITIL) infected :) For me the top of the top priority is to have a stable DSPAM. I personally would love to have the possibility to store more then 65kb data but if I have to choose between stability and more data on the storage then I choose stability.
BTW: DSPAM saves in PostgreSQL in bigger BLOB's then MySQL: $ psql dspam dspam=# select uid,signature,octet_length(data),length,created_on from dspam_signature_data where signature='500,47566afe216271527717022'; uid | signature | octet_length | length | created_on -----+-----------------------------+--------------+---------+------------ 500 | 500,47566afe216271527717022 | 1925400 | 1925400 | 2007-12-05 (1 row) So pushing MySQL to be on the same level is not so far fetched. What do you think?
Okay, have it your way. ;)
(In reply to comment #22) > Okay, have it your way. ;) > My way? LOL! I don't do all this just for me. It is for US Gentoo users :)
BTW: Could you emerge DSPAM without debug USE flag and try if your installation works more stable then with debug USE flag?
(In reply to comment #24) > BTW: Could you emerge DSPAM without debug USE flag and try if your installation > works more stable then with debug USE flag? You mean me? :) Well, initially it was not my intent to debug DSPAM but to use it... So when I first experienced the crash it was without the debug USE flag. By the way in the beginning I've tried to do some research about which backend I should use. I'm not a fan of putting every database in the system into MySQL and also it was stated that several backends are not reentrant so I chose hdb because it was claimed to be reentrant which is a requirement for --daemon mode. Based on my own experiences and after endless g00gling about the topic I suspect some weaknesses in this Piece of Software - but I could be wrong because I haven't reviewed it thoroughly.
(In reply to comment #25) > (In reply to comment #24) > > BTW: Could you emerge DSPAM without debug USE flag and try if your installation > > works more stable then with debug USE flag? > > You mean me? :) > Yes you :) > Well, initially it was not my intent to debug DSPAM but to use it... > So when I first experienced the crash it was without the debug USE flag. > Okay. In the beginning no debug flag and you had a crash. Hmm... Could it be because your dspam.conf lacks the following entries: # # StorageDriver: Specifies the storage driver backend (library) to use. # You'll only need to set this if you are using dynamic storage driver plugins # from a binary distribution. The default build statically links the storage # driver (when only one is specified at configure time), overriding this # setting, which only comes into play if multiple storage drivers are specified # at configure time. When using dynamic linking, be sure to include the path # to the library if necessary, and some systems may use an extension other # than .so (e.g. OSX uses .dylib). # # Options include: # # libmysql_drv.so libpgsql_drv.so libsqlite_drv.so # libsqlite3_drv.so libhash_drv.so # # IMPORTANT: Switching storage drivers requires more than merely changing # this option. If you do not wish to lose all of your data, you will need to # migrate it to the new backend before making this change. # StorageDriver /usr/lib/dspam/libhash_drv.so > By the way in the beginning I've tried to do some research about which > backend I should use. I'm not a fan of putting every database in the > system into MySQL and also it was stated that several backends are not > reentrant so I chose hdb because it was claimed to be reentrant which > is a requirement for --daemon mode. Based on my own experiences and > after endless g00gling about the topic I suspect some weaknesses in > this Piece of Software - but I could be wrong because I haven't > reviewed it thoroughly. > What is hdb? I don't know that storage backend for DSPAM. However... DSPAM is not bad. If you look around then you will see that DSPAM ranks among CRM114 and OSBF-Lua on the top when doing benchmarks at TREC. At TREC07 CRM114 was one of the best at learning having a below 1% ROCA. Two other filters shared the 1st place with CRM114. DSPAM was not on board since Jonathan has left DSPAM and Sensory Network is/was not interested going to TREC07 (this is my assumption. I don't know why they did not go). However: The hash driver is new in DSPAM. Jonathan ported it from CRM114 to DSPAM. It is relative new and has some issues: - locking is not done the right way. - the tools are trying to rename a file they create in /tmp (this path is hardcoded!) to the destination. On my system this does not work since /tmp is a separate LVM volume. - etc... If you want to use something fast as hell and you want to use the hash driver, then I would suggest you to look at CRM114 (for fast go with HYPERSPACE in CRM114). From the viewpoint of tokenizers and algorithms CRM114 beats DSPAM in every aspect. CRM114 has every tokenizer/algorithm DSPAM has and more. But DSPAM is easier to manage and has some other nice functions which makes life for the operator (and user) much more easy. I am hosting over 200 domains and while I love (and use for my self) CRM114; I am still using DSPAM for the other users. As mentioned before: DSPAM is much easier to handle then CRM114. If you are going to use DSPAM for a lot of users, then I think MySQL or PostgreSQL offers you the most. Both have preference extensions and this is a really huge benefit for managing users. Currently I am fixing the mysql_drv in DSPAM and while doing that I have found some really nasty things in the other modules. But I am first making the mysql_drv finished and rock solid and then I go after PostgreSQL and then after SQLite/SQLite3 and then Hash. If you can't wait, then I would suggest to switch to MySQL (even if you don't like to have everything in one DB) or PostgreSQL or the SQLite backend. All of them are enough stable for production. Just don't go with hash if you have a lot of traffic and especially don't go with hash if you are plaining to put the hash database on a shared storage which does not support proper locking. // SteveB
@Rumi: I strongly suspect the debug flags to be the problem. Could you completly disable the debug USE flag in DSAPM and remerge DSPAM without debug flags set and then try again if DSPAM is crashing? // SteveB
Hi, I'm using mail-filter/dspam-3.8.0-r9 with daemon mysql syslog and virtual-users flags and FEATURES="splitdebug" on Actually, dspam was crashing randomly in daemon as well as in client mode and has led to some mail lost. It could be really difficult to find out where the memory overflow sit. So, i tried to stabilize dspam by modifying dspam.conf file. With the new dspam.conf joined in attachment dspam won't crash anymore. I believe the issue could be related with some uninitialized variables. I didn't look in the source code right now but I suspect the following options Serverxx and Clientxx are needed even when dspam is working in client mode (I was not able to use dspam in daemon mode with qmail as expected. It doesn't behave well with vpopmail - vpopmail user has no shell). ServerPass.Relay1 "secret" ServerParameters "--deliver=innocent" ServerIdent "localhost" ServerDomainSocketPath "/var/run/dspam/dspam.sock" ClientHost "/var/run/dspam/dspam.sock" ClientIdent "secret@Relay1" I'm now able to use debug flag, -O3 as well -O2 optimization, dspam simply doesn't crash anymore. dspam uses --stdout to deliver the mail to the virtual mailboxes. cat /var/vpopmail/domains/../.qmail-default | /usr/bin/dspam --deliver=innocent --user $EXT@$HOST --process --debug --stdout | /var/vpopmail/bin/vdelivermail '' bounce-no-mailbox I hope i'm not dreaming :) Jj
Created attachment 139317 [details] usable configuration file
(In reply to comment #28) > Hi, > > I'm using mail-filter/dspam-3.8.0-r9 with daemon mysql syslog and > virtual-users flags and FEATURES="splitdebug" on > > Actually, dspam was crashing randomly in daemon as well as in client mode and > has led to some mail lost. > Was it crashing with 3.8.0-r9 or with an older version? You know that we fixed a crash issue in the 3.8.0-r9 release? Anything older then -r9 still has this issue. > It could be really difficult to find out where the > memory overflow sit. So, i tried to stabilize dspam by modifying dspam.conf > file. > With the new dspam.conf joined in attachment dspam won't crash anymore. > Do you have by accident the old dspam.conf where your DSPAM would crash with it? If so: Could you attach it to this bug report? > I > believe the issue could be related with some uninitialized variables. I didn't > look in the source code right now but I suspect the following options Serverxx > and Clientxx are needed even when dspam is working in client mode (I was not > able to use dspam in daemon mode with qmail as expected. It doesn't behave well > with vpopmail - vpopmail user has no shell). > > ServerPass.Relay1 "secret" > ServerParameters "--deliver=innocent" > ServerIdent "localhost" > ServerDomainSocketPath "/var/run/dspam/dspam.sock" > ClientHost "/var/run/dspam/dspam.sock" > ClientIdent "secret@Relay1" > > I'm now able to use debug flag, -O3 as well -O2 optimization, dspam simply > doesn't crash anymore. > It is not surprising that it works with -r9. We fixed a crash with turned on debugging and pushed that patch into -r9. > dspam uses --stdout to deliver the mail to the virtual mailboxes. > > cat /var/vpopmail/domains/../.qmail-default > | /usr/bin/dspam --deliver=innocent --user $EXT@$HOST --process --debug > --stdout | /var/vpopmail/bin/vdelivermail '' bounce-no-mailbox > > I hope i'm not dreaming :) > Why should you be dreaming? Because of no crash? If I understand you right, then your issue is solved? Can Alin close this bug then? > > Jj > Steve
Seems that reporter loosed interest for this package. Closed as NEEDINFO.
