Bugzilla Bug 172929

From the DSPAM mailing list:
> Greetings,
> 
> I know it's been a long time since the last DSPAM release, and I've  
> decided - after much testing over the past year - that it was good  
> and appropriate to release the changes in CVS. Because the  
> configuration has changed to some degree, and new features have been  
> added, I thought it best to call this release 3.8.0 instead of 3.6.9,  
> as I had been originally planning. Don't be fooled, though - it's  
> very similar to 3.6, but has been optimized significantly and some  
> important bugfixes have been made. It also sports some new features  
> such as Bill Yerazunis' Orthogonal Sparse Bigram <OSB> tokenizer.
> 
> There are some very positive things happening with the project, and  
> I'll be making an announcement soon that I think will answer a lot of  
> questions and help the project move forward again out of the small  
> slump it's been in. Until all ducks are in a row, though, I'm going  
> to release 3.8.0 as an unofficial release. You can download it here:
> 
> http://www.zdziarski.com/projects/dspam/sources/
> MD5 (dspam-3.8.0.tar.gz) = 056b8c8b3ad9415a52c01b22ff1e64cf
> 
> Or grab it from cvs.zdziarski.com (I haven't yet committed every last  
> detail into the production CVS).
> 
>   Jonathan

Reproducible: Always

Created an attachment (id=115094) [details]
mail-filter/dspam/dspam-3.8.0.ebuild

Created an attachment (id=115095) [details]
mail-filter/dspam/files/3.8.0-dspam.cron

Created an attachment (id=115096) [details]
mail-filter/dspam/files/dspam.rc

Created an attachment (id=115098) [details]
mail-filter/dspam/files/logrotate.dspam

Created an attachment (id=115099) [details]
mysql_purge-4.1-optimized.sql

If some one wants to have faster purging with MySQL then use this SQL file and
execute this inside the DSPAM database:
alter table dspam_token_data add index(spam_hits);
alter table dspam_token_data add index(innocent_hits);
alter table dspam_token_data add index(last_hit);

Copy the SQL file to your /etc/mail/dspam/ directory. The included cron script
will pick up the proper SQL file when purging.

More info about the script:
http://securitydot.net/txt/id/32/type/articles/

Created an attachment (id=115101) [details]
dspam-3.8.0-patches-20070331.tar.gz

I know, I know... binary files should be avoided in Bugzilla.
The binary has the same patches as the 3.6.8 version in Portage except that
patch 11 is not needed anymore and a new patch (number 16) was added.

I prefer to wait for this to become official (plus I'm very busy atm), but if
someone else wanna perform this bump, it's okay by me.

Careful though, there are other open dspam bugs in my input tray.

(In reply to comment #7)
> I prefer to wait for this to become official (plus I'm very busy atm), but if
> someone else wanna perform this bump, it's okay by me.
> 
> Careful though, there are other open dspam bugs in my input tray.
> 

Okay. I understand your point. What open DSPAM bugs do you have in your input
tray? Maybe I could help?

I was talking about bug 172781. Nothing to be done there except applying the
patch and maybe modifying the cron script.

You forgot about www-apps/dspam-web. This should also be bumped, although I
don't know if there is something changed in it.

(In reply to comment #9)
> I was talking about bug 172781. Nothing to be done there except applying the
> patch and maybe modifying the cron script.
> 
> You forgot about www-apps/dspam-web. This should also be bumped, although I
> don't know if there is something changed in it.
> 
So you want the cron script to use /bin/sh or another shell instead of bash? Is
that right?

Created an attachment (id=115556) [details]
3.8.0-dspam.cron

- uses /bin/sh instead of /bin/bash
- handles correctly missing *.data files
- processes SQLite and SQLite3 databases

Created an attachment (id=115578) [details]
mail-filter/dspam/files/3.8.0-dspam.cron

- added support for remote MySQL server

Created an attachment (id=115579) [details]
mail-filter/dspam/files/3.8.0-dspam.cron

- added remote MySQL support when running "OPTIMIZE TABLE" commands

Created an attachment (id=115582) [details]
mail-filter/dspam/files/3.8.0-dspam.cron

- fixed typo in SQLite3 purge part

Created an attachment (id=115584) [details]
www-apps\dspam-web\dspam-web-3.8.0.ebuild

The requested dspam-web 3.8.0 ebuild

Created an attachment (id=115637) [details]
mail-filter/dspam/dspam-3.8.0.ebuild

- fixed naming issue with cron script
- added cron script for purging old user log entries

Created an attachment (id=115676) [details]
mail-filter/dspam/files/3.8.0-dspam.cron

Added dspam_logrotate instead of deploying two cron scripts.

Created an attachment (id=115678) [details]
mail-filter/dspam/dspam-3.8.0.ebuild

Removed separate cron job for dspam_logrotate (dspam_logrotate is now included
in 3.8.0-dspam.cron).

What else is missing now? Is there anything other needed to be fixed or
included?

Created an attachment (id=116391) [details]
distfiles/dspam-3.8.0-patches-20070416.tar.gz

Created an attachment (id=116392) [details]
mail-filter/dspam/dspam-3.8.0.ebuild

Created an attachment (id=116394) [details]
www-apps/dspam-web/dspam-web-3.8.0.ebuild

BTW: DSPAM 3.8.0 is now official ->
http://dspam.nuclearelephant.com/download.shtml


Version 3.8.0
-------------

[20061210.1435] jonz: fixed message corruption problems with direct delivery

when using direct delivery (e.g. DeliveryHost), certain servers require a
linefeed after carriage return otherwise the message will become malformatted.

[20060818.0700] jonz: added msg tagging support

added ability to add tagline to messages based on their classification; see
tagSpam and tagNonspam preferences in README

[20060607.1200] jonz: removed depricated oracle driver

removed outdated oracle driver; no maintainer, lack of interest

[20060606.0000] jonz: added ldap client to build

added ldap client headers to makefile, would not build on some systems

[20060601.0500] jonz: fix for dynamic storage drivers api

fixed _ds_pref_del call to storage library

[20060601.0300] jonz: webui history fix for 12:00 noon

bugfix to display 12 noon as 12p, not 12a

[20060530.0145] jonz: added connect check for pgsql

added a connection check for pgsql, to reconnect on failure in daemon mode

[20060530.0130] jonz: added logging of viruses

added logging of viruses (and the source) to agent

[20060527.1700] jonz: added HashPctIncrease option in dspam.conf

HashPctIncrease: Increase the next extent size by n% from the size of the
last extent. The default behavior, when HashPctIncrease is not used, is to
always use HashExtentSize with no increase. This is useful in accommodating
systems where the default HashExtentSize can be too small for certain
high-volume users. 

[20060527.1530] jonz: cache runtime user information

added caching of runtime user information, so this information is not polled
every message when running in daemon mode. also elimiates the need for
getpwuid_r when running in daemon mode (unless using mysql or pgsql), 
which some operating systems do not have.

[20060527.1530] jonz: moved TIME_ME into DEBUG

when debug is active, TIME_ME automatically runs, reporting processing time
to debug

[20060526.1900] jonz: fix for library TIME_ME measurements

fixed bug where negative processing times were reported using TIME_ME

[20060526.1600] jonz: turned off locking when not using syslog or logging

no need to lock on LOG() when not logging

[20060526.0230] jonz: rewrite for hash_drv offset caching

rewrote offset caching in hash_drv; fixed some bugs which may have caused
a crash on extent addition

[20060525.1100] jonz: fix for segfault on undefined DeliveryHost or ClientHost

fix for segfault in daemon mode when DeliveryHost or ClientHost is not 
specified

[20060524.0300] jonz: added --client support for dspam_train

use --client after username

[20060523.0300] jonz: more code optimizations

various optimizations to:
- tokenizer core
- hash_drv driver (store offset for writes)
- libdspam (preference lookups)
- optimizations for osb/sbph

[20060522.0300] jonz: added ProcessorURLContext

ProcessorURLContext creates Url* context-specific tokens for URLs; this is
the default in previous (and current) versions

[20060522.0300] jonz: optimized osb/sbph tokenizer

replaced several strlcat's with simple len counting to eliminate thousands of
unnecessary calls to strlen() and speed up osb/sbph tokenization process

[20060519.0300] jonz: fix for segfault in vsyslog()

fix segfault caused by bad use of va_args when vsyslog is called

[20060519.0130] jonz: fix for segfault in dlopen() failure

fixed bug causing segfault when dlopen() to storage driver library fails.
dspam still won't work any better if dlopen is failing but huzzah.

[20060519.0100] jonz: fix for performance template / local domain

added fix to display correct local domain in performance template, and only
display local domain if the username doesn't include an @ sign

[20060517.0700] jonz: fix for preference delete

fixed infinite loop on all non-preference-extension calls to delete a
preference

[20060516.0200] jonz: changed SupressWebStats

SupressWebStats is now WebStats in dspam.conf, and setting is inverted.

[20060516.0200] jonz: fix for agent flags

discovered that agent flags required a 64-bit variable to hold all flags, but
only 32-bit variable was being used; this may have caused unpredictable
behavior when using SBPH, "unlearning" a message, or processing summaries.

[20060516.0200] jonz: added OSB tokenizer

osb (orthogonal sparse bigram) is similar to sbph, however only bigrams are
used to form sparse tokens; this uses far fewer resources than sbph with
very similar results

[20060516.0200] jonz: interface change: added tokenizer variable

added tokenizer variable to DSPAM_CTX and added following tokenizer flags:

      DSZ_WORD                Use WORD (uniGram) tokenizer
      DSZ_CHAIN               Use CHAIN (biGram) tokenizer
      DSZ_SBPH                Use SBPH (Sparse BP Hashing) tokenizer
      DSZ_OSB                 Use OSB (Orthogonal Sparse biGram)

WARNING: This is an API change and constitutes a new major version. Third
         party applications may fail to compile/run against this.

[20060414.1145] jonz: fix for segfault on log write err

when using --with-logfile, if file cannot be opened, dspam segfaulted

[20060513.1100] jonz: fixed compiler warnings on sqlite drivers

signed-ness warnings, nothing significant

[20060514.0900] jonz: discontinued support for berkeley db

deprecated bdb drivers finally removed from distribution

[20060512.2105] jonz: copyright modifications

reassignment to Jonathan Zdziarski instead of using my corporate face

[20060512.2100] jonz: removed some legacy piecess

- removed dspam_corpus (replaced by newer dspam_train)
- removed dspam_genaliases (replaced by parse-to-headers, virtual users, etc)

[20060512.0100] jonz: segfault fix for UIDInSignature

fixed a critical bug that can cause segfaults when correcting messages using
UIDInSignature options. database handle is refreshed, but new pointer is never
used.

[20060510.0800] jonz: fix to recognize trainPristine "off" in preferences

preference turned "off" should override config turned "on"

Created an attachment (id=116401) [details]
mail-filter/dspam/dspam-3.8.0.ebuild

- Changed download URL
- Added msgtag.spam and msgtag.nonspam

Created an attachment (id=116403) [details]
www-apps/dspam-web/dspam-web-3.8.0.ebuild

Changed download URL

Created an attachment (id=116404) [details]
3.8.0-dspam.cron.patch

Steve, you have done a great work with this.

I saw atleast one thing that would not work on dash:

$((foo + bar)) should be $(($foo + $bar))

The rest is mostly nitpicks, removing useless use of grep/cat etc.

The patch needs testing.

(In reply to comment #26)
> Created an attachment (id=116404) [edit] [details]
> 3.8.0-dspam.cron.patch
> 
> Steve, you have done a great work with this.
> 
Thanks


> I saw atleast one thing that would not work on dash:
> 
> $((foo + bar)) should be $(($foo + $bar))
> 
Perfect. This will work in bash and sh as well.


> The rest is mostly nitpicks, removing useless use of grep/cat etc.
> 
No, no, no.... Don't lower your great work. It is good. Using just one tool for
the job instead of cat, pipe to grep, grep, pipe to sed, etc... is much better.
Thanks!


> The patch needs testing.
> 
Patched cron job works without problems on my DSPAM 3.8.0 and MySQL 5.0.38.

Created an attachment (id=116462) [details]
mail-filter/dspam/files/3.8.0-dspam.cron

New cron job. Patched with the changes from Natanael Copa.

Stop posting cron scripts!
I already work on the new script ;)

(In reply to comment #29)
> Stop posting cron scripts!
> I already work on the new script ;)
> 
Post it! POST IT!

I want to test it :)

Fixed in cvs, thanks guys!

I've rewritten the cron script starting from your version. I hope it works well
for everyone.

The analyzer graph still has problems. Maybe one of you could take care of this
in a future bug?

(In reply to comment #31)
> Fixed in cvs, thanks guys!
> 
> I've rewritten the cron script starting from your version. I hope it works well
> for everyone.
> 
> The analyzer graph still has problems. Maybe one of you could take care of this
> in a future bug?
> 
What is the problem with the analyzer graph? On my setup I have no problem with
the analyzer graph.

Created an attachment (id=116513) [details]
dspam-web weird graphs

A picture is better than 1000 words.