Peter Farrell wrote:
> Hi all.
> Testing new setup:
> CentOS 4.4
> amavisd-new-2.5.1
> SpamAssassin version 3.2.1
> running on Perl version 5.8.5
> +RulesDuJour
> Quad proc Dell PE w/ 4 GB RAM.

Point blank. In general, *NOBODY* should use WS's blacklist file's for
ANYTHING. It is most unfortunate that RDJ has a built-in configuration
for this file.

Just take a look at the size of the files. sa-blacklist is over 24 MB!

1) the uri blacklist is redundant with SURBL. SURBL is lightweight and
reasonably fast, while the uri blacklist is a heavy memory burden and
relatively slow.

2) the email address blacklist is interesting for research purposes, but
it's real-world use is almost pointless. spammers rotate domains in from
addresses so often that the gains of this blacklist are limited, and the
memory consumption is absurd.

The files add something like 500MB to an instance of SA. That's *HUGE*.
Check your memory usage and see if the blacklist file is making your box
page. your box *might* be enough to handle the sa-blacklist, but
personally I'd consider your box kinda borderline stats-wise for running
sa-blacklist. I'd generally think more on the scale of 8GB of ram unless
I was going to constrain SA to only existing in 1 or 2 instances.
> So my questions are:
> 1. is the timing 'normal' when using the blacklist rules called
> through 'spamassassin'? Is it just a storm in a teacup? When it's
> called from Perl will it all be loaded into memory and the timing will
> drop down?

Well, calling 'spamassassin' with sa-blacklist loaded is going to be
very painful. sa-blacklist will cause SA to initialize around 500MB of
memory, that's not quick.

Or were those multi-minute times from amavis? That would be a bit much,
and I'd be checking to see if you're thrashing your swap partition.

Even so, I'd still expect it to take a least 60 seconds to scan a
message with these blacklist files loaded, on a very fast CPU.

> 2. are the rules compatible w/ the 3.2 branch of SA?

Yes, both of WS's blacklist files are technically compatible with most
any version of SA, save very, very old ones that don't support the uri
keyword. (at the very least, both will work with anything from 2.40 and
higher. digging back futher than 2.40 is an archaeological dig I'm not
really interested in at the moment).

However, in practice, sa-blacklist is not practical for real-world use,
so you could also say it's incompatible with every version of SA.

> 3. if it's 'wrong' how does one debug further? I've enabled level 5 in
> amavisd.conf & 'smtpd -v' at the top of my Am I looking in
> the wrong place? Am I missing some sort of Perl module that would
> mitigate this in some way? (I'll list these at the end)

Nope. sa-blacklist is just too huge for practical purposes. SA is
designed to efficiently support hundreds, even thousands of
blacklist_from's, but sa-blacklist has hundreds of thousands of them.
(691,372 in fact).