new anti spam feature: "zombielist", help wanted - SpamAssassin
This is a discussion on new anti spam feature: "zombielist", help wanted - SpamAssassin ; Dear spam haters,
we've developed an idea and made a quick "proof of concept" hack and
tested it during the last months, and it works very successfully. The
idea was to decrease the load on the mailgate running spamassassin +
...
-
new anti spam feature: "zombielist", help wanted
Dear spam haters,
we've developed an idea and made a quick "proof of concept" hack and
tested it during the last months, and it works very successfully. The
idea was to decrease the load on the mailgate running spamassassin +
postfix, by automatically creating a "zombielist" of IPs that are known
to be bad. These IPs are then blacklisted in postfix, this file is used
before any checks of RBLs, URIBLs, etc.
Bad in our case are
- no reverse lookup for that IP at all
- "lost connection after" messages
- RBL listing of that IP (in order to prevent future lookups)
- "day old bread": sender domains that are younger than 5 days (which
kills a lot of those http://domainnamekiting.com/ domains spams.
This simple table alone helps us reject about 30% of all spam before any
spamassassin or DNS lookups are needed. It currently lists about
250.000 single IPs for us.
The current implementation also has some downsides:
- we used the python script "blockhosts" which is horribly slow. It
reads the maillog and greps the strings from there
- the script does not run as a daemon but from cron, which means a
several minute delay from the first access of a zombie until it is
listed
- because of that delay, we set the time until deletion of an IP very
high, which makes good hosts that once had no reverse IP lookup but now
have not go off the list very quickly, meaning manual interaction
sometimes
- it's only possible to list or not list. A bias depending on which
error you got would be good in order to make a difference between "is
listed on RBL" and "junkdomain <5d old".
We've had lots of ideas for a new daemon, but we're no hackers and have
nobody to do it.
Roughly it should be able to do this:
- run as a permanent daemon
- be high speed in reading mail logs and updating records
- use an SQL db (postgreSQL at least) for managing records
- finer control over how many errors of which type are allowed
- manual whitelist IPs (e.g. known ISPs)
- a simple web interface to manage IPs
- GNU GPL licence
Anybody willing to help? I crosspost to users@spamassassin.apache.org
also, but please keep discussions on the postfix list if possible. It's
not really about postfix, but even less about spamassassin, it's just
another tool to get better filtering. If we start development, we'll
make a sf.net project anyway.
mfg zmi
--
// Michael Monnerie, Ing.BSc ----- http://it-management.at
// Tel: 0676/846 914 666 .network.your.ideas.
// PGP Key: "curl -s http://zmi.at/zmi.asc | gpg --import"
// Fingerprint: AC19 F9D5 36ED CD8A EF38 500E CE14 91F7 1C12 09B4
// Keyserver: www.keyserver.net Key-ID: 1C1209B4
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.4-svn0 (GNU/Linux)
iD8DBQBIOsEAzhSR9xwSCbQRAkKCAKCJTF3Uk1b4fb1zUlV+QX sQn1hH8ACg4PCZ
U84R9U/tuLnMy6v1LOHDGc0=
=3tjf
-----END PGP SIGNATURE-----
-
Re: new anti spam feature: "zombielist", help wanted
On Mon, May 26, 2008 15:54, Michael Monnerie wrote:
> Dear spam haters,
i am a spam lower 
if i get no spam i cant block it
# /etc/postfix/amavisd_defer_if_permit_maps.cf
# using amavisd sql logs
user = foo
password = foo
hosts = localhost
dbname = foo
query = SELECT CONCAT('DEFER_IF_PERMIT') FROM msgs WHERE client_addr = '%s'
AND content = 'S' AND spam_level > '0' ORDER BY 'spam_level' ASC LIMIT 1
# main.cf
smtpd_client_restrictions =
check_client_access mysql:/etc/postfix/amavisd_defer_if_permit_maps.cf
if one can improve it please tell me
simple 
Benny Pedersen
Need more webspace ? http://www.servage.net/?coupon=cust37098
-
Message-ID:Reply-To:References:MIME-Version:Content-Type:In-Reply-To; b=f9/IJvoztnr+eOVHVhJtGLPA5dmuHNRB8T89/0XpSRLB6AEqGHFMiqur997NpzZlozcKn6pMTpk+q4hllqlEvQJ tse9Lv+5+HwsG9MnGzHStMVDJLE6/APf/95Ziy59xdg7KOr8HXufEppGQU5muayiv3KuKD/cpIRAPuwt231g=
On Tue, May 27, 2008 at 01:45:23AM +0200, Benny Pedersen wrote:
>
> On Mon, May 26, 2008 15:54, Michael Monnerie wrote:
> > Dear spam haters,
>
> i am a spam lower 
>
> if i get no spam i cant block it
>
> # /etc/postfix/amavisd_defer_if_permit_maps.cf
> # using amavisd sql logs
> user = foo
> password = foo
> hosts = localhost
> dbname = foo
> query = SELECT CONCAT('DEFER_IF_PERMIT') FROM msgs WHERE client_addr = '%s'
> AND content = 'S' AND spam_level > '0' ORDER BY 'spam_level' ASC LIMIT 1
>
> # main.cf
> smtpd_client_restrictions =
> check_client_access mysql:/etc/postfix/amavisd_defer_if_permit_maps.cf
>
>
> if one can improve it please tell me
I hope you have a good whitelist.
What do you do when you get a single spam from hotmail/gmail etc? You are
going to defer a host that sends you mostly legimate mails?
You should modify it atleast to take account of how much ham has been sent.
I would continue this on amavis mailing list, since it's a bit offtopic.
-
Re: new anti spam feature: "zombielist", help wanted
On Dienstag, 27. Mai 2008 Henrik K wrote:
> What do you do when you get a single spam from hotmail/gmail etc? You
> are going to defer a host that sends you mostly legimate mails?
No, it's not even about spam currently. It's plainly one of the three
options I mentioned in my first e-mail (see also below) than can get
you on the list.
Also, there's a counter, and only after several errors a host get's
listed.
With the new daemon, it should be possible to count for a sender IP the
number of hams/spams and then you could say "if a sender sends 50% spam
during the last 24h, block it".
> You should modify it atleast to take account of how much ham has been
> sent.
That is currently not implemented as we do not look at this at all.
> I would continue this on amavis mailing list, since it's a bit
> offtopic.
It's also not an amavis thingy, but a good idea I will drop the message
there also. It best fits on postfix-users I guess, but we will see.
[From another post:]
> what's the goal here exactly?
* reduce the CPU and network load:
- you don't need to ask RBLs too often
- if the sender server does not have a rDNS, you block it for a while
- sender's domains younger than 5 days are blocked
- and with the new daemon I want to be able to have finer control, so
you can also block senders who constantly connect and then "lost
connection after ..." (we've had this once from lot's of hosts, looked
like a dDoS). The current implementation only gives one possible
timeout value for delisting, while the new should have a timeout per
rule (eg. delist IP after 6h if listed because of RBL, but 1d if listed
because of no reverse IP).
[From another post:]
>> - no reverse lookup for that IP at all
> Good luck. I'd reject heaps of legimate mail with that.
We've started implementing that early 2007. By then, we've had some
issues, but they are constantly reducing. We're in Austria/Europe, and
one big advantage is that GMX is also rejecting in case of missing
rDNS: http://faq.gmx.de/optionen/email/antispam/4.html
If nobody does start to enforce stricter rules, the SPAM problem won't
go away. But anyway, it's of course configurable if you want to reject
on a missing rDNS or not.
One solution could be to make a "transition phase", in that you greylist
hosts without rDNS or reject e-mails for an hour with a message that
they should create a rDNS entry and then allow for an hour... but not
doing anything because that could possibly upset someone with a
not-so-clean setup is not on our mind.
Currently, when we have a valid customer w/o rDNS then we whitelist then
for 14 days, and send an info e-mail to them and their postmaster@,
with the explanation what to do. Works quite well, very seldom we hear
again after 14 days that they want to be whitelisted again - and only
one I had somebody 3x whitelisted ;-)
> I just use a simple perl-script that blocks hosts with iptables. It
> uses File::Tail to monitor log, so blocking is done almost without
> delays. I don't see much point in using postfix to block things,
> since it takes more resources.
> http://hege.li/contrib/smtp_block.pl
That way you cannot even see when a host tried to send an e-mail, which
makes looking for missing e-mails impossible. If you have a false
positive, you cannot even look it up in mail logs. That's absolutely a
no-go for us. Either we accept an e-mail or we reject it - on SMTP
level. It might be a good idea for hosts that behave very badly though
(say, both listed on RBLs and no rDNS), but the little overhead
involved in rejecting at SMTP level is no real ressource pain.
But it's a good start for a daemon :-)
mfg zmi
--
// Michael Monnerie, Ing.BSc ----- http://it-management.at
// Tel: 0676/846 914 666 .network.your.ideas.
// PGP Key: "curl -s http://zmi.at/zmi.asc | gpg --import"
// Fingerprint: AC19 F9D5 36ED CD8A EF38 500E CE14 91F7 1C12 09B4
// Keyserver: www.keyserver.net Key-ID: 1C1209B4
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.4-svn0 (GNU/Linux)
iD8DBQBIO8bAzhSR9xwSCbQRAruxAJ9c+lHKP+8XLICCsEUr36 FQsXphkACZAdGK
uQxik7NX3eDEQ73Ha1tva9s=
=16Sj
-----END PGP SIGNATURE-----