new anti spam feature: "zombielist", help wanted - SpamAssassin

This is a discussion on new anti spam feature: "zombielist", help wanted - SpamAssassin ; Dear spam haters, we've developed an idea and made a quick "proof of concept" hack and tested it during the last months, and it works very successfully. The idea was to decrease the load on the mailgate running spamassassin + ...

+ Reply to Thread
Results 1 to 4 of 4

Thread: new anti spam feature: "zombielist", help wanted

  1. new anti spam feature: "zombielist", help wanted

    Dear spam haters,

    we've developed an idea and made a quick "proof of concept" hack and
    tested it during the last months, and it works very successfully. The
    idea was to decrease the load on the mailgate running spamassassin +
    postfix, by automatically creating a "zombielist" of IPs that are known
    to be bad. These IPs are then blacklisted in postfix, this file is used
    before any checks of RBLs, URIBLs, etc.
    Bad in our case are
    - no reverse lookup for that IP at all
    - "lost connection after" messages
    - RBL listing of that IP (in order to prevent future lookups)
    - "day old bread": sender domains that are younger than 5 days (which
    kills a lot of those http://domainnamekiting.com/ domains spams.

    This simple table alone helps us reject about 30% of all spam before any
    spamassassin or DNS lookups are needed. It currently lists about
    250.000 single IPs for us.

    The current implementation also has some downsides:
    - we used the python script "blockhosts" which is horribly slow. It
    reads the maillog and greps the strings from there
    - the script does not run as a daemon but from cron, which means a
    several minute delay from the first access of a zombie until it is
    listed
    - because of that delay, we set the time until deletion of an IP very
    high, which makes good hosts that once had no reverse IP lookup but now
    have not go off the list very quickly, meaning manual interaction
    sometimes
    - it's only possible to list or not list. A bias depending on which
    error you got would be good in order to make a difference between "is
    listed on RBL" and "junkdomain <5d old".

    We've had lots of ideas for a new daemon, but we're no hackers and have
    nobody to do it.

    Roughly it should be able to do this:
    - run as a permanent daemon
    - be high speed in reading mail logs and updating records
    - use an SQL db (postgreSQL at least) for managing records
    - finer control over how many errors of which type are allowed
    - manual whitelist IPs (e.g. known ISPs)
    - a simple web interface to manage IPs
    - GNU GPL licence

    Anybody willing to help? I crosspost to users@spamassassin.apache.org
    also, but please keep discussions on the postfix list if possible. It's
    not really about postfix, but even less about spamassassin, it's just
    another tool to get better filtering. If we start development, we'll
    make a sf.net project anyway.

    mfg zmi
    --
    // Michael Monnerie, Ing.BSc ----- http://it-management.at
    // Tel: 0676/846 914 666 .network.your.ideas.
    // PGP Key: "curl -s http://zmi.at/zmi.asc | gpg --import"
    // Fingerprint: AC19 F9D5 36ED CD8A EF38 500E CE14 91F7 1C12 09B4
    // Keyserver: www.keyserver.net Key-ID: 1C1209B4

    -----BEGIN PGP SIGNATURE-----
    Version: GnuPG v2.0.4-svn0 (GNU/Linux)

    iD8DBQBIOsEAzhSR9xwSCbQRAkKCAKCJTF3Uk1b4fb1zUlV+QX sQn1hH8ACg4PCZ
    U84R9U/tuLnMy6v1LOHDGc0=
    =3tjf
    -----END PGP SIGNATURE-----


  2. Re: new anti spam feature: "zombielist", help wanted


    On Mon, May 26, 2008 15:54, Michael Monnerie wrote:
    > Dear spam haters,


    i am a spam lower

    if i get no spam i cant block it

    # /etc/postfix/amavisd_defer_if_permit_maps.cf
    # using amavisd sql logs
    user = foo
    password = foo
    hosts = localhost
    dbname = foo
    query = SELECT CONCAT('DEFER_IF_PERMIT') FROM msgs WHERE client_addr = '%s'
    AND content = 'S' AND spam_level > '0' ORDER BY 'spam_level' ASC LIMIT 1

    # main.cf
    smtpd_client_restrictions =
    check_client_access mysql:/etc/postfix/amavisd_defer_if_permit_maps.cf


    if one can improve it please tell me


    simple


    Benny Pedersen
    Need more webspace ? http://www.servage.net/?coupon=cust37098


  3. Message-ID:Reply-To:References:MIME-Version:Content-Type:In-Reply-To; b=f9/IJvoztnr+eOVHVhJtGLPA5dmuHNRB8T89/0XpSRLB6AEqGHFMiqur997NpzZlozcKn6pMTpk+q4hllqlEvQJ tse9Lv+5+HwsG9MnGzHStMVDJLE6/APf/95Ziy59xdg7KOr8HXufEppGQU5muayiv3KuKD/cpIRAPuwt231g=

    On Tue, May 27, 2008 at 01:45:23AM +0200, Benny Pedersen wrote:
    >
    > On Mon, May 26, 2008 15:54, Michael Monnerie wrote:
    > > Dear spam haters,

    >
    > i am a spam lower
    >
    > if i get no spam i cant block it
    >
    > # /etc/postfix/amavisd_defer_if_permit_maps.cf
    > # using amavisd sql logs
    > user = foo
    > password = foo
    > hosts = localhost
    > dbname = foo
    > query = SELECT CONCAT('DEFER_IF_PERMIT') FROM msgs WHERE client_addr = '%s'
    > AND content = 'S' AND spam_level > '0' ORDER BY 'spam_level' ASC LIMIT 1
    >
    > # main.cf
    > smtpd_client_restrictions =
    > check_client_access mysql:/etc/postfix/amavisd_defer_if_permit_maps.cf
    >
    >
    > if one can improve it please tell me


    I hope you have a good whitelist.

    What do you do when you get a single spam from hotmail/gmail etc? You are
    going to defer a host that sends you mostly legimate mails?

    You should modify it atleast to take account of how much ham has been sent.

    I would continue this on amavis mailing list, since it's a bit offtopic.


  4. Re: new anti spam feature: "zombielist", help wanted

    On Dienstag, 27. Mai 2008 Henrik K wrote:
    > What do you do when you get a single spam from hotmail/gmail etc? You
    > are going to defer a host that sends you mostly legimate mails?


    No, it's not even about spam currently. It's plainly one of the three
    options I mentioned in my first e-mail (see also below) than can get
    you on the list.
    Also, there's a counter, and only after several errors a host get's
    listed.
    With the new daemon, it should be possible to count for a sender IP the
    number of hams/spams and then you could say "if a sender sends 50% spam
    during the last 24h, block it".

    > You should modify it atleast to take account of how much ham has been
    > sent.


    That is currently not implemented as we do not look at this at all.

    > I would continue this on amavis mailing list, since it's a bit
    > offtopic.


    It's also not an amavis thingy, but a good idea I will drop the message
    there also. It best fits on postfix-users I guess, but we will see.

    [From another post:]
    > what's the goal here exactly?


    * reduce the CPU and network load:
    - you don't need to ask RBLs too often
    - if the sender server does not have a rDNS, you block it for a while
    - sender's domains younger than 5 days are blocked
    - and with the new daemon I want to be able to have finer control, so
    you can also block senders who constantly connect and then "lost
    connection after ..." (we've had this once from lot's of hosts, looked
    like a dDoS). The current implementation only gives one possible
    timeout value for delisting, while the new should have a timeout per
    rule (eg. delist IP after 6h if listed because of RBL, but 1d if listed
    because of no reverse IP).

    [From another post:]
    >> - no reverse lookup for that IP at all

    > Good luck. I'd reject heaps of legimate mail with that.


    We've started implementing that early 2007. By then, we've had some
    issues, but they are constantly reducing. We're in Austria/Europe, and
    one big advantage is that GMX is also rejecting in case of missing
    rDNS: http://faq.gmx.de/optionen/email/antispam/4.html
    If nobody does start to enforce stricter rules, the SPAM problem won't
    go away. But anyway, it's of course configurable if you want to reject
    on a missing rDNS or not.
    One solution could be to make a "transition phase", in that you greylist
    hosts without rDNS or reject e-mails for an hour with a message that
    they should create a rDNS entry and then allow for an hour... but not
    doing anything because that could possibly upset someone with a
    not-so-clean setup is not on our mind.
    Currently, when we have a valid customer w/o rDNS then we whitelist then
    for 14 days, and send an info e-mail to them and their postmaster@,
    with the explanation what to do. Works quite well, very seldom we hear
    again after 14 days that they want to be whitelisted again - and only
    one I had somebody 3x whitelisted ;-)

    > I just use a simple perl-script that blocks hosts with iptables. It
    > uses File::Tail to monitor log, so blocking is done almost without
    > delays. I don't see much point in using postfix to block things,
    > since it takes more resources.
    > http://hege.li/contrib/smtp_block.pl


    That way you cannot even see when a host tried to send an e-mail, which
    makes looking for missing e-mails impossible. If you have a false
    positive, you cannot even look it up in mail logs. That's absolutely a
    no-go for us. Either we accept an e-mail or we reject it - on SMTP
    level. It might be a good idea for hosts that behave very badly though
    (say, both listed on RBLs and no rDNS), but the little overhead
    involved in rejecting at SMTP level is no real ressource pain.

    But it's a good start for a daemon :-)

    mfg zmi
    --
    // Michael Monnerie, Ing.BSc ----- http://it-management.at
    // Tel: 0676/846 914 666 .network.your.ideas.
    // PGP Key: "curl -s http://zmi.at/zmi.asc | gpg --import"
    // Fingerprint: AC19 F9D5 36ED CD8A EF38 500E CE14 91F7 1C12 09B4
    // Keyserver: www.keyserver.net Key-ID: 1C1209B4

    -----BEGIN PGP SIGNATURE-----
    Version: GnuPG v2.0.4-svn0 (GNU/Linux)

    iD8DBQBIO8bAzhSR9xwSCbQRAruxAJ9c+lHKP+8XLICCsEUr36 FQsXphkACZAdGK
    uQxik7NX3eDEQ73Ha1tva9s=
    =16Sj
    -----END PGP SIGNATURE-----


+ Reply to Thread