How HOTMAIL authenticate received messages? - TCP-IP

This is a discussion on How HOTMAIL authenticate received messages? - TCP-IP ; Does anyone know which mechanism does HOTMAIL use to authenticate received emails? Recently I received in my inbox a spam from viff@viff.org . And tried to reproduce it by sending the following SMTP message to HOTMAIL, but although HOTMAIL queued ...

+ Reply to Thread
Results 1 to 18 of 18

Thread: How HOTMAIL authenticate received messages?

  1. How HOTMAIL authenticate received messages?

    Does anyone know which mechanism does HOTMAIL use to authenticate
    received emails?
    Recently I received in my inbox a spam from viff@viff.org. And tried
    to reproduce it by sending the following SMTP message to HOTMAIL, but
    although HOTMAIL queued for delivery, it never showed up in my inbox.
    Could HOTMAIL be comparing the IP address of the sender with the
    address returned by a DNS query for the mail server of the sender's
    domain (nslookup -querytype=mx viff@org) stated in the EHLO header to
    authenticate these messages? In this case a spam sent from the
    viff.org email server would go through. While another spam send from
    another domain, but claiming to come from viff.org would be blocked?
    If so, is there any documentation on the net about it?

    Thanks in advance,

    Andre

    ************************************************
    andre@kirchner:~$ telnet mx4.hotmail.com 25
    Trying 65.54.244.232...
    Connected to mx4.hotmail.com.
    Escape character is '^]'.
    220 bay0-mc7-f2.bay0.hotmail.com Sending unsolicited commercial or
    bulk e-mail to Microsoft's computer network is prohibited. Other
    restrictions are found at http://privacy.msn.com/Anti-spam/.
    Violations will result in use of equipment located in California and
    other states. Sat, 2 Feb 2008 12:16:29 -0800

    EHLO testing.com
    250-bay0-mc7-f2.bay0.hotmail.com (3.5.0.22) Hello [76.77.66.100]
    250-SIZE 29696000
    250-PIPELINING
    250-8bitmime
    250-BINARYMIME
    250-CHUNKING
    250-AUTH LOGIN
    250-AUTH=LOGIN
    250 OK

    MAIL FROM: viff@viff.org
    250 viff@viff.org....Sender OK

    RCPT TO: my_email@hotmail.com
    250 my_email@hotmail.com

    DATA
    354 Start mail input; end with .
    From:
    To:
    Subject: SMTP test

    SMTP test body

    ..
    250 Queued
    mail for delivery

    QUIT
    221 bay0-mc7-f2.bay0.hotmail.com Service closing transmission channel
    Connection closed by foreign host.

    ************************************************

  2. Re: How HOTMAIL authenticate received messages?

    In article
    ,
    Andre wrote:

    > Does anyone know which mechanism does HOTMAIL use to authenticate
    > received emails?
    > Recently I received in my inbox a spam from viff@viff.org. And tried
    > to reproduce it by sending the following SMTP message to HOTMAIL, but
    > although HOTMAIL queued for delivery, it never showed up in my inbox.
    > Could HOTMAIL be comparing the IP address of the sender with the
    > address returned by a DNS query for the mail server of the sender's
    > domain (nslookup -querytype=mx viff@org) stated in the EHLO header to
    > authenticate these messages? In this case a spam sent from the
    > viff.org email server would go through. While another spam send from
    > another domain, but claiming to come from viff.org would be blocked?
    > If so, is there any documentation on the net about it?


    Checking the sending address against the MX would be a BAD idea, since
    many organizations (including almost all large ISPs) use different
    servers for incoming and outgoing mail.

    If viff.org had an SPF record they might check against that, but it
    doesn't.

    They might be checking your IP address against a list of residential,
    dynamically-assigned addresses. Many mail systems will not accept
    messages from these -- you're expected to send through your ISP's SMTP
    server, not directly from home. But if this is what Hotmail is doing,
    it wouldn't matter what sender address you used. Are you able to send a
    test message like this when you use a different sender address?

    >
    > Thanks in advance,
    >
    > Andre
    >
    > ************************************************
    > andre@kirchner:~$ telnet mx4.hotmail.com 25
    > Trying 65.54.244.232...
    > Connected to mx4.hotmail.com.
    > Escape character is '^]'.
    > 220 bay0-mc7-f2.bay0.hotmail.com Sending unsolicited commercial or
    > bulk e-mail to Microsoft's computer network is prohibited. Other
    > restrictions are found at http://privacy.msn.com/Anti-spam/.
    > Violations will result in use of equipment located in California and
    > other states. Sat, 2 Feb 2008 12:16:29 -0800
    >
    > EHLO testing.com
    > 250-bay0-mc7-f2.bay0.hotmail.com (3.5.0.22) Hello [76.77.66.100]
    > 250-SIZE 29696000
    > 250-PIPELINING
    > 250-8bitmime
    > 250-BINARYMIME
    > 250-CHUNKING
    > 250-AUTH LOGIN
    > 250-AUTH=LOGIN
    > 250 OK
    >
    > MAIL FROM: viff@viff.org
    > 250 viff@viff.org....Sender OK
    >
    > RCPT TO: my_email@hotmail.com
    > 250 my_email@hotmail.com
    >
    > DATA
    > 354 Start mail input; end with .
    > From:
    > To:
    > Subject: SMTP test
    >
    > SMTP test body
    >
    > .
    > 250 Queued
    > mail for delivery
    >
    > QUIT
    > 221 bay0-mc7-f2.bay0.hotmail.com Service closing transmission channel
    > Connection closed by foreign host.
    >
    > ************************************************


    --
    Barry Margolin, barmar@alum.mit.edu
    Arlington, MA
    *** PLEASE post questions in newsgroups, not directly to me ***
    *** PLEASE don't copy me on replies, I'll read them in the group ***

  3. Re: How HOTMAIL authenticate received messages?

    In article ,
    Barry Margolin wrote:

    >> Does anyone know which mechanism does HOTMAIL use to authenticate
    >> received emails?
    >> Recently I received in my inbox a spam from viff@viff.org. And tried
    >> to reproduce it by sending the following SMTP message to HOTMAIL, but
    >> although HOTMAIL queued for delivery, it never showed up in my inbox.
    >> Could HOTMAIL be comparing the IP address of the sender with the
    >> address returned by a DNS query for the mail server of the sender's
    >> domain (nslookup -querytype=mx viff@org) stated in the EHLO header to
    >> authenticate these messages? In this case a spam sent from the


    >Checking the sending address against the MX would be a BAD idea, since
    >many organizations (including almost all large ISPs) use different
    >servers for incoming and outgoing mail.


    agreed.
    The obvious wrongness of the idea that doesn't keep people from perpetually
    re-inventing that FUSSP (see http://www.google.com/search?q=fussp )
    and even installing it on their personal mailboxes.
    However, Hotmail is big enough that such wrongheadedness would not last
    there long enough to be noticed.


    >If viff.org had an SPF record they might check against that, but it
    >doesn't.


    It's been 3 or maybe 4 months since I last tested after one of the periodic
    announcements from the usual suspects that SPF, DKIM, and/or some other
    flavor of sender authentication was finally in use at all of the major
    ISPs. At that time Hotmail was ignoring conflicts with even obligatory
    SPF "-all" records, not to mention the wishy washy "~all".
    When I do those tests, I turn on "-all" SPF TXT RRs for rhyolite.com,
    send to Hotmail and some other free mail providers that supposedly care
    about SPF, and find no effects on deliverly. To ensure that my RRs
    were valid, I bounce mail off some test SMTP servers.
    (I've also tried SPF as a defense against the bursts of as many as a few
    1000 spam backscatter per day sent in my direction without seeing
    anything but new kinds of backscatter due to SPF itself.)


    >They might be checking your IP address against a list of residential,
    >dynamically-assigned addresses. Many mail systems will not accept
    >messages from these -- you're expected to send through your ISP's SMTP
    >server, not directly from home. But if this is what Hotmail is doing,
    >it wouldn't matter what sender address you used. Are you able to send a
    >test message like this when you use a different sender address?


    For various reasons, I'd bet against Hotmail doing that. If I'm wrong,
    it is likely that a check of the SMTP client (mail sending) IP address
    in the usual "dynamic" or similar DNS blacklists would turn up a problem.
    See http://openrbl.org/ or especially
    http://www.spamhaus.org/query/bl?ip=x.y.z.w
    where x.y.z.w is the client's IP address.


    When I poked at Hotmail last fall, I found that mail without some headers
    was accepted but silently discarded. I think Subject was one, which
    struck me me as strange, because most spam that I see has Subject headers.

    If you have a thick skin and effective filters against poseurs,
    fools, children with delusions, legions of Cliff Clavens, etc. even
    worse than those that infest comp.protocols.tcp-ip, it would probably
    be more effective to ask the question in news.admin.net-abuse.email.


    Disclaimer: dealing with spam is a hobby of mine. See
    http://www.google.com/search?q=dcc+checksum


    Vernon Schryver vjs@rhyolite.com

  4. Re: How HOTMAIL authenticate received messages?

    On Feb 4, 7:08 pm, Barry Margolin wrote:
    > In article
    > ,
    >
    > Andre wrote:
    > > Does anyone know which mechanism does HOTMAIL use to authenticate
    > > received emails?
    > > Recently I received in my inbox a spam from v...@viff.org. And tried
    > > to reproduce it by sending the following SMTP message to HOTMAIL, but
    > > although HOTMAIL queued for delivery, it never showed up in my inbox.
    > > Could HOTMAIL be comparing the IP address of the sender with the
    > > address returned by a DNS query for the mail server of the sender's
    > > domain (nslookup -querytype=mx viff@org) stated in the EHLO header to
    > > authenticate these messages? In this case a spam sent from the
    > > viff.org email server would go through. While another spam send from
    > > another domain, but claiming to come from viff.org would be blocked?
    > > If so, is there any documentation on the net about it?

    >
    > Checking the sending address against the MX would be a BAD idea, since
    > many organizations (including almost all large ISPs) use different
    > servers for incoming and outgoing mail.
    >
    > If viff.org had an SPF record they might check against that, but it
    > doesn't.
    >
    > They might be checking your IP address against a list of residential,
    > dynamically-assigned addresses. Many mail systems will not accept
    > messages from these -- you're expected to send through your ISP's SMTP
    > server, not directly from home. But if this is what Hotmail is doing,
    > it wouldn't matter what sender address you used. Are you able to send a
    > test message like this when you use a different sender address?
    >

    Yes, I was able to send this message to my work email address, and
    received it.
    >
    >
    >
    >
    > > Thanks in advance,

    >
    > > Andre

    >
    > > ************************************************
    > > andre@kirchner:~$ telnet mx4.hotmail.com 25
    > > Trying 65.54.244.232...
    > > Connected to mx4.hotmail.com.
    > > Escape character is '^]'.
    > > 220 bay0-mc7-f2.bay0.hotmail.com Sending unsolicited commercial or
    > > bulk e-mail to Microsoft's computer network is prohibited. Other
    > > restrictions are found athttp://privacy.msn.com/Anti-spam/.
    > > Violations will result in use of equipment located in California and
    > > other states. Sat, 2 Feb 2008 12:16:29 -0800

    >
    > > EHLO testing.com
    > > 250-bay0-mc7-f2.bay0.hotmail.com (3.5.0.22) Hello [76.77.66.100]
    > > 250-SIZE 29696000
    > > 250-PIPELINING
    > > 250-8bitmime
    > > 250-BINARYMIME
    > > 250-CHUNKING
    > > 250-AUTH LOGIN
    > > 250-AUTH=LOGIN
    > > 250 OK

    >
    > > MAIL FROM: v...@viff.org
    > > 250 v...@viff.org....Sender OK

    >
    > > RCPT TO: my_em...@hotmail.com
    > > 250 my_em...@hotmail.com

    >
    > > DATA
    > > 354 Start mail input; end with .
    > > From:
    > > To:
    > > Subject: SMTP test

    >
    > > SMTP test body

    >
    > > .
    > > 250 Queued
    > > mail for delivery

    >
    > > QUIT
    > > 221 bay0-mc7-f2.bay0.hotmail.com Service closing transmission channel
    > > Connection closed by foreign host.

    >
    > > ************************************************

    >
    > --
    > Barry Margolin, bar...@alum.mit.edu
    > Arlington, MA
    > *** PLEASE post questions in newsgroups, not directly to me ***
    > *** PLEASE don't copy me on replies, I'll read them in the group ***



  5. Re: How HOTMAIL authenticate received messages?

    On Feb 4, 7:08 pm, Barry Margolin wrote:

    > They might be checking your IP address against a list of residential,
    > dynamically-assigned addresses. Many mail systems will not accept
    > messages from these -- you're expected to send through your ISP's SMTP
    > server, not directly from home. But if this is what Hotmail is doing,
    > it wouldn't matter what sender address you used. Are you able to send a
    > test message like this when you use a different sender address?


    > > 250 Queued
    > > mail for delivery


    But they did accept it.

    This would be enough to make me not use hotmail (if I were dumb enough
    to use it in the first place). If they accept a message for delivery,
    they are accepting responsibility for delivering it. If they fail in
    that responsibility, they are unreliable.

    They are welcome to deliver it low priority or put it in a special
    folder if they want, but not delivering mail one has accepted is
    unacceptable. At least, in my opinion.

    DS

  6. Re: How HOTMAIL authenticate received messages?

    In article
    ,
    Andre wrote:

    > On Feb 4, 7:08 pm, Barry Margolin wrote:
    > > In article
    > > ,
    > >
    > > Andre wrote:
    > > > Does anyone know which mechanism does HOTMAIL use to authenticate
    > > > received emails?
    > > > Recently I received in my inbox a spam from v...@viff.org. And tried
    > > > to reproduce it by sending the following SMTP message to HOTMAIL, but
    > > > although HOTMAIL queued for delivery, it never showed up in my inbox.
    > > > Could HOTMAIL be comparing the IP address of the sender with the
    > > > address returned by a DNS query for the mail server of the sender's
    > > > domain (nslookup -querytype=mx viff@org) stated in the EHLO header to
    > > > authenticate these messages? In this case a spam sent from the
    > > > viff.org email server would go through. While another spam send from
    > > > another domain, but claiming to come from viff.org would be blocked?
    > > > If so, is there any documentation on the net about it?

    > >
    > > Checking the sending address against the MX would be a BAD idea, since
    > > many organizations (including almost all large ISPs) use different
    > > servers for incoming and outgoing mail.
    > >
    > > If viff.org had an SPF record they might check against that, but it
    > > doesn't.
    > >
    > > They might be checking your IP address against a list of residential,
    > > dynamically-assigned addresses. Many mail systems will not accept
    > > messages from these -- you're expected to send through your ISP's SMTP
    > > server, not directly from home. But if this is what Hotmail is doing,
    > > it wouldn't matter what sender address you used. Are you able to send a
    > > test message like this when you use a different sender address?
    > >

    > Yes, I was able to send this message to my work email address, and
    > received it.


    Do you mean FROM your work email address? The presumed problem is the
    sender address, not recipient. And how would you send to your work
    address via the hotmail MX server, anyway (unless your work address is a
    hotmail.com address)?

    --
    Barry Margolin, barmar@alum.mit.edu
    Arlington, MA
    *** PLEASE post questions in newsgroups, not directly to me ***
    *** PLEASE don't copy me on replies, I'll read them in the group ***

  7. Re: How HOTMAIL authenticate received messages?

    In article
    ,
    David Schwartz wrote:

    > On Feb 4, 7:08 pm, Barry Margolin wrote:
    >
    > > They might be checking your IP address against a list of residential,
    > > dynamically-assigned addresses. Many mail systems will not accept
    > > messages from these -- you're expected to send through your ISP's SMTP
    > > server, not directly from home. But if this is what Hotmail is doing,
    > > it wouldn't matter what sender address you used. Are you able to send a
    > > test message like this when you use a different sender address?

    >
    > > > 250 Queued
    > > > mail for delivery

    >
    > But they did accept it.
    >
    > This would be enough to make me not use hotmail (if I were dumb enough
    > to use it in the first place). If they accept a message for delivery,
    > they are accepting responsibility for delivering it. If they fail in
    > that responsibility, they are unreliable.
    >
    > They are welcome to deliver it low priority or put it in a special
    > folder if they want, but not delivering mail one has accepted is
    > unacceptable. At least, in my opinion.


    Lots of mail systems will accept and then silently discard or quarantine
    mail that triggers their antivirus filter.

    --
    Barry Margolin, barmar@alum.mit.edu
    Arlington, MA
    *** PLEASE post questions in newsgroups, not directly to me ***
    *** PLEASE don't copy me on replies, I'll read them in the group ***

  8. Re: How HOTMAIL authenticate received messages?

    In article ,
    Barry Margolin wrote:

    >Lots of mail systems will accept and then silently discard or quarantine
    >mail that triggers their antivirus filter.


    Yes, and systems that silently discard are broken by design, and generally
    not only in discarding mail. Their "design" often includes dumb
    implementation choices such as wrong programming languates that make
    their filtering slow and rationalize filtering after the end of the
    SMTP transaction. Those rationalizations are not sound, because a SMTP
    server cannot keep up with the stream of incoming mail by checking
    during transactions is unlikely to be helped merely be delaying the
    work and so suffering more bookkeeping other overhead.

    Reasonable SMTP servers do filtering during the transaction before
    saying "250 OK" to the DATA command. That keeps false positives from
    falling into blackholes. It also removes the excuse for backscatter.


    Vernon Schryver vjs@rhyolite.com

  9. Re: How HOTMAIL authenticate received messages?

    In article ,
    vjs@calcite.rhyolite.com (Vernon Schryver) wrote:

    > In article ,
    > Barry Margolin wrote:
    >
    > >Lots of mail systems will accept and then silently discard or quarantine
    > >mail that triggers their antivirus filter.

    >
    > Yes, and systems that silently discard are broken by design, and generally
    > not only in discarding mail. Their "design" often includes dumb
    > implementation choices such as wrong programming languates that make
    > their filtering slow and rationalize filtering after the end of the
    > SMTP transaction. Those rationalizations are not sound, because a SMTP
    > server cannot keep up with the stream of incoming mail by checking
    > during transactions is unlikely to be helped merely be delaying the
    > work and so suffering more bookkeeping other overhead.
    >
    > Reasonable SMTP servers do filtering during the transaction before
    > saying "250 OK" to the DATA command. That keeps false positives from
    > falling into blackholes. It also removes the excuse for backscatter.


    Often the sending system has a timeout waiting for the "250 OK". Doing
    a virus check on a multi-megabyte attachment can exceed this timeout.
    We used to get calls about this when I worked tech support for Symantec
    Gateway Security firewalls.

    And even if you don't time out, inline filtering means that the sending
    system is slowed down, because it has to wait for this processing to
    complete before it can use this thread to send another message.

    --
    Barry Margolin, barmar@alum.mit.edu
    Arlington, MA
    *** PLEASE post questions in newsgroups, not directly to me ***
    *** PLEASE don't copy me on replies, I'll read them in the group ***

  10. Re: How HOTMAIL authenticate received messages?

    On Feb 5, 9:36 pm, Barry Margolin wrote:

    > Lots of mail systems will accept and then silently discard or quarantine
    > mail that triggers their antivirus filter.


    This is, IMO, totally unacceptable behavior. What if the email is from
    your sister, sending her a copy of the new game she got from a friend
    and is about to install and mail to a whole bunch of other people?

    What if it's a copy of a program you sent them first, along with a
    text warning that it contains a virus? What if it's a warning not
    including a copy of the virus that your compromised computer is
    sending to thousands of people?

    It is perfectly acceptable to test inline. It is perfectly acceptable
    to specially mark the attachment. It is even acceptable to remove the
    attachment and put the email in a spam folder.

    However, accepting an email for delivery and then silently completely
    discarding it is, IMO, *never* acceptable. If you wish to completely
    refuse to deliver a message, you must do so without accepting it for
    delivery.

    DS

  11. Re: How HOTMAIL authenticate received messages?

    In article ,
    Barry Margolin wrote:

    >> Yes, and systems that silently discard are broken by design, and generally
    >> not only in discarding mail. Their "design" often includes dumb
    >> implementation choices such as wrong programming languates that make
    >> their filtering slow and rationalize filtering after the end of the
    >> SMTP transaction. Those rationalizations are not sound, because a SMTP
    >> server cannot keep up with the stream of incoming mail by checking
    >> during transactions is unlikely to be helped merely be delaying the
    >> work and so suffering more bookkeeping other overhead.
    >>
    >> Reasonable SMTP servers do filtering during the transaction before
    >> saying "250 OK" to the DATA command. That keeps false positives from
    >> falling into blackholes. It also removes the excuse for backscatter.

    >
    >Often the sending system has a timeout waiting for the "250 OK". Doing
    >a virus check on a multi-megabyte attachment can exceed this timeout.
    >We used to get calls about this when I worked tech support for Symantec
    >Gateway Security firewalls.


    That is true only for junk filters that are too slow for professional
    use, often because, as I wrote above, "dumb implementation choices such
    as wrong programming languates that make their filtering slow." Section
    4.5.3.2 of RFC 2821 recommends a timeout of 10 minutes for the final
    DATA chunk, but that is about 100 times too long. If your virus filter
    cannot figure out whether a mail message is a virus or worm in far less
    time than is required to do a DNS blacklist check, then it is evidence
    of exceptional incompetence.

    Of course, there are plenty of such proofs available.

    Besides, as I also tried to say above, postponing the checking can only
    increase the total time spent, at least for competent code. If your
    virus filter needs minutes or even just seconds to check each mail
    message, then whether it does the checking during the SMTP transaction
    or after, then your mail system cannot receive more than a smaller than
    trivially tiny number of mail messages per day.


    >And even if you don't time out, inline filtering means that the sending
    >system is slowed down, because it has to wait for this processing to
    >complete before it can use this thread to send another message.


    That is true only of junk SMTP clients that cannot keep up with non-trivial
    amounts of outgoing email. A mail system that send or receives 10 mail
    messages per second is nothing to brag about. You won't get any respect
    even for 100 mail messages/second today.

    And again, if your filtering needs more than a fraction of a second of
    CPU time on average, then either your mail SMPT server (mail receiving
    system) cannot keep up or it is a tiny slow toy.


    Vernon Schryver vjs@rhyolite.com

  12. Re: How HOTMAIL authenticate received messages?

    In article ,
    vjs@calcite.rhyolite.com (Vernon Schryver) wrote:

    > In article ,
    > Barry Margolin wrote:
    >
    > >> Yes, and systems that silently discard are broken by design, and generally
    > >> not only in discarding mail. Their "design" often includes dumb
    > >> implementation choices such as wrong programming languates that make
    > >> their filtering slow and rationalize filtering after the end of the
    > >> SMTP transaction. Those rationalizations are not sound, because a SMTP
    > >> server cannot keep up with the stream of incoming mail by checking
    > >> during transactions is unlikely to be helped merely be delaying the
    > >> work and so suffering more bookkeeping other overhead.
    > >>
    > >> Reasonable SMTP servers do filtering during the transaction before
    > >> saying "250 OK" to the DATA command. That keeps false positives from
    > >> falling into blackholes. It also removes the excuse for backscatter.

    > >
    > >Often the sending system has a timeout waiting for the "250 OK". Doing
    > >a virus check on a multi-megabyte attachment can exceed this timeout.
    > >We used to get calls about this when I worked tech support for Symantec
    > >Gateway Security firewalls.

    >
    > That is true only for junk filters that are too slow for professional
    > use, often because, as I wrote above, "dumb implementation choices such
    > as wrong programming languates that make their filtering slow." Section
    > 4.5.3.2 of RFC 2821 recommends a timeout of 10 minutes for the final
    > DATA chunk, but that is about 100 times too long. If your virus filter
    > cannot figure out whether a mail message is a virus or worm in far less
    > time than is required to do a DNS blacklist check, then it is evidence
    > of exceptional incompetence.


    Suppose the virus is nested in several levels of gzip'ing and archiving?
    The filter has to uncompress and extract all of these.

    No matter how fast your filter is, someone can always make the
    attachment bigger and more complex, to make it take longer to check it.

    >
    > Of course, there are plenty of such proofs available.
    >
    > Besides, as I also tried to say above, postponing the checking can only
    > increase the total time spent, at least for competent code. If your
    > virus filter needs minutes or even just seconds to check each mail
    > message, then whether it does the checking during the SMTP transaction
    > or after, then your mail system cannot receive more than a smaller than
    > trivially tiny number of mail messages per day.
    >
    >
    > >And even if you don't time out, inline filtering means that the sending
    > >system is slowed down, because it has to wait for this processing to
    > >complete before it can use this thread to send another message.

    >
    > That is true only of junk SMTP clients that cannot keep up with non-trivial
    > amounts of outgoing email. A mail system that send or receives 10 mail
    > messages per second is nothing to brag about. You won't get any respect
    > even for 100 mail messages/second today.
    >
    > And again, if your filtering needs more than a fraction of a second of
    > CPU time on average, then either your mail SMPT server (mail receiving
    > system) cannot keep up or it is a tiny slow toy.


    Even if everything you say is true, what can be done about it? ISPs
    have to use the products that are available, and many of them are junk.
    Suppose they want features X and Y, and the SMTP servers that have them
    have slow AV filters -- they have to make settle.

    In an ideal world we wouldn't have to deal with spam and viruses in the
    first place. But the world isn't ideal, and tradeoffs have to be made.
    Should ISPs sacrifice features just to get the best spam/virus
    filtering? If they do that, the terrorists win.

    --
    Barry Margolin, barmar@alum.mit.edu
    Arlington, MA
    *** PLEASE post questions in newsgroups, not directly to me ***
    *** PLEASE don't copy me on replies, I'll read them in the group ***

  13. Re: How HOTMAIL authenticate received messages?

    On Feb 6, 5:08 pm, Barry Margolin wrote:

    > Suppose the virus is nested in several levels of gzip'ing and archiving?
    > The filter has to uncompress and extract all of these.


    > No matter how fast your filter is, someone can always make the
    > attachment bigger and more complex, to make it take longer to check it.


    True, and you have a lot of choices. You could refuse large
    attachments. You could scan them inline. You could do any number of
    things -- the only thing you can't do is accept a message for delivery
    and then silently discard it.

    > Even if everything you say is true, what can be done about it? ISPs
    > have to use the products that are available, and many of them are junk.
    > Suppose they want features X and Y, and the SMTP servers that have them
    > have slow AV filters -- they have to make settle.


    That's fine, they just can't accept someone else's mail for delivery
    and then drop it on the floor.

    > In an ideal world we wouldn't have to deal with spam and viruses in the
    > first place. But the world isn't ideal, and tradeoffs have to be made.
    > Should ISPs sacrifice features just to get the best spam/virus
    > filtering? If they do that, the terrorists win.


    Accepting mail for delivery and then intentionally dropping it on the
    floor is unacceptable. I won't use any email service that does that.
    They are welcome to handle it any number of other ways, including
    removing the attachment and/or moving the mail to a spam folder.

    Creating backscatter is prohibited. Silently dropping mail on the
    floor is prohibited. There are reasonable and unreasonable solutions.

    DS

  14. Re: How HOTMAIL authenticate received messages?

    In article ,
    Barry Margolin wrote:

    >> >Often the sending system has a timeout waiting for the "250 OK". Doing
    >> >a virus check on a multi-megabyte attachment can exceed this timeout.
    >> >We used to get calls about this when I worked tech support for Symantec
    >> >Gateway Security firewalls.

    >>
    >> That is true only for junk filters that are too slow for professional
    >> use, often because, as I wrote above, "dumb implementation choices such


    >Suppose the virus is nested in several levels of gzip'ing and archiving?
    >The filter has to uncompress and extract all of these.
    >
    >No matter how fast your filter is, someone can always make the
    >attachment bigger and more complex, to make it take longer to check it.


    That is still wrong, because it conflicts with an overriding design
    requirement for any mail system. If your filter is not fast enough to
    handle almost all incoming mail messages within seconds, it is not fast
    enough for professional use.

    If your AV or spam filter or any part of your overall mail system needs
    more than 20 or 30 seconds per mail message, not to mention the 10
    minutes of RFC 2821 or the 3 hours of classic sendmail, then your overall
    mail system is good only for an individual with very low expectations.
    Doing the filtering after saying "250 OK" to the SMTP data command does
    not and cannot make your mail system go faster. If your mail system
    can't keep up in real time, then delaying the work can and will only
    make it slower.

    Professional mail systems handle millions to billions (with a 'B') mail
    messages per day, and there are only 86400 seconds in a day.

    Decompressing gzip and other archive formats is *fast*, provided you
    don't write the result to a disk or some other slow destination. If
    your filter is reasonable, it will decompress as fast as the chunks of
    message arrive. All that remains is the anti-virus or spam filtering,
    which should be fast enough.

    Consider the implications is the default sendmail milter timeout on how
    long virus scanning takes, provided you're not using junk AV code.

    Note that you cannot increase the size of the worst case mail message
    indefinitely without encountering message size limits. Consider why
    the implications of the ESMTP messages size extension exists.

    A reasonable filter dealing with arbitrarily complicated formats (I
    don't think there are any, but never mind) will limit the time it spends
    working on any single message and declare the message "BAD" or "QUARANTINE"
    if it exceeds that limit.

    Note also that if your anti-virus filter is too slow to handle a real
    world mail stream at mere Mbits/sec, then it is too slow to scan a PC's
    disk of GBYtes in less time than a user will tolerate. If your virus
    scanning can scan a 30 GByte disk in an hour, then it averages 66 Mbit/sec.



    >> And again, if your filtering needs more than a fraction of a second of
    >> CPU time on average, then either your mail SMPT server (mail receiving
    >> system) cannot keep up or it is a tiny slow toy.

    >
    >Even if everything you say is true, what can be done about it? ISPs
    >have to use the products that are available, and many of them are junk.
    >Suppose they want features X and Y, and the SMTP servers that have them
    >have slow AV filters -- they have to make settle.


    That is true only if you restrict your attention to the second and third
    tier SMTP servers and the junk filters flogged by some salescritters.


    >In an ideal world we wouldn't have to deal with spam and viruses in the
    >first place. But the world isn't ideal, and tradeoffs have to be made.
    >Should ISPs sacrifice features just to get the best spam/virus
    >filtering? If they do that, the terrorists win.


    That is a false choice. You don't need to choose between good virus
    filtering and slow mail systems. You *certainly* do not need to choose
    between slow and bad spam filtering. That given a task, you can spend
    more than any fixed number of CPU seconds doing the task is irrelevant.
    What incompetent ISPs choose to do wrong implies nothing interesting
    except to other incompetents and to the salesslime that prey on them.


    I'm not talking only from theory or about idle worlds. My spam filtering
    code is used at close to 200,000 sites (distinct IP addresses of DCC
    clients) ranging from individuals to modest ISPs (more than 10 but fewer
    than 100 million msgs/day). It saw more than 366 million mail messages
    in the 24 hours ending 03:00 UTC Feb 7.

    It is irrelevant that if ISPs cared to terminate their spamming
    customers, including the lusers running botnets on their Windows
    PCs, there would no spam problem.

    It is also irrelevant that there would be no spam or virus/worm problems
    (and so no virus scanning industry) if users would demand reasonable
    software, including systems that don't require running as "administrator"
    all of the time.


    Vernon Schryver vjs@rhyolite.com

  15. Re: How HOTMAIL authenticate received messages?

    In article ,
    vjs@calcite.rhyolite.com (Vernon Schryver) wrote:

    > >In an ideal world we wouldn't have to deal with spam and viruses in the
    > >first place. But the world isn't ideal, and tradeoffs have to be made.
    > >Should ISPs sacrifice features just to get the best spam/virus
    > >filtering? If they do that, the terrorists win.

    >
    > That is a false choice. You don't need to choose between good virus
    > filtering and slow mail systems.


    That's not the tradeoff I was talking about. I said the tradeoff is
    between mail system features and good+fast filtering.

    --
    Barry Margolin, barmar@alum.mit.edu
    Arlington, MA
    *** PLEASE post questions in newsgroups, not directly to me ***
    *** PLEASE don't copy me on replies, I'll read them in the group ***

  16. Re: How HOTMAIL authenticate received messages?

    In article ,
    Barry Margolin wrote:

    >> >In an ideal world we wouldn't have to deal with spam and viruses in the
    >> >first place. But the world isn't ideal, and tradeoffs have to be made.
    >> >Should ISPs sacrifice features just to get the best spam/virus
    >> >filtering? If they do that, the terrorists win.

    >>
    >> That is a false choice. You don't need to choose between good virus
    >> filtering and slow mail systems.

    >
    >That's not the tradeoff I was talking about. I said the tradeoff is
    >between mail system features and good+fast filtering.


    That is also or still a false choice. Real world fancy mail system
    features are in the MUA or perhaps MSA, not the MTA. The MTA or SMTP
    server need only run the SMTP state machine, accept incoming mail at a
    good speed, and let or help the filtering see the messages to say yea
    or nay. The SMTP state machine is for handling the DATA command is
    *trivial*.

    The only likely fancy feature of an MTA or SMTP server that might cause
    speed problems is dealing with TLS, but the same mechanisms that let
    HTTP servers handle zillions of HTTPS hits per day are applicable to
    SMTP. I don't know of any SMTP servers that use those mechanisms, but
    that may be because so little email uses STARTTLS.


    Arguing that you must do spam or virus filtering after the end of the
    SMTP transaction because the system you've seen do it that way and are
    still not quite able to keep up is like the position of some programmers
    in the 1980s who claimed that any non-trivial network application must
    use raw Ethernet packets. They felt TCP/IP is inevitably far too slow,
    because all of the network applications they'd seen using other protocols
    were incredibly slow. Then they saw some off-host TCP implementations
    (e.g. Excelan or the IBM box) that validated their worst speed predictions.
    They probably still don't believe Van Jacobson's value for the number
    of CPU cycles per TCP segment.

    Its like the position of the people who are convinced that UDP is the
    solution to fast file transfers because they just know based on their
    wide experience with TCP and UDP that TCP is too slow. Never mind that
    their experience is so wide that they don't know where to find TFTP and
    FTP client/server pairs to do benchmarking.


    Vernon Schryver vjs@rhyolite.com

  17. Re: How HOTMAIL authenticate received messages?

    In article ,
    vjs@calcite.rhyolite.com (Vernon Schryver) wrote:

    > In article ,
    > Barry Margolin wrote:
    >
    > >> >In an ideal world we wouldn't have to deal with spam and viruses in the
    > >> >first place. But the world isn't ideal, and tradeoffs have to be made.
    > >> >Should ISPs sacrifice features just to get the best spam/virus
    > >> >filtering? If they do that, the terrorists win.
    > >>
    > >> That is a false choice. You don't need to choose between good virus
    > >> filtering and slow mail systems.

    > >
    > >That's not the tradeoff I was talking about. I said the tradeoff is
    > >between mail system features and good+fast filtering.

    >
    > That is also or still a false choice.


    Is there really any point to this argument? It hardly matters what the
    choices SHOULD be. As customers, we don't get to choose the software
    used on the mail servers.

    I was trying to describe how things actually work, and the
    justifications for them. Maybe those justifications aren't valid, it's
    still the reasoning taking place at the service providers.

    If business decisions were always based on the best technical merit,
    would MS Windows be on 90% of office desktops?

    --
    Barry Margolin, barmar@alum.mit.edu
    Arlington, MA
    *** PLEASE post questions in newsgroups, not directly to me ***
    *** PLEASE don't copy me on replies, I'll read them in the group ***

  18. Re: How HOTMAIL authenticate received messages?

    In article ,
    Barry Margolin wrote:

    >> >> That is a false choice. You don't need to choose between good virus
    >> >> filtering and slow mail systems.
    >> >
    >> >That's not the tradeoff I was talking about. I said the tradeoff is
    >> >between mail system features and good+fast filtering.

    >>
    >> That is also or still a false choice.

    >
    >Is there really any point to this argument? It hardly matters what the
    >choices SHOULD be. As customers, we don't get to choose the software
    >used on the mail servers.
    >
    >I was trying to describe how things actually work, and the
    >justifications for them. Maybe those justifications aren't valid, it's
    >still the reasoning taking place at the service providers.
    >
    >If business decisions were always based on the best technical merit,
    >would MS Windows be on 90% of office desktops?


    One of the main (but certainly not the only) reasons Microsoft software
    has so much marketshare and has the characteristics that it is the
    is the endless repetitions of false statements. If people who know
    or should know bettter had not spent 25 years agreeing that:

    - it is normal to need to reboot a computer every few hours
    - virus are normal and inevitiable, and never mind the implications
    of lack of any notion of file ownership or permissions in the
    almost UNIX-like file system in DOS
    - "user friendliness" is more important than any notion of security,
    and so
    + running as an ordinary user instead of administrator is
    too "unfriendly"
    + it is good to execute anything from anywhere that looks
    like code in Internet Explorer, and never mind MIME types,
    which not at all incidentally is much of what justifies
    deep, slow scanning by SMTP anti-virus filters
    + the lack of a sandbox in ActiveX is a feature instead
    of a stupid mistake, which also justifies AV deep scanning

    then either Microsoft would have been doing a better job (as it finally
    seems to havee begun) or would not have been so successful.

    Similarly, if people who know better would not claim that junk SMTP
    filters are the right, best and only way to do things, they wouldn't
    be so common. A big irony is that unlike the desktop market, not one
    of the really bad SMTP server filters has a dominent position in the
    market. Much of the worst of SMTP filters are locally created by people
    telling each other that they have no alternative and being willfully
    ignorant of better solutions.


    Vernon Schryver vjs@rhyolite.com

+ Reply to Thread