In article
,
Andre wrote:

> Does anyone know which mechanism does HOTMAIL use to authenticate
> received emails?
> Recently I received in my inbox a spam from viff@viff.org. And tried
> to reproduce it by sending the following SMTP message to HOTMAIL, but
> although HOTMAIL queued for delivery, it never showed up in my inbox.
> Could HOTMAIL be comparing the IP address of the sender with the
> address returned by a DNS query for the mail server of the sender's
> domain (nslookup -querytype=mx viff@org) stated in the EHLO header to
> authenticate these messages? In this case a spam sent from the
> viff.org email server would go through. While another spam send from
> another domain, but claiming to come from viff.org would be blocked?
> If so, is there any documentation on the net about it?

Checking the sending address against the MX would be a BAD idea, since
many organizations (including almost all large ISPs) use different
servers for incoming and outgoing mail.

If viff.org had an SPF record they might check against that, but it
doesn't.

They might be checking your IP address against a list of residential,
dynamically-assigned addresses. Many mail systems will not accept
messages from these -- you're expected to send through your ISP's SMTP
server, not directly from home. But if this is what Hotmail is doing,
it wouldn't matter what sender address you used. Are you able to send a
test message like this when you use a different sender address?

>
> Thanks in advance,
>
> Andre
>
> ************************************************
> andre@kirchner:~$ telnet mx4.hotmail.com 25
> Trying 65.54.244.232...
> Connected to mx4.hotmail.com.
> Escape character is '^]'.
> 220 bay0-mc7-f2.bay0.hotmail.com Sending unsolicited commercial or
> bulk e-mail to Microsoft's computer network is prohibited. Other
> restrictions are found at http://privacy.msn.com/Anti-spam/.
> Violations will result in use of equipment located in California and
> other states. Sat, 2 Feb 2008 12:16:29 -0800
>
> EHLO testing.com
> 250-bay0-mc7-f2.bay0.hotmail.com (3.5.0.22) Hello [76.77.66.100]
> 250-SIZE 29696000
> 250-PIPELINING
> 250-8bitmime
> 250-BINARYMIME
> 250-CHUNKING
> 250-AUTH LOGIN
> 250-AUTH=LOGIN
> 250 OK
>
> MAIL FROM: viff@viff.org
> 250 viff@viff.org....Sender OK
>
> RCPT TO: my_email@hotmail.com
> 250 my_email@hotmail.com
>
> DATA
> 354 Start mail input; end with .
> From:
> To:
> Subject: SMTP test
>
> SMTP test body
>
> .
> 250 Queued
> mail for delivery
>
> QUIT
> 221 bay0-mc7-f2.bay0.hotmail.com Service closing transmission channel
> Connection closed by foreign host.
>
> ************************************************

--
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***
*** PLEASE don't copy me on replies, I'll read them in the group ***

In article ,
Barry Margolin wrote:

>> Does anyone know which mechanism does HOTMAIL use to authenticate
>> received emails?
>> Recently I received in my inbox a spam from viff@viff.org. And tried
>> to reproduce it by sending the following SMTP message to HOTMAIL, but
>> although HOTMAIL queued for delivery, it never showed up in my inbox.
>> Could HOTMAIL be comparing the IP address of the sender with the
>> address returned by a DNS query for the mail server of the sender's
>> domain (nslookup -querytype=mx viff@org) stated in the EHLO header to
>> authenticate these messages? In this case a spam sent from the

>Checking the sending address against the MX would be a BAD idea, since
>many organizations (including almost all large ISPs) use different
>servers for incoming and outgoing mail.

agreed.
The obvious wrongness of the idea that doesn't keep people from perpetually
re-inventing that FUSSP (see http://www.google.com/search?q=fussp )
and even installing it on their personal mailboxes.
However, Hotmail is big enough that such wrongheadedness would not last
there long enough to be noticed.


>If viff.org had an SPF record they might check against that, but it
>doesn't.

It's been 3 or maybe 4 months since I last tested after one of the periodic
announcements from the usual suspects that SPF, DKIM, and/or some other
flavor of sender authentication was finally in use at all of the major
ISPs. At that time Hotmail was ignoring conflicts with even obligatory
SPF "-all" records, not to mention the wishy washy "~all".
When I do those tests, I turn on "-all" SPF TXT RRs for rhyolite.com,
send to Hotmail and some other free mail providers that supposedly care
about SPF, and find no effects on deliverly. To ensure that my RRs
were valid, I bounce mail off some test SMTP servers.
(I've also tried SPF as a defense against the bursts of as many as a few
1000 spam backscatter per day sent in my direction without seeing
anything but new kinds of backscatter due to SPF itself.)


>They might be checking your IP address against a list of residential,
>dynamically-assigned addresses. Many mail systems will not accept
>messages from these -- you're expected to send through your ISP's SMTP
>server, not directly from home. But if this is what Hotmail is doing,
>it wouldn't matter what sender address you used. Are you able to send a
>test message like this when you use a different sender address?

For various reasons, I'd bet against Hotmail doing that. If I'm wrong,
it is likely that a check of the SMTP client (mail sending) IP address
in the usual "dynamic" or similar DNS blacklists would turn up a problem.
See http://openrbl.org/ or especially
http://www.spamhaus.org/query/bl?ip=x.y.z.w
where x.y.z.w is the client's IP address.


When I poked at Hotmail last fall, I found that mail without some headers
was accepted but silently discarded. I think Subject was one, which
struck me me as strange, because most spam that I see has Subject headers.

If you have a thick skin and effective filters against poseurs,
fools, children with delusions, legions of Cliff Clavens, etc. even
worse than those that infest comp.protocols.tcp-ip, it would probably
be more effective to ask the question in news.admin.net-abuse.email.


Disclaimer: dealing with spam is a hobby of mine. See
http://www.google.com/search?q=dcc+checksum


Vernon Schryver vjs@rhyolite.com

On Feb 4, 7:08 pm, Barry Margolin wrote:
> In article
> ,
>
> Andre wrote:
> > Does anyone know which mechanism does HOTMAIL use to authenticate
> > received emails?
> > Recently I received in my inbox a spam from v...@viff.org. And tried
> > to reproduce it by sending the following SMTP message to HOTMAIL, but
> > although HOTMAIL queued for delivery, it never showed up in my inbox.
> > Could HOTMAIL be comparing the IP address of the sender with the
> > address returned by a DNS query for the mail server of the sender's
> > domain (nslookup -querytype=mx viff@org) stated in the EHLO header to
> > authenticate these messages? In this case a spam sent from the
> > viff.org email server would go through. While another spam send from
> > another domain, but claiming to come from viff.org would be blocked?
> > If so, is there any documentation on the net about it?
>
> Checking the sending address against the MX would be a BAD idea, since
> many organizations (including almost all large ISPs) use different
> servers for incoming and outgoing mail.
>
> If viff.org had an SPF record they might check against that, but it
> doesn't.
>
> They might be checking your IP address against a list of residential,
> dynamically-assigned addresses. Many mail systems will not accept
> messages from these -- you're expected to send through your ISP's SMTP
> server, not directly from home. But if this is what Hotmail is doing,
> it wouldn't matter what sender address you used. Are you able to send a
> test message like this when you use a different sender address?
>
Yes, I was able to send this message to my work email address, and
received it.
>
>
>
>
> > Thanks in advance,
>
> > Andre
>
> > ************************************************
> > andre@kirchner:~$ telnet mx4.hotmail.com 25
> > Trying 65.54.244.232...
> > Connected to mx4.hotmail.com.
> > Escape character is '^]'.
> > 220 bay0-mc7-f2.bay0.hotmail.com Sending unsolicited commercial or
> > bulk e-mail to Microsoft's computer network is prohibited. Other
> > restrictions are found athttp://privacy.msn.com/Anti-spam/.
> > Violations will result in use of equipment located in California and
> > other states. Sat, 2 Feb 2008 12:16:29 -0800
>
> > EHLO testing.com
> > 250-bay0-mc7-f2.bay0.hotmail.com (3.5.0.22) Hello [76.77.66.100]
> > 250-SIZE 29696000
> > 250-PIPELINING
> > 250-8bitmime
> > 250-BINARYMIME
> > 250-CHUNKING
> > 250-AUTH LOGIN
> > 250-AUTH=LOGIN
> > 250 OK
>
> > MAIL FROM: v...@viff.org
> > 250 v...@viff.org....Sender OK
>
> > RCPT TO: my_em...@hotmail.com
> > 250 my_em...@hotmail.com
>
> > DATA
> > 354 Start mail input; end with .
> > From:
> > To:
> > Subject: SMTP test
>
> > SMTP test body
>
> > .
> > 250 Queued
> > mail for delivery
>
> > QUIT
> > 221 bay0-mc7-f2.bay0.hotmail.com Service closing transmission channel
> > Connection closed by foreign host.
>
> > ************************************************
>
> --
> Barry Margolin, bar...@alum.mit.edu
> Arlington, MA
> *** PLEASE post questions in newsgroups, not directly to me ***
> *** PLEASE don't copy me on replies, I'll read them in the group ***

On Feb 4, 7:08 pm, Barry Margolin wrote:

> They might be checking your IP address against a list of residential,
> dynamically-assigned addresses. Many mail systems will not accept
> messages from these -- you're expected to send through your ISP's SMTP
> server, not directly from home. But if this is what Hotmail is doing,
> it wouldn't matter what sender address you used. Are you able to send a
> test message like this when you use a different sender address?

> > 250 Queued
> > mail for delivery

But they did accept it.

This would be enough to make me not use hotmail (if I were dumb enough
to use it in the first place). If they accept a message for delivery,
they are accepting responsibility for delivering it. If they fail in
that responsibility, they are unreliable.

They are welcome to deliver it low priority or put it in a special
folder if they want, but not delivering mail one has accepted is
unacceptable. At least, in my opinion.

DS

In article
,
Andre wrote:

> On Feb 4, 7:08 pm, Barry Margolin wrote:
> > In article
> > ,
> >
> > Andre wrote:
> > > Does anyone know which mechanism does HOTMAIL use to authenticate
> > > received emails?
> > > Recently I received in my inbox a spam from v...@viff.org. And tried
> > > to reproduce it by sending the following SMTP message to HOTMAIL, but
> > > although HOTMAIL queued for delivery, it never showed up in my inbox.
> > > Could HOTMAIL be comparing the IP address of the sender with the
> > > address returned by a DNS query for the mail server of the sender's
> > > domain (nslookup -querytype=mx viff@org) stated in the EHLO header to
> > > authenticate these messages? In this case a spam sent from the
> > > viff.org email server would go through. While another spam send from
> > > another domain, but claiming to come from viff.org would be blocked?
> > > If so, is there any documentation on the net about it?
> >
> > Checking the sending address against the MX would be a BAD idea, since
> > many organizations (including almost all large ISPs) use different
> > servers for incoming and outgoing mail.
> >
> > If viff.org had an SPF record they might check against that, but it
> > doesn't.
> >
> > They might be checking your IP address against a list of residential,
> > dynamically-assigned addresses. Many mail systems will not accept
> > messages from these -- you're expected to send through your ISP's SMTP
> > server, not directly from home. But if this is what Hotmail is doing,
> > it wouldn't matter what sender address you used. Are you able to send a
> > test message like this when you use a different sender address?
> >
> Yes, I was able to send this message to my work email address, and
> received it.

Do you mean FROM your work email address? The presumed problem is the
sender address, not recipient. And how would you send to your work
address via the hotmail MX server, anyway (unless your work address is a
hotmail.com address)?

--
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***
*** PLEASE don't copy me on replies, I'll read them in the group ***

In article
,
David Schwartz wrote:

> On Feb 4, 7:08 pm, Barry Margolin wrote:
>
> > They might be checking your IP address against a list of residential,
> > dynamically-assigned addresses. Many mail systems will not accept
> > messages from these -- you're expected to send through your ISP's SMTP
> > server, not directly from home. But if this is what Hotmail is doing,
> > it wouldn't matter what sender address you used. Are you able to send a
> > test message like this when you use a different sender address?
>
> > > 250 Queued
> > > mail for delivery
>
> But they did accept it.
>
> This would be enough to make me not use hotmail (if I were dumb enough
> to use it in the first place). If they accept a message for delivery,
> they are accepting responsibility for delivering it. If they fail in
> that responsibility, they are unreliable.
>
> They are welcome to deliver it low priority or put it in a special
> folder if they want, but not delivering mail one has accepted is
> unacceptable. At least, in my opinion.

Lots of mail systems will accept and then silently discard or quarantine
mail that triggers their antivirus filter.

--
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***
*** PLEASE don't copy me on replies, I'll read them in the group ***

In article ,
Barry Margolin wrote:

>Lots of mail systems will accept and then silently discard or quarantine
>mail that triggers their antivirus filter.

Yes, and systems that silently discard are broken by design, and generally
not only in discarding mail. Their "design" often includes dumb
implementation choices such as wrong programming languates that make
their filtering slow and rationalize filtering after the end of the
SMTP transaction. Those rationalizations are not sound, because a SMTP
server cannot keep up with the stream of incoming mail by checking
during transactions is unlikely to be helped merely be delaying the
work and so suffering more bookkeeping other overhead.

Reasonable SMTP servers do filtering during the transaction before
saying "250 OK" to the DATA command. That keeps false positives from
falling into blackholes. It also removes the excuse for backscatter.


Vernon Schryver vjs@rhyolite.com

In article ,
vjs@calcite.rhyolite.com (Vernon Schryver) wrote:

> In article ,
> Barry Margolin wrote:
>
> >Lots of mail systems will accept and then silently discard or quarantine
> >mail that triggers their antivirus filter.
>
> Yes, and systems that silently discard are broken by design, and generally
> not only in discarding mail. Their "design" often includes dumb
> implementation choices such as wrong programming languates that make
> their filtering slow and rationalize filtering after the end of the
> SMTP transaction. Those rationalizations are not sound, because a SMTP
> server cannot keep up with the stream of incoming mail by checking
> during transactions is unlikely to be helped merely be delaying the
> work and so suffering more bookkeeping other overhead.
>
> Reasonable SMTP servers do filtering during the transaction before
> saying "250 OK" to the DATA command. That keeps false positives from
> falling into blackholes. It also removes the excuse for backscatter.

Often the sending system has a timeout waiting for the "250 OK". Doing
a virus check on a multi-megabyte attachment can exceed this timeout.
We used to get calls about this when I worked tech support for Symantec
Gateway Security firewalls.

And even if you don't time out, inline filtering means that the sending
system is slowed down, because it has to wait for this processing to
complete before it can use this thread to send another message.

--
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***
*** PLEASE don't copy me on replies, I'll read them in the group ***

On Feb 5, 9:36 pm, Barry Margolin wrote:

> Lots of mail systems will accept and then silently discard or quarantine
> mail that triggers their antivirus filter.

This is, IMO, totally unacceptable behavior. What if the email is from
your sister, sending her a copy of the new game she got from a friend
and is about to install and mail to a whole bunch of other people?

What if it's a copy of a program you sent them first, along with a
text warning that it contains a virus? What if it's a warning not
including a copy of the virus that your compromised computer is
sending to thousands of people?

It is perfectly acceptable to test inline. It is perfectly acceptable
to specially mark the attachment. It is even acceptable to remove the
attachment and put the email in a spam folder.

However, accepting an email for delivery and then silently completely
discarding it is, IMO, *never* acceptable. If you wish to completely
refuse to deliver a message, you must do so without accepting it for
delivery.

DS

In article ,
Barry Margolin wrote:

>> Yes, and systems that silently discard are broken by design, and generally
>> not only in discarding mail. Their "design" often includes dumb
>> implementation choices such as wrong programming languates that make
>> their filtering slow and rationalize filtering after the end of the
>> SMTP transaction. Those rationalizations are not sound, because a SMTP
>> server cannot keep up with the stream of incoming mail by checking
>> during transactions is unlikely to be helped merely be delaying the
>> work and so suffering more bookkeeping other overhead.
>>
>> Reasonable SMTP servers do filtering during the transaction before
>> saying "250 OK" to the DATA command. That keeps false positives from
>> falling into blackholes. It also removes the excuse for backscatter.
>
>Often the sending system has a timeout waiting for the "250 OK". Doing
>a virus check on a multi-megabyte attachment can exceed this timeout.
>We used to get calls about this when I worked tech support for Symantec
>Gateway Security firewalls.

That is true only for junk filters that are too slow for professional
use, often because, as I wrote above, "dumb implementation choices such
as wrong programming languates that make their filtering slow." Section
4.5.3.2 of RFC 2821 recommends a timeout of 10 minutes for the final
DATA chunk, but that is about 100 times too long. If your virus filter
cannot figure out whether a mail message is a virus or worm in far less
time than is required to do a DNS blacklist check, then it is evidence
of exceptional incompetence.

Of course, there are plenty of such proofs available.

Besides, as I also tried to say above, postponing the checking can only
increase the total time spent, at least for competent code. If your
virus filter needs minutes or even just seconds to check each mail
message, then whether it does the checking during the SMTP transaction
or after, then your mail system cannot receive more than a smaller than
trivially tiny number of mail messages per day.


>And even if you don't time out, inline filtering means that the sending
>system is slowed down, because it has to wait for this processing to
>complete before it can use this thread to send another message.

That is true only of junk SMTP clients that cannot keep up with non-trivial
amounts of outgoing email. A mail system that send or receives 10 mail
messages per second is nothing to brag about. You won't get any respect
even for 100 mail messages/second today.

And again, if your filtering needs more than a fraction of a second of
CPU time on average, then either your mail SMPT server (mail receiving
system) cannot keep up or it is a tiny slow toy.


Vernon Schryver vjs@rhyolite.com

In article ,
vjs@calcite.rhyolite.com (Vernon Schryver) wrote:

> In article ,
> Barry Margolin wrote:
>
> >> Yes, and systems that silently discard are broken by design, and generally
> >> not only in discarding mail. Their "design" often includes dumb
> >> implementation choices such as wrong programming languates that make
> >> their filtering slow and rationalize filtering after the end of the
> >> SMTP transaction. Those rationalizations are not sound, because a SMTP
> >> server cannot keep up with the stream of incoming mail by checking
> >> during transactions is unlikely to be helped merely be delaying the
> >> work and so suffering more bookkeeping other overhead.
> >>
> >> Reasonable SMTP servers do filtering during the transaction before
> >> saying "250 OK" to the DATA command. That keeps false positives from
> >> falling into blackholes. It also removes the excuse for backscatter.
> >
> >Often the sending system has a timeout waiting for the "250 OK". Doing
> >a virus check on a multi-megabyte attachment can exceed this timeout.
> >We used to get calls about this when I worked tech support for Symantec
> >Gateway Security firewalls.
>
> That is true only for junk filters that are too slow for professional
> use, often because, as I wrote above, "dumb implementation choices such
> as wrong programming languates that make their filtering slow." Section
> 4.5.3.2 of RFC 2821 recommends a timeout of 10 minutes for the final
> DATA chunk, but that is about 100 times too long. If your virus filter
> cannot figure out whether a mail message is a virus or worm in far less
> time than is required to do a DNS blacklist check, then it is evidence
> of exceptional incompetence.

Suppose the virus is nested in several levels of gzip'ing and archiving?
The filter has to uncompress and extract all of these.

No matter how fast your filter is, someone can always make the
attachment bigger and more complex, to make it take longer to check it.

>
> Of course, there are plenty of such proofs available.
>
> Besides, as I also tried to say above, postponing the checking can only
> increase the total time spent, at least for competent code. If your
> virus filter needs minutes or even just seconds to check each mail
> message, then whether it does the checking during the SMTP transaction
> or after, then your mail system cannot receive more than a smaller than
> trivially tiny number of mail messages per day.
>
>
> >And even if you don't time out, inline filtering means that the sending
> >system is slowed down, because it has to wait for this processing to
> >complete before it can use this thread to send another message.
>
> That is true only of junk SMTP clients that cannot keep up with non-trivial
> amounts of outgoing email. A mail system that send or receives 10 mail
> messages per second is nothing to brag about. You won't get any respect
> even for 100 mail messages/second today.
>
> And again, if your filtering needs more than a fraction of a second of
> CPU time on average, then either your mail SMPT server (mail receiving
> system) cannot keep up or it is a tiny slow toy.

Even if everything you say is true, what can be done about it? ISPs
have to use the products that are available, and many of them are junk.
Suppose they want features X and Y, and the SMTP servers that have them
have slow AV filters -- they have to make settle.

In an ideal world we wouldn't have to deal with spam and viruses in the
first place. But the world isn't ideal, and tradeoffs have to be made.
Should ISPs sacrifice features just to get the best spam/virus
filtering? If they do that, the terrorists win.

--
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***
*** PLEASE don't copy me on replies, I'll read them in the group ***

On Feb 6, 5:08 pm, Barry Margolin wrote:

> Suppose the virus is nested in several levels of gzip'ing and archiving?
> The filter has to uncompress and extract all of these.

> No matter how fast your filter is, someone can always make the
> attachment bigger and more complex, to make it take longer to check it.

True, and you have a lot of choices. You could refuse large
attachments. You could scan them inline. You could do any number of
things -- the only thing you can't do is accept a message for delivery
and then silently discard it.

> Even if everything you say is true, what can be done about it? ISPs
> have to use the products that are available, and many of them are junk.
> Suppose they want features X and Y, and the SMTP servers that have them
> have slow AV filters -- they have to make settle.

That's fine, they just can't accept someone else's mail for delivery
and then drop it on the floor.

> In an ideal world we wouldn't have to deal with spam and viruses in the
> first place. But the world isn't ideal, and tradeoffs have to be made.
> Should ISPs sacrifice features just to get the best spam/virus
> filtering? If they do that, the terrorists win.

Accepting mail for delivery and then intentionally dropping it on the
floor is unacceptable. I won't use any email service that does that.
They are welcome to handle it any number of other ways, including
removing the attachment and/or moving the mail to a spam folder.

Creating backscatter is prohibited. Silently dropping mail on the
floor is prohibited. There are reasonable and unreasonable solutions.

DS

In article ,
Barry Margolin wrote:

>> >Often the sending system has a timeout waiting for the "250 OK". Doing
>> >a virus check on a multi-megabyte attachment can exceed this timeout.
>> >We used to get calls about this when I worked tech support for Symantec
>> >Gateway Security firewalls.
>>
>> That is true only for junk filters that are too slow for professional
>> use, often because, as I wrote above, "dumb implementation choices such

>Suppose the virus is nested in several levels of gzip'ing and archiving?
>The filter has to uncompress and extract all of these.
>
>No matter how fast your filter is, someone can always make the
>attachment bigger and more complex, to make it take longer to check it.

That is still wrong, because it conflicts with an overriding design
requirement for any mail system. If your filter is not fast enough to
handle almost all incoming mail messages within seconds, it is not fast
enough for professional use.

If your AV or spam filter or any part of your overall mail system needs
more than 20 or 30 seconds per mail message, not to mention the 10
minutes of RFC 2821 or the 3 hours of classic sendmail, then your overall
mail system is good only for an individual with very low expectations.
Doing the filtering after saying "250 OK" to the SMTP data command does
not and cannot make your mail system go faster. If your mail system
can't keep up in real time, then delaying the work can and will only
make it slower.

Professional mail systems handle millions to billions (with a 'B') mail
messages per day, and there are only 86400 seconds in a day.

Decompressing gzip and other archive formats is *fast*, provided you
don't write the result to a disk or some other slow destination. If
your filter is reasonable, it will decompress as fast as the chunks of
message arrive. All that remains is the anti-virus or spam filtering,
which should be fast enough.

Consider the implications is the default sendmail milter timeout on how
long virus scanning takes, provided you're not using junk AV code.

Note that you cannot increase the size of the worst case mail message
indefinitely without encountering message size limits. Consider why
the implications of the ESMTP messages size extension exists.

A reasonable filter dealing with arbitrarily complicated formats (I
don't think there are any, but never mind) will limit the time it spends
working on any single message and declare the message "BAD" or "QUARANTINE"
if it exceeds that limit.

Note also that if your anti-virus filter is too slow to handle a real
world mail stream at mere Mbits/sec, then it is too slow to scan a PC's
disk of GBYtes in less time than a user will tolerate. If your virus
scanning can scan a 30 GByte disk in an hour, then it averages 66 Mbit/sec.



>> And again, if your filtering needs more than a fraction of a second of
>> CPU time on average, then either your mail SMPT server (mail receiving
>> system) cannot keep up or it is a tiny slow toy.
>
>Even if everything you say is true, what can be done about it? ISPs
>have to use the products that are available, and many of them are junk.
>Suppose they want features X and Y, and the SMTP servers that have them
>have slow AV filters -- they have to make settle.

That is true only if you restrict your attention to the second and third
tier SMTP servers and the junk filters flogged by some salescritters.


>In an ideal world we wouldn't have to deal with spam and viruses in the
>first place. But the world isn't ideal, and tradeoffs have to be made.
>Should ISPs sacrifice features just to get the best spam/virus
>filtering? If they do that, the terrorists win.

That is a false choice. You don't need to choose between good virus
filtering and slow mail systems. You *certainly* do not need to choose
between slow and bad spam filtering. That given a task, you can spend
more than any fixed number of CPU seconds doing the task is irrelevant.
What incompetent ISPs choose to do wrong implies nothing interesting
except to other incompetents and to the salesslime that prey on them.


I'm not talking only from theory or about idle worlds. My spam filtering
code is used at close to 200,000 sites (distinct IP addresses of DCC
clients) ranging from individuals to modest ISPs (more than 10 but fewer
than 100 million msgs/day). It saw more than 366 million mail messages
in the 24 hours ending 03:00 UTC Feb 7.

It is irrelevant that if ISPs cared to terminate their spamming
customers, including the lusers running botnets on their Windows
PCs, there would no spam problem.

It is also irrelevant that there would be no spam or virus/worm problems
(and so no virus scanning industry) if users would demand reasonable
software, including systems that don't require running as "administrator"
all of the time.


Vernon Schryver vjs@rhyolite.com

In article ,
vjs@calcite.rhyolite.com (Vernon Schryver) wrote:

> >In an ideal world we wouldn't have to deal with spam and viruses in the
> >first place. But the world isn't ideal, and tradeoffs have to be made.
> >Should ISPs sacrifice features just to get the best spam/virus
> >filtering? If they do that, the terrorists win.
>
> That is a false choice. You don't need to choose between good virus
> filtering and slow mail systems.

That's not the tradeoff I was talking about. I said the tradeoff is
between mail system features and good+fast filtering.

--
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***
*** PLEASE don't copy me on replies, I'll read them in the group ***

In article ,
Barry Margolin wrote:

>> >In an ideal world we wouldn't have to deal with spam and viruses in the
>> >first place. But the world isn't ideal, and tradeoffs have to be made.
>> >Should ISPs sacrifice features just to get the best spam/virus
>> >filtering? If they do that, the terrorists win.
>>
>> That is a false choice. You don't need to choose between good virus
>> filtering and slow mail systems.
>
>That's not the tradeoff I was talking about. I said the tradeoff is
>between mail system features and good+fast filtering.

That is also or still a false choice. Real world fancy mail system
features are in the MUA or perhaps MSA, not the MTA. The MTA or SMTP
server need only run the SMTP state machine, accept incoming mail at a
good speed, and let or help the filtering see the messages to say yea
or nay. The SMTP state machine is for handling the DATA command is
*trivial*.

The only likely fancy feature of an MTA or SMTP server that might cause
speed problems is dealing with TLS, but the same mechanisms that let
HTTP servers handle zillions of HTTPS hits per day are applicable to
SMTP. I don't know of any SMTP servers that use those mechanisms, but
that may be because so little email uses STARTTLS.


Arguing that you must do spam or virus filtering after the end of the
SMTP transaction because the system you've seen do it that way and are
still not quite able to keep up is like the position of some programmers
in the 1980s who claimed that any non-trivial network application must
use raw Ethernet packets. They felt TCP/IP is inevitably far too slow,
because all of the network applications they'd seen using other protocols
were incredibly slow. Then they saw some off-host TCP implementations
(e.g. Excelan or the IBM box) that validated their worst speed predictions.
They probably still don't believe Van Jacobson's value for the number
of CPU cycles per TCP segment.

Its like the position of the people who are convinced that UDP is the
solution to fast file transfers because they just know based on their
wide experience with TCP and UDP that TCP is too slow. Never mind that
their experience is so wide that they don't know where to find TFTP and
FTP client/server pairs to do benchmarking.


Vernon Schryver vjs@rhyolite.com

In article ,
vjs@calcite.rhyolite.com (Vernon Schryver) wrote:

> In article ,
> Barry Margolin wrote:
>
> >> >In an ideal world we wouldn't have to deal with spam and viruses in the
> >> >first place. But the world isn't ideal, and tradeoffs have to be made.
> >> >Should ISPs sacrifice features just to get the best spam/virus
> >> >filtering? If they do that, the terrorists win.
> >>
> >> That is a false choice. You don't need to choose between good virus
> >> filtering and slow mail systems.
> >
> >That's not the tradeoff I was talking about. I said the tradeoff is
> >between mail system features and good+fast filtering.
>
> That is also or still a false choice.

Is there really any point to this argument? It hardly matters what the
choices SHOULD be. As customers, we don't get to choose the software
used on the mail servers.

I was trying to describe how things actually work, and the
justifications for them. Maybe those justifications aren't valid, it's
still the reasoning taking place at the service providers.

If business decisions were always based on the best technical merit,
would MS Windows be on 90% of office desktops?

--
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***
*** PLEASE don't copy me on replies, I'll read them in the group ***

In article ,
Barry Margolin wrote:

>> >> That is a false choice. You don't need to choose between good virus
>> >> filtering and slow mail systems.
>> >
>> >That's not the tradeoff I was talking about. I said the tradeoff is
>> >between mail system features and good+fast filtering.
>>
>> That is also or still a false choice.
>
>Is there really any point to this argument? It hardly matters what the
>choices SHOULD be. As customers, we don't get to choose the software
>used on the mail servers.
>
>I was trying to describe how things actually work, and the
>justifications for them. Maybe those justifications aren't valid, it's
>still the reasoning taking place at the service providers.
>
>If business decisions were always based on the best technical merit,
>would MS Windows be on 90% of office desktops?

One of the main (but certainly not the only) reasons Microsoft software
has so much marketshare and has the characteristics that it is the
is the endless repetitions of false statements. If people who know
or should know bettter had not spent 25 years agreeing that:

- it is normal to need to reboot a computer every few hours
- virus are normal and inevitiable, and never mind the implications
of lack of any notion of file ownership or permissions in the
almost UNIX-like file system in DOS
- "user friendliness" is more important than any notion of security,
and so
+ running as an ordinary user instead of administrator is
too "unfriendly"
+ it is good to execute anything from anywhere that looks
like code in Internet Explorer, and never mind MIME types,
which not at all incidentally is much of what justifies
deep, slow scanning by SMTP anti-virus filters
+ the lack of a sandbox in ActiveX is a feature instead
of a stupid mistake, which also justifies AV deep scanning

then either Microsoft would have been doing a better job (as it finally
seems to havee begun) or would not have been so successful.

Similarly, if people who know better would not claim that junk SMTP
filters are the right, best and only way to do things, they wouldn't
be so common. A big irony is that unlike the desktop market, not one
of the really bad SMTP server filters has a dominent position in the
market. Much of the worst of SMTP filters are locally created by people
telling each other that they have no alternative and being willfully
ignorant of better solutions.


Vernon Schryver vjs@rhyolite.com