A strange thing with long SMTP message lines - VMS

This is a discussion on A strange thing with long SMTP message lines - VMS ; I've run into an interesting thing with PMDF 6.4 receiving SMTP messages with long text lines. The "standard" maximum text line length in SMTP is 1000 bytes, 998 when one accounts for the (as per section 4.5.3.1.6 of RFC 5321). ...

+ Reply to Thread
Results 1 to 12 of 12

Thread: A strange thing with long SMTP message lines

  1. A strange thing with long SMTP message lines

    I've run into an interesting thing with PMDF 6.4 receiving SMTP messages
    with long text lines.



    The "standard" maximum text line length in SMTP is 1000 bytes, 998 when
    one accounts for the (as per section 4.5.3.1.6 of RFC 5321).
    Servers are free to accept longer lines.



    My testing indicates that PMDF 6.4 will accept SMTP text lines up to
    1023 characters (not counting the ).



    If the text line is > 1024 bytes then PMDF will truncate it and create a
    "PMDF-SMTP-Warning" header



    However if the text line is exactly 1024 characters then PMDF will
    truncate it to 1023, but not create a "PMDF-SMTP-Warning" header. This
    would appear to be a bug. It would seem that PMDF should create the
    warning header for text lines of 1024 characters so that there would be
    a record that truncation to 1023 characters had occurred.



    Of course any client (in this case Apple Mail 2.753) that generates
    lines > 1000 characters is broken, but then living with broken clients
    is a cross we all have to bear. It might be better if PMDF rejected
    messages with excessively long lines by returning a 500 error ( as per
    section 4.5.3.1.9 of RFC 5351 ) or at least had an option to do so. This
    may be preferable to delivering a message that may be corrupted due to
    one or more lines being truncated.



    Thoughts?







  2. Re: A strange thing with long SMTP message lines

    PMDF seems to be one of a very few mailers that doesn't gleefully accept
    arbitrarily long lines. I'm still running 6.2-1 here (long, sad story)
    and I believe that with that version, lines greater than 1000 bytes are
    truncated at 1000 bytes and a header stating that long lines have been
    truncated is added.

    - ken

    Malcolm Dunnett wrote:
    >
    > Iíve run into an interesting thing with PMDF 6.4 receiving SMTP
    > messages with long text lines.
    >
    > The "standard" maximum text line length in SMTP is 1000 bytes, 998
    > when one accounts for the (as per section 4.5.3.1.6 of RFC
    > 5321). Servers are free to accept longer lines.
    >
    > My testing indicates that PMDF 6.4 will accept SMTP text lines up to
    > 1023 characters (not counting the ).
    >
    > If the text line is > 1024 bytes then PMDF will truncate it and create
    > a "PMDF-SMTP-Warning" header
    >
    > However if the text line is exactly 1024 characters then PMDF will
    > truncate it to 1023, but not create a "PMDF-SMTP-Warning" header. This
    > would appear to be a bug. It would seem that PMDF should create the
    > warning header for text lines of 1024 characters so that there would
    > be a record that truncation to 1023 characters had occurred.
    >
    > Of course any client (in this case Apple Mail 2.753) that generates
    > lines > 1000 characters is broken, but then living with broken clients
    > is a cross we all have to bear. It might be better if PMDF rejected
    > messages with excessively long lines by returning a 500 error ( as per
    > section 4.5.3.1.9 of RFC 5351 ) or at least had an option to do so.
    > This may be preferable to delivering a message that may be corrupted
    > due to one or more lines being truncated.
    >
    > Thoughts?
    >


    --
    - Ken
    ================================================== ===============
    Ken Connelly Associate Director, Security and Systems
    ITS Network Services University of Northern Iowa
    email: Ken.Connelly@uni.edu p: (319) 273-5850 f: (319) 273-7373



  3. RE: A strange thing with long SMTP message lines

    Ken Connelly wrote:

    >PMDF seems to be one of a very few mailers that doesn't gleefully

    accept
    >arbitrarily long lines. I'm still running 6.2-1 here (long, sad story)
    >and I believe that with that version, lines greater than 1000 bytes are


    >truncated at 1000 bytes and a header stating that long lines have been
    >truncated is added.


    I agree that PMDF accepting arbitrarily long lines (or at least a much
    longer size such as 32767 bytes) would be the ideal solution, but I
    don't know if that might entail significant re-engineering. The two
    suggestions I made seem like rather simple fixes that would at least be
    better than the current situation.


  4. Re: A strange thing with long SMTP message lines

    Malcolm Dunnett wrote:
    >
    > I agree that PMDF accepting arbitrarily long lines (or at least a much
    > longer size such as 32767 bytes) would be the ideal solution, but I
    > don't know if that might entail significant re-engineering. The two
    > suggestions I made seem like rather simple fixes that would at least be
    > better than the current situation.
    >
    >

    I disagree. While more consistency would be an improvement, PMDF is
    following the standards by truncating long lines. What *would* be a
    good thing is fixing the plethora of broken clients that emit mail
    messages which violate the standards.

    --
    - Ken
    ================================================== ===============
    Ken Connelly Associate Director, Security and Systems
    ITS Network Services University of Northern Iowa
    email: Ken.Connelly@uni.edu p: (319) 273-5850 f: (319) 273-7373


  5. RE: A strange thing with long SMTP message lines


    Ken Connelly wrote:

    >I disagree. While more consistency would be an improvement, PMDF is
    >following the standards by truncating long lines.


    PMDF is following the "minimum" standard by truncating long lines,
    however
    the standard also says:

    "To the maximum extent possible, implementation techniques which impose
    no limits on the length of these objects should be used."

    So it's not a violation of the standard to accept longer lines, it's
    only a violation to reject lines shorter than 1000 characters. In fact
    PMDF "should" (in the meaning of the RFC) accept longer lines ( I
    would argue that the "maximum extent possible" would be to accept
    lines up to 32767 characters, this being the limit on RMS records
    and the maximum one can describe in a VMS string descriptor - however
    I will grant there may be other technical issues which would impose
    a shorter limit)

    > What *would* be a
    >good thing is fixing the plethora of broken clients that emit mail
    >messages which violate the standards.


    Yes, it would - but I know of no way to influence that, whereas
    discussions in this group *may* influence Process Software to
    modify PMDF to accept longer lines. As an admin I just want the
    mail to go through with minimum hassle, so if PMDF can be modified
    to be more tolerant of these broken clients that is a good thing.
    If not I still don't like the current model of truncating longer
    lines, I would prefer PMDF to reject the message, to make it
    clear to the broken client that their message did not get through
    intact.


  6. RE: A strange thing with long SMTP message lines


    > Ken Connelly wrote:


    > >I disagree. While more consistency would be an improvement, PMDF is
    > >following the standards by truncating long lines.


    > PMDF is following the "minimum" standard by truncating long lines, however
    > the standard also says:


    > "To the maximum extent possible, implementation techniques which impose
    > no limits on the length of these objects should be used."


    The standard you are quoting here is RFC 821 (section 4.5.3 to be specific).
    Problem is, it, along with RFC 822, are now obsolete, having been replaced by
    RFCs 5321 and 5322. (Yes, I'm aware that these are current at draft and 821/822
    are full standards, but progression to full standard seems inevitable at this
    point so any attempt to cling to the older specifications for guidance has a
    lifespan best measured in months at this point.) These new specifications are
    much more strict about this stuff. RFC 5322 in particular has this to say
    (section 2.1.1):

    There are two limits that this specification places on the number of
    characters in a line. Each line of characters MUST be no more than
    998 characters, and SHOULD be no more than 78 characters, excluding
    the CRLF.

    The 998 character limit is due to limitations in many implementations
    that send, receive, or store IMF messages which simply cannot handle
    more than 998 characters on a line. Receiving implementations would
    do well to handle an arbitrarily large number of characters in a line
    for robustness sake. However, there are so many implementations that
    (in compliance with the transport requirements of [RFC5321]) do not
    accept messages containing more than 1000 characters including the CR
    and LF per line, it is important for implementations not to create
    such messages.

    This really could not be more clear, especially the part about "not creating
    such messages".

    > So it's not a violation of the standard to accept longer lines, it's
    > only a violation to reject lines shorter than 1000 characters.


    That's quite simply false. Messages containing lines longer that 1000
    characters are incompliant with current standards and may be handled in any
    manner an implementation chooses, including forcible line wrapping or even
    summary rejection.

    > In fact PMDF "should" (in the meaning of the RFC) accept longer lines ( I
    > would argue that the "maximum extent possible" would be to accept
    > lines up to 32767 characters, this being the limit on RMS records
    > and the maximum one can describe in a VMS string descriptor - however
    > I will grant there may be other technical issues which would impose
    > a shorter limit)


    The argument that PMDF "should" do this doesn't come close to washing given
    what the standards currently say (and have said for quite a while - this
    language or something close to it also appeared in RFCs 2821 and 2822).

    And while tweaking PMDF so that all lines could be, say, 2048 characters or
    some other fixed value, changing things so that arbitrarily long lines could be
    accomodated would be EXTREMELY difficult, and if done would probably compromise
    performance quite significantly. And any fixed limit won't help with clients
    that insist on spewing out megabytes of HTML on a single line.

    What would help, and is the appproach we adopted in iMS some time back, is to
    forcibly wrap such lines. We also added an option to reject messages with
    overly long lines because believe it or not, quite a few people demanded it.
    So now sites can choose between truncation, wrapping, or rejection.

    Oh, and we also set the wrap/truncate/reject point to 998 instead of 1023/1024
    as it is in PMDF. And the reason we did this is - wait for it - sending out
    messages with lines even slightly past the limit was causing interop problems
    with quite a few other mail systems. So much for the notion that everybody out
    there accepts arbitrarily long lines.

    Wrapping, if done, seems to work best if the heuristics assume HTML-ish input,
    since a significant number of these broken messages are text/html or something
    similar.

    I can provide the code to Process to do line wraps if they want it, however,
    the SMTP server code has diverged so from the PMDF original I cannot provide a
    complete retrofit in this case.

    > > What *would* be a
    > > good thing is fixing the plethora of broken clients that emit mail
    > > messages which violate the standards.


    > Yes, it would - but I know of no way to influence that,


    Well, summary rejection of such messages would definitely "send" a "message",
    especially if lots of sites did it. But I doubt if enought sites have the
    stomach for that.

    > whereas
    > discussions in this group *may* influence Process Software to
    > modify PMDF to accept longer lines. As an admin I just want the
    > mail to go through with minimum hassle, so if PMDF can be modified
    > to be more tolerant of these broken clients that is a good thing.


    I'm sorry, but I have to say I see this attitude as quite dangerous and in fact
    at least partly responsible for much of the mess that email is now in.

    Nondelivery notification handling is a particularly good example of this.
    Blowback spam is a fairly significant problem for a lot of sites, so much so
    that legitimate notifications can be swamped with garbage. Applying the
    "minimum hassle" metric in this case leads to an obvious conclusion: Throw them
    all away, or better still, simply reject all messages with a empty MAIL FROM
    right there in the SMTP dialogue. (I know of sites that have done this so this
    is not an academic example.) Trouble is, every once in a while a nondelivery
    notification does convey critical information, and you've just screwed the user
    who was depending on that. All in the interests of "minimum hassle".

    Now, AFAIK the situation with long lines in messages hasn't resulted in this
    sort of damage. But there are a variety of ways in which it could easily reach
    that point. (The obvious one is a massive influx of spam that looks completely
    legitimate other than containing very long lines. Think what the likely
    consequence of that scenario will be on the "just send it along unchanged"
    crowd.) At least if the message has been truncated or rejected someone will be
    made aware of what happened, and can figure out why. But wrapping long lines is
    really the best option IMO, even though it's arguably the biggest hassle
    (because if the wrapping breaks the message it's hard to avoid owning the
    breakage).

    > If not I still don't like the current model of truncating longer
    > lines, I would prefer PMDF to reject the message, to make it
    > clear to the broken client that their message did not get through
    > intact.


    Adding an option to reject such messages would be a good idea IMO, but
    adding one to forcibly wrap would be even better.

    Ned

    P.S. Yet another option is forcible encoding of messages. The only problem with
    this is it's an implementation nightmare.

  7. RE: A strange thing with long SMTP message lines

    Ned Freed wrote:

    Thanks for your thoughts. I respect that you know a whole
    lot more about the issue than I do, but I do have to
    disagree with some of what you said.

    >The standard you are quoting here is RFC 821 (section 4.5.3 to be

    specific).
    >Problem is, it, along with RFC 822, are now obsolete, having been

    replaced by
    >RFCs 5321 and 5322.


    Actually I was quoting section 4.5.3.1 of RFC 5321. While the example
    they give is an address and not a text line it seems to me from the
    context of the section numbering that they are talking about all
    the objects within the section, including "4.5.3.1.6 Text Line"

    > more than 998 characters on a line. Receiving implementations would
    > do well to handle an arbitrarily large number of characters in a

    line
    > for robustness sake.


    That's what I'm asking for, the receiving implementation to handle
    an arbitrarily large number of characters (for robustness sake.).
    Realistically I'm not asking for arbitrarily large, just "as large
    as is reasonable to implement given the existing code base"

    > However, there are so many implementations that
    > (in compliance with the transport requirements of [RFC5321]) do not
    > accept messages containing more than 1000 characters including the

    CR
    > and LF per line, it is important for implementations not to create
    > such messages.


    >This really could not be more clear, especially the part about "not

    creating
    >such messages".


    Precisely. I never suggested anyone should create such messages (the
    actual
    creation would be a UA issue, not an MTA issue). However I was
    suggesting
    that if presented with one the MTA should (or "would do well to" if
    you prefer) accommodate it (as per the quote above).

    >> So it's not a violation of the standard to accept longer lines, it's
    >> only a violation to reject lines shorter than 1000 characters.


    >That's quite simply false. Messages containing lines longer that 1000
    >characters are incompliant with current standards and may be handled in

    any
    >manner an implementation chooses, including forcible line wrapping or

    even
    >summary rejection.


    That doesn't make what I said false, it merely indicates (as I said)
    that
    nothing in the standard *prevents* PMDF from receiving longer lines.
    I will grant you that it's not *required* to.

    > In fact PMDF "should" (in the meaning of the RFC) accept longer lines

    ( I
    > would argue that the "maximum extent possible" would be to accept
    > lines up to 32767 characters, this being the limit on RMS records
    > and the maximum one can describe in a VMS string descriptor - however
    > I will grant there may be other technical issues which would impose
    > a shorter limit)


    >The argument that PMDF "should" do this doesn't come close to washing

    given
    >what the standards currently say (and have said for quite a while -

    this
    >language or something close to it also appeared in RFCs 2821 and 2822).


    As I was quoting from the current standard I can't see that your
    argument
    is valid.

    >And while tweaking PMDF so that all lines could be, say, 2048

    characters or
    >some other fixed value, changing things so that arbitrarily long lines

    could be
    >accomodated would be EXTREMELY difficult, and if done would probably

    compromise
    >performance quite significantly. And any fixed limit won't help with

    clients
    >that insist on spewing out megabytes of HTML on a single line.


    I will certainly bow to your superior knowledge of what is feasible
    in PMDF. AS I said earlier, there may well be implementation reasons
    that preclude accepting longer lines. My 32767 example was merely
    noting an example of an upper bound that the current implementation
    may impose.

    >What would help, and is the appproach we adopted in iMS some time back,

    is to
    >forcibly wrap such lines.


    That sounds like an elegant solution (assuming it wraps them
    intelligently
    so as to not corrupt the syntax of the message)

    >We also added an option to reject messages with
    >overly long lines because believe it or not, quite a few people

    demanded it.

    Why wouldn't I believe it, if you read my earlier message you'll see
    I asked for that as a second choice. If the non-conforming message
    can't be delivered intact by PMDF I think it would be better to reject
    it (along with a suitable message to the effect that the format
    is non-conforming).

    As an aside, if I create a message with the PMDF API and submit
    message body lines of more than 1024 characters they are
    truncated, but no "lines truncated header" is created, nor is
    an error status returned by PMDF_write_text. Another example
    of where PMDF can silently corrupt the message.

    >Wrapping, if done, seems to work best if the heuristics assume HTML-ish

    input,
    >since a significant number of these broken messages are text/html or

    something
    >similar.


    The cases I ran into were HTML code, so I would concur with that
    statement.

    >I can provide the code to Process to do line wraps if they want it,

    however,
    >the SMTP server code has diverged so from the PMDF original I cannot

    provide a
    >complete retrofit in this case.


    That of course is up to Process, however I would certainly applaud
    them adopting that change.

    >Well, summary rejection of such messages would definitely "send" a

    "message",
    >especially if lots of sites did it. But I doubt if enought sites have

    the
    >stomach for that.


    Probably true - and something I can understand. End users tend
    to assume that if we can't accept a message from their Macintosh,
    or can't send one to Hotmail, that it must be our fault and
    demand that we "fix it". However I would still prefer to
    justify to a client why I rejected his message rather than why I
    delivered a version that was "corrupted" due to line truncation.

    >> modify PMDF to accept longer lines. As an admin I just want the
    >> mail to go through with minimum hassle, so if PMDF can be modified
    >> to be more tolerant of these broken clients that is a good thing.

    >
    >I'm sorry, but I have to say I see this attitude as quite dangerous and

    in fact
    >at least partly responsible for much of the mess that email is now in.


    Well that may be true - if nobody accepted lines longer than 1000
    characters then the broken UAs would probably get fixed. But in
    the real world that isn't going to happen, is it?

    >Now, AFAIK the situation with long lines in messages hasn't resulted in

    this
    >sort of damage. But there are a variety of ways in which it could

    easily reach
    >that point. (The obvious one is a massive influx of spam that looks

    completely
    >legitimate other than containing very long lines. Think what the likely
    >consequence of that scenario will be on the "just send it along

    unchanged"
    >crowd.)


    Why is this worse than any other kind of spam? Are you thinking about
    buffer overflow attacks in the MTA or UA?

    > At least if the message has been truncated or rejected someone will be
    >made aware of what happened, and can figure out why. But wrapping long

    lines is
    >really the best option IMO, even though it's arguably the biggest

    hassle
    >(because if the wrapping breaks the message it's hard to avoid owning

    the
    >breakage).


    Well wrapping is certainly less likely to break the message than
    truncation is, so overall it's far more likely to be an
    acceptable solution.


  8. RE: A strange thing with long SMTP message lines

    In article <509D66DBD103FE47BA04384CC1F850FC0219AEA2@james.cap itan.mala.bc.ca>, Malcolm Dunnett writes:
    >Ned Freed wrote:
    >
    >Thanks for your thoughts. I respect that you know a whole
    >lot more about the issue than I do, but I do have to
    >disagree with some of what you said.
    >
    >>The standard you are quoting here is RFC 821 (section 4.5.3 to be

    >specific).
    >>Problem is, it, along with RFC 822, are now obsolete, having been

    >replaced by
    >>RFCs 5321 and 5322.

    >
    >Actually I was quoting section 4.5.3.1 of RFC 5321. While the example
    >they give is an address and not a text line it seems to me from the
    >context of the section numbering that they are talking about all
    >the objects within the section, including "4.5.3.1.6 Text Line"
    >
    >> more than 998 characters on a line. Receiving implementations would
    >> do well to handle an arbitrarily large number of characters in a

    >line
    >> for robustness sake.

    >
    > That's what I'm asking for, the receiving implementation to handle
    >an arbitrarily large number of characters (for robustness sake.).
    >Realistically I'm not asking for arbitrarily large, just "as large
    >as is reasonable to implement given the existing code base"
    >

    If you were going to do this then I'd suggest that the best method would be to
    increase the buffer size for the PMDF SMTP server so that it could receive
    messages with larger line lengths but then to apply a default linelength
    channel keyword of 1024 which would result in messages with long
    linelengths being MIME encoded - thereby removing any need to alter other
    parts of PMDF.

    David Webb
    Security team leader
    CCSS
    Middlesex University




    >> However, there are so many implementations that
    >> (in compliance with the transport requirements of [RFC5321]) do not
    >> accept messages containing more than 1000 characters including the

    >CR
    >> and LF per line, it is important for implementations not to create
    >> such messages.

    >
    >>This really could not be more clear, especially the part about "not

    >creating
    >>such messages".

    >
    >Precisely. I never suggested anyone should create such messages (the
    >actual
    >creation would be a UA issue, not an MTA issue). However I was
    >suggesting
    >that if presented with one the MTA should (or "would do well to" if
    >you prefer) accommodate it (as per the quote above).
    >
    >>> So it's not a violation of the standard to accept longer lines, it's
    >>> only a violation to reject lines shorter than 1000 characters.

    >
    >>That's quite simply false. Messages containing lines longer that 1000
    >>characters are incompliant with current standards and may be handled in

    >any
    >>manner an implementation chooses, including forcible line wrapping or

    >even
    >>summary rejection.

    >
    >That doesn't make what I said false, it merely indicates (as I said)
    >that
    >nothing in the standard *prevents* PMDF from receiving longer lines.
    >I will grant you that it's not *required* to.
    >
    >> In fact PMDF "should" (in the meaning of the RFC) accept longer lines

    >( I
    >> would argue that the "maximum extent possible" would be to accept
    >> lines up to 32767 characters, this being the limit on RMS records
    >> and the maximum one can describe in a VMS string descriptor - however
    >> I will grant there may be other technical issues which would impose
    >> a shorter limit)

    >
    >>The argument that PMDF "should" do this doesn't come close to washing

    >given
    >>what the standards currently say (and have said for quite a while -

    >this
    >>language or something close to it also appeared in RFCs 2821 and 2822).

    >
    >As I was quoting from the current standard I can't see that your
    >argument
    >is valid.
    >
    >>And while tweaking PMDF so that all lines could be, say, 2048

    >characters or
    >>some other fixed value, changing things so that arbitrarily long lines

    >could be
    >>accomodated would be EXTREMELY difficult, and if done would probably

    >compromise
    >>performance quite significantly. And any fixed limit won't help with

    >clients
    >>that insist on spewing out megabytes of HTML on a single line.

    >
    > I will certainly bow to your superior knowledge of what is feasible
    >in PMDF. AS I said earlier, there may well be implementation reasons
    >that preclude accepting longer lines. My 32767 example was merely
    >noting an example of an upper bound that the current implementation
    >may impose.=20
    >
    >>What would help, and is the appproach we adopted in iMS some time back,

    >is to
    >>forcibly wrap such lines.=20

    >
    >That sounds like an elegant solution (assuming it wraps them
    >intelligently
    >so as to not corrupt the syntax of the message)
    >
    >>We also added an option to reject messages with
    >>overly long lines because believe it or not, quite a few people

    >demanded it.
    >
    >Why wouldn't I believe it, if you read my earlier message you'll see
    >I asked for that as a second choice. If the non-conforming message
    >can't be delivered intact by PMDF I think it would be better to reject
    >it (along with a suitable message to the effect that the format
    >is non-conforming).
    >
    >As an aside, if I create a message with the PMDF API and submit
    >message body lines of more than 1024 characters they are
    >truncated, but no "lines truncated header" is created, nor is
    >an error status returned by PMDF_write_text. Another example
    >of where PMDF can silently corrupt the message.
    >
    >>Wrapping, if done, seems to work best if the heuristics assume HTML-ish

    >input,
    >>since a significant number of these broken messages are text/html or

    >something
    >>similar.

    >
    >The cases I ran into were HTML code, so I would concur with that
    >statement.
    >
    >>I can provide the code to Process to do line wraps if they want it,

    >however,
    >>the SMTP server code has diverged so from the PMDF original I cannot

    >provide a
    >>complete retrofit in this case.

    >
    >That of course is up to Process, however I would certainly applaud
    >them adopting that change.
    >
    >>Well, summary rejection of such messages would definitely "send" a

    >"message",
    >>especially if lots of sites did it. But I doubt if enought sites have

    >the
    >>stomach for that.

    >
    >Probably true - and something I can understand. End users tend
    >to assume that if we can't accept a message from their Macintosh,
    >or can't send one to Hotmail, that it must be our fault and
    >demand that we "fix it". However I would still prefer to
    >justify to a client why I rejected his message rather than why I
    >delivered a version that was "corrupted" due to line truncation.
    >
    >>> modify PMDF to accept longer lines. As an admin I just want the
    >>> mail to go through with minimum hassle, so if PMDF can be modified
    >>> to be more tolerant of these broken clients that is a good thing.

    >>
    >>I'm sorry, but I have to say I see this attitude as quite dangerous and

    >in fact
    >>at least partly responsible for much of the mess that email is now in.

    >
    >Well that may be true - if nobody accepted lines longer than 1000
    >characters then the broken UAs would probably get fixed. But in
    >the real world that isn't going to happen, is it?
    >
    >>Now, AFAIK the situation with long lines in messages hasn't resulted in

    >this
    >>sort of damage. But there are a variety of ways in which it could

    >easily reach
    >>that point. (The obvious one is a massive influx of spam that looks

    >completely
    >>legitimate other than containing very long lines. Think what the likely
    >>consequence of that scenario will be on the "just send it along

    >unchanged"
    >>crowd.)

    >
    >Why is this worse than any other kind of spam? Are you thinking about
    >buffer overflow attacks in the MTA or UA?=20
    >
    >> At least if the message has been truncated or rejected someone will be
    >>made aware of what happened, and can figure out why. But wrapping long

    >lines is
    >>really the best option IMO, even though it's arguably the biggest

    >hassle
    >>(because if the wrapping breaks the message it's hard to avoid owning

    >the
    >>breakage).

    >
    >Well wrapping is certainly less likely to break the message than
    >truncation is, so overall it's far more likely to be an
    >acceptable solution.
    >


  9. Re: A strange thing with long SMTP message lines

    In an ideal world, people with broken mailers like Microsoft would fix
    their products, and ensure they adhere to the common standard.

    We don't live in an ideal world.

    Vendors whose products occupy a dominant position have a vested interest
    in ensuring that they do diverge from the standard, and provide extra
    functionality. Users then perceive other products as being inferior
    because what works with "most of the stuff out" there doesn't work with
    such-and-such product, and that provides migratory pressure on the user
    base. This is why Outlook has a 'richtext' encoding option (TNEF) that
    makes its attachments incompatible with everyone elses - it applies a
    steady pressure to get people to standardize on Outlook (it even seems
    to be the default setting). This can only be the case when the errant
    mailer already has a huge share of the market.

    Microsoft didn't invent this. IBM were probably the first to adopt the
    "we're-big-enough-to-be-the-standard", and so did Unix in the early days
    of the Internet (which is why Multinet to this day as an FTP server that
    can pretend to be a Unix system to satisfy many of the clients out there
    that think everything is either Windows or Unix).

    The hard commercial fact is if you are trying to sell a mailer that must
    compete with Exchange, then if something works fine with Exchange and
    fails with your mailer, it is no use pointing out that their's is out of
    step, and your's is the correct standard. The paying punters don't see
    it that way, and will vote with their wallets.

    Because of this, mailers such as PMDF have a lot of features
    ('msexchange' keyword, MIME header correction capabilities etc) designed
    to cope with the way the dominant player strays from the standard when
    it suits them. The issue is whether the effort is worth the gain, and
    this is primarily a commercial decision, not a standards decision, and
    certainly not one based on notions of 'fair' or 'correct'.

    I suspect it was for commercial reasons that Sun decided to make iMS
    able to handle lines that were longer than the standard. There is also
    the tried and trusted "be conversative in what you send, and liberal in
    what you accept" principle, which means you adhere to the letter of the
    standard with what you emit, and try to allow as much as possible for
    the many broken mailers when accepting traffic in.

    I accept Ned's assertion that trying to increase this limit in PMDF will
    be a big upheaval and could seriously impact performance, as nobody
    knows more about the innards of PMDF. I would suggest to Process that
    Ned's suggestion of automatically wrapping long lines along white space
    boundaries should be considered as an urgent enhancement. Yes, this is
    not a 100% correct way of handling a message, but is a much superior
    option to simply chopping the line, and if it does mess up a bodypart,
    then (A) it was already broken to start with and (B) it wouldn't fare
    much better going to an existing competing mailer.

    I find it difficult to explain to paying customers that something that
    works fine with competitor products (based on Exchange or say iMS) and
    not on ours (based on PMDF) is not our fault - they simply see it was
    one less reason to go with PMDF, which is a great pity, as the last
    thing we need is to facilitate migrations away from that great product.


    ---------------------------------------------------------
    Tom Wade | EMail: tee dot wade at eurokom dot ie
    EuroKom | Tel: +353 (1) 296-9696
    A2, Nutgrove Office Park | Fax: +353 (1) 296-9697
    Rathfarnham | Disclaimer: This is not a disclaimer
    Dublin 14 | Tip: "Friends don't let friends do Unix !"
    Ireland


  10. Re: A strange thing with long SMTP message lines

    I totally agree with Tom, and I encourage Process to try to maintain
    up to date the PMDF as much as possible.
    (For example, I think that one of the next big effort for Internet application
    developers will be IPv6 support).

    Thanks, Tom.

    Francesco


    > In an ideal world, people with broken mailers like Microsoft would fix
    > their products, and ensure they adhere to the common standard.


    > We don't live in an ideal world.


    > Vendors whose products occupy a dominant position have a vested interest
    > in ensuring that they do diverge from the standard, and provide extra
    > functionality. Users then perceive other products as being inferior
    > because what works with "most of the stuff out" there doesn't work with
    > such-and-such product, and that provides migratory pressure on the user
    > base. This is why Outlook has a 'richtext' encoding option (TNEF) that
    > makes its attachments incompatible with everyone elses - it applies a
    > steady pressure to get people to standardize on Outlook (it even seems
    > to be the default setting). This can only be the case when the errant
    > mailer already has a huge share of the market.


    > Microsoft didn't invent this. IBM were probably the first to adopt the
    > "we're-big-enough-to-be-the-standard", and so did Unix in the early days
    > of the Internet (which is why Multinet to this day as an FTP server that
    > can pretend to be a Unix system to satisfy many of the clients out there
    > that think everything is either Windows or Unix).


    > The hard commercial fact is if you are trying to sell a mailer that must
    > compete with Exchange, then if something works fine with Exchange and
    > fails with your mailer, it is no use pointing out that their's is out of
    > step, and your's is the correct standard. The paying punters don't see
    > it that way, and will vote with their wallets.


    > Because of this, mailers such as PMDF have a lot of features
    > ('msexchange' keyword, MIME header correction capabilities etc) designed
    > to cope with the way the dominant player strays from the standard when
    > it suits them. The issue is whether the effort is worth the gain, and
    > this is primarily a commercial decision, not a standards decision, and
    > certainly not one based on notions of 'fair' or 'correct'.


    > I suspect it was for commercial reasons that Sun decided to make iMS
    > able to handle lines that were longer than the standard. There is also
    > the tried and trusted "be conversative in what you send, and liberal in
    > what you accept" principle, which means you adhere to the letter of the
    > standard with what you emit, and try to allow as much as possible for
    > the many broken mailers when accepting traffic in.


    > I accept Ned's assertion that trying to increase this limit in PMDF will
    > be a big upheaval and could seriously impact performance, as nobody
    > knows more about the innards of PMDF. I would suggest to Process that
    > Ned's suggestion of automatically wrapping long lines along white space
    > boundaries should be considered as an urgent enhancement. Yes, this is
    > not a 100% correct way of handling a message, but is a much superior
    > option to simply chopping the line, and if it does mess up a bodypart,
    > then (A) it was already broken to start with and (B) it wouldn't fare
    > much better going to an existing competing mailer.


    > I find it difficult to explain to paying customers that something that
    > works fine with competitor products (based on Exchange or say iMS) and
    > not on ours (based on PMDF) is not our fault - they simply see it was
    > one less reason to go with PMDF, which is a great pity, as the last
    > thing we need is to facilitate migrations away from that great product.



    > ---------------------------------------------------------
    > Tom Wade | EMail: tee dot wade at eurokom dot ie
    > EuroKom | Tel: +353 (1) 296-9696
    > A2, Nutgrove Office Park | Fax: +353 (1) 296-9697
    > Rathfarnham | Disclaimer: This is not a disclaimer
    > Dublin 14 | Tip: "Friends don't let friends do Unix !"
    > Ireland



  11. RE: A strange thing with long SMTP message lines

    > >The standard you are quoting here is RFC 821 (section 4.5.3 to be specific).
    > >Problem is, it, along with RFC 822, are now obsolete, having been replaced by
    > >RFCs 5321 and 5322.


    > Actually I was quoting section 4.5.3.1 of RFC 5321. While the example
    > they give is an address and not a text line it seems to me from the
    > context of the section numbering that they are talking about all
    > the objects within the section, including "4.5.3.1.6 Text Line"


    Hmm. I thought that text had vanished. It is quite unfortunate it has been
    left in. (I have similar issues with some of the language surrounding CRLF
    handling that are causing far more interop problems than they are solving, but
    that's a story for another day.)

    Regardless, the fact remains that emitting a message with lines longer than 998
    violates a MUST elsewhere. A mild recommendation that implementations should
    not limit things unnecessarily (note the lack of compliance language) does not
    suffice to override a MUST. That means any implementation that accepts such
    lines also has to figure out what it's going to do with them once it has them.
    Sending them out unchanged is NOT an acceptable option.

    > > more than 998 characters on a line. Receiving implementations would
    > > do well to handle an arbitrarily large number of characters in a

    > line
    > > for robustness sake.


    > That's what I'm asking for, the receiving implementation to handle
    > an arbitrarily large number of characters (for robustness sake.).
    > Realistically I'm not asking for arbitrarily large, just "as large
    > as is reasonable to implement given the existing code base"


    Actually, there is no reason not to handle arbitraily long lines if what you do
    is modify them either by folding, truncating or rejecting them right away.
    Sending them through unchanged is what's hard. And the problem with any fixed
    buffer size is something out there is almost guaranteed to exceed it. (OK,
    maybe if you set it to a couple of megabytes the number of messages that will
    exceed it will be negligable, but that's the sort of fixed limit you have to
    have if you expect this to actually salvage the real-world malformed messages
    people send. But that big of a buffer size would be extremely inefficient, even
    in this time of ever-increasing memory.)

    > > However, there are so many implementations that
    > > (in compliance with the transport requirements of [RFC5321]) do not
    > > accept messages containing more than 1000 characters including the CR
    > > and LF per line, it is important for implementations not to create
    > > such messages.


    > >This really could not be more clear, especially the part about "not

    > creating
    > >such messages".


    > Precisely. I never suggested anyone should create such messages (the
    > actual
    > creation would be a UA issue, not an MTA issue). However I was
    > suggesting
    > that if presented with one the MTA should (or "would do well to" if
    > you prefer) accommodate it (as per the quote above).


    I disagree. "Handling" these lines without alteration, in addition to violating
    that MUST, would mean either changing the fixed buffer to a completely
    unreasonable value or rewriting a significant fraction of the MTA to work in a
    very different way. While such a rewrite is possible, it would unavoidably lead
    to a performance loss and given the extent of the changes regressions are
    certain to occur.

    All this just to cater to clearly broken behavior. The costs just aren't in
    line with the benefits, especially since there's a viable alternative that
    solves most of the problem without having to "handle" such material.

    > >> So it's not a violation of the standard to accept longer lines, it's
    > >> only a violation to reject lines shorter than 1000 characters.


    > >That's quite simply false. Messages containing lines longer that 1000
    > >characters are incompliant with current standards and may be handled in

    > any
    > >manner an implementation chooses, including forcible line wrapping or

    > even
    > >summary rejection.


    > That doesn't make what I said false, it merely indicates (as I said) that
    > nothing in the standard *prevents* PMDF from receiving longer lines.


    Sorry, I misread your statement and thought you were saying it was
    required.

    > I will grant you that it's not *required* to.


    On that we agree.

    > > In fact PMDF "should" (in the meaning of the RFC) accept longer lines

    > ( I
    > > would argue that the "maximum extent possible" would be to accept
    > > lines up to 32767 characters, this being the limit on RMS records
    > > and the maximum one can describe in a VMS string descriptor - however
    > > I will grant there may be other technical issues which would impose
    > > a shorter limit)


    > > The argument that PMDF "should" do this doesn't come close to washing given
    > > what the standards currently say (and have said for quite a while - this
    > > language or something close to it also appeared in RFCs 2821 and 2822).


    > As I was quoting from the current standard I can't see that your
    > argument is valid.


    Again, a mild recommendations that limits not be imposed unnecessarily does not
    come close to overriding MUSTs elsewhere.

    > > What would help, and is the appproach we adopted in iMS some time back, s to
    > > forcibly wrap such lines.


    > That sounds like an elegant solution (assuming it wraps them intelligently
    > so as to not corrupt the syntax of the message)


    That's effectively impossible, if for no other reaons than there are plenty of
    text formats that employ extremely long lines and which are totally intolerant
    of ANY line breaks being inserted. (CSV is a good exmaple of such a format
    that's in common use.)

    The best that can be done here is to try and not put the breaks in bad spots.
    But the reality is a few things are still going to break. And they'll also
    break if you encode the material, for that matter - even at this late date not
    everyone does MIME/uuencode/whatever decoding.

    > Why wouldn't I believe it, if you read my earlier message you'll see
    > I asked for that as a second choice. If the non-conforming message
    > can't be delivered intact by PMDF I think it would be better to reject
    > it (along with a suitable message to the effect that the format
    > is non-conforming).


    > As an aside, if I create a message with the PMDF API and submit
    > message body lines of more than 1024 characters they are
    > truncated, but no "lines truncated header" is created, nor is
    > an error status returned by PMDF_write_text. Another example
    > of where PMDF can silently corrupt the message.


    Yep, and in this case I think the API's behavior is reasonable and should not
    be changed. The PMDF API is not bound by any standard to behave in any
    particular way.

    > >Wrapping, if done, seems to work best if the heuristics assume HTML-ish

    > input,
    > >since a significant number of these broken messages are text/html or

    > something
    > >similar.


    > The cases I ran into were HTML code, so I would concur with that statement.


    It is almost always possible to insert breaks into HTML in such a way that
    things don't break. Of course finding the right points is another matter. A
    fairly simple algorithm seems to mostly work, though.

    > ...


    > >> modify PMDF to accept longer lines. As an admin I just want the
    > >> mail to go through with minimum hassle, so if PMDF can be modified
    > >> to be more tolerant of these broken clients that is a good thing.

    > >
    > >I'm sorry, but I have to say I see this attitude as quite dangerous and

    > in fact
    > >at least partly responsible for much of the mess that email is now in.


    > Well that may be true - if nobody accepted lines longer than 1000
    > characters then the broken UAs would probably get fixed. But in
    > the real world that isn't going to happen, is it?


    > >Now, AFAIK the situation with long lines in messages hasn't resulted in

    > this
    > >sort of damage. But there are a variety of ways in which it could

    > easily reach
    > >that point. (The obvious one is a massive influx of spam that looks

    > completely
    > >legitimate other than containing very long lines. Think what the likely
    > >consequence of that scenario will be on the "just send it along

    > unchanged"
    > >crowd.)


    > Why is this worse than any other kind of spam?


    It's not the spam but the behavior such spam would cause among spam-blockers.
    If such spam messages were to become common spam blockers will start using line
    length as a criteria for deciding that messages are spam. And if that happens
    you can kiss your ability to send messages with overly long lines goodbye no
    matter how big the population of misbehaving MUAs happens to be. Like it or
    not, the final arbiter os what's acceptable in email is now determined by the
    antispam tools.

    And this is not idle theorizing on my part - there are several past exanmples
    of this sort of thing. The one I remeber best is when a particularly virulent
    spambot spread a few years ago whose signature characteristic was that it's
    SMTP client said "HELO host" instead of "HELO host.something". Before that time
    very few sites enforced the restriction that HELO arguments have to be fully
    qualified domain names and a huge number of sites happily used shortform named
    in HELO, but after the spambot showed up things changed rapidly so that if you
    broke this rule you stood a good chance of having your mail bounce, or worse,
    silently lost. And yes, there was lots of complaining, but when you're being
    buried in spam somehow the license of others to break the rules doesn't seem
    like that big a deal.

    Ned

  12. Re: A strange thing with long SMTP message lines

    ned+info-pmdf@mauve.mrochek.com wrote:

    >...
    > Actually, there is no reason not to handle arbitraily long lines if what you do
    > is modify them either by folding, truncating or rejecting them right away.
    > Sending them through unchanged is what's hard. And the problem with any fixed
    > buffer size is something out there is almost guaranteed to exceed it. (OK,
    > maybe if you set it to a couple of megabytes the number of messages that will
    > exceed it will be negligable, but that's the sort of fixed limit you have to
    > have if you expect this to actually salvage the real-world malformed messages
    > people send. But that big of a buffer size would be extremely inefficient, even
    > in this time of ever-increasing memory.)


    Even if you increase PMDF's limit you'll have other limits (like RMS) to
    deal with as evidenced by recent bounce:

    %PMDF-E-MESFILERR, Error in accessing intermediate message file
    -RMS-W-RTB, 4152564 byte record too large for user's buffer

    The client was:

    X-Mailer: Apple Mail (2.929.2)

    --
    -- Carl Karcher, Waisman Computing Services, Waisman Center, UW-Madison
    -- karcher@waisman.wisc.edu

+ Reply to Thread