TCP Segment of a Reassembled PDU - TCP-IP

This is a discussion on TCP Segment of a Reassembled PDU - TCP-IP ; One of my servers gives the continuous message "TCP segment of a reassembled PDU" in a sniffer trace of its conversations with other machines on the same ethernet segment. Some brief research suggested that there may be a problem with ...

+ Reply to Thread
Results 1 to 8 of 8

Thread: TCP Segment of a Reassembled PDU

  1. TCP Segment of a Reassembled PDU

    One of my servers gives the continuous message "TCP segment of a reassembled
    PDU" in a sniffer trace of its conversations with other machines on the same
    ethernet segment. Some brief research suggested that there may be a
    problem with the TCP MTU size on either end.

    As a test, on one of the Windows 2000 workstations that has performance
    problems with this server, I hardcoded through the registry the DWORD MTU in
    the adapter TCP/IP Properties, and then set this to decimal 1400 (default is
    1500 for ethernet). The workstation has a copper GigE adapter. The
    server has an optical GigE adapter.

    We rebooted the Windows 2000 workstation and instantly performance was great
    with the server. Opening a message on that server took 4.5 seconds before
    the MTU change now took 1/2 second after the MTU change. So clearly MTU
    setting of 1400 did something good here. However:

    1) The sniffer trace still shows the reassembled PDU messages.

    2) The workstation immediately had severe problems using the file server and
    was not able to use the Internet at all.

    I backed out the MTU change and all resumed to its "normal" state.

    Are there other settings that must be changed when you lower the MTU setting
    on Windows 2000? What about Windows XP/Windows 2003?

    What are some possible causes of these symptoms. Ultimately I would like
    to know what changes would:

    1) Make conversations with the server fast.

    2) Remove fragmentation suggested by the reassembled PDU sniffer messages.

    3) Allow normal communication through routers and onto the Internet.

    --
    Will



  2. Re: TCP Segment of a Reassembled PDU

    Will wrote:

    > Are there other settings that must be changed when you lower the MTU setting
    > on Windows 2000? What about Windows XP/Windows 2003?


    Don't know - but poeple often use a little free util called drtcp.exe to
    save regediting.

    One thing to check is whether the card/drivers are doing tcp
    segmentation offload - maybe there is some messup to do with that.

    Andy.

  3. Re: TCP Segment of a Reassembled PDU

    In article ,
    "Will" wrote:

    > One of my servers gives the continuous message "TCP segment of a reassembled
    > PDU" in a sniffer trace of its conversations with other machines on the same
    > ethernet segment. Some brief research suggested that there may be a
    > problem with the TCP MTU size on either end.


    Are you talking about Wireshark (formerly Ethereal)? That message
    doesn't indicate a problem, and it's not talking about IP
    fragmentation/reassembly.

    What Wireshark does is automatically combine all the TCP segments of an
    application-layer message into a single row in the display. For
    instance, it will show an HTTP download as a single "Reassembled PDU".
    All the segments that contribute to it are marked with the above
    message, to indicate that they're duplicated in the merged row.

    --
    Barry Margolin, barmar@alum.mit.edu
    Arlington, MA
    *** PLEASE post questions in newsgroups, not directly to me ***
    *** PLEASE don't copy me on replies, I'll read them in the group ***

  4. Re: TCP Segment of a Reassembled PDU

    "Barry Margolin" wrote in message
    news:barmar-534748.13131402122006@comcast.dca.giganews.com...
    > In article ,
    > "Will" wrote:
    > > One of my servers gives the continuous message "TCP segment of a

    reassembled
    > > PDU" in a sniffer trace of its conversations with other machines on the

    same
    > > ethernet segment. Some brief research suggested that there may be a
    > > problem with the TCP MTU size on either end.

    >
    > Are you talking about Wireshark (formerly Ethereal)? That message
    > doesn't indicate a problem, and it's not talking about IP
    > fragmentation/reassembly.
    >
    > What Wireshark does is automatically combine all the TCP segments of an
    > application-layer message into a single row in the display. For
    > instance, it will show an HTTP download as a single "Reassembled PDU".
    > All the segments that contribute to it are marked with the above
    > message, to indicate that they're duplicated in the merged row.


    Well, that certainly explains the message. It doesn't however explain why
    the workstation with bad performance had a query to one server go from 4.5
    seconds to 0.5 seconds when the MTU was lowered to 1400.

    The problem is what other settings need to be changed because now the
    workstation doesn't work to servers that require it to send packets through
    routers.

    --
    Will



  5. Re: TCP Segment of a Reassembled PDU

    "Andy Furniss" wrote in message
    news:4571c56f.0@entanet...
    > Will wrote:
    >
    > > Are there other settings that must be changed when you lower the MTU

    setting
    > > on Windows 2000? What about Windows XP/Windows 2003?

    >
    > Don't know - but poeple often use a little free util called drtcp.exe to
    > save regediting.
    >
    > One thing to check is whether the card/drivers are doing tcp
    > segmentation offload - maybe there is some messup to do with that.


    I've seen that utility, but it sure looks like it was written for older
    versions of Windows, and I guess I'm not sure if I should trust it for
    Windows 2003.

    I would also prefer to have the knowledge of what it is doing.

    --
    Will



  6. Re: TCP Segment of a Reassembled PDU

    On Sat, 02 Dec 2006 17:20:06 -0800, Will wrote:

    > Well, that certainly explains the message. It doesn't however explain why
    > the workstation with bad performance had a query to one server go from 4.5
    > seconds to 0.5 seconds when the MTU was lowered to 1400.


    You have to look further on the wire. Your problem sounds pretty weird.
    This performance gain does not correspond to "normal" MTU problems. Either
    a transmission succeeds or it doesn't at all. "Normal" MTU problems occur
    when a datagram is send with a size larger than the path MTU (PMTU) and
    the datagram has the Don't Fragment bit set (PMTUD discovery is on, this
    is the default for most modern systems) and something blocks the ICMP
    "Fragmentation needed but Don't Fragment bit set" error. The datagram is
    discarded, resend, discarded, repeat until time out. This is called a PMTU
    blackhole.

    But I seem to recall that Windows sometimes lowers the MTU on retries to
    get around such blackholes, maybe this explains the behaviour you are
    seeing.

    Yes, the more I think about it, that would explain your problem. That
    assumes a MTU problem somewhere which would also explain your other
    problems (not being able to access the Internet and fileservers after
    lowering the MTU). Something is blocking the ICMP "Fragmentation needed
    but Don't Fragment bit set" error. And although I advice to check the
    firewall settings on both client and server, this is probably happening
    somewhere along the path.

    Luckily for you, this is often easy to diagnose, even if it is somewhere
    on the path. First check the firewalls on the client and server. If that
    doesn't solve it, use tracert.exe with the appropriate options. Start
    with a large MTU, and lower it until things start to work (the trace gets
    through). The hop that lies one behind the one that is not responding with
    the large MTU is the one that blocks the error (in some cases it may also
    be the last one that DID respond).

    If you cannot trace because something allows only certain traffic through,
    that would be the first point to investigate. That point probably blocks
    the ICMP error as well. You can also try ping with different MTUs and the
    "record route" option. This assumes ping is allowed when traceroute isn't
    and doesn't always work as record route is often seen as a security risk
    and filtered out.

    A whole other aproach would be to look at the network path the packets
    take and see where the MTU is lower and where ICMP "Fragmentation needed
    but Don't Fragment bit set" errors are blocked. Or get in someone who
    really knows networking and have that person have a look at the network
    (there more often than not are other problems when you have PMTU
    blackholes).

    Or start sniffing on various parts of the path. Find out that way where
    the packets are dropped. Works best with two persons and two (time
    synchronized!) sniffers and do a kind of binary search along the path.

    Last resort would be to boot the 2000 workstation of a Linux live-CD that
    has hping2 (Auditor comes to mind, but beware that you need to enable
    networking manually after booting, that trips up everyone starting out
    with Auditor). Hping2 can do a traceroute using tcp packets on the same
    port you use for your client server communication. That is virtually
    guarenteed to work (firewalls might drop packets with a to low TTL, but I
    haven't seen it in practice) as whatever blocks those packets is already
    allowing that client-server protocol. (You could also install a free
    version of VMWare and install Linux in that).

    I sincerely hope you don't work at one of those "all icmp is evil" shops.
    Many icmp packets pose a real security risk, while adding virtually no
    real value. Those can be blocked. Some can be blocked with certain side
    effects. But "Fragmentation needed but Don't Fragment bit set" errors
    should NEVER, NEVER, NEVER, EVER be blocked. It lands you in this kind of
    trouble. I know, I once worked at such a shop. Even after one and a half
    years of continious troubles they would not change that policy. The truth
    is, anyone who advocates blocking of "Fragmentation needed but Don't
    Fragment bit set" errors does not know enough about TCP/IP to be allowed
    to make such decissions. It's simply wrong. Search Google for "PMTU
    blackholes" if you need additional ammunition to convince people they
    should not do this.

    > The problem is what other settings need to be changed because now the
    > workstation doesn't work to servers that require it to send packets
    > through routers.


    Besides the local firewalls, you are probably not going to solve this with
    local settings. A workaround, if Windows allows this which I think it
    does, is to set an explicit route with an explicit MTU on the
    workstation for that particular server. But I advice against it, the
    problems will come back to bite you again and again and again.

    But this whole "solution" hinges on the assumption that Windows does retry
    with a lower MTU. I'm actually pretty sure about that and the more I think
    about it, the more I am convinced I'm right. Still it may be something
    completely different. Time to get out that sniffer again and validate or
    disprove my theory. Let us know how it went.

    HTH,
    M4
    --
    Redundancy is a great way to introduce more single points of failure.


  7. Re: TCP Segment of a Reassembled PDU

    On Sun, 03 Dec 2006 21:53:53 +0100, Martijn Lievaart wrote:

    > Luckily for you, this is often easy to diagnose, even if it is somewhere
    > on the path. First check the firewalls on the client and server. If that
    > doesn't solve it, use tracert.exe with the appropriate options. Start
    > with a large MTU, and lower it until things start to work (the trace gets
    > through). The hop that lies one behind the one that is not responding with
    > the large MTU is the one that blocks the error (in some cases it may also
    > be the last one that DID respond).


    Correction, that last is completely bull. The MTU problem is
    immediately after the last hop that responded. The blocking of the ICMP
    error occurs somehere between (inclusive) the client and that hop.

    Also be sure to turn on the Don't Fragment bit on any trace you do,
    otherwise you won't see the problem.

    A last quick fix, turn of PMTUD at the client completely. Fixes all your
    problems, but is not the greatest of solutions. However, IF this solves
    all your problems you are sure it is a PMTU blackhole problem.

    M4
    --
    Redundancy is a great way to introduce more single points of failure.


  8. Re: TCP Segment of a Reassembled PDU

    In article , Martijn Lievaart writes:
    > On Sat, 02 Dec 2006 17:20:06 -0800, Will wrote:
    >
    >> Well, that certainly explains the message. It doesn't however explain why
    >> the workstation with bad performance had a query to one server go from 4.5
    >> seconds to 0.5 seconds when the MTU was lowered to 1400.

    >
    > You have to look further on the wire. Your problem sounds pretty weird.
    > This performance gain does not correspond to "normal" MTU problems. Either
    > a transmission succeeds or it doesn't at all. "Normal" MTU problems occur
    > when a datagram is send with a size larger than the path MTU (PMTU) and
    > the datagram has the Don't Fragment bit set (PMTUD discovery is on, this
    > is the default for most modern systems) and something blocks the ICMP
    > "Fragmentation needed but Don't Fragment bit set" error. The datagram is
    > discarded, resend, discarded, repeat until time out. This is called a PMTU
    > blackhole.
    >
    > But I seem to recall that Windows sometimes lowers the MTU on retries to
    > get around such blackholes, maybe this explains the behaviour you are
    > seeing.


    Yes. "PMTU black hole router detection"

    e.g. http://support.microsoft.com/kb/925280

    That article describes the response as reducing the MTU to 536 and
    clearing the DF bit. [It is plausible that the MTU is a typo
    for MSS or that 536 is a typo for 576 -- 536 seems like a strange
    number]

    Packet capture should provide conclusive evidence whether this is
    taking place.

+ Reply to Thread