TCP connections hang after 4k data sent - TCP-IP

This is a discussion on TCP connections hang after 4k data sent - TCP-IP ; Hi, I am unable to maintain a TCP connection across the internet to a couple of servers for more than a few KB data sent. After googling the problem, it appears to be very similar to reports of 'mtu' issues, ...

+ Reply to Thread
Results 1 to 17 of 17

Thread: TCP connections hang after 4k data sent

  1. TCP connections hang after 4k data sent

    Hi,

    I am unable to maintain a TCP connection across the internet to a
    couple of servers for more than a few KB data sent.

    After googling the problem, it appears to be very similar to reports
    of 'mtu' issues, however reducing the mtu setting of my local machine,
    the router, and the remote machine seems to make no difference. I'd
    really appreciate any suggestions people can offer...

    My setup:
    The local host: Linux 2.4.27
    ADSL router: USR 9003
    The remote hosts: OpenVZ VPS, running Linux 2.6.x

    The problem (most obvious symptoms):
    With ssh I can connect to either remote host. Everything seems fine
    until at some point the connection hangs (usually, it seems, while
    typing fast). With scp I can copy small files from local host to
    remote host, but for files >=4k (ish) the connection stalls.

    To diagnose the problem, I enabled TCP chargen, echo & discard on one
    of the remote boxes and did a certain amount of testing with netcat.

    I found that for data sent from the remote box to the local box I was
    able to transfer several MB without problem. However, for data sent
    from local box to the remote, the connection hung at around 4k-6k data
    sent.

    A friend opened TCP echo service on his box for me, and I was able to
    echo several MB through it without issue (i.e. both send & receive).

    He also repeated some tests and found no issue with sending from his
    box data to the remote box.

    I have two servers (both VPS's), on different networks but both in the
    USA, and both exhibit the same problem. I am in the UK, as is the
    friend who helped with testing.

    I do not have any other (noticable) problems with the connection on my
    local machine. It is online 24/7 (via the ADSL router), serves a
    'hobby' web-site, and I ssh to it from elsewhere on a regular basis.

    Based on a google search for similar problems,I have tried reducing
    the 'mtu' setting for eth1 on the local box, as well as the eth
    interface on the router, as well as all eth interfaces on the remote
    VPS. None of this seemed to make any difference, although tweaking
    was done with a certain amount of ignorance so I could still be
    persuaded it is an mtu issue...

  2. Re: TCP connections hang after 4k data sent

    Hi,

    This sounds like a TCP receive buffer issue with OpenVZ.

    What is the advertised window size by your VPS ? You can run tcpdump /
    ethereal to get this information.




    news-03@foobar.clara.co.uk wrote
    > Hi,
    >
    > I am unable to maintain a TCP connection across the internet to a
    > couple of servers for more than a few KB data sent.
    >
    > After googling the problem, it appears to be very similar to reports
    > of 'mtu' issues, however reducing the mtu setting of my local machine,
    > the router, and the remote machine seems to make no difference. I'd
    > really appreciate any suggestions people can offer...
    >
    > My setup:
    > The local host: Linux 2.4.27
    > ADSL router: USR 9003
    > The remote hosts: OpenVZ VPS, running Linux 2.6.x
    >
    > The problem (most obvious symptoms):
    > With ssh I can connect to either remote host. Everything seems fine
    > until at some point the connection hangs (usually, it seems, while
    > typing fast). With scp I can copy small files from local host to
    > remote host, but for files >=4k (ish) the connection stalls.
    >
    > To diagnose the problem, I enabled TCP chargen, echo & discard on one
    > of the remote boxes and did a certain amount of testing with netcat.
    >
    > I found that for data sent from the remote box to the local box I was
    > able to transfer several MB without problem. However, for data sent
    > from local box to the remote, the connection hung at around 4k-6k data
    > sent.
    >
    > A friend opened TCP echo service on his box for me, and I was able to
    > echo several MB through it without issue (i.e. both send & receive).
    >
    > He also repeated some tests and found no issue with sending from his
    > box data to the remote box.
    >
    > I have two servers (both VPS's), on different networks but both in the
    > USA, and both exhibit the same problem. I am in the UK, as is the
    > friend who helped with testing.
    >
    > I do not have any other (noticable) problems with the connection on my
    > local machine. It is online 24/7 (via the ADSL router), serves a
    > 'hobby' web-site, and I ssh to it from elsewhere on a regular basis.
    >
    > Based on a google search for similar problems,I have tried reducing
    > the 'mtu' setting for eth1 on the local box, as well as the eth
    > interface on the router, as well as all eth interfaces on the remote
    > VPS. None of this seemed to make any difference, although tweaking
    > was done with a certain amount of ignorance so I could still be
    > persuaded it is an mtu issue...



  3. Re: TCP connections hang after 4k data sent

    On Mon, 05 Jun 2006 00:09:03 +0100, news-03 wrote:

    ....

    > I found that for data sent from the remote box to the local box I was
    > able to transfer several MB without problem. However, for data sent
    > from local box to the remote, the connection hung at around 4k-6k data
    > sent.


    ....

    > Based on a google search for similar problems,I have tried reducing
    > the 'mtu' setting for eth1 on the local box, as well as the eth
    > interface on the router, as well as all eth interfaces on the remote
    > VPS. None of this seemed to make any difference, although tweaking
    > was done with a certain amount of ignorance so I could still be
    > persuaded it is an mtu issue...


    Ethernet has a MTU of 1500. If you have MTU problems, you would not be
    able to get 4K across. I think you should look elsewhere.

    But first thing to do is to get a packet sniffer installed. As most
    (all) of your systems use linux, install ethereal and have a look what
    happens on the packet level.

    If the box doesn't have Xm just use tcpdump (with -s 1500!!!) and -w the
    trace to a file. Then use ethereal on any other box (can even be Windows)
    to look at the trace.

    M4
    --
    Redundancy is a great way to introduce more single points of failure.


  4. Re: TCP connections hang after 4k data sent

    Martijn Lievaart wrote:
    > On Mon, 05 Jun 2006 00:09:03 +0100, news-03 wrote:
    > ...
    >> I found that for data sent from the remote box to the local box I was
    >> able to transfer several MB without problem. However, for data sent
    >> from local box to the remote, the connection hung at around 4k-6k data
    >> sent.

    > ...
    >> Based on a google search for similar problems,I have tried reducing
    >> the 'mtu' setting for eth1 on the local box, as well as the eth
    >> interface on the router, as well as all eth interfaces on the remote
    >> VPS. None of this seemed to make any difference, although tweaking
    >> was done with a certain amount of ignorance so I could still be
    >> persuaded it is an mtu issue...


    > Ethernet has a MTU of 1500. If you have MTU problems, you would not be
    > able to get 4K across. I think you should look elsewhere.


    Well, _IP_ over Ethernet has an MTU of 1500 bytes Indeed they
    probably need to look elsewhere. It is interesting (to me anyway)
    that the limit on the initial congestion window over a 1500 byte MTU
    IP netowrk is also very close to 4K - at 4380 bytes.

    > If the box doesn't have Xm just use tcpdump (with -s 1500!!!) and -w
    > the trace to a file. Then use ethereal on any other box (can even be
    > Windows) to look at the trace.


    Why the snapsize of 1500? I would think that anything beyond the TCP
    headers wuldn't be all that interesting here.

    rick jones
    --
    The glass is neither half-empty nor half-full. The glass has a leak.
    The real question is "Can it be patched?"
    these opinions are mine, all mine; HP might not want them anyway...
    feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

  5. Re: TCP connections hang after 4k data sent

    In article ,
    Rick Jones wrote:
    >Martijn Lievaart wrote:
    >> On Mon, 05 Jun 2006 00:09:03 +0100, news-03 wrote:


    >> Ethernet has a MTU of 1500. If you have MTU problems, you would not be
    >> able to get 4K across. I think you should look elsewhere.

    >
    >Well, _IP_ over Ethernet has an MTU of 1500 bytes Indeed they
    >probably need to look elsewhere.


    What about jumbo frames?

    Despite the reservations of most contributors to the IETF pppext
    mailing list, there is talk about using jumbo frames to patch the
    minor braindamage of the PPPoE 1492 MTU. (The major braindamage
    of PPPoE is that PPPoE was ever written down, not to mention allowed
    to escape into the wild.)


    Vernon Schryver vjs@rhyolite.com

  6. Re: TCP connections hang after 4k data sent

    Vernon Schryver wrote:
    > In article ,
    > Rick Jones wrote:
    >>Martijn Lievaart wrote:
    >>> On Mon, 05 Jun 2006 00:09:03 +0100, news-03 wrote:


    >>> Ethernet has a MTU of 1500. If you have MTU problems, you would
    >>> not be able to get 4K across. I think you should look elsewhere.

    >>
    >>Well, _IP_ over Ethernet has an MTU of 1500 bytes Indeed they
    >>probably need to look elsewhere.


    > What about jumbo frames?


    What about them?-) I think they are fine and dandy, and consider a
    9000 byte IP MTU to be the de facto definition of Jumbo Frames, but
    since there isn't any de jure standard - esp the IEEE - it seems that
    one can have as many definitions of "Jubmo Frames" as there are
    vendors.

    And, to harken back to other fun sore points of mine if it
    ain't in the IEEE spec it ain't "Ethernet" right?-)

    > Despite the reservations of most contributors to the IETF pppext
    > mailing list, there is talk about using jumbo frames to patch the
    > minor braindamage of the PPPoE 1492 MTU. (The major braindamage of
    > PPPoE is that PPPoE was ever written down, not to mention allowed to
    > escape into the wild.)


    Somehow I wonder if that might not be adding fuel to the fire - as I
    doubt they would have "Jumbo Frame => 9000 byte MTU" and so _further_
    confuse things wrt "What is the size of a Jumbo Frame."

    rick jones
    --
    oxymoron n, commuter in a gas-guzzling luxury SUV with an American flag
    these opinions are mine, all mine; HP might not want them anyway...
    feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

  7. Re: TCP connections hang after 4k data sent

    On Tue, 13 Jun 2006 17:06:26 +0000, Rick Jones wrote:

    >> Ethernet has a MTU of 1500. If you have MTU problems, you would not be
    >> able to get 4K across. I think you should look elsewhere.

    >
    > Well, _IP_ over Ethernet has an MTU of 1500 bytes Indeed they


    Looking at the name for this group made me assume IP Actually an
    interesting question, what's the MTU for other protocols on ethernet?

    >> If the box doesn't have Xm just use tcpdump (with -s 1500!!!) and -w
    >> the trace to a file. Then use ethereal on any other box (can even be
    >> Windows) to look at the trace.

    >
    > Why the snapsize of 1500? I would think that anything beyond the TCP
    > headers wuldn't be all that interesting here.


    Probably not, but I've more than once needed to see the data in the stream
    (wshich I could not foresee), so I make a habit of capturing the whole
    frame. Actually I should just use 0 (captrue whole packet), but I always
    forget that.

    M4
    --
    Redundancy is a great way to introduce more single points of failure.


  8. Re: TCP connections hang after 4k data sent

    Martijn Lievaart wrote:
    > On Tue, 13 Jun 2006 17:06:26 +0000, Rick Jones wrote:


    >>> Ethernet has a MTU of 1500. If you have MTU problems, you would not be
    >>> able to get 4K across. I think you should look elsewhere.

    >>
    >> Well, _IP_ over Ethernet has an MTU of 1500 bytes Indeed they


    > Looking at the name for this group made me assume IP


    Not that I actually follow his advice consistently, but a wise old
    engineer once taught me the correct spelling is ass-u-me

    > Actually an
    > interesting question, what's the MTU for other protocols on ethernet?


    Good question, I think IP has "suppressed" most other protocols

    rick jones
    --
    The glass is neither half-empty nor half-full. The glass has a leak.
    The real question is "Can it be patched?"
    these opinions are mine, all mine; HP might not want them anyway...
    feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

  9. Re: TCP connections hang after 4k data sent

    "Martijn Lievaart" wrote:

    > Looking at the name for this group made me assume IP Actually an
    > interesting question, what's the MTU for other protocols on ethernet?


    The 1500 byte limit *is* Ethernet's, so any protocol running over
    Ethernet, in principle, has that limit.

    However, many (not all by any means) Ethernet switches permit the use of
    jumbo frames. These are vendor specific enhancements, all non-standard.
    I guess you'd have to stick with the Ethernet "type" format to use jumbo
    frames, though, else there's no way to indicate "length" beyond 1535
    bytes, so LLC/SNAP encapsulation is out, I think.

    There ought to be a "type" that indicates LLC/SNAP, but I don't see it
    in

    http://www.iana.org/assignments/ethernet-numbers

    Bert


  10. Re: TCP connections hang after 4k data sent

    Albert Manfredi wrote:
    > "Martijn Lievaart" wrote:


    >> Looking at the name for this group made me assume IP Actually an
    >> interesting question, what's the MTU for other protocols on ethernet?


    > The 1500 byte limit *is* Ethernet's, so any protocol running over
    > Ethernet, in principle, has that limit.


    Indeed, my mistake.

    rick jones
    --
    denial, anger, bargaining, depression, acceptance, rebirth...
    where do you want to be today?
    these opinions are mine, all mine; HP might not want them anyway...
    feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

  11. Re: TCP connections hang after 4k data sent

    In article ,
    Albert Manfredi wrote:

    >There ought to be a "type" that indicates LLC/SNAP, but I don't see it
    >in
    >
    >http://www.iana.org/assignments/ethernet-numbers


    There are a few Ethernet and even a couple of LLC/SNAP numbers that
    IANA can speak authoritatively about, but for general ITU or IEEE
    numbers, I'd look elsewhere. The IANA and IETF are not the IEEE, ITU
    or ISO, which might be why that page says:

    The following list of EtherTypes is contributed unverified information
    from various sources.

    Besides, wasn't IP/LLC/SNAP/802.3 officially deprecated by the IETF a
    few years ago? Perhaps not, because I can't seem to find it in
    rfc-index.txt


    Vernon Schryver vjs@rhyolite.com

  12. Re: TCP connections hang after 4k data sent

    On Wed, 14 Jun 2006 21:46:45 +0000, Albert Manfredi wrote:

    > "Martijn Lievaart" wrote:
    >
    >> Looking at the name for this group made me assume IP Actually an
    >> interesting question, what's the MTU for other protocols on ethernet?

    >
    > The 1500 byte limit *is* Ethernet's, so any protocol running over
    > Ethernet, in principle, has that limit.


    But 802.1q? How does that work? Hmmm.

    = IEEE Std 802.3ac-1998 defines an extension to the normal 802.3 maximum
    = frame size for the specific pur- pose of accommodating the additional
    = octets of the VLAN tag header. The example frame translations in this
    = annex make use of this extension to the 802.3 frame size.

    So it seems the limit for payload is 1500, the limit for header+payload
    has been expanded to allow for VLANs?

    M4
    --
    Redundancy is a great way to introduce more single points of failure.


  13. Re: TCP connections hang after 4k data sent

    Hello,

    Martijn Lievaart a écrit :
    >>
    >>The 1500 byte limit *is* Ethernet's, so any protocol running over
    >>Ethernet, in principle, has that limit.

    >
    > But 802.1q? How does that work? Hmmm.


    By inserting 4 extra bytes into the ethernet header while the payload
    maximum length remains 1500.

    [...]
    > So it seems the limit for payload is 1500, the limit for header+payload
    > has been expanded to allow for VLANs?


    Correct.

  14. Re: TCP connections hang after 4k data sent

    Vernon Schryver a écrit :
    >
    > Despite the reservations of most contributors to the IETF pppext
    > mailing list, there is talk about using jumbo frames to patch the
    > minor braindamage of the PPPoE 1492 MTU. (The major braindamage
    > of PPPoE is that PPPoE was ever written down, not to mention allowed
    > to escape into the wild.)


    What's so wrong with PPPoE and the 1492 MTU limit ? Dial-up connections
    on POTS lines have usually a MTU of 576 and just work fine. IP *has* to
    accomodate the fact that the MTU may be lower than 1500.
    IMHO the braindamage is on the side of ISP who "bridge" PPPoE with
    another PPP encapsulation type such as L2TP while advertising a 1500 MTU
    on the L2TP side.

  15. Re: TCP connections hang after 4k data sent

    Vernon Schryver wrote:
    > In article ,
    > Albert Manfredi wrote:
    >
    > >There ought to be a "type" that indicates LLC/SNAP, but I don't see it
    > >in
    > >
    > >http://www.iana.org/assignments/ethernet-numbers

    >
    > There are a few Ethernet and even a couple of LLC/SNAP numbers that
    > IANA can speak authoritatively about, but for general ITU or IEEE
    > numbers, I'd look elsewhere. The IANA and IETF are not the IEEE, ITU
    > or ISO, which might be why that page says:
    >
    > The following list of EtherTypes is contributed unverified information
    > from various sources.


    This discussion ought to be in the Ethernet list, but this is where the
    thread was. So ...

    You will see here:

    http://grouper.ieee.org/groups/802/s.../msg00695.html

    that back in 1999, draft-kaplan-isis-ext-eth-02.txt was written
    proposing to do exactly what I aksed, and for the same purpose. Which
    is, provide an Ethertype which specifies the LLC/SNAP encapsulation, to
    permit individual organizations to ID their own protocol without
    incurring the 1500-byte payload limit of Ethernet.

    The Ethertype proposed back then was 0x8870.

    I haven't seen this Ethertype defined anywhere in current
    documentation, so I assume the effort died on the vine.

    > Besides, wasn't IP/LLC/SNAP/802.3 officially deprecated by the IETF a
    > few years ago? Perhaps not, because I can't seem to find it in
    > rfc-index.txt


    I don't know about "deprecated," but I've been told that IP over
    LLC/SNAP is never used with Ethernet. (Of course, it has to be used
    with Token Ring or FDDI.)

    However, I'm not interested in doing IP over LLC/SNAP. I'm interested
    (and am doing) a special-purpose protocol over LLC/SNAP, but am limited
    to a 1500 byte payload when implementing this over Ethernet.

    It seems silly to have to register every special-purpose protocol with
    the IEEE, or avoid this but then incur the 1500-byte limitation.

    Bert


  16. Re: TCP connections hang after 4k data sent

    In article ,
    Pascal Hambourg wrote:

    >> Despite the reservations of most contributors to the IETF pppext
    >> mailing list, there is talk about using jumbo frames to patch the
    >> minor braindamage of the PPPoE 1492 MTU. (The major braindamage
    >> of PPPoE is that PPPoE was ever written down, not to mention allowed
    >> to escape into the wild.)

    >
    >What's so wrong with PPPoE and the 1492 MTU limit ?


    The trouble is that your home computer is likely to talk to your DSL
    or cable modem with ordinary 802.3 Ethernet and so send 1500 byte
    TCP/IP/Ethernet packets after negotiating a TCP MSS of 1460 with a
    remote TCP host. If there is a cursed PPPoE link in the path, such as
    between your DSL or cable modem and your ISP's real routers, your TCP
    packets will need to be fragmented.

    The ills that come with IP fragmentation include:
    - reduced performance due to
    - lost IP fragments
    - bandwidth spent on increased IP and link layer headers
    - more bandwidth spent on PMTU probing
    - broken applications due to filtering of ICMP packets

    I've not understood enough about the problem at the root of this thread
    to say, but as others have pointed out, it sounds like the ever popular
    result of the PPPoE MTU mixed with ICMP filtering or other things.


    > Dial-up connections
    >on POTS lines have usually a MTU of 576 and just work fine.


    That is as irrelevant as the fact that FDDI rings have a 4K MTU and
    ATM uses 9K.

    Besides, what sort of "dial-up connections on POTS lines" have a 575
    MTU? The default PPP MTU is 1500 and except for PPPoE, is rarely
    otherwise in practice. Perhaps there is confusion between the default
    TCP MSS to distant hosts and the MTU of a PPPoE link.

    > IP *has* to
    >accomodate the fact that the MTU may be lower than 1500.


    Yes, and in the case of PPPoE with IP fragmentation, reduced performance,
    and more serious problems that are often described as with phrases like
    "connections hang after 4k data sent."


    >IMHO the braindamage is on the side of ISP who "bridge" PPPoE with
    >another PPP encapsulation type such as L2TP while advertising a 1500 MTU
    >on the L2TP side.


    That sentence does not make sense to me. There are some things that it
    might mean, but I'm not sure which.

    PPPoE is based on the lazy notion that making a router is too hard, and
    so boxes that shuffle packets between a consumer's Ethernet and a serial
    link or between a serial link and an ISP's real routers ought to be
    stupid, broken (e.g. non-learning) bridges.
    --


    Vernon Schryver vjs@rhyolite.com

  17. Re: TCP connections hang after 4k data sent

    In article <1150406376.086223.5890@y41g2000cwy.googlegroups.co m>,
    Albert Manfredi wrote:

    >> Besides, wasn't IP/LLC/SNAP/802.3 officially deprecated by the IETF a
    >> few years ago? Perhaps not, because I can't seem to find it in
    >> rfc-index.txt

    >
    >I don't know about "deprecated," but I've been told that IP over
    >LLC/SNAP is never used with Ethernet. (Of course, it has to be used
    >with Token Ring or FDDI.)


    Perhaps Rick Jones will say something about the 3000 series.
    --


    Vernon Schryver vjs@rhyolite.com

+ Reply to Thread