tcp window size of 1 - Networking

This is a discussion on tcp window size of 1 - Networking ; Hi guys, I have a strange behavior on a Centos 5 machine. Whenever I connect from the internet to that machine using ssh I get a tcp window size of 1. But when I connect from the local network it ...

+ Reply to Thread
Results 1 to 11 of 11

Thread: tcp window size of 1

  1. tcp window size of 1

    Hi guys,
    I have a strange behavior on a Centos 5 machine. Whenever I connect from
    the internet to that machine using ssh I get a tcp window size of 1. But
    when I connect from the local network it is fine (it's around 30).
    The machine is behind a firewall using NAT to forward any traffic from www
    to the same network card on which the local network resides.
    Any kernel parameter is the default which comes with Centos 5.

    net.ipv4.tcp_rmem = 4096 87380 16777216
    net.ipv4.tcp_wmem = 4096 65536 16777216
    net.ipv4.tcp_mem = 786432 1048576 1572864
    net.core.rmem_default = 110592
    net.core.wmem_default = 110592
    net.core.rmem_max = 16777216
    net.core.wmem_max = 16777216
    net.ipv4.tcp_sack = 1
    net.ipv4.tcp_window_scaling = 1

    Do you have any clue why this happens?
    Thanks for your attention.
    Ciao
    Thorsten

  2. Re: tcp window size of 1

    On Jan 25, 8:34 am, Thorsten Kohlhepp wrote:

    > I have a strange behavior on a Centos 5 machine. Whenever I connect from
    > the internet to that machine using ssh I get a tcp window size of 1. But
    > when I connect from the local network it is fine (it's around 30).
    > The machine is behind a firewall using NAT to forward any traffic from www
    > to the same network card on which the local network resides.
    > Any kernel parameter is the default which comes with Centos 5.


    Sounds like the firewall/NAT device is broken and doesn't understand
    TCP window scaling. Turning off TCP window scaling on the Centos
    machine may workaround the problem.

    DS

  3. Re: tcp window size of 1

    David Schwartz wrote:
    > On Jan 25, 8:34 am, Thorsten Kohlhepp wrote:
    >
    >> I have a strange behavior on a Centos 5 machine. Whenever I connect from
    >> the internet to that machine using ssh I get a tcp window size of 1. But
    >> when I connect from the local network it is fine (it's around 30).
    >> The machine is behind a firewall using NAT to forward any traffic from www
    >> to the same network card on which the local network resides.
    >> Any kernel parameter is the default which comes with Centos 5.

    >
    > Sounds like the firewall/NAT device is broken and doesn't understand
    > TCP window scaling. Turning off TCP window scaling on the Centos
    > machine may workaround the problem.
    >
    > DS

    There are several other Centos machines behind the same firewall and on
    those it is working. I checked if a firewall is running on the machine
    which has that issue and no there is no firewall running.
    If I turn off window scaling I get a window size of 2 and the internet
    connections stop working. It is really weird, I guess it is a kernel
    problem. The machine is running on 2.6.9-42.0.8.ELsmp. Does anyone have
    the same experience?
    Ciao
    thorko

  4. Re: tcp window size of 1

    What is the value of the window scale in the TCP SYNchronize segments?

    rick jones
    --
    firebug n, the idiot who tosses a lit cigarette out his car window
    these opinions are mine, all mine; HP might not want them anyway...
    feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

  5. Re: tcp window size of 1

    Rick Jones wrote:
    > What is the value of the window scale in the TCP SYNchronize segments?
    >
    > rick jones

    From the Centos Machine to the internet wscale of 9 and vice versa 2.
    Is there anything wrong about it? Because on other machines I always get
    2. If it is wrong how can I change it?
    Thanks
    thorko

  6. Re: tcp window size of 1

    Thorsten Kohlhepp wrote:
    > Rick Jones wrote:
    > > What is the value of the window scale in the TCP SYNchronize
    > > segments?

    > From the Centos Machine to the internet wscale of 9 and vice versa
    > 2. Is there anything wrong about it? Because on other machines I
    > always get 2. If it is wrong how can I change it?


    So, if the Centos system has said it is using a WSCALE of 9, and then
    puts a value of 1 in the window field, the actual window will be 1 <<
    9, or 512 bytes. Still seems a little low, especially if the MSS
    exchanged was > 512 (what was the exchanged MSS?). However, in theory
    at least some data should flow, and given Linux's penchant for
    moderating the recvbuf, starting with a small value, I'd expect it to
    grow as data was received.

    On the centos system, what are the sysctl values for:

    net.ipv4.tcp_rmem
    net.ipv4.tcp_wmem
    net.ipv4.tcp_mem
    net.core.rmem_default
    net.core.wmem_default
    net.core.rmem_max
    net.core.wmem_max

    And is ssh making any explicit setsockopt() calls setting SO_SNDBUF
    and/or SO_RCVBUF? That would require taking an strace.

    rick jones
    --
    The computing industry isn't as much a game of "Follow The Leader" as
    it is one of "Ring Around the Rosy" or perhaps "Duck Duck Goose."
    - Rick Jones
    these opinions are mine, all mine; HP might not want them anyway...
    feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

  7. Re: tcp window size of 1

    Rick Jones wrote:
    > On the centos system, what are the sysctl values for:


    > net.ipv4.tcp_rmem
    > net.ipv4.tcp_wmem
    > net.ipv4.tcp_mem
    > net.core.rmem_default
    > net.core.wmem_default
    > net.core.rmem_max
    > net.core.wmem_max


    Belay that, I found your original post.

    > And is ssh making any explicit setsockopt() calls setting SO_SNDBUF
    > and/or SO_RCVBUF? That would require taking an strace.


    That would still be interesting to see.

    For grins, disabling window scaling might be an interesting experiment.

    rick jones
    --
    No need to believe in either side, or any side. There is no cause.
    There's only yourself. The belief is in your own precision. - Jobert
    these opinions are mine, all mine; HP might not want them anyway...
    feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

  8. Re: tcp window size of 1

    Rick Jones wrote:
    > Rick Jones wrote:
    >> On the centos system, what are the sysctl values for:

    >
    >> net.ipv4.tcp_rmem
    >> net.ipv4.tcp_wmem
    >> net.ipv4.tcp_mem
    >> net.core.rmem_default
    >> net.core.wmem_default
    >> net.core.rmem_max
    >> net.core.wmem_max

    >
    > Belay that, I found your original post.
    >
    >> And is ssh making any explicit setsockopt() calls setting SO_SNDBUF
    >> and/or SO_RCVBUF? That would require taking an strace.

    I did an strace on both server and client. It is making an setsockopt
    call but it doesn't set the SNDBUF and RCVBUF. It sets the KEEPALIVE and
    the NODELAY flag but that's it.
    >
    > That would still be interesting to see.
    >
    > For grins, disabling window scaling might be an interesting experiment.
    >
    > rick jones


    The MSS is 1460 and of course it should increase the window size during
    the connection but it doesn't. It stays always at exactly 1 and wscale
    9. It looks like it isn't negotiating anything. AFAIK it should change
    the window size because it is receiving and processing data which will
    fill and clear the buffer during the connection. Correct me if I'm wrong
    but it would mean the buffer on the receiver side can only handle a
    small amount of data before getting jammed and losing data which isn't
    true because the receiving buffer is high enough to keep more than 512
    bytes.
    I've read that some broken routers are rewriting the wcale factor which
    causes the sender and receiver to assume a different window size. This
    ends up in a very slow connection which would make sense in my case,
    because the local network connections are using a much higher window size.

    But I'm not sure how to fix the firewall / router in this case. The
    firewall in between is also a Centos 4.3 machine. I don't know how to
    disable the option to rewrite the wscale factor.

    If I turn off the wscale option on the Centos server which is behind the
    firewall it stops working. I don't get any connection from the internet
    to that machine anymore while the LAN connection stays normally.
    I fear to disable the wscale option on the firewall machine because if
    it drops the internet connection it will be really bad.

    Do you know how I can trace down if this issue is related to the
    firewall or to the server itself?

    Thanks for pointing me into the right direction.
    Ciao
    thorko

  9. Re: tcp window size of 1

    Thorsten Kohlhepp wrote:
    > The MSS is 1460 and of course it should increase the window size during
    > the connection but it doesn't. It stays always at exactly 1 and wscale
    > 9. It looks like it isn't negotiating anything. AFAIK it should change
    > the window size because it is receiving and processing data which will
    > fill and clear the buffer during the connection. Correct me if I'm wrong
    > but it would mean the buffer on the receiver side can only handle a
    > small amount of data before getting jammed and losing data which isn't
    > true because the receiving buffer is high enough to keep more than 512
    > bytes.


    The "linux" stack likes to try to calculate (guess, apply heuristic,
    whatever) what the optimium receive window should be. It does that
    based on the arriving traffic stream. Perhaps there is a Catch-22
    situation.

    IIRC one can disable the "moderation" of the window with a sysctl:

    net.ipv4.tcp_moderate_rcvbuf

    if you set it to zero.

    > I've read that some broken routers are rewriting the wcale factor
    > which causes the sender and receiver to assume a different window
    > size. This ends up in a very slow connection which would make sense
    > in my case, because the local network connections are using a much
    > higher window size.


    I just _love_ firewalls...

    > But I'm not sure how to fix the firewall / router in this case. The
    > firewall in between is also a Centos 4.3 machine. I don't know how
    > to disable the option to rewrite the wscale factor.


    Well as much as I'd like to say point a shotgun at it (did I mention I
    don't like firewalls?-) it would first be best to see whether it is
    actually rewriting the wcale option.

    > If I turn off the wscale option on the Centos server which is behind
    > the firewall it stops working.


    ?!? That is _really_ weird and suggests something is really broken.
    Window scaling is optional - TCP connections should still be able to
    be established even if it is disabled. A trace of that would be good
    to see.

    > I don't get any connection from the internet to that machine anymore
    > while the LAN connection stays normally. I fear to disable the
    > wscale option on the firewall machine because if it drops the
    > internet connection it will be really bad.


    > Do you know how I can trace down if this issue is related to the
    > firewall or to the server itself?


    Well, one experiment is to keep window scale on on your system and
    disable the receive buffer moderation.

    After that, I would suggest tracing on both sides of the firewall to
    see just what it is doing to the headers.

    rick jones
    --
    Wisdom Teeth are impacted, people are affected by the effects of events.
    these opinions are mine, all mine; HP might not want them anyway...
    feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

  10. Re: tcp window size of 1

    Rick Jones wrote:
    > Thorsten Kohlhepp wrote:
    >> The MSS is 1460 and of course it should increase the window size during
    >> the connection but it doesn't. It stays always at exactly 1 and wscale
    >> 9. It looks like it isn't negotiating anything. AFAIK it should change
    >> the window size because it is receiving and processing data which will
    >> fill and clear the buffer during the connection. Correct me if I'm wrong
    >> but it would mean the buffer on the receiver side can only handle a
    >> small amount of data before getting jammed and losing data which isn't
    >> true because the receiving buffer is high enough to keep more than 512
    >> bytes.

    >
    > The "linux" stack likes to try to calculate (guess, apply heuristic,
    > whatever) what the optimium receive window should be. It does that
    > based on the arriving traffic stream. Perhaps there is a Catch-22
    > situation.
    >
    > IIRC one can disable the "moderation" of the window with a sysctl:
    >
    > net.ipv4.tcp_moderate_rcvbuf
    >
    > if you set it to zero.
    >

    That didn't help
    >> I've read that some broken routers are rewriting the wcale factor
    >> which causes the sender and receiver to assume a different window
    >> size. This ends up in a very slow connection which would make sense
    >> in my case, because the local network connections are using a much
    >> higher window size.

    >
    > I just _love_ firewalls...
    >
    >> But I'm not sure how to fix the firewall / router in this case. The
    >> firewall in between is also a Centos 4.3 machine. I don't know how
    >> to disable the option to rewrite the wscale factor.

    >
    > Well as much as I'd like to say point a shotgun at it (did I mention I
    > don't like firewalls?-) it would first be best to see whether it is
    > actually rewriting the wcale option.
    >
    >> If I turn off the wscale option on the Centos server which is behind
    >> the firewall it stops working.

    >
    > ?!? That is _really_ weird and suggests something is really broken.
    > Window scaling is optional - TCP connections should still be able to
    > be established even if it is disabled. A trace of that would be good
    > to see.
    >
    >> I don't get any connection from the internet to that machine anymore
    >> while the LAN connection stays normally. I fear to disable the
    >> wscale option on the firewall machine because if it drops the
    >> internet connection it will be really bad.

    >
    >> Do you know how I can trace down if this issue is related to the
    >> firewall or to the server itself?

    >
    > Well, one experiment is to keep window scale on on your system and
    > disable the receive buffer moderation.
    >
    > After that, I would suggest tracing on both sides of the firewall to
    > see just what it is doing to the headers.
    >
    > rick jones

    I traced it down to the misbehaving Centos server (not the firewall).
    After doing some tcpdump's I saw the misbehaving interface on the Centos
    server. It was a virtual ip on one of the physical interfaces. After
    restarting this specific virtual interface it went back to normal. The
    tcpdump on the firewall didn't indicate that it was rewriting the window
    scale, but it pointed me to the right interface. At the beginning I was
    always thinking it would use the physical interface on the Centos server
    as the outside interface, but it didn't. It was using the virtual
    interface attached to that physical one.

    Anyway I appreciate all your help and your patience, you were pointing
    me into the right direction.
    Thanks a lot
    Ciao
    thorko

  11. Re: tcp window size of 1

    Thorsten Kohlhepp wrote:
    > I traced it down to the misbehaving Centos server (not the
    > firewall). After doing some tcpdump's I saw the misbehaving
    > interface on the Centos server. It was a virtual ip on one of the
    > physical interfaces. After restarting this specific virtual
    > interface it went back to normal. The tcpdump on the firewall didn't
    > indicate that it was rewriting the window scale, but it pointed me
    > to the right interface. At the beginning I was always thinking it
    > would use the physical interface on the Centos server as the outside
    > interface, but it didn't. It was using the virtual interface
    > attached to that physical one.


    > Anyway I appreciate all your help and your patience, you were
    > pointing me into the right direction.


    You're welcome.

    rick jones
    --
    firebug n, the idiot who tosses a lit cigarette out his car window
    these opinions are mine, all mine; HP might not want them anyway...
    feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

+ Reply to Thread