Delayed acks slowing TCP bulk transfer? - TCP-IP

This is a discussion on Delayed acks slowing TCP bulk transfer? - TCP-IP ; I'm seeing a problem where TCP bulk transfers (SMB file transfers) are extremely slow (~70 KB/sec) because sends are being repeatedly held off waiting for delayed acks. I can see in packet captures that it sends 4 packets quickly then ...

+ Reply to Thread
Results 1 to 10 of 10

Thread: Delayed acks slowing TCP bulk transfer?

  1. Delayed acks slowing TCP bulk transfer?

    I'm seeing a problem where TCP bulk transfers (SMB file transfers) are
    extremely slow (~70 KB/sec) because sends are being repeatedly held off
    waiting for delayed acks. I can see in packet captures that it sends 4
    packets quickly then waits roughly 200 ms before continuing the send.
    >From the application layer, it's doing a single send of much more data

    than that (about 23KB per send). If I disable delayed acks on the
    receiver, performance is fine (several MB/sec). The app essentially
    does the following:

    App receives SMB read request (this packet is less than 64 bytes)
    App does a socket send of ~23 KB and the capture shows:
    #1 1448 payload TCP segment sent
    #2 1448 payload TCP segment sent
    #3 1448 payload TCP segment sent
    #4 1448 payload TCP segment sent
    ~200ms delay (this is what slows everything down)
    ACK received
    TCP segments start sending again

    What I don't understand is why the sender's TCP stack (both sides are
    BSD Unix) is waiting for an ack when the receivers TCP window has
    plenty of room (over 50 KB). Is this expected behavior with TCP?


  2. Re: Delayed acks slowing TCP bulk transfer?

    Hi!
    Track the cwnd on the senders side, that should give you some idea?

    Rajkumar...

    skillzero@gmail.com wrote:
    > I'm seeing a problem where TCP bulk transfers (SMB file transfers) are
    > extremely slow (~70 KB/sec) because sends are being repeatedly held off
    > waiting for delayed acks. I can see in packet captures that it sends 4
    > packets quickly then waits roughly 200 ms before continuing the send.
    > >From the application layer, it's doing a single send of much more data

    > than that (about 23KB per send). If I disable delayed acks on the
    > receiver, performance is fine (several MB/sec). The app essentially
    > does the following:
    >
    > App receives SMB read request (this packet is less than 64 bytes)
    > App does a socket send of ~23 KB and the capture shows:
    > #1 1448 payload TCP segment sent
    > #2 1448 payload TCP segment sent
    > #3 1448 payload TCP segment sent
    > #4 1448 payload TCP segment sent
    > ~200ms delay (this is what slows everything down)
    > ACK received
    > TCP segments start sending again
    >
    > What I don't understand is why the sender's TCP stack (both sides are
    > BSD Unix) is waiting for an ack when the receivers TCP window has
    > plenty of room (over 50 KB). Is this expected behavior with TCP?



  3. Re: Delayed acks slowing TCP bulk transfer?


    skillzero@gmail.com wrote:

    > App receives SMB read request (this packet is less than 64 bytes)
    > App does a socket send of ~23 KB and the capture shows:
    > #1 1448 payload TCP segment sent
    > #2 1448 payload TCP segment sent
    > #3 1448 payload TCP segment sent
    > #4 1448 payload TCP segment sent
    > ~200ms delay (this is what slows everything down)
    > ACK received
    > TCP segments start sending again
    >
    > What I don't understand is why the sender's TCP stack (both sides are
    > BSD Unix) is waiting for an ack when the receivers TCP window has
    > plenty of room (over 50 KB). Is this expected behavior with TCP?


    TCP will not fill the window at full speed because this fails miserably
    over slow connections. There is no reason to think the window size
    tells you anything about the pipe.

    What I don't understand is why the other side does not respond to the
    second packet with an ACK. This seems to be a bug in the other end.

    Both sides can disable Nagle in the hopes that this will help. However,
    this doesn't look like the typical Nagle issue. If both sides send in
    large chunks, and sides strictly alternate transmissions, Nagle
    shouldn't really ever kick in.

    DS


  4. Re: Delayed acks slowing TCP bulk transfer?


    David Schwartz wrote:
    > skillzero@gmail.com wrote:
    >
    > > App receives SMB read request (this packet is less than 64 bytes)
    > > App does a socket send of ~23 KB and the capture shows:
    > > #1 1448 payload TCP segment sent
    > > #2 1448 payload TCP segment sent
    > > #3 1448 payload TCP segment sent
    > > #4 1448 payload TCP segment sent
    > > ~200ms delay (this is what slows everything down)
    > > ACK received
    > > TCP segments start sending again
    > >
    > > What I don't understand is why the sender's TCP stack (both sides are
    > > BSD Unix) is waiting for an ack when the receivers TCP window has
    > > plenty of room (over 50 KB). Is this expected behavior with TCP?

    >
    > TCP will not fill the window at full speed because this fails miserably
    > over slow connections. There is no reason to think the window size
    > tells you anything about the pipe.
    >
    > What I don't understand is why the other side does not respond to the
    > second packet with an ACK. This seems to be a bug in the other end.


    The deferred ack mechanism is usually controlled by both a timer and a
    deferred ack counter.

    Perhaps the receiver's deferred ack mechanism is being triggered by the
    timer (200ms is typical), and not by the deferred ack counter (> 4 is
    typical).

    As the OP notes, the sender's failure to send a 5th segment is a
    critical question.

    If there has been segment loss, or if we're very early in the
    connection (for bulk transfer scenarios), then cwin type stuff would
    apply. Give it time, it should speed up.

    If the transfer continues at 4 segments / 200ms, then I'd start to
    wonder about the sender's transmission window.

    > Both sides can disable Nagle in the hopes that this will help. However,
    > this doesn't look like the typical Nagle issue. If both sides send in
    > large chunks, and sides strictly alternate transmissions, Nagle
    > shouldn't really ever kick in.


    Agreed, this doesn't look like Nagle.


    /chris marget


  5. Re: Delayed acks slowing TCP bulk transfer?

    skillzero@gmail.com wrote:
    > I'm seeing a problem where TCP bulk transfers (SMB file transfers) are
    > extremely slow (~70 KB/sec) because sends are being repeatedly held off
    > waiting for delayed acks. I can see in packet captures that it sends 4
    > packets quickly then waits roughly 200 ms before continuing the send.
    >>From the application layer, it's doing a single send of much more data

    > than that (about 23KB per send). If I disable delayed acks on the
    > receiver, performance is fine (several MB/sec). The app essentially
    > does the following:


    > App receives SMB read request (this packet is less than 64 bytes)
    > App does a socket send of ~23 KB and the capture shows:
    > #1 1448 payload TCP segment sent
    > #2 1448 payload TCP segment sent
    > #3 1448 payload TCP segment sent
    > #4 1448 payload TCP segment sent
    > ~200ms delay (this is what slows everything down)
    > ACK received
    > TCP segments start sending again


    > What I don't understand is why the sender's TCP stack (both sides
    > are BSD Unix) is waiting for an ack when the receivers TCP window
    > has plenty of room (over 50 KB). Is this expected behavior with TCP?


    A few questions:

    *) Which rev of BSD?

    *) How long between subsequent requests? There are newer RFCs where
    people got toghether and decided it would be goodness to reset or
    at least shrink the congestion window (cwnd) if the connection is
    idle for some period of time. That length of time is O(RTT) or
    perhaps O(RTO). If the SMB requests were spread-out enough (in
    time) that the connection goes idle long enough... well, then it
    may always be in some variant of slow-start. Doubleplusungood.

    I could see the same thing happening with NFS over TCP.

    *) Heck, it may be the case that the "is the connection idle"
    heuristic for shrinking cwnd hit once while waiting for a delayed
    ACK, and subsequent delayed ACK delays is just perpetuating it. I
    certainly hope that no stack has that "idleness" measure running
    while there is still data to be sent on the connection...

    *) Can you see the same behaviour with a netperf TCP_RR test with
    appropriate values for -r ? Might be worthwhile to ./configure
    --enable-histogram and --enable-intervals and then you can "pace"
    the transactions to find the point when the cwnd shrinkage kicks-in
    and also its effect on response time.

    rick jones

    given CKO and copyavoidance, as a performance guy I am rather fond of
    ACK avoidance heuristics...in that context, an ACK is just as
    expensive (CPU) as a data segment.

    --
    denial, anger, bargaining, depression, acceptance, rebirth...
    where do you want to be today?
    these opinions are mine, all mine; HP might not want them anyway...
    feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

  6. Re: Delayed acks slowing TCP bulk transfer?


    googlegroups@marget.com wrote:

    > Perhaps the receiver's deferred ack mechanism is being triggered by the
    > timer (200ms is typical), and not by the deferred ack counter (> 4 is
    > typical).


    I was under the impression that during a ramp up, the deferred ack
    counter started at two, precisely to avoid bad interactions with
    slow-start.

    DS


  7. Re: Delayed acks slowing TCP bulk transfer?

    David Schwartz wrote:
    > googlegroups@marget.com wrote:


    >> Perhaps the receiver's deferred ack mechanism is being triggered by
    >> the timer (200ms is typical), and not by the deferred ack counter
    >> (> 4 is typical).


    > I was under the impression that during a ramp up, the deferred ack
    > counter started at two, precisely to avoid bad interactions with
    > slow-start.


    It should, in a "good" implementation of ACK avoidance. Of course, if
    the sending side is using the "reset cwnd after idle" business, and
    the receiver with the ACK avoidance heuristic doesn't have a similar
    analog...

    rick jones
    --
    firebug n, the idiot who tosses a lit cigarette out his car window
    these opinions are mine, all mine; HP might not want them anyway...
    feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

  8. Re: Delayed acks slowing TCP bulk transfer?

    >From what I've read, it looks like I might be running into a "stretch
    ack violation" with the receiving stack.

    Since I can't change the receiving TCP stack, is there anything I can
    do in the sending TCP stack to work better with TCP stacks with the
    "stretch ack violation"?


  9. Re: Delayed acks slowing TCP bulk transfer?


    skillzero@gmail.com wrote:

    > >From what I've read, it looks like I might be running into a "stretch
    > > ack violation" with the receiving stack.


    This seems to be correct. It is also possible that the ACK was sent but
    dropped.

    It also seems strange that the sender waited for the ACK rather than
    filling the window (perhaps with a slowdown, but not a .2s wait).
    Perhaps the sending stack also has some deficiencies, such as throwing
    away congestion and rtt information if the connection goes idle.

    > Since I can't change the receiving TCP stack, is there anything I can
    > do in the sending TCP stack to work better with TCP stacks with the
    > "stretch ack violation"?


    Not easily. I know some coding tricks that you can do in the receiver's
    application code (for example, sending a byte after each chunk is
    received to allow the ACK to piggyback on that data byte). You can turn
    off Nagle (and should, I think) but I doubt it will help much.

    You may be able to change your stack's TCP tuning parameters. This may
    cause worse behavior over low-capacity and/or high-delay links but
    might improve things over high-capacity low-delay links.

    You could also use UDP instead. This puts you in full control over
    transmit pacing and the like, but has obvious disadvantages as well.

    DS


  10. Re: Delayed acks slowing TCP bulk transfer?

    > It also seems strange that the sender waited for the ACK rather than
    > filling the window (perhaps with a slowdown, but not a .2s wait).


    This is also what confuses me. It's very predictable though. With the
    current buffering scheme, it always shows the following pattern
    (timestamps on the left):

    0.401319 Receive SMB read request for 24768 bytes
    0.403291 Send 6 1448 byte TCP segments
    0.403933 Receives ACK
    0.404020 Sends 6 1448 byte TCP segments
    0.404967 Receives ACK
    0.405617 Sends 4 1448 byte TCP segments (last at 0.405927)
    0.600468 Finally receives ACK (~200ms late)...This is the slowdown
    0.601208 Send 1 1448 byte TCP segment (last full one)
    0.601213 Send final 270 byte segment to complete SMB message (PSH set)
    0.601297 Receive ACK

    I see hundreds of back-to-back transactions just like this. I don't see
    why the sender seems to wait for an ACK after only 4 segments near the
    end of each SMB frame when it previously sent 6 segments before each
    ACK earlier in the SMB frame.

    If I can just trick it into sending the final 2 segments of the
    transaction, I think the receiver would deliver the ACK for the final 6
    segments without the delay. I tried increasing the buffer in the
    sending app slightly, but that just moves when it waits for the delayed
    ACK.

    I'm pretty new to looking at the TCP stack code (it's NetBSD 3.0), but
    maybe looking at the congestion window stuff will shed some light on
    why it's seemingly stopping too soon.


+ Reply to Thread