200ms delay sending with WinCE - TCP-IP

This is a discussion on 200ms delay sending with WinCE - TCP-IP ; Hello, I have an application, where a server is hosted on a WinCE 6.0 computer and a client on a WinXP computer. Both are connected via an ethernet cross cable. The client connects to the server and the server begins ...

+ Reply to Thread
Results 1 to 11 of 11

Thread: 200ms delay sending with WinCE

  1. 200ms delay sending with WinCE

    Hello,
    I have an application, where a server is hosted on a WinCE 6.0 computer
    and a client on a WinXP computer. Both are connected via an ethernet
    cross cable. The client connects to the server and the server begins to
    send tiny message (average size ~10 bytes). The server would like to
    send those messages at a frequency of ~20hz. However, the maximum
    frequency the server can send is 5hz because the WinSock API function
    blocks for 200ms while sending. When I modify the client to send a small
    reply after receiving a message, the time it takes for the server to
    send a message drops down to ~1ms. After doing some googleing, I think
    what happens here is that the client receives a tcp package and delays
    the ack of that packet for 200ms and the server waits for the ack of a
    previous send message before sending a second tpc packet.

    Is this waiting for the acknowledge before sending a second package a
    usual behaviour for a tcp stack? If not does someone know a way to alter
    this behaviour?

    best regards,
    Torsten

    --
    kostenlose Wirtschaftssimulation: http://www.financial-rumors.de


  2. Re: 200ms delay sending with WinCE

    Torsten Robitzki wrote:
    > I have an application, where a server is hosted on a WinCE 6.0
    > computer and a client on a WinXP computer. Both are connected via an
    > ethernet cross cable. The client connects to the server and the
    > server begins to send tiny message (average size ~10 bytes). The
    > server would like to send those messages at a frequency of
    > ~20hz. However, the maximum frequency the server can send is 5hz
    > because the WinSock API function blocks for 200ms while
    > sending. When I modify the client to send a small reply after
    > receiving a message, the time it takes for the server to send a
    > message drops down to ~1ms. After doing some googleing, I think what
    > happens here is that the client receives a tcp package and delays
    > the ack of that packet for 200ms and the server waits for the ack of
    > a previous send message before sending a second tpc packet.


    > Is this waiting for the acknowledge before sending a second package
    > a usual behaviour for a tcp stack? If not does someone know a way to
    > alter this behaviour?


    Yes, it is normal, and yet, there is a way to alter the behaviour.
    Some related boilerplate I trot-out from time to time:

    > I'm not familiar with this issue, and I'm mostly ignorant about what
    > tcp does below the sockets interface. Can anybody briefly explain what
    > "nagle" is, and how and when to turn it off? Or point me to the
    > appropriate manual.


    In broad terms, whenever an application does a send() call, the logic
    of the Nagle algorithm is supposed to go something like this:

    1) Is the quantity of data in this send, plus any queued, unsent data,
    greater than the MSS (Maximum Segment Size) for this connection? If
    yes, send the data in the user's send now (modulo any other
    constraints such as receiver's advertised window and the TCP
    congestion window). If no, go to 2.

    2) Is the connection to the remote otherwise idle? That is, is there
    no unACKed data outstanding on the network. If yes, send the data in
    the user's send now. If no, queue the data and wait. Either the
    application will continue to call send() with enough data to get to a
    full MSS-worth of data, or the remote will ACK all the currently sent,
    unACKed data, or our retransmission timer will expire.

    Now, where applications run into trouble is when they have what might
    be described as "write, write, read" behaviour, where they present
    logically associated data to the transport in separate 'send' calls
    and those sends are typically less than the MSS for the connection.
    It isn't so much that they run afoul of Nagle as they run into issues
    with the interaction of Nagle and the other heuristics operating on
    the remote. In particular, the delayed ACK heuristics.

    When a receiving TCP is deciding whether or not to send an ACK back to
    the sender, in broad handwaving terms it goes through logic similar to
    this:

    a) is there data being sent back to the sender? if yes, piggy-back the
    ACK on the data segment.

    b) is there a window update being sent back to the sender? if yes,
    piggy-back the ACK on the window update.

    c) has the standalone ACK timer expired.

    Window updates are generally triggered by the following heuristics:

    i) would the window update be for a non-trivial fraction of the window
    - typically somewhere at or above 1/4 the window, that is, has the
    application "consumed" at least that much data? if yes, send a
    window update. if no, check ii.

    ii) would the window update be for, the application "consumed," at
    least 2*MSS worth of data? if yes, send a window update, if no wait.

    Now, going back to that write, write, read application, on the sending
    side, the first write will be transmitted by TCP via logic rule 2 -
    the connection is otherwise idle. However, the second small send will
    be delayed as there is at that point unACKnowledged data outstanding
    on the connection.

    At the receiver, that small TCP segment will arrive and will be passed
    to the application. The application does not have the entire app-level
    message, so it will not send a reply (data to TCP) back. The typical
    TCP window is much much larger than the MSS, so no window update would
    be triggered by heuristic i. The data just arrived is < 2*MSS, so no
    window update from heuristic ii. Since there is no window update, no
    ACK is sent by heuristic b.

    So, that leaves heuristic c - the standalone ACK timer. That ranges
    anywhere between 50 and 200 milliseconds depending on the TCP stack in
    use.

    If you've read this far now we can take a look at the effect of
    various things touted as "fixes" to applications experiencing this
    interaction. We take as our example a client-server application where
    both the client and the server are implemented with a write of a small
    application header, followed by application data. First, the
    "default" case which is with Nagle enabled (TCP_NODELAY _NOT_ set) and
    with standard ACK behaviour:

    Client Server
    Req Header ->
    <- Standalone ACK after Nms
    Req Data ->
    <- Possible standalone ACK
    <- Rsp Header
    Standalone ACK ->
    <- Rsp Data
    Possible standalone ACK ->


    For two "messages" we end-up with at least six segments on the wire.
    The possible standalone ACKs will depend on whether the server's
    response time, or client's think time is longer than the standalone
    ACK interval on their respective sides. Now, if TCP_NODELAY is set we
    see:


    Client Server
    Req Header ->
    Req Data ->
    <- Possible Standalone ACK after Nms
    <- Rsp Header
    <- Rsp Data
    Possible Standalone ACK ->

    In theory, we are down two four segments on the wire which seems good,
    but frankly we can do better. First though, consider what happens
    when someone disables delayed ACKs

    Client Server
    Req Header ->
    <- Immediate Standalone ACK
    Req Data ->
    <- Immediate Standalone ACK
    <- Rsp Header
    Immediate Standalone ACK ->
    <- Rsp Data
    Immediate Standalone ACK ->

    Now we definitly see 8 segments on the wire. It will also be that way
    if both TCP_NODELAY is set and delayed ACKs are disabled.

    How about if the application did the "right" think in the first place?
    That is sent the logically associated data at the same time:


    Client Server
    Request ->
    <- Possible Standalone ACK
    <- Response
    Possible Standalone ACK ->

    We are down to two segments on the wire.

    For "small" packets, the CPU cost is about the same regardless of data
    or ACK. This means that the application which is making the propper
    gathering send call will spend far fewer CPU cycles in the networking
    stack.



    Now, in your case the boilerplate is not complete - since your's is a
    paced unidirectional stream of logically unrelated small messages,
    setting TCP_NODELAY is the correct course of action. Keep in mind
    though that things like lost TCP segments may put some serious
    "hiccups" into your data stream...

    rick jones
    --
    No need to believe in either side, or any side. There is no cause.
    There's only yourself. The belief is in your own precision. - Jobert
    these opinions are mine, all mine; HP might not want them anyway...
    feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

  3. Re: 200ms delay sending with WinCE

    Hi Rick,
    thank you for taking the time to send such a complete reply. Seems like
    there is still some of the "old" DEC spirit left at hp ;-)


    Rick Jones wrote:

    > Torsten Robitzki wrote:


    >>Is this waiting for the acknowledge before sending a second package
    >>a usual behaviour for a tcp stack? If not does someone know a way to
    >>alter this behaviour?


    <...>

    > In broad terms, whenever an application does a send() call, the logic
    > of the Nagle algorithm is supposed to go something like this:
    >
    > 1) <...>
    >
    > 2) Is the connection to the remote otherwise idle? That is, is there
    > no unACKed data outstanding on the network. If yes, send the data in
    > the user's send now. If no, queue the data and wait. Either the
    > application will continue to call send() with enough data to get to a
    > full MSS-worth of data, or the remote will ACK all the currently sent,
    > unACKed data, or our retransmission timer will expire.


    I think I'm observing the "If no" part. But how can the application
    queue more data by calling send(), if the previous call to send() blocks
    for 200ms?

    > If you've read this far


    I did ;-)

    <...>

    > Now, in your case the boilerplate is not complete - since your's is a
    > paced unidirectional stream of logically unrelated small messages,
    > setting TCP_NODELAY is the correct course of action. Keep in mind
    > though that things like lost TCP segments may put some serious
    > "hiccups" into your data stream...


    Sorry, I forgot to mention, that I've disabled the nagle algorithm
    already. It was just a test, but it should be still disabled when I've
    did my last test, with the client sending a little message in response
    to a received message. I will do the same test with nagle enabled tomorrow.

    best regards
    Torsten

    --
    kostenlose Wirtschaftssimulation: http://www.financial-rumors.de


  4. Re: 200ms delay sending with WinCE

    On Oct 28, 12:33*pm, Torsten Robitzki wrote:

    > I have an application, where a server is hosted on a WinCE 6.0 computer
    > and a client on a WinXP computer. Both are connected via an ethernet
    > cross cable. The client connects to the server and the server begins to
    > send tiny message (average size ~10 bytes). The server would like to
    > send those messages at a frequency of ~20hz. However, the maximum
    > frequency the server can send is 5hz because the WinSock API function
    > blocks for 200ms while sending. When I modify the client to send a small
    > reply after receiving a message, the time it takes for the server to
    > send a message drops down to ~1ms. After doing some googleing, I think
    > what happens here is that the client receives a tcp package and delays
    > the ack of that packet for 200ms and the server waits for the ack of a
    > previous send message before sending a second tpc packet.


    This is good. It results in a smaller number of large packets which is
    superior to a large number of small packets.

    > Is this waiting for the acknowledge before sending a second package a
    > usual behaviour for a tcp stack? If not does someone know a way to alter
    > this behaviour?


    This is correct and usual behavior. Why do you think you want to
    change it?

    If you must change it, the best solution depends on a lot of details
    about your application that you haven't told us. Does it have to run
    over WANs or just LANs? How would you like it to handle congestion?
    Would you prefer congestion cause data to be delayed? Or would you
    prefer that data be lost in exchange for getting newer data faster?

    The solution might be (in order of probability):

    1) Leave it as is.

    2) Send a reply to each small message.

    3) Switch to UDP.

    4) Disable Nagle.

    DS

  5. Re: 200ms delay sending with WinCE

    Torsten Robitzki wrote:
    > Hi Rick,
    > thank you for taking the time to send such a complete reply.


    You are welcome.

    > Seems like there is still some of the "old" DEC spirit left at hp
    > ;-)


    There may be, I however am pre-merger HP not DEC

    rick jones
    --
    oxymoron n, Hummer H2 with California Save Our Coasts and Oceans plates
    these opinions are mine, all mine; HP might not want them anyway...
    feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

  6. Re: 200ms delay sending with WinCE

    Hi David,

    David Schwartz wrote:

    > On Oct 28, 12:33 pm, Torsten Robitzki wrote:
    >>Is this waiting for the acknowledge before sending a second package a
    >>usual behaviour for a tcp stack? If not does someone know a way to alter
    >>this behaviour?

    >
    >
    > This is correct and usual behavior. Why do you think you want to
    > change it?


    My problem was/is that a call to send() blocks for 200ms and this allows
    a maximum of 5 send per second. I guess that wasn't clear from my
    initial post. A delay of 200ms is acceptable.

    But the full truth is that it's not send() that blocks for 200ms but my
    self written send() wrapper is blocking for 200ms. The wrapper calls
    WSASend, with the passed socket being an overlapped socket and an
    overlapped structure is passed. When there is an io pending, the
    function returns with an error code of WSA_IO_PENDING. Then the function
    waits for the event that was passed to WSASend to become active by
    calling WSAWaitForMultipleEvents(). And this call to
    WSAWaitForMultipleEvents() is then blocking for 200ms. I use this
    approach because I want to be able to kick a thread blocking on send()
    out of this blocking function call. So I add a second event to
    WSAWaitForMultipleEvents() and set this event if the thread should be
    unblocked.

    So it boils down to the problem of how to cancel a blocking call to send().

    > If you must change it, the best solution depends on a lot of details
    > about your application that you haven't told us. Does it have to run
    > over WANs or just LANs?


    Just LANs. This is the connection between an auto pilot for ships and
    the control unit for that auto pilot.

    > How would you like it to handle congestion?
    > Would you prefer congestion cause data to be delayed? Or would you
    > prefer that data be lost in exchange for getting newer data faster?


    The later would be better.

    best regards
    Torsten

    --
    kostenlose Wirtschaftssimulation: http://www.financial-rumors.de


  7. Re: 200ms delay sending with WinCE

    Torsten Robitzki wrote:
    > Hi David,


    > David Schwartz wrote:


    > > On Oct 28, 12:33 pm, Torsten Robitzki wrote:
    > >>Is this waiting for the acknowledge before sending a second package a
    > >>usual behaviour for a tcp stack? If not does someone know a way to alter
    > >>this behaviour?

    > >
    > >
    > > This is correct and usual behavior. Why do you think you want to
    > > change it?


    > My problem was/is that a call to send() blocks for 200ms and this allows
    > a maximum of 5 send per second. I guess that wasn't clear from my
    > initial post. A delay of 200ms is acceptable.


    > But the full truth is that it's not send() that blocks for 200ms but my
    > self written send() wrapper is blocking for 200ms. The wrapper calls
    > WSASend, with the passed socket being an overlapped socket and an
    > overlapped structure is passed. When there is an io pending, the
    > function returns with an error code of WSA_IO_PENDING. Then the function
    > waits for the event that was passed to WSASend to become active by
    > calling WSAWaitForMultipleEvents(). And this call to
    > WSAWaitForMultipleEvents() is then blocking for 200ms. I use this
    > approach because I want to be able to kick a thread blocking on send()
    > out of this blocking function call. So I add a second event to
    > WSAWaitForMultipleEvents() and set this event if the thread should be
    > unblocked.


    Hmm, then I'm not sure that setting TCP_NODELAY will help if the calls
    you are making end-up waiting for the previous send to be "completed"
    and that is defined by the remote's ACK arriving. In that case, apart
    from switching to UDP (which I suspect David will suggest based on
    what you say below) your only recourse is to either find a way to
    allow multiple "sends" to be outstanding, or restore your
    application-layer ACK message.

    > So it boils down to the problem of how to cancel a blocking call to
    > send().


    > > If you must change it, the best solution depends on a lot of details
    > > about your application that you haven't told us. Does it have to run
    > > over WANs or just LANs?


    > Just LANs. This is the connection between an auto pilot for ships and
    > the control unit for that auto pilot.


    > > How would you like it to handle congestion?
    > > Would you prefer congestion cause data to be delayed? Or would you
    > > prefer that data be lost in exchange for getting newer data faster?


    > The later would be better.


    Connection between autopilot and control unit sounds rather "mission
    critical..."

    What would the control unit do if it received ~1460/sizeof(message)
    (or more) messages "all at once?" That can happen with TCP -
    TCP_NODELAY or not if there is an interruption in the link between the
    two connection endpoints. Are these messages "Go to full power" or
    "Increase power 5%" similarly are they "Come to heading N" or "Come
    [left|right] N degrees?" I guess the fancy way to ask it is are all
    the messages being sent to the control unit idempotent and don't rely
    on previous messages?

    What would the control unit do if it stopped receiving any of your
    messages?

    What would happen if the control unit received two or more copies of
    the same message?

    Does the autopilot need to have some idea that it is no longer in
    effective communication with the control unit?

    The answers to all of those will factor into a TCP, TCP with
    application ACK, UDP or UDP with application ACK decision... I'm
    betting that SCTP is not available in WinCE.

    rick jones
    --
    firebug n, the idiot who tosses a lit cigarette out his car window
    these opinions are mine, all mine; HP might not want them anyway...
    feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

  8. Re: 200ms delay sending with WinCE


    Torsten Robitzki wrote:

    > But the full truth is that it's not send() that blocks for 200ms but my
    > self written send() wrapper is blocking for 200ms. The wrapper calls
    > WSASend, with the passed socket being an overlapped socket and an
    > overlapped structure is passed. When there is an io pending, the
    > function returns with an error code of WSA_IO_PENDING. Then the function
    > waits for the event that was passed to WSASend to become active by
    > calling WSAWaitForMultipleEvents(). And this call to
    > WSAWaitForMultipleEvents() is then blocking for 200ms. I use this
    > approach because I want to be able to kick a thread blocking on send()
    > out of this blocking function call. So I add a second event to
    > WSAWaitForMultipleEvents() and set this event if the thread should be
    > unblocked.


    Yuck. Do you change the send buffer size?

    Either way, write a sensible wrapper.

    DS

  9. Re: 200ms delay sending with WinCE

    Rick Jones wrote:
    > Connection between autopilot and control unit sounds rather "mission
    > critical..."
    >
    > What would the control unit do if it received ~1460/sizeof(message)
    > (or more) messages "all at once?" That can happen with TCP -
    > TCP_NODELAY or not if there is an interruption in the link between the
    > two connection endpoints.


    I'm quit sure, that the current control unit can handle this.

    > Are these messages "Go to full power" or
    > "Increase power 5%" similarly are they "Come to heading N" or "Come
    > [left|right] N degrees?" I guess the fancy way to ask it is are all
    > the messages being sent to the control unit idempotent and don't rely
    > on previous messages?


    Messages contain absolute information and information that have to be
    displayed by the control unit are send out when they change and
    periodically.

    > What would the control unit do if it stopped receiving any of your
    > messages?


    Showing an alert message.

    > What would happen if the control unit received two or more copies of
    > the same message?


    That wouldn't be a problem as long as these messages are received in the
    order they where send. If they where received in the opposite order,
    there would be an old value displayed for the period this value is send in.

    > Does the autopilot need to have some idea that it is no longer in
    > effective communication with the control unit?


    Yes, therefor sign of life messages are send from the auto pilot to the
    control unit and from the control unit to the auto pilot. If the auto
    pilot detects that it's active control unit fails, an alarm is generated.

    I did the same test today with the auto pilot software running on a
    remote WinXP PC. And exactly the same send() wrapper didn't caused any
    problem. This doesn't solve my problem but let me believe, that the tcp
    implementation of WinCE is at least unusual.

    best regards
    Torsten

    --
    kostenlose Wirtschaftssimulation: http://www.financial-rumors.de


  10. Re: 200ms delay sending with WinCE

    David Schwartz wrote:

    > Torsten Robitzki wrote:


    > Yuck. Do you change the send buffer size?


    Nope, but I will give it a try.

    > Either way, write a sensible wrapper.


    I will do my very best ;-)

    best regards
    Torsten

    --
    kostenlose Wirtschaftssimulation: http://www.financial-rumors.de


  11. Re: 200ms delay sending with WinCE

    Torsten Robitzki wrote:
    > Rick Jones wrote:


    > > What would happen if the control unit received two or more copies
    > > of the same message?


    > That wouldn't be a problem as long as these messages are received in
    > the order they where send. If they where received in the opposite
    > order, there would be an old value displayed for the period this
    > value is send in.


    I was asking a slightly different question - IP is allowed to
    "duplicate" IP datagrams. TCP and its sequence number will deal with
    this if it should happen. UDP has no sequence number and so dealing
    with duplicate datagrams is left to the application. If you were say
    to use UDP you would need some sort of message ID that could be used
    to detect duplicates.

    > I did the same test today with the auto pilot software running on a
    > remote WinXP PC. And exactly the same send() wrapper didn't caused any
    > problem. This doesn't solve my problem but let me believe, that the tcp
    > implementation of WinCE is at least unusual.


    It is entirely possible that the WinCE stack, being designed for
    rather "limited" resources (relatively speaking) is transmitting from
    your applications buffer and may be "limited" in how many sends it can
    process at one time.

    rick jones
    --
    oxymoron n, commuter in a gas-guzzling luxury SUV with an American flag
    these opinions are mine, all mine; HP might not want them anyway...
    feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

+ Reply to Thread