Measuring latency - TCP-IP

This is a discussion on Measuring latency - TCP-IP ; Hi Guys Has anybody here had any success measuring latency for TCP connections on Linux(2.6.9)? It appears that the SO_TIMESTAMP option combined with recvmsg doesn't work for SOCK_STREAM connections. My ultimate goal is to be able to measure the latency ...

+ Reply to Thread
Results 1 to 9 of 9

Thread: Measuring latency

  1. Measuring latency

    Hi Guys

    Has anybody here had any success measuring latency for TCP connections
    on Linux(2.6.9)? It appears that the SO_TIMESTAMP option combined with
    recvmsg doesn't work for SOCK_STREAM connections. My ultimate goal is
    to be able to measure the latency between a packet arriving at a
    socket and my application reading the data so I can determine whether
    the app is network or cpu bound. Currently I attempting to infer this
    by measuring how full the receive buffer is using ioctl and FIONREAD
    immediately after a select call unblocks. My logic is that if the
    level of the socket buffer is gradually increasing then the app is
    probably cpu bound and if the socket is pretty much empty then it is
    network bound.

    Does anybody have any other suggestions, please?

    TIA

    AJ

  2. Re: Measuring latency

    ajcppmod@gmail.com wrote:
    > Has anybody here had any success measuring latency for TCP connections
    > on Linux(2.6.9)? It appears that the SO_TIMESTAMP option combined with
    > recvmsg doesn't work for SOCK_STREAM connections. My ultimate goal is
    > to be able to measure the latency between a packet arriving at a
    > socket and my application reading the data so I can determine whether
    > the app is network or cpu bound. Currently I attempting to infer this
    > by measuring how full the receive buffer is using ioctl and FIONREAD
    > immediately after a select call unblocks. My logic is that if the
    > level of the socket buffer is gradually increasing then the app is
    > probably cpu bound and if the socket is pretty much empty then it is
    > network bound.


    > Does anybody have any other suggestions, please?


    If you were to "speculatively" read from the socket(s) (non-blocking)
    before ever going into select(), the simple act of calling select()
    could be interpreted as being network bound.

    Eg:

    begin:
    do
    readfromsockets
    while (atleastonehaddata)
    select()
    goto begin:

    (or something along those lines).

    If it does not go into select() at all you could assume it was not
    network bound and perhaps bound somehow else (CPU, disc, whatever).

    rick jones
    --
    a wide gulf separates "what if" from "if only"
    these opinions are mine, all mine; HP might not want them anyway...
    feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

  3. Re: Measuring latency

    On Jun 9, 12:10*pm, ajcpp...@gmail.com wrote:
    > Hi Guys
    >
    > Has anybody here had any success measuring latency for TCP connections
    > on Linux(2.6.9)? It appears that the SO_TIMESTAMP option combined with
    > recvmsg doesn't work for SOCK_STREAM connections. My ultimate goal is
    > to be able to measure the latency between a packet arriving at a
    > socket and my application reading the data so I can determine whether
    > the app is network or cpu bound. Currently I attempting to infer this
    > by measuring how full the receive buffer is using ioctl and FIONREAD
    > immediately after a select call unblocks. My logic is that if the
    > level of the socket buffer is gradually increasing then the app is
    > probably cpu bound and if the socket is pretty much empty then it is
    > network bound.
    >
    > Does anybody have any other suggestions, please?
    >
    > TIA
    >
    > AJ


    I would simply use the following heuristic: if the receive buffer is
    almost never empty, I'm CPU bound.

    DS

  4. Re: Measuring latency

    David Schwartz wrote:
    > I would simply use the following heuristic: if the receive buffer is
    > almost never empty, I'm CPU bound.


    Don't you mean 'almost *always* empty'?

    It's impossible to get this information in some environments. In Java
    all you can see is that OP_WRITE doesn't fire, which tells you you are
    network-bound, or receiver-bound.

  5. Re: Measuring latency

    On Jun 10, 8:06*pm, EJP wrote:
    > David Schwartz wrote:
    > > I would simply use the following heuristic: if the receive buffer is
    > > almost never empty, I'm CPU bound.

    >
    > Don't you mean 'almost *always* empty'?


    Yeah, exactly.

    > It's impossible to get this information in some environments. In Java
    > all you can see is that OP_WRITE doesn't fire, which tells you you are
    > network-bound, or receiver-bound.


    True, I guess in that case the thing to look at would be whether the
    receive queue is full when OP_WRITE does fire. If so, your program is
    probably holding up the TCP implementation.

    Of course, if the protocol is query/response, and the other side is
    waiting for you and so can't fill the receive buffer anyway, this
    won't work.

    I guess there's no good general way.

    DS

  6. Re: Measuring latency

    David Schwartz wrote:
    > True, I guess in that case the thing to look at would be whether the
    > receive queue is full when OP_WRITE does fire.


    Sorry, I don't understand that either. Do you mean whether the socket
    receive buffer is full? which is another thing you can't see in Java.
    All you can see is whether OP_READ & OP_WRITE fire.

    > If so, your program is probably holding up the TCP implementation.


    If OP_WRITE isn't firing, you are output bound.

    If OP_READ isn't firing, you are input bound.

    If OP_READ fires and you don't get zero length writes you are not output
    bound: either the reader is fast enough to keep up with you, or you are
    too CPU bound to discover its maximum rate. You can't tell the
    difference between these two situations: all it means is that the
    receiver is >= as fast as you.

    If OP_READ fires and you get a zero length write, you are now output
    bound, and should start selecting for OP_WRITE *instead* of OP_READ.

    If OP_WRITE is firing and OP_READ is also firing you have in my opinion
    a design error. If you're waiting for OP_WRITE it's because you have
    something to write that won't write, and you don't really have any
    business reading from that socket until you've finished the write.
    Otherwise you just piling up data inside your application that would be
    better left in the receive buffer and ignored until you've finished the
    write. It's like letting the passengers off the bus first before
    admitting new ones.

  7. Re: Measuring latency

    On Jun 11, 7:04*pm, EJP wrote:

    > Sorry, I don't understand that either. Do you mean whether the socket
    > receive buffer is full? which is another thing you can't see in Java.
    > All you can see is whether OP_READ & OP_WRITE fire.


    You cannot tell that it was ever full with 100% certainty, but if you
    go to read the socket and get a number of bytes equal to the receive
    buffer size, you can be sure enough for statistical purposes.


    > If OP_WRITE is firing and OP_READ is also firing you have in my opinion
    > a design error. If you're waiting for OP_WRITE it's because you have
    > something to write that won't write, and you don't really have any
    > business reading from that socket until you've finished the write.


    WHAT?! NO!!

    You *MUST* read from the socket.

    > Otherwise you just piling up data inside your application that would be
    > better left in the receive buffer and ignored until you've finished the
    > write. It's like letting the passengers off the bus first before
    > admitting new ones.


    Nonsense! That's insanity. If everyone did that, deadlock would be
    rampant. Imagine if each side has data to send and all stack buffers
    are full. Both sides would be waiting for the other side to receive so
    it could finish sending.

    You *MUST* receive all data, you *MUST NOT* ever wait for the other
    side to receive before you receive unless your protocol explicitly
    allows one side to do this and you are implementing that side *only*.

    DS

  8. Re: Measuring latency

    On 2008-06-10 07:22:29 -0400, David Schwartz said:

    > I would simply use the following heuristic: if the receive buffer is
    > almost never empty, I'm CPU bound.



    Depending on the criticality of the application you're writing (its
    latency characteristics / requirements), I don't trust the application
    itself, to measure itself. The one notable exception to this is the
    smart folks @ 29-West who make a low-latency message bus infrastructure
    that actually does do this quite reliably.

    In other cases, I insist on having packet capture products around to
    take accurate measurements and report back to my infrastructure what's
    up. This is also a good way to build reasonable measurement into your
    software and have a solid basis of comparison. There's too many
    variables in OS implementations, compiler output, and TCP/IP stack
    tweaks, NIC drivers, etc. to just trust a software measurement.

    /dmfh

    --
    _ __ _
    __| |_ __ / _| |_ 01100100 01101101
    / _` | ' \| _| ' \ 01100110 01101000
    \__,_|_|_|_|_| |_||_| dmfh(-2)dmfh.cx


  9. Re: Measuring latency

    David Schwartz wrote:
    > Nonsense! That's insanity. If everyone did that, deadlock would be
    > rampant. Imagine if each side has data to send and all stack buffers
    > are full. Both sides would be waiting for the other side to receive so
    > it could finish sending.


    So both sides should keep reading and run out of memory? Where does this
    policy stop?

    If one peer is writing blindly and not reading whatever it should be
    reading, why should either the network or the other peer have to cope
    with that?

+ Reply to Thread