Unexpected CPU Usage - TCP-IP

This is a discussion on Unexpected CPU Usage - TCP-IP ; All: I'm having some troubles explaining what I consider high CPU usage (as reported from time, prstat, or top) when running a very simple UDP network program: #include #include #include #include #include int main(int argc, char ** argv) { char ...

+ Reply to Thread
Results 1 to 7 of 7

Thread: Unexpected CPU Usage

  1. Unexpected CPU Usage

    All:

    I'm having some troubles explaining what I consider high CPU usage (as
    reported from time, prstat, or top) when running a very simple UDP
    network program:

    #include
    #include
    #include
    #include
    #include

    int main(int argc, char ** argv)
    {
    char * aPacket = malloc(4 * (1 << 20));
    socklen_t socklen = sizeof(struct sockaddr_in);

    /* Create our socket to send data */
    int s = socket(PF_INET, SOCK_DGRAM, IPPROTO_UDP);
    if (s == -1)
    {
    printf("ERROR: Unable to establish socket.\n");;
    }

    /* Setup out destination address */
    struct sockaddr_in sock_server;
    memset(&sock_server, 0, socklen);
    sock_server.sin_family = AF_INET;
    sock_server.sin_port = htons(50000);
    sock_server.sin_addr.s_addr = inet_addr("192.168.1.110");

    for (;
    {
    if (sendto(s, aPacket, 65500, 0, &sock_server, socklen) ==
    -1)
    {
    printf("ERROR: Could not send packet\n");
    }

    usleep(10000); /* 10 ms sleep */
    }

    return 0;
    }

    On my Pentium 4 running Solaris 10, this program reports about 4.0%
    CPU usage for this extremely simple program. When I change the
    '192.168.1.110' to an address that doesn't exist on my network, the
    usage is about 0.3% CPU usage which is closer to what I'd expect. I
    don't have alot of experience working with *nix networking or UDP, so
    I was hoping someone could shed some light on what's going on here. I
    see similar high numbers running Ubuntu on my Athlon 64. Is this
    expected?

    Thanks,

    - Bryan


  2. Re: Unexpected CPU Usage

    On May 27, 11:51 pm, Bryan.Be...@gmail.com wrote:
    > All:
    >
    > I'm having some troubles explaining what I consider high CPU usage (as
    > reported from time, prstat, or top) when running a very simple UDP
    > network program:
    >
    > #include
    > #include
    > #include
    > #include
    > #include
    >
    > int main(int argc, char ** argv)
    > {
    > char * aPacket = malloc(4 * (1 << 20));
    > socklen_t socklen = sizeof(struct sockaddr_in);
    >
    > /* Create our socket to send data */
    > int s = socket(PF_INET, SOCK_DGRAM, IPPROTO_UDP);
    > if (s == -1)
    > {
    > printf("ERROR: Unable to establish socket.\n");;
    > }
    >
    > /* Setup out destination address */
    > struct sockaddr_in sock_server;
    > memset(&sock_server, 0, socklen);
    > sock_server.sin_family = AF_INET;
    > sock_server.sin_port = htons(50000);
    > sock_server.sin_addr.s_addr = inet_addr("192.168.1.110");
    >
    > for (;
    > {
    > if (sendto(s, aPacket, 65500, 0, &sock_server, socklen) ==
    > -1)
    > {
    > printf("ERROR: Could not send packet\n");
    > }
    >
    > usleep(10000); /* 10 ms sleep */
    > }
    >
    > return 0;
    >
    > }
    >
    > On my Pentium 4 running Solaris 10, this program reports about 4.0%
    > CPU usage for this extremely simple program. When I change the
    > '192.168.1.110' to an address that doesn't exist on my network, the
    > usage is about 0.3% CPU usage which is closer to what I'd expect. I
    > don't have alot of experience working with *nix networking or UDP, so
    > I was hoping someone could shed some light on what's going on here. I
    > see similar high numbers running Ubuntu on my Athlon 64. Is this
    > expected?
    >
    > Thanks,
    >
    > - Bryan


    could you figure that out?? i am facing a similar problem


  3. Re: Unexpected CPU Usage

    On 27 May 2007, Bryan.Berns@gmail.com wrote:

    > I'm having some troubles explaining what I consider high CPU usage
    > (as reported from time, prstat, or top) when running a very simple
    > UDP network program:


    [code snipped]

    > On my Pentium 4 running Solaris 10, this program reports about
    > 4.0% CPU usage for this extremely simple program. When I change
    > the '192.168.1.110' to an address that doesn't exist on my
    > network, the usage is about 0.3% CPU usage which is closer to what
    > I'd expect. I don't have alot of experience working with *nix
    > networking or UDP, so I was hoping someone could shed some light
    > on what's going on here. I see similar high numbers running
    > Ubuntu on my Athlon 64. Is this expected?


    Why do you think code simplicity should translate into low CPU usage?
    Could you say how much CPU you expected this program to use and why?

    But to answer your question, let's start with why sending to a
    nonexistent address takes less CPU time. If the nonexistent address
    is on the same subnet (as appears to be the case here), no UDP packet
    will ever be sent on the network! The sending machine will ARP and
    get no reply, then just start throwing away the data. Not much
    processing involved in that.

    If there *is* a machine at the address, you're generating 100 * 65500
    * 8 = 52400000 bits per seconds = 52.4 Mbps. That's not an
    insignificant amount of traffic. Here's a rough estimate of what
    sending that involves. Assuming an MTU of 1500 with IP header = 20,
    65500 byte sends + 8 byte UDP header, you get 1480 bytes of data per
    transmission unit or 45 packets per sendto() call, 100 calls per
    second = 4500 packets per second. On many systems, that will
    generate just about 4500 transmit interrupts per second, and
    interrupt handling is quite costly in terms of CPU time. If there's
    no program listening on the port on the target machine, you'll also
    likely receive some number of ICMP port unreachable messages from the
    target which must also be received and processed. Is that more work
    than you thought there'd be from your simple program?

    BTW, I don't know how system processing time (like interrupt handling
    and IP stack processing) is assigned to each program on Solaris using
    prstat or top, so I don't know if what I've said really explains what
    you're actually seeing ;-)

    Dave

    --
    D.a.v.i.d T.i.k.t.i.n
    t.i.k.t.i.n [at] a.d.v.a.n.c.e.d.r.e.l.a.y [dot] c.o.m

  4. Re: Unexpected CPU Usage

    On May 27, 11:51 am, Bryan.Be...@gmail.com wrote:

    > On my Pentium 4 running Solaris 10, this program reports about 4.0%
    > CPU usage for this extremely simple program. When I change the
    > '192.168.1.110' to an address that doesn't exist on my network, the
    > usage is about 0.3% CPU usage which is closer to what I'd expect. I
    > don't have alot of experience working with *nix networking or UDP, so
    > I was hoping someone could shed some light on what's going on here. I
    > see similar high numbers running Ubuntu on my Athlon 64. Is this
    > expected?


    Calculate how much work this program is doing. Since this program is
    using 4% of the CPU, that means that the CPU as a whole can do about
    25 times that much work. So multiply your first calcuation by 25. Then
    ask if that's a reasonable amount of data to transfer to a network
    card across whatever bus the network card is connected to.

    DS


  5. Re: Unexpected CPU Usage

    > If there *is* a machine at the address, you're generating 100 * 65500
    > * 8 = 52400000 bits per seconds = 52.4 Mbps. That's not an
    > insignificant amount of traffic. Here's a rough estimate of what
    > sending that involves. Assuming an MTU of 1500 with IP header = 20,
    > 65500 byte sends + 8 byte UDP header, you get 1480 bytes of data per
    > transmission unit or 45 packets per sendto() call, 100 calls per
    > second = 4500 packets per second. On many systems, that will
    > generate just about 4500 transmit interrupts per second, and
    > interrupt handling is quite costly in terms ofCPUtime. If there's
    > no program listening on the port on the target machine, you'll also
    > likely receive some number of ICMP port unreachable messages from the
    > target which must also be received and processed. Is that more work
    > than you thought there'd be from your simple program?


    Thanks for your reply.

    While your path of reasoning makes sense to me, it confuses me why
    other operating systems like Windows seem unhindered when performing
    intense network file transfers. I would have expected solaris / linux
    to operate similiarly.

    I've programmed firmware for VME serial cards and my hope would have
    been that packet transmission over ethernet would have been similar in
    terms of controller involvement (i.e. basically sending a transfer
    request with a DMA address, size, protocol, and let the card do all
    the work). It's absurb that the kernel needs to receive an interrupt
    for every transmission unit.


  6. Re: Unexpected CPU Usage

    In article <1182060974.815844.147770@n60g2000hse.googlegroups. com>,
    wrote:

    >> second = 4500 packets per second. On many systems, that will
    >> generate just about 4500 transmit interrupts per second, and


    The words "on many systems" are critical.

    >While your path of reasoning makes sense to me, it confuses me why
    >other operating systems like Windows seem unhindered when performing
    >intense network file transfers. I would have expected solaris / linux
    >to operate similiarly.


    If you think that all Solaris or Linux systems are slower than all
    Windows systems, then you are wrong. All three run on a wide variety
    of hardware. Besides, similar hardware often has quite different drivers.

    >I've programmed firmware for VME serial cards and my hope would have
    >been that packet transmission over ethernet would have been similar in
    >terms of controller involvement (i.e. basically sending a transfer
    >request with a DMA address, size, protocol, and let the card do all
    >the work). It's absurb that the kernel needs to receive an interrupt
    >for every transmission unit.


    It's absurb to over generalize.

    If the kernel cannot get an interrupt for an individual PDU, it must
    either waste too many cycles in polling loops or suffer unacceptable
    latency when is running faster than the wire.

    A lot of people have done a lot of work in UNIX-like systems over the
    decades to miminize network latency and maximize throughput. One
    important tactic is reducing the average number of interrupts per user
    data byte without reducing it too much. I've spent a lot of my time
    on such games, including work on VME RS-232, FDDI, and Ethernet cards
    with and without CPUs. It's easier when the the card has a CPU and
    software or at least writable firmware. It is harder when the hardware
    is stupid or simplistic.


    Vernon Schryver vjs@rhyolite.com

  7. Re: Unexpected CPU Usage

    On 16 Jun 2007, Bryan.Berns@gmail.com wrote:

    >> If there *is* a machine at the address, you're generating 100 *
    >> 65500 * 8 = 52400000 bits per seconds = 52.4 Mbps. That's not an
    >> insignificant amount of traffic. Here's a rough estimate of what
    >> sending that involves. Assuming an MTU of 1500 with IP header =
    >> 20, 65500 byte sends + 8 byte UDP header, you get 1480 bytes of
    >> data per transmission unit or 45 packets per sendto() call, 100
    >> calls per second = 4500 packets per second. On many systems,
    >> that will generate just about 4500 transmit interrupts per
    >> second, and interrupt handling is quite costly in terms
    >> ofCPUtime. If there's no program listening on the port on the
    >> target machine, you'll also likely receive some number of ICMP
    >> port unreachable messages from the target which must also be
    >> received and processed. Is that more work than you thought
    >> there'd be from your simple program?

    >
    > Thanks for your reply.
    >
    > While your path of reasoning makes sense to me, it confuses me why
    > other operating systems like Windows seem unhindered when
    > performing intense network file transfers. I would have expected
    > solaris / linux to operate similiarly.


    I'm not sure what you mean by "unhindered." (4% doesn't seem like
    much a hinderance in a multitasking OS.) Did you run the same test
    on a (similarly configured?) Windows machine and did it use less than
    4% of the CPU?

    > I've programmed firmware for VME serial cards and my hope would
    > have been that packet transmission over ethernet would have been
    > similar in terms of controller involvement (i.e. basically sending
    > a transfer request with a DMA address, size, protocol, and let the
    > card do all the work). It's absurb that the kernel needs to
    > receive an interrupt for every transmission unit.


    I've done some driver writing myself, and one interrupt per PDU is
    common, assuming you actually do have DMA. (Often, the interrupt
    lets the driver immediately reuse the buffer for another PDU.) But I
    want to emphasize that I have *no real information* about the system
    you're running on or whether you're getting 4500 interrupts per
    second or even whether 4500 interrupts per second would use 4% of the
    CPU on Solaris 10 with your particular hardware (whew!). I'm just
    asking why *you* think the result you got seemed anomalous. Why did
    you think your program should use an order of magnitude less CPU than
    it did? What calculation did you do? What assumptions did you make?
    You surely made *some* assumptions. I'm just asking you to state
    them so we can examine them.

    OK, I'm guessing about the interrupts, but Solaris may keep track of
    (or can be made to keep track of) how many interrupts have occurred
    on each IRQ. Why not check?

    Dave

    --
    D.a.v.i.d T.i.k.t.i.n
    t.i.k.t.i.n [at] a.d.v.a.n.c.e.d.r.e.l.a.y [dot] c.o.m

+ Reply to Thread