Gigabit ethernet And Windows Stack Efficiency - TCP-IP

This is a discussion on Gigabit ethernet And Windows Stack Efficiency - TCP-IP ; Hi, I'm investigating methods of moving large amounts of data (say 70Mbytes/sec) between PCs over Gigabit Ethernet. Using the traditional socket programming methods, this incurrs a large (and unacceptable) CPU overhead. As I don't need the ethernet connection for any ...

+ Reply to Thread
Results 1 to 6 of 6

Thread: Gigabit ethernet And Windows Stack Efficiency

  1. Gigabit ethernet And Windows Stack Efficiency

    Hi,
    I'm investigating methods of moving large amounts of data (say
    70Mbytes/sec) between PCs over Gigabit Ethernet. Using the traditional
    socket programming methods, this incurrs a large (and unacceptable)
    CPU overhead.

    As I don't need the ethernet connection for any other use, I assumed
    that I could somehow rewrite the ethernet card driver to avoid the
    Windows Stack code (may using TOE also) to improve things. However, as
    I have no idea how to do this (and I know it's complicated) I wondered
    whether there was already a low level driver available that I could
    use. I assume that the Windows Stack code must sit on a standard
    interface to the ethernet card, so presumably I could interface with
    this interface directly?.

    Any ideas would be appreciated.
    Ta,
    Dave


  2. Re: Gigabit ethernet And Windows Stack Efficiency

    dave_m_moore@post2me.freeserve.co.uk wrote:
    > I'm investigating methods of moving large amounts of data (say
    > 70Mbytes/sec) between PCs over Gigabit Ethernet. Using the
    > traditional socket programming methods, this incurrs a large (and
    > unacceptable) CPU overhead.


    What are some of the constants you are using with the traditional
    socket programming methods? How much data are you presenting to the
    stack at one time? What is the window size you are using? What sort
    of Gigabit Ethernet NIC(s) are you using? Are you using JumboFrames?
    etc etc etc.

    > I assume that the Windows Stack code must sit on a standard
    > interface to the ethernet card, so presumably I could interface with
    > this interface directly?.


    Presumably. You could, and while it may be just between two systems on
    a private network, your replacement for the systems' TCP stacks would
    need to have _some_ sort of flow control and error recovery (lost
    packets) mechanism. You might be able to get away with something very
    simplistic but then again...

    rick jones
    --
    denial, anger, bargaining, depression, acceptance, rebirth...
    where do you want to be today?
    these opinions are mine, all mine; HP might not want them anyway...
    feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

  3. Re: Gigabit ethernet And Windows Stack Efficiency

    On Mar 9, 2:27 pm, dave_m_mo...@post2me.freeserve.co.uk wrote:
    > As I don't need the ethernet connection for any other use, I assumed
    > that I could somehow rewrite the ethernet card driver to avoid the
    > Windows Stack code (may using TOE also) to improve things. However, as
    > I have no idea how to do this (and I know it's complicated) I wondered
    > whether there was already a low level driver available that I could
    > use. I assume that the Windows Stack code must sit on a standard
    > interface to the ethernet card, so presumably I could interface with
    > this interface directly?


    Yes you can, but unfortunately, being able to bind directly to the
    device driver that the TCP/IP stack uses appears to be not possible,
    though I have not looked to see what interface is presented. The
    reason I say this that others, including experts, who want to do what
    you are doing, simply write a new driver that sits on top of the
    driver that binds to Ethernet interface, then have their user-mode
    applications bind to the upper driver.

    The most prominent do-it-yourself-just-getting-started driver is
    called NDISPROT, and comes with the Window Server 2003 DDK as one of
    the samples. When this driver is loaded by your user-mode
    application, it binds to all Ethernet drivers (miniports) in the
    system. You can then use standard open/close/read/write/ioctl to send
    and receive frames. There is precursor to this driver called
    NDISUIO. Use NDISPROT Instead. You can installed the NDISPROT driver
    by using the .INF file that comes with it, or you can install it
    programmatically using CreateService/etc. You use OpenService/
    StartService, etc. to load the driver, and CreateFile, ReadFile,
    WriteFile, CloseHandle to interact with the driver while it is loaded.

    To get an idea of what drivers are in your system, browse under
    registry key:
    HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Contro l\Network

    To get an idea of where the drives are located on hard disk, search
    for NDIS under this key:
    HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Servic es

    On the matter of performance:

    If you study how NDISPROT and other "protocol" drivers are designed,
    they do a lot of queuing of frames, both incoming and outgoing, to
    avoid frame loss. You would redesign this driver to not only
    eliminate this queuing, but also minimize transitions between user-
    mode and kernel-mode. I am using a shared memory model that will
    essentially avoid frame queuing and essentially slam all incoming
    packets directly from Ethernet interface driver to user-mode, and vice-
    versa. I have not tested it yet, but looking at the alternative way
    that is done now, it appears it will show some performance
    improvement.

    -Le Chaud Lapin-


  4. Re: Gigabit ethernet And Windows Stack Efficiency

    On 9 Mar, 21:27, dave_m_mo...@post2me.freeserve.co.uk wrote:
    > Hi,
    > I'm investigating methods of moving large amounts of data (say
    > 70Mbytes/sec) between PCs over Gigabit Ethernet. Using the traditional
    > socket programming methods, this incurrs a large (and unacceptable)
    > CPU overhead.
    >
    > As I don't need the ethernet connection for any other use, I assumed
    > that I could somehow rewrite the ethernet card driver to avoid the
    > Windows Stack code (may using TOE also) to improve things. However, as
    > I have no idea how to do this (and I know it's complicated) I wondered
    > whether there was already a low level driver available that I could
    > use. I assume that the Windows Stack code must sit on a standard
    > interface to the ethernet card, so presumably I could interface with
    > this interface directly?.
    >
    > Any ideas would be appreciated.
    > Ta,
    > Dave



    70MB/s does not show up any higer CPU-consumption when ftp-ing on
    a BSD or Linux box. Using rsync ( which is probably what you should
    use if
    you repliacte large amounts of data) will consume more cpu, but it
    will
    prevent a lot of network traffic.

    You code might do inefficient things. Have it reviewd by someone.


  5. Re: Gigabit ethernet And Windows Stack Efficiency


    dave_m_moore@post2me.freeserve.co.uk wrote:
    > Hi,
    > I'm investigating methods of moving large amounts of data (say
    > 70Mbytes/sec) between PCs over Gigabit Ethernet. Using the traditional
    > socket programming methods, this incurrs a large (and unacceptable)
    > CPU overhead.
    >


    You could use a NIC that supports TCP processing offloading to the
    NIC. This will reduce the load on the main CPU.

    --
    EventStudio 4.0 - http://www.EventHelix.com/EventStudio
    Design and Document Protocol Interactions as Sequence Diagrams


  6. Re: Gigabit ethernet And Windows Stack Efficiency

    In article <1173833712.089436.276290@e1g2000hsg.googlegroups.c om>,
    EventHelix.com wrote:

    >> I'm investigating methods of moving large amounts of data (say
    >> 70Mbytes/sec) between PCs over Gigabit Ethernet. Using the traditional
    >> socket programming methods, this incurrs a large (and unacceptable)
    >> CPU overhead.

    >
    >You could use a NIC that supports TCP processing offloading to the
    >NIC. This will reduce the load on the main CPU.


    Before accepting marketing claims, always run your own, relevant
    benchmarks on relevant hardware and software.

    For the most of the almost 3 decades that people have been selling "TCP
    offload processors" (e.g. Excelan and Interlan in the early 1980s), it
    has been common to spend more host CPU cycles jumping through the hoops
    needed to offload the TCP processing than to just do the TCP processing
    in the host. Synchronizing two computers to handle the single job of
    pushing data over a wire needs uncommon trickery, plenty of CPU cycles,
    or both. The classic "ring buffer" solutions involve no trickery and
    plenty of the worst kind of cache-missing, I/O bus CPU cycles.

    The exceptions to this general rule have generally been special cases,
    such as the "vector" computers once called "super computers" that could
    not do general comparing and deciding as fast as other CPUs. Even in
    those special cases, other ways of looking at the problem yielded good
    speed purely in the host (e.g. using vector instructions to compute the
    TCP checksum).

    As legions of people have pointed out over the decades, TCP processing
    by itself costs a trivial number of CPU cycles. I've forgotten Van
    Jacobson's classic number from the early 1990s except that it was fewer
    than 200 cycles per TCP segment exclusive of copying the user data among
    buffers or computing the TCP checksum. The TCP checksum on a 32-bit
    system with carry bit (e.g. 32-bit 80*86) costs about 1 CPU cycle per
    byte. Thus, the overall cost of TCP processing, except for shuffling
    data among buffers, should be on the order of about 1 CPU cycle per
    byte.

    A speed of 70 MByte/sec gives about 38 CPU cycles/byte on a 2.67 GHz
    80*86, so that TCP processing including the checksum needs less than
    10% of the CPU, except for shuffling data among buffers.

    If your application is written sanely (e.g. reasonable buffer sizes and
    sane TCP window sizes), there is probably not enough TCP processing to
    "off load," except for moving data among buffers.

    Shuffling data among buffers is the main cost of TCP, because of the
    cost in memory bandwidth including cache misses. Sometimes "TCP
    processing offloading" can reduce those costs, but it often makes things
    worse, such as by forcing the CPU to copy data even more times and by
    requiring cache missing, painfully slow, I/O bus chitchat with the
    specialized TCP processor.

    The main trouble with TCP at speed is that real applications differ
    from artificial benchmarks such as ttcp and Netperf. Real applications
    need to get the data from somewhere or put it somewhere. That shuffling
    of data among buffers is what kills TCP speed. If this application
    does more than the nothing that `ttcp` does with data, do not expect
    to save the day "TCP processing offloading."

    My views were tutored in UNIX-like systems intended to go fast as opposed
    to consumer grade, "user friendly" systems intended for whatever Microsoft
    intends, but the principle is likely to hold. It is possible to make
    anything slower than anyone would expect, but my guess is that you will
    spend far more cycles on Windows threading, DLL rigamarole, and so forth
    than on TCP processing even in TCP/IP code from Microsoft.


    Vernon Schryver vjs@rhyolite.com

+ Reply to Thread