eth network seems way too slow - Networking

This is a discussion on eth network seems way too slow - Networking ; Rahul wrote: > Rick Jones wrote in news:ga97d2$raa$1 > @usenet01.boi.hp.com: > >> Looks like you have NICs and drivers which favor bulk throughput over >> minimizing latency: >> > > In which case my latency might be horrible but I ...

+ Reply to Thread
Page 2 of 2 FirstFirst 1 2
Results 21 to 30 of 30

Thread: eth network seems way too slow

  1. Re: eth network seems way too slow

    Rahul wrote:
    > Rick Jones wrote in news:ga97d2$raa$1
    > @usenet01.boi.hp.com:
    >
    >> Looks like you have NICs and drivers which favor bulk throughput over
    >> minimizing latency:
    >>

    >
    > In which case my latency might be horrible but I ought to still expect
    > throughput close to 2 Gbps post bonding, right? At least with mode=4
    > (802.3ad) that truely aggregates the channels even for a peer-to-single-
    > peer?
    >


    I don't think 802.3ad could use the two channel for a peer to single peer

  2. Re: eth network seems way too slow

    Rick Jones wrote in news:ga9r33$pr7$1
    @usenet01.boi.hp.com:

    > If you have three systems involved and two netperfs I would expect to
    > see the sum of the netperfs to be greater than a single link.



    Hi Rick. Thanks for that lead. You were absolutely right. If I start 2
    netperfs on a node but point them to netservers on two different machines
    then each netperf reports 1Gbps. Then if I kill a eth link performance
    degrades to 0.5 Gbps. All makes sense. Doesn't matter which mode (4 or 6) I
    use.

    The only thing I do not understand is this: None of the modes give me 2
    Gbps for a node-to-single-node connection. Not even if I start 2 netperf
    processes (just like you said!). But I still don't see this. I reread the
    802.3ad spec. and the only advantage of this mode over alb seems that this
    ought to support 2 Gbps even for a peer-to-single-peer. Not in a single
    "conversation", true. But if I started 2 netperfs in parallel they ought to
    since then the packet-reordering etc. problems do not arise since there are
    two independent packet-sources. So far as I understand that is the
    advantage of the 802.3ad over alb. You don't think so? You think there's no
    way other than mode_rr of duplexing traffic peer-to-peer even if all the
    traffic came from a bunch of different protocols and processes?

    --
    Rahul

  3. Re: eth network seems way too slow

    On Thu, 11 Sep 2008, Rahul wrote:

    > Steve Thompson wrote in
    > news:alpine.LRH.0.9999.0809101654230.20117@honker. vgersoft.com:
    >>> In which case my latency might be horrible but I ought to still expect
    >>> throughput close to 2 Gbps post bonding, right?

    >>
    >> No.

    > Why not? I know that bandwidth and latency can in some cases be coupled.
    > But from what I know of how technologies like ISDN work bonding two (or
    > more) channels (albeit of bad latency) does result in a bandwidth
    > multiplication. Or not? I thought latency was something one is stuck with
    > but bandwidth is "more easily" multiplied? Especially with modern Linux
    > bonding drivers.


    Bonding will only give you extra bandwidth if there are multiple
    applications using it (and even then you may be at the mercy of what the
    switch does at the other end); a single application will only get at most
    the bandwidth of one of the links (unless it is playing games with
    multiple I/O's via nonblocking sockets). You might be able to get more
    than 1 GB/s by running more than one netperf at once, for example.

    Steve

  4. Re: eth network seems way too slow

    Steve Thompson wrote in
    news:alpine.LRH.0.9999.0809120950150.8650@honker.v gersoft.com:

    > Bonding will only give you extra bandwidth if there are multiple
    > applications using it (and even then you may be at the mercy of what
    > the switch does at the other end); a single application will only get
    > at most the bandwidth of one of the links (unless it is playing games
    > with multiple I/O's via nonblocking sockets). You might be able to get
    > more than 1 GB/s by running more than one netperf at once, for
    > example.
    >


    Exactly. I had expected two netperfs to give me 1 Gbps each even on a
    machine-to-machine basis. But it does not happen. They fall back to
    0.5Gbps.

    --
    Rahul

  5. Re: eth network seems way too slow

    On Fri, 12 Sep 2008, Rahul wrote:

    > Steve Thompson wrote in
    > news:alpine.LRH.0.9999.0809120950150.8650@honker.v gersoft.com:
    >
    >> Bonding will only give you extra bandwidth if there are multiple
    >> applications using it (and even then you may be at the mercy of what
    >> the switch does at the other end); a single application will only get
    >> at most the bandwidth of one of the links (unless it is playing games
    >> with multiple I/O's via nonblocking sockets). You might be able to get
    >> more than 1 GB/s by running more than one netperf at once, for
    >> example.

    > Exactly. I had expected two netperfs to give me 1 Gbps each even on a
    > machine-to-machine basis. But it does not happen. They fall back to
    > 0.5Gbps.


    That depends on the hashing method being used by your bonding mode. Try
    two netperf's to two different machines.

    Steve

  6. Re: eth network seems way too slow

    Steve Thompson wrote in
    news:alpine.LRH.0.9999.0809121459350.5841@honker.v gersoft.com:

    > That depends on the hashing method being used by your bonding mode. Try
    > two netperf's to two different machines.
    >


    Two different machines always gives me 2Gbps in total (1 Gbps each). But I
    would very much like to get the same performance multiplication on machine-
    to-machine communications too. Any recommendations what hash I should be
    forcing it to? So far I only know I can tweak the "modes". Can one fine-
    tune the hash-choice too? I've seen some stuff in my switch configs about
    selecting what to hash on (MAC, IP etc.) but no more idea.

    Point is, these are pretty fast 8 cpu servers. They are going to run
    parallel MPI jobs. So it is very likely that even a fairly large 16 cpu job
    runs accross 2 machines. Then it would be very wasteful if I could not use
    both my eth cards. In fact, I might have to force jobs to run on cpus
    spanning 3 servers just to get bandwidth multiplication. That seems very
    counter-intutive and unproductive.

    --
    Rahul

  7. Re: eth network seems way too slow

    In comp.os.linux.networking Rahul wrote:
    > The only thing I do not understand is this: None of the modes give
    > me 2 Gbps for a node-to-single-node connection. Not even if I start
    > 2 netperf processes (just like you said!). But I still don't see
    > this. I reread the 802.3ad spec. and the only advantage of this mode
    > over alb seems that this ought to support 2 Gbps even for a
    > peer-to-single-peer. Not in a single "conversation", true. But if I
    > started 2 netperfs in parallel they ought to since then the
    > packet-reordering etc. problems do not arise since there are two
    > independent packet-sources.


    Depends on how a "conversation" is defined.

    In a "pure" IEEE context a conversation (aka flow) is probably defined
    by the src and dst MAC (ethernet) address. Two nodes there are then
    only one converstation no matter how many TCP connections.

    A "flow" could also be defined as a unique src/dst IP pair. Same
    result as above.

    Only when a "conversation" is defined as a unique four-tuple of
    src/dst IP/port will there be more than one between two nodes.

    I'm not sure if 802.3ad specifies the definition of a "converstation"
    or if it leaves it to interpretation. What switches can do probably
    depends on the switch/vendor and their decisions.

    rick jones
    --
    Process shall set you free from the need for rational thought.
    these opinions are mine, all mine; HP might not want them anyway...
    feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

  8. Re: eth network seems way too slow

    In comp.os.linux.networking Steve Thompson wrote:
    > Bonding will only give you extra bandwidth if there are multiple
    > applications using it (and even then you may be at the mercy of what
    > the switch does at the other end)


    It isn't a question of how many applications or processes or even
    threads. The aggregation/bonding/trunking doesn't care about those
    sorts of things. It is entirely how many "flows" are seen, where a
    "flow" might be src/dst MAC, or src/dst IP or a full four-tuple of
    src/dst IP/port.

    rick jones
    --
    oxymoron n, commuter in a gas-guzzling luxury SUV with an American flag
    these opinions are mine, all mine; HP might not want them anyway...
    feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

  9. Re: eth network seems way too slow

    Rick Jones wrote in news:gama7d$2uq$2
    @usenet01.boi.hp.com:

    > Depends on how a "conversation" is defined.
    >
    > In a "pure" IEEE context a conversation (aka flow) is probably defined
    > by the src and dst MAC (ethernet) address. Two nodes there are then
    > only one converstation no matter how many TCP connections.
    >
    > A "flow" could also be defined as a unique src/dst IP pair. Same
    > result as above.
    >
    > Only when a "conversation" is defined as a unique four-tuple of
    > src/dst IP/port will there be more than one between two nodes.
    >
    > I'm not sure if 802.3ad specifies the definition of a "converstation"
    > or if it leaves it to interpretation. What switches can do probably
    > depends on the switch/vendor and their decisions.
    >
    >


    Thanks for those comments Rick. But that opens ways to get around that
    are pretty strange, in my mind:

    Consider this scenario:

    I have two scp's (or netperfs) "talking" from Machine B to Machine A.
    Both A and B have twin bonded eth cards. I still get just 1 Gbps,
    frustrating my attempt to get both A and B to use both their eth cards.

    If I add an intermediary machine C which just accepts traffic on a port
    and forwards it to A then I _could_ get both cards on Machines A and B
    into action.

    Do you think I can "fool" the machines into using both cards in this
    manner. If I can, then I think its somewhat a limitation that this
    "intelligence" was not factored into the card-bonding drivers in the
    first place. Are there any other considerations? I mean, there is no
    rationale in worrying about ordering traffice coming from different
    machine processes, is there?

    Or perhaps this is not implimented so as not to violate the sanctity of
    each network layer not interfering with the ones above it?

    --
    Rahul

  10. Re: eth network seems way too slow

    In comp.os.linux.networking Rahul wrote:
    > Thanks for those comments Rick. But that opens ways to get around
    > that are pretty strange, in my mind:


    > Consider this scenario:


    > I have two scp's (or netperfs) "talking" from Machine B to Machine
    > A. Both A and B have twin bonded eth cards. I still get just 1
    > Gbps, frustrating my attempt to get both A and B to use both their
    > eth cards.


    > If I add an intermediary machine C which just accepts traffic on a
    > port and forwards it to A then I _could_ get both cards on Machines
    > A and B into action.


    > Do you think I can "fool" the machines into using both cards in this
    > manner. If I can, then I think its somewhat a limitation that this
    > "intelligence" was not factored into the card-bonding drivers in the
    > first place. Are there any other considerations? I mean, there is no
    > rationale in worrying about ordering traffice coming from different
    > machine processes, is there?


    > Or perhaps this is not implimented so as not to violate the sanctity
    > of each network layer not interfering with the ones above it?


    Burried in the bonding.txt file I think you might be able to tell the
    end systems to schedule packets on links in the bond/trunk/aggregate
    based on the four-tuple - soething about an xor comes to mind.
    Certainly the HP-UX Auto Port Aggregation (APA) software can do that.

    The real issue IMO is getting the _switch_ to do that so you still
    have spread of multiple connections between two systems on the way
    _in_ to the host. Some switches may be willing and able to look that
    deep into the headers, some may not.

    rick jones
    --
    The computing industry isn't as much a game of "Follow The Leader" as
    it is one of "Ring Around the Rosy" or perhaps "Duck Duck Goose."
    - Rick Jones
    these opinions are mine, all mine; HP might not want them anyway...
    feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

+ Reply to Thread
Page 2 of 2 FirstFirst 1 2