Max Ethernet ports - Networking

This is a discussion on Max Ethernet ports - Networking ; Hello, Is there an inherent limit to the number of Ethernet interfaces Linux will support? I'm looking to build a switch like appliance with custom software/protocols for up to 48 10/100 downstream ports, convert and upload over two upstream ports ...

+ Reply to Thread
Results 1 to 14 of 14

Thread: Max Ethernet ports

  1. Max Ethernet ports

    Hello,

    Is there an inherent limit to the number of Ethernet interfaces Linux will
    support? I'm looking to build a switch like appliance with custom
    software/protocols for up to 48 10/100 downstream ports, convert and upload
    over two upstream ports 10/100/1Gbe. I realize the throughput will be no
    where close to a real switch however I don't need much as each device uses
    minimal bandwidth and they hardly talk amongst themselves. Ethernet is used
    more of a ease of install than a bandwidth requirement.

    Any ideas?

    Thanks




  2. Re: Max Ethernet ports

    > Is there an inherent limit to the number of Ethernet interfaces Linux will
    > support?


    Somewhere, long, long ago, an official-ish statement was made on that
    subject, to the effect that there IS a limit - but that it would be all but
    impossible to connect that many to a single machine anyway. If memory
    serves, the number was in the thousands, or tens of thousands.

    steve



  3. Re: Max Ethernet ports

    Thanks so now the bigger question is any ideas short of building a custom
    board as to how to get these many ports on a single linux box?

    Mike

    "Steve Wolfe" wrote in message
    news:g_-dndt9qYtgoorbnZ2dnUVZ_qemnZ2d@comcast.com...
    >> Is there an inherent limit to the number of Ethernet interfaces Linux
    >> will
    >> support?

    >
    > Somewhere, long, long ago, an official-ish statement was made on that
    > subject, to the effect that there IS a limit - but that it would be all
    > but impossible to connect that many to a single machine anyway. If memory
    > serves, the number was in the thousands, or tens of thousands.
    >
    > steve
    >




  4. Re: Max Ethernet ports

    On Sat, 07 Apr 2007 18:36:37 +0000, Mike rearranged some electrons to
    form:

    > Thanks so now the bigger question is any ideas short of building a custom
    > board as to how to get these many ports on a single linux box?
    >
    > Mike


    Ethernet switch...?



    --
    David M (dmacchiarolo)
    http://home.triad.rr.com/redsled
    T/S 53
    sled351 Linux 2.4.18-14 has been up 5 days 22:49


  5. Re: Max Ethernet ports

    On Fri, 06 Apr 2007, in the Usenet newsgroup comp.os.linux.networking, in
    article , Mike
    wrote:

    >Is there an inherent limit to the number of Ethernet interfaces Linux
    >will support?


    It's more a hardware limit than O/S specific. Go price out a D-Link
    DFE-570TX, DFE-580TX, AEI P430TX, 3Com has one as well, but I can't think
    of the number - four port Ethernet cards - PCI. Then look at your PCI
    motherboard, and tell me how many sockets you see where you can plug in
    such a card. Five? That gives you 20 interfaces, and they're not
    going to be cheap.

    >I'm looking to build a switch like appliance with custom software/
    >protocols for up to 48 10/100 downstream ports, convert and upload
    >over two upstream ports 10/100/1Gbe.


    Not sure I follow you here - Ethernet is a pretty dumb protocol, and
    while the 'type' block in the frame is 16 bits, there are only about 180
    protocol numbers currently assigned. See
    http://www.iana.org/assignments/ethernet-numbers

    >I realize the throughput will be no where close to a real switch however
    >I don't need much as each device uses minimal bandwidth and they hardly
    >talk amongst themselves. Ethernet is used more of a ease of install
    >than a bandwidth requirement.


    I'm not sure what problem you are trying to solve, but using a PC as a
    switch is heading in a completely different direction. Spamazon is showing
    a SMC 8748MINT 48-PORT 10/100/1000 10G switch for US$1713, but you can
    likely cascade 8-10 port SOHO grade switches for even less. What makes a
    switch work (basically a 64 bit delay line per input, to allow the smarts
    to look at the destination MAC and switch accordingly) is quite different
    from a general purpose compute like a PC.

    Old guy

  6. Re: Max Ethernet ports

    On Sat, 7 Apr 2007, in the Usenet newsgroup comp.os.linux.networking, in article
    , Steve Wolfe wrote:

    >> Is there an inherent limit to the number of Ethernet interfaces Linux
    >> will support?

    >
    > Somewhere, long, long ago, an official-ish statement was made on
    >that subject, to the effect that there IS a limit - but that it would be
    >all but impossible to connect that many to a single machine anyway.


    That's the real key - even assuming 5 PCI slots with Quad Ethernet cards
    (and perhaps a built-in interface on the motherboard), you're a long way
    from what a cheap SOHO Ethernet Switch can do. Then there's the fact
    that the switch and a general purpose computer need quite different
    designs.

    >If memory serves, the number was in the thousands, or tens of thousands.


    I'd find that a bit unusual. Other than the hardware problem, the most
    likely problem I can think of is 'Neighbor Table Overflow' which can be
    a function of the connection tracking limits in the firewall code, and
    the size of the ARP cache - both of which can be tweaked while the
    system is running. There might also be a limit in the size of the
    routing table.

    Old guy

  7. Re: Max Ethernet ports

    Hmm. I was browsing through the bridge code in Linux and it seemed to what
    I was needing. The protocol layer I'm converting runs on top of Ethernet &
    TCP/IP (Level 7 data transfers).

    Does anyone make a switch where an OEM or developer can embed an application
    rides in between the uplink ports and the downstream switch ports

    Thanks again for the assistance in this,
    Mike

    "Moe Trin" wrote in message
    news:slrnf1g6a3.h6r.ibuprofin@compton.phx.az.us...
    > On Fri, 06 Apr 2007, in the Usenet newsgroup comp.os.linux.networking, in
    > article , Mike
    > wrote:
    >
    >>Is there an inherent limit to the number of Ethernet interfaces Linux
    >>will support?

    >
    > It's more a hardware limit than O/S specific. Go price out a D-Link
    > DFE-570TX, DFE-580TX, AEI P430TX, 3Com has one as well, but I can't think
    > of the number - four port Ethernet cards - PCI. Then look at your PCI
    > motherboard, and tell me how many sockets you see where you can plug in
    > such a card. Five? That gives you 20 interfaces, and they're not
    > going to be cheap.
    >
    >>I'm looking to build a switch like appliance with custom software/
    >>protocols for up to 48 10/100 downstream ports, convert and upload
    >>over two upstream ports 10/100/1Gbe.

    >
    > Not sure I follow you here - Ethernet is a pretty dumb protocol, and
    > while the 'type' block in the frame is 16 bits, there are only about 180
    > protocol numbers currently assigned. See
    > http://www.iana.org/assignments/ethernet-numbers
    >
    >>I realize the throughput will be no where close to a real switch however
    >>I don't need much as each device uses minimal bandwidth and they hardly
    >>talk amongst themselves. Ethernet is used more of a ease of install
    >>than a bandwidth requirement.

    >
    > I'm not sure what problem you are trying to solve, but using a PC as a
    > switch is heading in a completely different direction. Spamazon is showing
    > a SMC 8748MINT 48-PORT 10/100/1000 10G switch for US$1713, but you can
    > likely cascade 8-10 port SOHO grade switches for even less. What makes a
    > switch work (basically a 64 bit delay line per input, to allow the smarts
    > to look at the destination MAC and switch accordingly) is quite different
    > from a general purpose compute like a PC.
    >
    > Old guy




  8. Re: Max Ethernet ports

    Mike wrote:
    > Hmm. I was browsing through the bridge code in Linux and it seemed to what
    > I was needing. The protocol layer I'm converting runs on top of Ethernet &
    > TCP/IP (Level 7 data transfers).


    If the Linux box needs to handle a zillion TCP/IP connections, then I
    expect that lack of resources for that will get you first. You don't
    need to add the overhead for a herd of NIC's to the mix. Take Moe's
    suggestion and use Ethernet hubs or switches to concentrate the traffic.
    A 24-port 10/100 switch costs under $100 these days.

  9. Re: Max Ethernet ports

    That's what I'm looking at however, I need to place my app (ok, now running
    on a Linux SBC) which is connected between the 24 port side of the switch
    and the 1 uplink port. I'd like this all in a 1U enclosure.
    I haven't found a switch I can insert a SBC in between these two ports and
    I'm trying to have this in a all in one 1U unit.

    I realize this could be done with two independent 1U boxes, but I'm trying
    to conserve some space.

    Thanks
    Mike

    "Allen McIntosh" wrote in message
    news:WlWRh.530$Dh.35@newsfe12.lga...
    > Mike wrote:
    >> Hmm. I was browsing through the bridge code in Linux and it seemed to
    >> what I was needing. The protocol layer I'm converting runs on top of
    >> Ethernet & TCP/IP (Level 7 data transfers).

    >
    > If the Linux box needs to handle a zillion TCP/IP connections, then I
    > expect that lack of resources for that will get you first. You don't need
    > to add the overhead for a herd of NIC's to the mix. Take Moe's suggestion
    > and use Ethernet hubs or switches to concentrate the traffic. A 24-port
    > 10/100 switch costs under $100 these days.




  10. Re: Max Ethernet ports

    On Sat, 07 Apr 2007, in the Usenet newsgroup comp.os.linux.networking, in
    article <9qVRh.24106$VU4.21538@bgtnsc05-news.ops.worldnet.att.net>, Mike wrote:

    >Hmm. I was browsing through the bridge code in Linux and it seemed to
    >what I was needing. The protocol layer I'm converting runs on top of
    >Ethernet & TCP/IP (Level 7 data transfers).


    I hate talking "levels" as people tend to disagree which is which and
    what is happening at the level they're referring to.

    "runs on top of Ethernet" - which is _normally_ were a switch tends
    to reside, shuffling packets based on the MAC address. The whole
    concept of a switch is to isolate the links as rapidly as possible.
    I mention the concept of a 64 bit delay line (often, a shift
    register running at the line speed) with taps such that the start of
    the frame can be recognized. At that point, the smarts reads the
    first 48 bits of the frame (destination MAC address) and sets a
    switch matrix so that the end of the shift register is fed into the
    "gozoutta" hose appropriate for that MAC address. (If the port is
    busy, the switch may dump the packet to short term memory so that it
    can be kicked out the door as soon as the line is free.) The switch
    doesn't have the time (read "bandwidth") to be doing anything else.
    In fact, the difference between the 48 bits of the address and the
    (example) 64 bits of the delay line is the amount of time that the
    smarts has to read the MAC, figure out which port that may be on,
    test to see if the port is free, and set the switch to route the
    "comezoutta" from the delay line to that appropriate port. For 100
    Megabit links, that might be 0.16 millionths of a second, or about
    one clock cycle on an old ISA bus. Don't you be late now!

    As for running above TCP/IP, that's three levels higher up the stack,
    and that likely means lots of CPU processing. "Switches switch" at the
    lowest protocol level for bandwidth. If you have to feed data to the
    CPU, you are running a bandwidth limited process. On a PCI based
    system, you _can_ move eight bytes (64 bits) at a chunk, but the
    top speed of the PCI bus is 66 MHz. and it takes more than one
    clock cycle to "inhale" and "exhale". Routers and similar can
    move bits rapidly, but only through a small number (compared to a
    switch) of interfaces.

    >Does anyone make a switch where an OEM or developer can embed an
    >application rides in between the uplink ports and the downstream
    >switch ports.


    I don't know of any, because you are leaving the realm of a switch
    (a device meant to route packets with a minimal delay and no data
    processing) and getting up into the realm where you are going to
    open the packet and dink with the contents. That's a completely
    incompatible process. Thats not a networking problem so much as a
    data processing application.

    In your original post, you spoke of "each device uses minimal
    bandwidth and they hardly talk amongst themselves". Still not
    enough information, but I'd suggest using cheap SOHO switches
    to concentrate your 48 devices into one/few Ethernet connections
    on the computing box, and do the processing serially, rather than
    try to parallel things.

    Old guy


  11. Re: Max Ethernet ports

    On Sun, 08 Apr 2007, in the Usenet newsgroup comp.os.linux.networking, in
    article , Mike wrote:

    >That's what I'm looking at however, I need to place my app (ok, now
    >running on a Linux SBC) which is connected between the 24 port side of
    >the switch and the 1 uplink port. I'd like this all in a 1U enclosure.


    Oh - you're talking about a packaging problem. Completely misunderstood
    your post.

    >I haven't found a switch I can insert a SBC in between these two ports
    >and I'm trying to have this in a all in one 1U unit.


    I suspect this to be highly unlikely. The switch manufacturer might
    have space in a 24 port switch if it's packaged in the same box they
    use for the 48 or larger port version, but you'll likely run into
    power and cooling problems.

    >I realize this could be done with two independent 1U boxes, but I'm
    >trying to conserve some space.


    Actually, I even try to avoid 1U boxes for energy density reasons.
    Yes I have seen the advertisements for $HUGE_NUMBER of quad-core boxes
    in this iddy-bitty rack, but I'm also in Arizona and getting a hundred
    cubic feet per second of zero Celsius bone dry air into the rack is a
    problem.

    Old guy

  12. Re: Max Ethernet ports

    >>If memory serves, the number was in the thousands, or tens of thousands.
    >
    > I'd find that a bit unusual. Other than the hardware problem, the most
    > likely problem I can think of is 'Neighbor Table Overflow' which can be
    > a function of the connection tracking limits in the firewall code, and
    > the size of the ARP cache - both of which can be tweaked while the
    > system is running. There might also be a limit in the size of the
    > routing table.


    If I recall correctly (and there's no guarantee that I do), it had to do
    with the small amount of memory that gets allocated for each port for
    buffering or whatever. In any event, no, there is no *practical* limit in
    Linux itself, you'd run into other constraints far earlier.

    steve



  13. Re: Max Ethernet ports

    > Thanks so now the bigger question is any ideas short of building a custom
    > board as to how to get these many ports on a single linux box?


    You can buy external PCI expansion chassis, which have their own PCI
    bridge. You usually get about 6 to 8 slots per chassis, but there are some
    are available with 13 slots as well. With quad-port cards (get the ones
    with four *controllers*, not just one controller and a four-port hub
    on-card), that's 52 per chassis.

    In order to keep contention down, ideally you'd use several PCI-E to PCI
    chassis, with fewer controllers per chassis. That would help alleviate
    bandwidth considerations, but you're still looking at having up to 100
    devices generating intterupts, which is another issue entirely. Using
    controllers which support interrupt coalescing or NAPI would help greatly
    with that.

    In any event, by the time you do something like that, it's going to be a
    lot of money, work, and rack space. Before you jump in with both feet, you
    might want to spend just a little more time thinking about whether a simpler
    approach might be made to work.

    As to the comment that the Linux machine might not handle a huge number of
    TCP sessions, I don't know where the limit lies, but they can handle a *lot*
    of traffic. The UK national web cache service has 40+ squid caches behind a
    Linux LVS load balancer, and they handle a HUGE amount of traffic and
    connections. From the (very) brief description that the original author
    gave, my impression was that he wouldn't be anywhere near that limit.

    steve



  14. Re: Max Ethernet ports

    I was trying to solve a product packaging problem by having the phyical
    Ethernet connects in the same box as the application level intelligence. I
    can buy a 24 port managed switch mate it with a 1U Linux box with enough
    Ethernet ports(2) to connect the uplink switch and then my application would
    uplink to the rest of the world on two other Ethernet ports. That pretty
    much solves the problem although it takes two boxes instead of just one to
    accomplish the end result. This moves it to the serial process I think you
    were talking about Moe.

    Do I have this correct?

    Mike


    "Moe Trin" wrote in message
    news:slrnf1go9e.l4v.ibuprofin@compton.phx.az.us...
    > On Sat, 07 Apr 2007, in the Usenet newsgroup comp.os.linux.networking, in
    > article <9qVRh.24106$VU4.21538@bgtnsc05-news.ops.worldnet.att.net>, Mike
    > wrote:
    >
    >>Hmm. I was browsing through the bridge code in Linux and it seemed to
    >>what I was needing. The protocol layer I'm converting runs on top of
    >>Ethernet & TCP/IP (Level 7 data transfers).

    >
    > I hate talking "levels" as people tend to disagree which is which and
    > what is happening at the level they're referring to.
    >
    > "runs on top of Ethernet" - which is _normally_ were a switch tends
    > to reside, shuffling packets based on the MAC address. The whole
    > concept of a switch is to isolate the links as rapidly as possible.
    > I mention the concept of a 64 bit delay line (often, a shift
    > register running at the line speed) with taps such that the start of
    > the frame can be recognized. At that point, the smarts reads the
    > first 48 bits of the frame (destination MAC address) and sets a
    > switch matrix so that the end of the shift register is fed into the
    > "gozoutta" hose appropriate for that MAC address. (If the port is
    > busy, the switch may dump the packet to short term memory so that it
    > can be kicked out the door as soon as the line is free.) The switch
    > doesn't have the time (read "bandwidth") to be doing anything else.
    > In fact, the difference between the 48 bits of the address and the
    > (example) 64 bits of the delay line is the amount of time that the
    > smarts has to read the MAC, figure out which port that may be on,
    > test to see if the port is free, and set the switch to route the
    > "comezoutta" from the delay line to that appropriate port. For 100
    > Megabit links, that might be 0.16 millionths of a second, or about
    > one clock cycle on an old ISA bus. Don't you be late now!
    >
    > As for running above TCP/IP, that's three levels higher up the stack,
    > and that likely means lots of CPU processing. "Switches switch" at the
    > lowest protocol level for bandwidth. If you have to feed data to the
    > CPU, you are running a bandwidth limited process. On a PCI based
    > system, you _can_ move eight bytes (64 bits) at a chunk, but the
    > top speed of the PCI bus is 66 MHz. and it takes more than one
    > clock cycle to "inhale" and "exhale". Routers and similar can
    > move bits rapidly, but only through a small number (compared to a
    > switch) of interfaces.
    >
    >>Does anyone make a switch where an OEM or developer can embed an
    >>application rides in between the uplink ports and the downstream
    >>switch ports.

    >
    > I don't know of any, because you are leaving the realm of a switch
    > (a device meant to route packets with a minimal delay and no data
    > processing) and getting up into the realm where you are going to
    > open the packet and dink with the contents. That's a completely
    > incompatible process. Thats not a networking problem so much as a
    > data processing application.
    >
    > In your original post, you spoke of "each device uses minimal
    > bandwidth and they hardly talk amongst themselves". Still not
    > enough information, but I'd suggest using cheap SOHO switches
    > to concentrate your 48 devices into one/few Ethernet connections
    > on the computing box, and do the processing serially, rather than
    > try to parallel things.
    >
    > Old guy
    >




+ Reply to Thread