IA-64 Linux Question on ProPack 3.0 - SGI

This is a discussion on IA-64 Linux Question on ProPack 3.0 - SGI ; Hello, we are contemplating installation of ProPack 3.0 on an 128 PE SGI Altix, which currently runs ProPack 2.4 + patches. To those who have already installed PP3.0, were there any problems with it? Is it as stable as PP2.4? ...

+ Reply to Thread
Page 1 of 2 1 2 LastLast
Results 1 to 20 of 31

Thread: IA-64 Linux Question on ProPack 3.0

  1. IA-64 Linux Question on ProPack 3.0

    Hello,

    we are contemplating installation of ProPack 3.0 on an 128 PE SGI Altix, which
    currently runs ProPack 2.4 + patches.

    To those who have already installed PP3.0, were there any problems with it? Is
    it as stable as PP2.4? Are the Intel compilers (v 7.x, 8.x) behaving properly
    with it, particularly MPI and OpenMP code? Any issues with using NPTL vs the
    Linux thread Libs?

    Are there any significant advantages to go with PP3.0 vs. staying with PP2.4?
    Or vice-versa?

    Any information would be greatly appreciated!

    Thanks
    Michael Thomadakis

  2. Re: IA-64 Linux Question on ProPack 3.0

    In article ,
    Michael E. Thomadakis wrote:
    ......
    >we are contemplating installation of ProPack 3.0 on an 128 PE SGI Altix, which
    >currently runs ProPack 2.4 + patches.


    >To those who have already installed PP3.0, were there any problems with it?...


    I too am comtemplating the "upgrade". It seems that there
    isn't any upgrade install, only an install operation. Anyone
    have experience with that aspect?


    --
    Daniel Packman
    NCAR/ACD
    pack@ucar.edu

  3. Re: IA-64 Linux Question on ProPack 3.0

    I've also heard that it is a reistall. PP3.0 is based on the AS 3.0 of RH
    which uses different source tree and NPTL threads.

    BTW, which Altix are you running and wich PP?

    -MT

    On Sun, 30 May 2004, Daniel Packman wrote:

    DP > Date: Sun, 30 May 2004 06:05:59 +0000 (UTC)
    DP > From: Daniel Packman
    DP > Newsgroups: comp.sys.sgi.admin, comp.os.linux.setup
    DP > Subject: Re: IA-64 Linux Question on ProPack 3.0
    DP >
    DP > In article ,
    DP > Michael E. Thomadakis wrote:
    DP > .....
    DP > >we are contemplating installation of ProPack 3.0 on an 128 PE SGI Altix, which
    DP > >currently runs ProPack 2.4 + patches.
    DP >
    DP > >To those who have already installed PP3.0, were there any problems with it?...
    DP >
    DP > I too am comtemplating the "upgrade". It seems that there
    DP > isn't any upgrade install, only an install operation. Anyone
    DP > have experience with that aspect?
    DP >
    DP >
    DP > --
    DP > Daniel Packman
    DP > NCAR/ACD
    DP > pack@ucar.edu
    DP >

  4. NFS on IA-64 Linux

    Hello all.

    We are maintaining an SGI Altix 3700 (128 PE, 10TB RAID storage) currently
    running PP2.4.

    Has anyone had experience with using this system as an NFS server? There is
    not much discussion for NFS in PP2.4. Can TCP be used as the transport for
    NFS? Is there any problem with large files/file systems? How is th
    throughput?

    Anyone with PP3.0 NFS experience?

    Thanks
    -MT

  5. Re: IA-64 Linux Question on ProPack 3.0

    Daniel Packman wrote:
    >
    > I too am comtemplating the "upgrade". It seems that there
    > isn't any upgrade install, only an install operation. Anyone
    > have experience with that aspect?


    ProPack 3 is a fresh install, no upgrade option. We provide some
    documentation on what to keep at hand during the process. Please provide
    feedback on what, if anything, might be added to the documentation.


  6. Re: IA-64 Linux Question on ProPack 3.0

    Michael E. Thomadakis wrote:

    > Hello,
    >
    > we are contemplating installation of ProPack 3.0 on an 128 PE SGI Altix, which
    > currently runs ProPack 2.4 + patches.
    >
    > To those who have already installed PP3.0, were there any problems with it?


    Not with it in se, but if you have commercial binaries, there are a few compatibility
    issues you can bump into - mainly because glibc is of a different rev.

    Not worse than going from RH ASE 2.1 to RH ASE 3.0 - same issues there...

    Is
    > it as stable as PP2.4?


    With the day one (or almost day one) patches, yes.

    Are the Intel compilers (v 7.x, 8.x) behaving properly
    > with it, particularly MPI and OpenMP code?


    Yes.

    Any issues with using NPTL vs the
    > Linux thread Libs?


    Sometimes (if your application is broken).

    An environment variable will set the Pthread library to
    revert to the old Linuxthreads, though - you just tell the library
    it has to assume an old Linux kernel.

    >
    > Are there any significant advantages to go with PP3.0 vs. staying with PP2.4?


    Yes. With patches, (slightly) better VM and buffer cache management,
    and compatibility with another version of RH and glibc -- which is a
    blessing or a curse depending on what your binaries support
    (unless you can recompile them all).

    May userland RPMs of a more recent date -- all the Propack 2.x flavours
    always tracked the already quite old RH ASE 2.1 RPMs.

    Read the release notes (on the Documentation CD's README.TXT).

    --
    Alexis Cousein Senior Systems Engineer
    alexis@sgi.com SGI/Silicon Graphics Brussels

    If I have seen further, it is by standing on reference manuals.


  7. Re: IA-64 Linux Question on ProPack 3.0

    Daniel Packman wrote:

    >
    >
    > I too am comtemplating the "upgrade". It seems that there
    > isn't any upgrade install, only an install operation. Anyone
    > have experience with that aspect?
    >
    >

    Yes. The rpms are structured in the same way as RedHat rpms,
    and there is no way to do an upgrade installation that works
    reliably: too many bits of code have found a new rpm home...

    For information on how to manage a fresh install while
    retaining such things as modified configuration files, see the
    release notes.

    --
    Alexis Cousein Senior Systems Engineer
    alexis@sgi.com SGI/Silicon Graphics Brussels

    If I have seen further, it is by standing on reference manuals.


  8. Re: NFS on IA-64 Linux

    Michael E. Thomadakis wrote:

    > Hello all.
    >
    > We are maintaining an SGI Altix 3700 (128 PE, 10TB RAID storage) currently
    > running PP2.4.
    >
    > Has anyone had experience with using this system as an NFS server?


    It works well.

    Provided the clients use read and write sizes that are
    a multiple of the base page size -- which, on this machine, is *16KB* -
    the default 8KB xfer sizes are a no-no (and an even bigger no-no for
    the NFS client, which means you do have to patch up many IA32 Linux
    servers quite heavily if they have to act as an NFS server for an
    Altix -- many with a 2.4 kernel do not really support sizes over 8KB,
    though Propack 2.4 does).

    The same advice holds for any other Linux server using a 2.4 kernel -
    but of course, 8KB is a multiple of a 4KB base page size on most
    IA32 Linux servers...

    As a consequence, NFS v2 is also a big no-no if you want performance,
    given its insistence on limiting xfer size to 8KB.


    > There is
    > not much discussion for NFS in PP2.4. Can TCP be used as the transport for
    > NFS?


    Not without patch 10054. Even with that patch, the 2.4 client code
    is noticeably faster over UDP - unless, of course, you have a dodgy
    network that can't get a 16KB transfer size over reliably, but
    manages to get 1500 byte MTUs over most of the time.

    Is there any problem with large files/file systems?

    Not unless you're mounting with the "soft" option. There are
    a few spots in the (generic) 2.4 client code that can trigger
    bogus timeouts propagated to the application as errors
    if you use this option.

    Don't export with the "async" option (this is generic advice) if
    all your clients are using NFS v3.

    There's a performance divot for non root copies of files with the group
    execute permission bit set if you don't have patch 10065.

    > How is th
    > throughput?


    Throughput is OK for one interface. Scalability for many GigE interfaces
    is poor if you're used to IRIX, and less good than on 2.6 kernels
    (by a huge margin), and even in 2.6 there's quite some work cut out to
    achieve IRIX-level scalability.

    --
    Alexis Cousein Senior Systems Engineer
    alexis@sgi.com SGI/Silicon Graphics Brussels

    If I have seen further, it is by standing on reference manuals.


  9. Re: NFS on IA-64 Linux

    Alexis,

    thanks for your informative reply. I believe that we've already applied all
    published ALTIX patches so far.

    -Michael

    On Tue, 1 Jun 2004, Alexis Cousein wrote:

    a| Date: Tue, 01 Jun 2004 16:22:21 +0200
    a| From: Alexis Cousein
    a| To: Michael E. Thomadakis
    a| Newsgroups: comp.sys.sgi.admin, comp.os.linux.setup
    a| Subject: Re: NFS on IA-64 Linux
    a|
    a| Michael E. Thomadakis wrote:
    a|
    a| > Hello all.
    a| >
    a| > We are maintaining an SGI Altix 3700 (128 PE, 10TB RAID storage) currently
    a| > running PP2.4.
    a| >
    a| > Has anyone had experience with using this system as an NFS server?
    a|
    a| It works well.
    a|
    a| Provided the clients use read and write sizes that are
    a| a multiple of the base page size -- which, on this machine, is *16KB* -
    a| the default 8KB xfer sizes are a no-no (and an even bigger no-no for
    a| the NFS client, which means you do have to patch up many IA32 Linux
    a| servers quite heavily if they have to act as an NFS server for an
    a| Altix -- many with a 2.4 kernel do not really support sizes over 8KB,
    a| though Propack 2.4 does).
    a|
    a| The same advice holds for any other Linux server using a 2.4 kernel -
    a| but of course, 8KB is a multiple of a 4KB base page size on most
    a| IA32 Linux servers...
    a|
    a| As a consequence, NFS v2 is also a big no-no if you want performance,
    a| given its insistence on limiting xfer size to 8KB.
    a|
    a|
    a| > There is
    a| > not much discussion for NFS in PP2.4. Can TCP be used as the transport for
    a| > NFS?
    a|
    a| Not without patch 10054. Even with that patch, the 2.4 client code
    a| is noticeably faster over UDP - unless, of course, you have a dodgy
    a| network that can't get a 16KB transfer size over reliably, but
    a| manages to get 1500 byte MTUs over most of the time.
    a|
    a| Is there any problem with large files/file systems?
    a|
    a| Not unless you're mounting with the "soft" option. There are
    a| a few spots in the (generic) 2.4 client code that can trigger
    a| bogus timeouts propagated to the application as errors
    a| if you use this option.
    a|
    a| Don't export with the "async" option (this is generic advice) if
    a| all your clients are using NFS v3.
    a|
    a| There's a performance divot for non root copies of files with the group
    a| execute permission bit set if you don't have patch 10065.
    a|
    a| > How is th
    a| > throughput?
    a|
    a| Throughput is OK for one interface. Scalability for many GigE interfaces
    a| is poor if you're used to IRIX, and less good than on 2.6 kernels
    a| (by a huge margin), and even in 2.6 there's quite some work cut out to
    a| achieve IRIX-level scalability.
    a|
    a| --
    a| Alexis Cousein Senior Systems Engineer
    a| alexis@sgi.com SGI/Silicon Graphics Brussels
    a|
    a| If I have seen further, it is by standing on reference manuals.
    a|
    a|

  10. Re: IA-64 Linux Question on ProPack 3.0

    Follow up on installing PP3.0

    The Intel license server that used to work on PP2.4 broke. It appears that the
    vendor's (Intel) license server cannot start at boot up time. The net result
    is that all compilers cannot run. Intel has replied that they are familiar
    with the problem and they are sending a fix any time now.

    -MT

    On Tue, 1 Jun 2004, Alexis Cousein wrote:

    a> Date: Tue, 01 Jun 2004 15:58:04 +0200
    a> From: Alexis Cousein
    a> Newsgroups: comp.sys.sgi.admin, comp.os.linux.setup
    a> Subject: Re: IA-64 Linux Question on ProPack 3.0
    a>
    a> Michael E. Thomadakis wrote:
    a>
    a> > Hello,
    a> >
    a> > we are contemplating installation of ProPack 3.0 on an 128 PE SGI Altix, which
    a> > currently runs ProPack 2.4 + patches.
    a> >
    a> > To those who have already installed PP3.0, were there any problems with it?
    a>
    a> Not with it in se, but if you have commercial binaries, there are a few compatibility
    a> issues you can bump into - mainly because glibc is of a different rev.
    a>
    a> Not worse than going from RH ASE 2.1 to RH ASE 3.0 - same issues there...
    a>
    a> Is
    a> > it as stable as PP2.4?
    a>
    a> With the day one (or almost day one) patches, yes.
    a>
    a> Are the Intel compilers (v 7.x, 8.x) behaving properly
    a> > with it, particularly MPI and OpenMP code?
    a>
    a> Yes.
    a>
    a> Any issues with using NPTL vs the
    a> > Linux thread Libs?
    a>
    a> Sometimes (if your application is broken).
    a>
    a> An environment variable will set the Pthread library to
    a> revert to the old Linuxthreads, though - you just tell the library
    a> it has to assume an old Linux kernel.
    a>
    a> >
    a> > Are there any significant advantages to go with PP3.0 vs. staying with PP2.4?
    a>
    a> Yes. With patches, (slightly) better VM and buffer cache management,
    a> and compatibility with another version of RH and glibc -- which is a
    a> blessing or a curse depending on what your binaries support
    a> (unless you can recompile them all).
    a>
    a> May userland RPMs of a more recent date -- all the Propack 2.x flavours
    a> always tracked the already quite old RH ASE 2.1 RPMs.
    a>
    a> Read the release notes (on the Documentation CD's README.TXT).
    a>
    a> --
    a> Alexis Cousein Senior Systems Engineer
    a> alexis@sgi.com SGI/Silicon Graphics Brussels
    a>
    a> If I have seen further, it is by standing on reference manuals.
    a>
    a>

  11. Re: IA-64 Linux Question on ProPack 3.0

    Michael E. Thomadakis wrote:
    > Follow up on installing PP3.0
    >
    > The Intel license server that used to work on PP2.4 broke. It appears that the
    > vendor's (Intel) license server cannot start at boot up time. The net result
    > is that all compilers cannot run. Intel has replied that they are familiar
    > with the problem and they are sending a fix any time now.
    >
    > -MT
    >


    Anything new on this? I have searched trough Intel's site
    and their Premium site but I couldn't find anything.

    /Per Andersson

    FOI, Sweden

  12. Re: IA-64 Linux Question on ProPack 3.0

    They are still 'investigating'. We received word that the person handling the
    license server issues is out on vacation. I hope he is having a great time
    while we are waiting anxiously to start compiling our programs....

    -mt

    On Tue, 15 Jun 2004, Per Andersson wrote:

    p_ Date: Tue, 15 Jun 2004 12:24:34 +0300
    p_ From: Per Andersson
    p_ Newsgroups: comp.sys.sgi.admin, comp.os.linux.setup
    p_ Subject: Re: IA-64 Linux Question on ProPack 3.0
    p_
    p_ Michael E. Thomadakis wrote:
    p_ > Follow up on installing PP3.0
    p_ >
    p_ > The Intel license server that used to work on PP2.4 broke. It appears that the
    p_ > vendor's (Intel) license server cannot start at boot up time. The net result
    p_ > is that all compilers cannot run. Intel has replied that they are familiar
    p_ > with the problem and they are sending a fix any time now.
    p_ >
    p_ > -MT
    p_ >
    p_
    p_ Anything new on this? I have searched trough Intel's site
    p_ and their Premium site but I couldn't find anything.
    p_
    p_ /Per Andersson
    p_
    p_ FOI, Sweden
    p_

  13. Re: IA-64 Linux Question on ProPack 3.0

    Michael E. Thomadakis wrote:
    > They are still 'investigating'. We received word that the person handling the
    > license server issues is out on vacation. I hope he is having a great time
    > while we are waiting anxiously to start compiling our programs....
    >
    > -mt
    >


    > p_ >
    > p_
    > p_ Anything new on this? I have searched trough Intel's site
    > p_ and their Premium site but I couldn't find anything.
    > p_
    > p_ /Per Andersson
    > p_
    > p_ FOI, Sweden
    > p_


    Actually, I finally got an answer from Intel this morning.
    Intel has recived a patch from Macrovision and they are
    testing it right now.

    /Per

  14. IA-64 Linux and IRIX 6.5.24 Jumbo Packet Questions

    Hello,

    We are running an SGI Altix (PP3.0) and SGI Origins (IRIX 6.5.23 and 6.5.24)
    and we are interested in using Jumbo packets and TCP off-loading on their
    Gigabit ethernets (tigon).

    Has anyone had experience with these two (or at least the jumbo packet)
    options on these OS / platform confgurations? The machines will be connected
    together via direct links or via a gigabit switch.

    How can we enable these options? Has anybody noticed buggy or unstable
    behavior with these options? We are transfering large amounts of archived data
    over the network but we have decided to avoid SAN/CXSF approach.

    Thanks
    -MT

  15. Re: IA-64 Linux Question on ProPack 3.0

    I see, that's good news. We've moved the license server to another machine
    (running IRIX) in order to get the compilers working again.

    thanks
    -m

    On Wed, 16 Jun 2004, Per Andersson wrote:

    p% Date: Wed, 16 Jun 2004 06:56:10 +0300
    p% From: Per Andersson
    p% Newsgroups: comp.sys.sgi.admin, comp.os.linux.setup
    p% Subject: Re: IA-64 Linux Question on ProPack 3.0
    p%
    p% Michael E. Thomadakis wrote:
    p% > They are still 'investigating'. We received word that the person handling the
    p% > license server issues is out on vacation. I hope he is having a great time
    p% > while we are waiting anxiously to start compiling our programs....
    p% >
    p% > -mt
    p% >
    p%
    p% > p_ >
    p% > p_
    p% > p_ Anything new on this? I have searched trough Intel's site
    p% > p_ and their Premium site but I couldn't find anything.
    p% > p_
    p% > p_ /Per Andersson
    p% > p_
    p% > p_ FOI, Sweden
    p% > p_
    p%
    p% Actually, I finally got an answer from Intel this morning.
    p% Intel has recived a patch from Macrovision and they are
    p% testing it right now.
    p%
    p% /Per
    p%

  16. Re: IA-64 Linux and IRIX 6.5.24 Jumbo Packet Questions

    In comp.sys.sgi.admin Michael E. Thomadakis wrote:
    > We are running an SGI Altix (PP3.0) and SGI Origins (IRIX 6.5.23 and
    > 6.5.24) and we are interested in using Jumbo packets and TCP
    > off-loading on their Gigabit ethernets (tigon).


    > Has anyone had experience with these two (or at least the jumbo
    > packet) options on these OS / platform confgurations? The machines
    > will be connected together via direct links or via a gigabit switch.


    Have you verified that the switch (and any other system in the
    broadcast domain) will understand Jumbo Frames? If it does not, your
    only options would be:

    1) new switch
    2) back-to-back only
    3) TSO only (large send, whatever different platforms want to call it)
    4) Plain 1500 byte MTU.

    You mentioned "tigon" - I trust you mean Tigon3 (BCM57XX)? I recall
    some experiments with TSO/largesend (on other platforms) with tigon2
    where it would "work" and would offload CPU cycles, but also left one
    with a half-speed NIC because the firmware (never put firmware in the
    datapath...) couldn't keep-up.

    I'd heard that tigon3 was better, but at the same time I thought that
    the Linux folks weren't going that way - I was under the perhaps
    mistaken impression that TSO was on the e1000's but not the tg3 cards.
    If that is different, I'd love to know about it.

    > How can we enable these options? Has anybody noticed buggy or
    > unstable behavior with these options? We are transfering large
    > amounts of archived data over the network but we have decided to
    > avoid SAN/CXSF approach.


    Under Linux 2.6 at least, it would be an ethtool -K thing to enable
    TSO/largesend.

    rick jones
    --
    firebug n, the idiot who tosses a lit cigarette out his car window
    these opinions are mine, all mine; HP might not want them anyway...
    feel free to post, OR email to raj in cup.hp.com but NOT BOTH...

  17. Re: IA-64 Linux and IRIX 6.5.24 Jumbo Packet Questions



    On Thu, 17 Jun 2004, Rick Jones wrote:

    r_ Date: Thu, 17 Jun 2004 21:37:00 GMT
    r_ From: Rick Jones
    r_ Newsgroups: comp.sys.sgi.admin, comp.os.linux.setup
    r_ Subject: Re: IA-64 Linux and IRIX 6.5.24 Jumbo Packet Questions
    r_
    r_ In comp.sys.sgi.admin Michael E. Thomadakis wrote:
    [...]
    r_
    r_ Have you verified that the switch (and any other system in the

    We would get a switch that supports Jumbo packets as soon as we can verify
    that the related hosts do support it.

    r_ broadcast domain) will understand Jumbo Frames? If it does not, your
    r_ only options would be:
    r_
    r_ 1) new switch
    r_ 2) back-to-back only

    We have a few spare ports for direct connections

    r_ 3) TSO only (large send, whatever different platforms want to call it)
    r_ 4) Plain 1500 byte MTU.
    r_
    r_ You mentioned "tigon" - I trust you mean Tigon3 (BCM57XX)? I recall
    r_ some experiments with TSO/largesend (on other platforms) with tigon2
    r_ where it would "work" and would offload CPU cycles, but also left one
    r_ with a half-speed NIC because the firmware (never put firmware in the
    r_ datapath...) couldn't keep-up.

    They are Tigon 3

    r_ I'd heard that tigon3 was better, but at the same time I thought that
    r_ the Linux folks weren't going that way - I was under the perhaps
    r_ mistaken impression that TSO was on the e1000's but not the tg3 cards.
    r_ If that is different, I'd love to know about it.

    I've seen documentation that talks only about the e1000 cards. I could not
    find other cards mentioned.

    r_
    r_ > How can we enable these options? Has anybody noticed buggy or
    r_ > unstable behavior with these options? We are transfering large
    r_ > amounts of archived data over the network but we have decided to
    r_ > avoid SAN/CXSF approach.
    r_
    r_ Under Linux 2.6 at least, it would be an ethtool -K thing to enable
    r_ TSO/largesend.

    Well, Altix Linux is based on 2.4.21 and the 2.6 kernel wont be usable for
    production until the end of this year....


    Thanks for you reply

    -MT

    r_
    r_ rick jones
    r_ --
    r_ firebug n, the idiot who tosses a lit cigarette out his car window
    r_ these opinions are mine, all mine; HP might not want them anyway...
    r_ feel free to post, OR email to raj in cup.hp.com but NOT BOTH...
    r_

  18. Re: IA-64 Linux and IRIX 6.5.24 Jumbo Packet Questions

    In comp.sys.sgi.admin Michael E. Thomadakis wrote:
    > On Thu, 17 Jun 2004, Rick Jones wrote:
    > r_ Have you verified that the switch (and any other system in the


    > We would get a switch that supports Jumbo packets as soon as we can verify
    > that the related hosts do support it.


    OK. Keep in mind that enabling JF on one host means all hosts in the
    broadcast domain (local IPsubnet perhaps) have to support it if you
    are going to do UDP - with TCP you can probably get away with mixed
    hosts thanks to the MSS exchange.

    Also, what some switch vendors call "jumbo frames" is not always the
    same. Some will consider support for frame sizes up to say 2K "jumbo
    frames" rather than the full 9XXX bytes on the link.

    > They are Tigon 3


    OK. IIRC those should do JF just fine.

    > r_ I'd heard that tigon3 was better, but at the same time I thought that
    > r_ the Linux folks weren't going that way - I was under the perhaps
    > r_ mistaken impression that TSO was on the e1000's but not the tg3 cards.
    > r_ If that is different, I'd love to know about it.


    > I've seen documentation that talks only about the e1000 cards. I could not
    > find other cards mentioned.


    That then is consistent with my understanding that TSO is really only
    available on the e1000 cards (or a subset thereof). For your TG3's
    you migth just use JF then.

    > r_ Under Linux 2.6 at least, it would be an ethtool -K thing to enable
    > r_ TSO/largesend.


    > Well, Altix Linux is based on 2.4.21 and the 2.6 kernel wont be
    > usable for production until the end of this year....


    well, there is that... (TSO being in 2.6 kernels)

    it is an "interesting" (as in the ancient chinese curse) situation.
    switch to TSO capable cards and keep the existing switch(es) or stick
    with the current NICs and switch the switch to go JF. I don't have
    absolute numbers at the tip of my fingers, but in broad handwaving
    terms, both would give nice boosts to bulk transfers.

    JF has the advantage of one always "offloading" a 9K, and then getting
    no more than one ACK for every 18 KB you send (the old
    ack-ever-other-segment) stuff. TSO will only "offload" the send side
    and the reciever will generate, and the sender recieve, just as many
    ACKs as before. Sometimes though, TSO may "offload" to more than 9K
    on the sender (at least I think it might but I'm not 100%)

    I think you mentioned some Irix as well? Is that going to do TSO?

    rick jones
    --
    The computing industry isn't as much a game of "Follow The Leader" as
    it is one of "Ring Around the Rosy" or perhaps "Duck Duck Goose."
    - Rick Jones
    these opinions are mine, all mine; HP might not want them anyway...
    feel free to post, OR email to raj in cup.hp.com but NOT BOTH...

  19. Re: IA-64 Linux and IRIX 6.5.24 Jumbo Packet Questions



    On Fri, 18 Jun 2004, Rick Jones wrote:

    r- Date: Fri, 18 Jun 2004 19:46:56 GMT
    r- From: Rick Jones
    r- Newsgroups: comp.sys.sgi.admin, comp.os.linux.setup
    r- Subject: Re: IA-64 Linux and IRIX 6.5.24 Jumbo Packet Questions
    r-
    r- In comp.sys.sgi.admin Michael E. Thomadakis wrote:
    r- > On Thu, 17 Jun 2004, Rick Jones wrote:
    r- > r_ Have you verified that the switch (and any other system in the
    r-

    [...]

    r- OK. Keep in mind that enabling JF on one host means all hosts in the
    r- broadcast domain (local IPsubnet perhaps) have to support it if you
    r- are going to do UDP - with TCP you can probably get away with mixed
    r- hosts thanks to the MSS exchange.

    We plan to enable Jumbo frames within an isolated LAN that connects
    all of our bigger hosts together. We don't want to use jumbo frames on a
    public subnet.

    r-
    r- Also, what some switch vendors call "jumbo frames" is not always the
    r- same. Some will consider support for frame sizes up to say 2K "jumbo
    r- frames" rather than the full 9XXX bytes on the link.

    Any suggestion on reliable gig-e switch that supports JF along possibly with
    other goodies, such as flow control/802.1q ? Ideally we would like to have one
    host feeding two ether fibers to the switch with connections then fanning out
    to the rest of our hosts. That host is an SGI O3800 with IRIX 6.5.24. I am
    trying to find documentation on how this can be achieved, especially if there
    are ether switches supporting the idea of trunked ports on the one side and
    single ports on the other. I may be looking for too much....

    r-
    r- > They are Tigon 3
    r-
    r- OK. IIRC those should do JF just fine.

    I didn't mention that one of the other hosts is an IBM p690 with GigE fiber
    that supposedly supports JF + TCP off-loading. The interoperability is a big ?


    r-
    r- > r_ I'd heard that tigon3 was better, but at the same time I thought that
    r- > r_ the Linux folks weren't going that way - I was under the perhaps

    I was recently at the SGI User's Group conference and SGI folks said that at
    least JF are supported on the ALTIX Linux. IRIX is more advanced in this
    (and other ;-) respects. The more fancy networking should come with the 2.6
    based kernels. Although we are not strapped for processors, it is nice to
    remove as much TCP/IP processing workload from the kernel. I've seen drastic
    decrease in CPU utilization when systems use JF. And even better one with full
    TCP off-loading.

    When 10GigE becomes mainstream, TCP offloading will be a major concern.

    r-
    r- > I've seen documentation that talks only about the e1000 cards. I could not
    r- > find other cards mentioned.
    r-
    r- That then is consistent with my understanding that TSO is really only
    r- available on the e1000 cards (or a subset thereof). For your TG3's
    r- you migth just use JF then.

    Alonmg with the ALTIX we have IRIX and AIX based machines which claim JF
    support. I will try to connect hosts point-to-point at first and see that at
    least they can use JFs smoothly.

    r-
    r- > r_ Under Linux 2.6 at least, it would be an ethtool -K thing to enable
    r- > r_ TSO/largesend.
    r-
    r- > Well, Altix Linux is based on 2.4.21 and the 2.6 kernel wont be
    r- > usable for production until the end of this year....
    r-
    r- well, there is that... (TSO being in 2.6 kernels)
    r-
    r- it is an "interesting" (as in the ancient chinese curse) situation.
    r- switch to TSO capable cards and keep the existing switch(es) or stick
    r- with the current NICs and switch the switch to go JF. I don't have
    r- absolute numbers at the tip of my fingers, but in broad handwaving
    r- terms, both would give nice boosts to bulk transfers.
    r-
    r- JF has the advantage of one always "offloading" a 9K, and then getting
    r- no more than one ACK for every 18 KB you send (the old
    r- ack-ever-other-segment) stuff. TSO will only "offload" the send side
    r- and the reciever will generate, and the sender recieve, just as many
    r- ACKs as before. Sometimes though, TSO may "offload" to more than 9K
    r- on the sender (at least I think it might but I'm not 100%)

    I think that in our specialized environment with isolated LANs and our large
    data tranfers, even JF by itself should give us a decent throughput.

    r-
    r- I think you mentioned some Irix as well? Is that going to do TSO?
    r-
    I am not sure if IRIX supports this, but it does support JF. IRIX also
    supports port stripping but I don't know if we can get a switch that can
    leverage this.

    BTW, are you still maintaining the TCP benchmarking tool?

    r- rick jones
    r- --
    r- The computing industry isn't as much a game of "Follow The Leader" as
    r- it is one of "Ring Around the Rosy" or perhaps "Duck Duck Goose."
    r- - Rick Jones
    r- these opinions are mine, all mine; HP might not want them anyway...
    r- feel free to post, OR email to raj in cup.hp.com but NOT BOTH...
    r-


    -MT


  20. Re: IA-64 Linux and IRIX 6.5.24 Jumbo Packet Questions

    In comp.protocols.tcp-ip Michael E. Thomadakis wrote:
    > On Fri, 18 Jun 2004, Rick Jones wrote:


    > Any suggestion on reliable gig-e switch that supports JF along
    > possibly with other goodies, such as flow control/802.1q ?


    Well, I believe that some of the HP ProCurve's support JumboFrame
    However, not all of them do, so do be sure to read all the specs
    first. I've not looked into 802.1q.

    > Ideally we would like to have one host feeding two ether fibers to
    > the switch with connections then fanning out to the rest of our
    > hosts. That host is an SGI O3800 with IRIX 6.5.24. I am trying to
    > find documentation on how this can be achieved, especially if there
    > are ether switches supporting the idea of trunked ports on the one
    > side and single ports on the other. I may be looking for too
    > much....


    Well, it may be a question better suited to comp.dcom.lans.ethernet,
    but I believe what you want is a swtich with LACP (Link Aggregation
    Control Protocol) support. You would then be able to
    bond/stripe/trunk links from one host to the swtich and have just a
    single link from another host to the switch.

    > r- > They are Tigon 3
    > r-
    > r- OK. IIRC those should do JF just fine.


    > I didn't mention that one of the other hosts is an IBM p690 with
    > GigE fiber that supposedly supports JF + TCP off-loading. The
    > interoperability is a big ?


    IIRC, the NICs used on the pSeries are Tigon[23]. IBM even did "large
    send" (what they called TSO) on the Tigon2's and (presumeably) lived
    with the real throughput limitatoins on the Tigon2 NICs in that mode
    (32 NICs on a SPECweb99 setup for a 16-CPU system back when 16 CPU
    systems didn't need more than 16 full GbE's worth of bandwidth, if
    even that) Since then I think that IBM has gone with either Tigon3 (I
    guess I should start calling that BCM570X...) or Intel Anvik mumble.
    I cannot remember which.

    I suspect that your biggest issue will be the switch and the trunking.

    > I was recently at the SGI User's Group conference and SGI folks said
    > that at least JF are supported on the ALTIX Linux. IRIX is more
    > advanced in this (and other ;-) respects. The more fancy networking
    > should come with the 2.6 based kernels. Although we are not strapped
    > for processors, it is nice to remove as much TCP/IP processing
    > workload from the kernel. I've seen drastic decrease in CPU
    > utilization when systems use JF. And even better one with full TCP
    > off-loading.


    Just remember there's no such thing as a free lunch, nor a good deed
    that goes unpunished

    > When 10GigE becomes mainstream, TCP offloading will be a major
    > concern.


    It might.


    > r- > I've seen documentation that talks only about the e1000
    > r- > cards. I could not find other cards mentioned.
    > r- That then is consistent with my understanding that TSO is really
    > r- only available on the e1000 cards (or a subset thereof). For
    > r- your TG3's you migth just use JF then.


    > Alonmg with the ALTIX we have IRIX and AIX based machines which
    > claim JF support. I will try to connect hosts point-to-point at
    > first and see that at least they can use JFs smoothly.


    > r-
    > r- I think you mentioned some Irix as well? Is that going to do TSO?
    > r-
    > I am not sure if IRIX supports this, but it does support JF. IRIX
    > also supports port stripping but I don't know if we can get a switch
    > that can leverage this.


    If the IRIX striping can speak IEEE LACP you should be OK. That, or
    if it will speak Cisco FEC (Fast EtherChannel).

    > BTW, are you still maintaining the TCP benchmarking tool?


    If you mean netperf, yes I'm "this close" to kicking netperf 2.3
    out the door.

    rick jones
    --
    firebug n, the idiot who tosses a lit cigarette out his car window
    these opinions are mine, all mine; HP might not want them anyway...
    feel free to post, OR email to raj in cup.hp.com but NOT BOTH...

+ Reply to Thread
Page 1 of 2 1 2 LastLast