Re: Best-performing disk I/O options for a DBMS server on 7-STABLE - FreeBSD

This is a discussion on Re: Best-performing disk I/O options for a DBMS server on 7-STABLE - FreeBSD ; On Thu, May 08, 2008 at 10:30:01AM +0100, Pete French wrote: > > I assume SCSI is the best path forward (either SA/SCSI or traditional) but > > have been out of the loop on the card(s) that work properly ...

+ Reply to Thread
Results 1 to 17 of 17

Thread: Re: Best-performing disk I/O options for a DBMS server on 7-STABLE

  1. Re: Best-performing disk I/O options for a DBMS server on 7-STABLE

    On Thu, May 08, 2008 at 10:30:01AM +0100, Pete French wrote:
    > > I assume SCSI is the best path forward (either SA/SCSI or traditional) but
    > > have been out of the loop on the card(s) that work properly for a good long
    > > while.

    >
    > HP P400 cards are PCI express and SAS - they work very well under FreeBSD.
    > I've also used the cheaper E200 and that appears to be fine too, though I
    > havent run it for as long as the 400's.
    >
    > http://h18004.www1.hp.com/products/s...400/index.html
    >
    > -pete.
    >


    Following upon my own stuff, I just laid hands on a Quad-Xeon machine with an
    Intel SRCSAS18E card in it, and well, the only way I can describe it is "oh
    my GOD!"

    Off a couple of older 250GB SATA drives, mirrored, it sustained 70MB/sec over
    the entire disk's volume space on reads and had ZERO visible impact on CPU
    OR another process doing mixed I/O at the same time (!).

    That's impressive.

    If its stable.

    The cards are not cheap, however.

    The only "gotcha" is that all configuration of the drive(s) appears to be
    done through the controller. I assume I COULD use something like gmirror,
    but see little reason to do so.

    -- Karl Denninger
    karl@denninger.net


    _______________________________________________
    freebsd-stable@freebsd.org mailing list
    http://lists.freebsd.org/mailman/lis...freebsd-stable
    To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"


  2. Management interface for cards powered by the "mfi" driver?

    On Mon, Jun 16, 2008 at 08:20:53PM -0500, Karl Denninger wrote:
    > On Thu, May 08, 2008 at 10:30:01AM +0100, Pete French wrote:
    > > > I assume SCSI is the best path forward (either SA/SCSI or traditional) but
    > > > have been out of the loop on the card(s) that work properly for a good long
    > > > while.

    > >
    > > HP P400 cards are PCI express and SAS - they work very well under FreeBSD.
    > > I've also used the cheaper E200 and that appears to be fine too, though I
    > > havent run it for as long as the 400's.
    > >
    > > http://h18004.www1.hp.com/products/s...400/index.html
    > >
    > > -pete.
    > >

    >
    > Following upon my own stuff, I just laid hands on a Quad-Xeon machine with an
    > Intel SRCSAS18E card in it, and well, the only way I can describe it is "oh
    > my GOD!"
    >
    > Off a couple of older 250GB SATA drives, mirrored, it sustained 70MB/sec over
    > the entire disk's volume space on reads and had ZERO visible impact on CPU
    > OR another process doing mixed I/O at the same time (!).
    >
    > That's impressive.
    >
    > If its stable.
    >
    > The cards are not cheap, however.
    >
    > The only "gotcha" is that all configuration of the drive(s) appears to be
    > done through the controller. I assume I COULD use something like gmirror,
    > but see little reason to do so.
    >
    > -- Karl Denninger
    > karl@denninger.net


    Ok, another followup!

    Is there a mangement interface program for the "mfi" driver somewhere? I
    rooted around in "ports" but didn't find one, nor on the base system.

    The Linux ones want to talk to the "amr" driver which is the older LSI Logic
    MegaRAID boards, not the newer ones that the mfi driver talks to.

    No management tool = el-sucko, because you can't rebuild a failed disk or
    even shut the alarm on the board off!

    -- Karl Denninger
    karl@denninger.net


    _______________________________________________
    freebsd-stable@freebsd.org mailing list
    http://lists.freebsd.org/mailman/lis...freebsd-stable
    To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"


  3. Re: Management interface for cards powered by the "mfi" driver?

    On Tue, Jun 17, 2008 at 11:15:30PM -0500, Karl Denninger wrote:
    > On Mon, Jun 16, 2008 at 08:20:53PM -0500, Karl Denninger wrote:
    > > On Thu, May 08, 2008 at 10:30:01AM +0100, Pete French wrote:
    > > > > I assume SCSI is the best path forward (either SA/SCSI or traditional) but
    > > > > have been out of the loop on the card(s) that work properly for a good long
    > > > > while.
    > > >
    > > > HP P400 cards are PCI express and SAS - they work very well under FreeBSD.
    > > > I've also used the cheaper E200 and that appears to be fine too, though I
    > > > havent run it for as long as the 400's.
    > > >
    > > > http://h18004.www1.hp.com/products/s...400/index.html
    > > >
    > > > -pete.
    > > >

    > >
    > > Following upon my own stuff, I just laid hands on a Quad-Xeon machine with an
    > > Intel SRCSAS18E card in it, and well, the only way I can describe it is "oh
    > > my GOD!"

    >
    > Ok, another followup!
    >
    > Is there a mangement interface program for the "mfi" driver somewhere? I
    > rooted around in "ports" but didn't find one, nor on the base system.
    >
    > The Linux ones want to talk to the "amr" driver which is the older LSI Logic
    > MegaRAID boards, not the newer ones that the mfi driver talks to.
    >
    > No management tool = el-sucko, because you can't rebuild a failed disk or
    > even shut the alarm on the board off!


    Its sysutils/linux-megacli.


    Andrew
    _______________________________________________
    freebsd-stable@freebsd.org mailing list
    http://lists.freebsd.org/mailman/lis...freebsd-stable
    To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"


  4. Re: Management interface for cards powered by the "mfi" driver?

    On Tue, Jun 17, 2008 at 11:15:30PM -0500, Karl Denninger wrote:
    > On Mon, Jun 16, 2008 at 08:20:53PM -0500, Karl Denninger wrote:
    > > On Thu, May 08, 2008 at 10:30:01AM +0100, Pete French wrote:
    > > > > I assume SCSI is the best path forward (either SA/SCSI or traditional) but
    > > > > have been out of the loop on the card(s) that work properly for a good long
    > > > > while.
    > > >
    > > > HP P400 cards are PCI express and SAS - they work very well under FreeBSD.
    > > > I've also used the cheaper E200 and that appears to be fine too, though I
    > > > havent run it for as long as the 400's.
    > > >
    > > > http://h18004.www1.hp.com/products/s...400/index.html
    > > >
    > > > -pete.
    > > >

    > >
    > > Following upon my own stuff, I just laid hands on a Quad-Xeon machine with an
    > > Intel SRCSAS18E card in it, and well, the only way I can describe it is "oh
    > > my GOD!"
    > >
    > > Off a couple of older 250GB SATA drives, mirrored, it sustained 70MB/sec over
    > > the entire disk's volume space on reads and had ZERO visible impact on CPU
    > > OR another process doing mixed I/O at the same time (!).
    > >
    > > That's impressive.
    > >
    > > If its stable.
    > >
    > > The cards are not cheap, however.
    > >
    > > The only "gotcha" is that all configuration of the drive(s) appears to be
    > > done through the controller. I assume I COULD use something like gmirror,
    > > but see little reason to do so.
    > >
    > > -- Karl Denninger
    > > karl@denninger.net

    >
    > Ok, another followup!
    >
    > Is there a mangement interface program for the "mfi" driver somewhere? I
    > rooted around in "ports" but didn't find one, nor on the base system.
    >
    > The Linux ones want to talk to the "amr" driver which is the older LSI Logic
    > MegaRAID boards, not the newer ones that the mfi driver talks to.
    >
    > No management tool = el-sucko, because you can't rebuild a failed disk or
    > even shut the alarm on the board off!



    The sysutil/linux-megacli port looks like it might do the trick.



    --

    Erik Trulsson
    ertr1013@student.uu.se
    _______________________________________________
    freebsd-stable@freebsd.org mailing list
    http://lists.freebsd.org/mailman/lis...freebsd-stable
    To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"


  5. Re: Management interface for cards powered by the "mfi" driver?

    On Tue, Jun 17, 2008 at 10:05:31PM -0700, Andrew Thompson wrote:
    > On Tue, Jun 17, 2008 at 11:15:30PM -0500, Karl Denninger wrote:
    > > On Mon, Jun 16, 2008 at 08:20:53PM -0500, Karl Denninger wrote:
    > > > On Thu, May 08, 2008 at 10:30:01AM +0100, Pete French wrote:
    > > > > > I assume SCSI is the best path forward (either SA/SCSI or traditional) but
    > > > > > have been out of the loop on the card(s) that work properly for a good long
    > > > > > while.
    > > > >
    > > > > HP P400 cards are PCI express and SAS - they work very well under FreeBSD.
    > > > > I've also used the cheaper E200 and that appears to be fine too, though I
    > > > > havent run it for as long as the 400's.
    > > > >
    > > > > http://h18004.www1.hp.com/products/s...400/index.html
    > > > >
    > > > > -pete.
    > > > >
    > > >
    > > > Following upon my own stuff, I just laid hands on a Quad-Xeon machine with an
    > > > Intel SRCSAS18E card in it, and well, the only way I can describe it is "oh
    > > > my GOD!"

    > >
    > > Ok, another followup!
    > >
    > > Is there a mangement interface program for the "mfi" driver somewhere? I
    > > rooted around in "ports" but didn't find one, nor on the base system.
    > >
    > > The Linux ones want to talk to the "amr" driver which is the older LSI Logic
    > > MegaRAID boards, not the newer ones that the mfi driver talks to.
    > >
    > > No management tool = el-sucko, because you can't rebuild a failed disk or
    > > even shut the alarm on the board off!

    >
    > Its sysutils/linux-megacli.


    Nope. It builds and installs but doesn't work (already tried this one.)

    Attempting to ask it for the number of boards returns "0".

    This may have something to do with changes made to the device structure(s)
    in FreeBSD-7 (I don't have a 6 machine to test on any more) but it
    definitely does not function.

    -- Karl Denninger
    karl@denninger.net


    _______________________________________________
    freebsd-stable@freebsd.org mailing list
    http://lists.freebsd.org/mailman/lis...freebsd-stable
    To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"


  6. Re: Management interface for cards powered by the "mfi" driver?



    Karl Denninger wrote:
    > On Tue, Jun 17, 2008 at 10:05:31PM -0700, Andrew Thompson wrote:
    >
    >> On Tue, Jun 17, 2008 at 11:15:30PM -0500, Karl Denninger wrote:
    >>
    >>> On Mon, Jun 16, 2008 at 08:20:53PM -0500, Karl Denninger wrote:
    >>>
    >>>> On Thu, May 08, 2008 at 10:30:01AM +0100, Pete French wrote:
    >>>>
    >>>>>> I assume SCSI is the best path forward (either SA/SCSI or traditional) but
    >>>>>> have been out of the loop on the card(s) that work properly for a good long
    >>>>>> while.
    >>>>>>
    >>>>> HP P400 cards are PCI express and SAS - they work very well under FreeBSD.
    >>>>> I've also used the cheaper E200 and that appears to be fine too, though I
    >>>>> havent run it for as long as the 400's.
    >>>>>
    >>>>> http://h18004.www1.hp.com/products/s...400/index.html
    >>>>>
    >>>>> -pete.
    >>>>>
    >>>>>
    >>>> Following upon my own stuff, I just laid hands on a Quad-Xeon machine with an
    >>>> Intel SRCSAS18E card in it, and well, the only way I can describe it is "oh
    >>>> my GOD!"
    >>>>
    >>> Ok, another followup!
    >>>
    >>> Is there a mangement interface program for the "mfi" driver somewhere? I
    >>> rooted around in "ports" but didn't find one, nor on the base system.
    >>>
    >>> The Linux ones want to talk to the "amr" driver which is the older LSI Logic
    >>> MegaRAID boards, not the newer ones that the mfi driver talks to.
    >>>
    >>> No management tool = el-sucko, because you can't rebuild a failed disk or
    >>> even shut the alarm on the board off!
    >>>

    >> Its sysutils/linux-megacli.
    >>

    >
    > Nope. It builds and installs but doesn't work (already tried this one.)
    >
    > Attempting to ask it for the number of boards returns "0".
    >
    > This may have something to do with changes made to the device structure(s)
    > in FreeBSD-7 (I don't have a 6 machine to test on any more) but it
    > definitely does not function.
    >

    It works fine for me on very recent 7-STABLE.

    megacli -CfgDsply -a0

    ================================================== ============================
    Adapter: 0
    Product Name: PERC 5/i Integrated
    Memory: 256MB
    BBU: Present
    Serial No: 12345
    ================================================== ============================
    -cut-

    megacli -AdpAllInfo -aALL also works without a problem.

    7.0-STABLE-200805 FreeBSD 7.0-STABLE-200805 #0: Sun May 11 11:55:15 UTC
    2008 root@driscoll.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64

    Did you follow all steps from ports/sysutils/linux-megacli/pkg-message ?
    Though I think there is a small bug in the port (maintainer CCed)
    It wants compat.linux.osrelease=2.6.12, but I think it should be
    compat.linux.osrelease=2.6.16 ?
    Just for reference I installed it with OVERRIDE_LINUX_BASE_PORT=f8
    I think you are confusing mega-cli with megamgr (which is the one that
    work with amr driver)

    --

    Best Wishes,
    Stefan Lambrev
    ICQ# 24134177

    _______________________________________________
    freebsd-stable@freebsd.org mailing list
    http://lists.freebsd.org/mailman/lis...freebsd-stable
    To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"


  7. Re: Management interface for cards powered by the "mfi" driver?


    Hi,

    On 18 Jun 2008, at 12:29, Stefan Lambrev wrote:

    > Did you follow all steps from ports/sysutils/linux-megacli/pkg-
    > message ?
    > Though I think there is a small bug in the port (maintainer CCed)
    > It wants compat.linux.osrelease=2.6.12, but I think it should be
    > compat.linux.osrelease=2.6.16 ?


    Last time I checked it was still 2.6.12. it still is set to that value
    on our 2950's (running 6.2 and linux_base-fc-4_9)

    > Just for reference I installed it with OVERRIDE_LINUX_BASE_PORT=f8
    > I think you are confusing mega-cli with megamgr (which is the one
    > that work with amr driver)


    People should only realise that they are running a linux binary under
    emulation in order to manage their mission critical servers. It is a
    thing you might not like.

    Regards,
    Ruben
    _______________________________________________
    freebsd-stable@freebsd.org mailing list
    http://lists.freebsd.org/mailman/lis...freebsd-stable
    To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"


  8. Re: Management interface for cards powered by the "mfi" driver?

    On Wed, Jun 18, 2008 at 01:29:56PM +0300, Stefan Lambrev wrote:
    > Karl Denninger wrote:
    > >On Tue, Jun 17, 2008 at 10:05:31PM -0700, Andrew Thompson wrote:
    > >
    > >>On Tue, Jun 17, 2008 at 11:15:30PM -0500, Karl Denninger wrote:
    > >>>>Intel SRCSAS18E card in it, and well, the only way I can describe it is
    > >>>>"oh
    > >>>>my GOD!"
    > >>>>
    > >>>Ok, another followup!
    > >>>
    > >>>Is there a mangement interface program for the "mfi" driver somewhere? I
    > >>>rooted around in "ports" but didn't find one, nor on the base system.
    > >>>
    > >>>The Linux ones want to talk to the "amr" driver which is the older LSI
    > >>>Logic
    > >>>MegaRAID boards, not the newer ones that the mfi driver talks to.
    > >>>
    > >>>No management tool = el-sucko, because you can't rebuild a failed disk or
    > >>>even shut the alarm on the board off!
    > >>>
    > >>Its sysutils/linux-megacli.
    > >>

    > >
    > >Nope. It builds and installs but doesn't work (already tried this one.)
    > >
    > >Attempting to ask it for the number of boards returns "0".
    > >
    > >This may have something to do with changes made to the device structure(s)
    > >in FreeBSD-7 (I don't have a 6 machine to test on any more) but it
    > >definitely does not function.
    > >

    > It works fine for me on very recent 7-STABLE.
    >
    > megacli -CfgDsply -a0
    >
    > ================================================== ============================
    > Adapter: 0
    > Product Name: PERC 5/i Integrated
    > Memory: 256MB
    > BBU: Present
    > Serial No: 12345
    > ================================================== ============================
    > -cut-
    >
    > megacli -AdpAllInfo -aALL also works without a problem.
    >
    > 7.0-STABLE-200805 FreeBSD 7.0-STABLE-200805 #0: Sun May 11 11:55:15 UTC
    > 2008 root@driscoll.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64
    >
    > Did you follow all steps from ports/sysutils/linux-megacli/pkg-message ?
    > Though I think there is a small bug in the port (maintainer CCed)
    > It wants compat.linux.osrelease=2.6.12, but I think it should be
    > compat.linux.osrelease=2.6.16 ?
    > Just for reference I installed it with OVERRIDE_LINUX_BASE_PORT=f8
    > I think you are confusing mega-cli with megamgr (which is the one that
    > work with amr driver)


    dbms# megacli -CfgDsply -a0

    Failed to get ControllerId List.
    Failed to get CpController object.
    dbms#

    But the card is definitely there; this is what it last logged when I manually
    yanked one of the mirrored drives, then shut down and told it to rebuild
    from the GUI before boot and let it go on its way:

    mfi0: 501 - PD 10(e252/s1) progress 99% seconds 7017s: Rebuild progress on
    PD 0a(e0xfc/s1) is 98.94%(7017s)
    mfi0: 502 - PD 10(e252/s1) progress 100% seconds 7158s: Rebuild progress on
    PD 0a(e0xfc/s1) is 99.94%(7158s)
    mfi0: 503 - PD 10(e252/s1) event: Rebuild complete on PD 0a(e0xfc/s1)
    mfi0: 504 - VD 00/0 state prior 2 new 3: State change on VD 00/0 from
    DEGRADED(2) to OPTIMAL(3)
    mfi0: 505 - VD 00/0 event: VD 00/0 is now OPTIMAL
    mfi0: 506 - PD 10(e252/s1) state prior 20 new 24: State change on PD
    0a(e0xfc/s1) from REBUILD(14) to ONLINE(18)

    Oook!

    So we have a card that the driver is perfectly happy with, but the
    management interface can't find.

    Note that this is the Intel board; I grabbed the Intel version of the
    software for Linux (which appears to be essentially the same code but under
    Intel's name), set the branding so the Linux emulator would play nice, and
    got the same thing:

    dbms# ./CmdTool2 -Cfgdsply -a0

    Failed to get ControllerId List.
    Failed to get CpController object.
    dbms#

    Hmmmm... this is on a new machine that I loaded 7-RELEASE on and then
    grabbed the CVS tree and pulled a -rRELENG_7 on and rebuilt.

    Let me dig around a bit..... I wonder if I got a "funny" in here...
    looking at kernel versions, it appears I might.

    If so, that's going to be an interesting problem (as in "how did that happen?")
    to figure out.

    uname -v is returning:
    FreeBSD 7.0-CURRENT-200608 #0: Mon Jun 16 20:54:25 UTC 2008 karl@dbms.denninger.net:/usr/obj/usr/src/sys/GENERIC

    How'd that happen with the tag "RELENG_7" in the tag? (and yes, it IS
    in there in the CVS directory in the "Tag" file....)

    -- Karl Denninger
    karl@denninger.net


    _______________________________________________
    freebsd-stable@freebsd.org mailing list
    http://lists.freebsd.org/mailman/lis...freebsd-stable
    To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"


  9. Re: Management interface for cards powered by the "mfi" driver?

    Hi,

    Ruben van Staveren wrote:
    >
    > Hi,
    >
    > On 18 Jun 2008, at 12:29, Stefan Lambrev wrote:
    >
    >> Did you follow all steps from ports/sysutils/linux-megacli/pkg-message ?
    >> Though I think there is a small bug in the port (maintainer CCed)
    >> It wants compat.linux.osrelease=2.6.12, but I think it should be
    >> compat.linux.osrelease=2.6.16 ?

    >
    > Last time I checked it was still 2.6.12. it still is set to that value
    > on our 2950's (running 6.2 and linux_base-fc-4_9)

    I never saw 2.6.12 in documentation, but you may be right. Tough in
    ports/UPDATING only 2.6.16 is mention.
    http://blogs.freebsdish.org/netchild...d/linuxolator/
    speaks also only for 2.6.16
    >
    >> Just for reference I installed it with OVERRIDE_LINUX_BASE_PORT=f8
    >> I think you are confusing mega-cli with megamgr (which is the one
    >> that work with amr driver)

    >
    > People should only realise that they are running a linux binary under
    > emulation in order to manage their mission critical servers. It is a
    > thing you might not like.

    It has been always this way with lsi ? Actually the difference now is
    that we are forced to use tools under beta/experimental linuxolator
    >
    > Regards,
    > Ruben


    --

    Best Wishes,
    Stefan Lambrev
    ICQ# 24134177

    _______________________________________________
    freebsd-stable@freebsd.org mailing list
    http://lists.freebsd.org/mailman/lis...freebsd-stable
    To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"


  10. Re: Management interface for cards powered by the "mfi" driver?

    Hi Karl,

    Karl Denninger wrote:
    > On Wed, Jun 18, 2008 at 01:29:56PM +0300, Stefan Lambrev wrote:
    >
    >> Karl Denninger wrote:
    >>
    >>> On Tue, Jun 17, 2008 at 10:05:31PM -0700, Andrew Thompson wrote:
    >>>
    >>>
    >>>> On Tue, Jun 17, 2008 at 11:15:30PM -0500, Karl Denninger wrote:
    >>>>
    >>>>>> Intel SRCSAS18E card in it, and well, the only way I can describe it is
    >>>>>> "oh
    >>>>>> my GOD!"
    >>>>>>
    >>>>>>
    >>>>> Ok, another followup!
    >>>>>
    >>>>> Is there a mangement interface program for the "mfi" driver somewhere? I
    >>>>> rooted around in "ports" but didn't find one, nor on the base system.
    >>>>>
    >>>>> The Linux ones want to talk to the "amr" driver which is the older LSI
    >>>>> Logic
    >>>>> MegaRAID boards, not the newer ones that the mfi driver talks to.
    >>>>>
    >>>>> No management tool = el-sucko, because you can't rebuild a failed disk or
    >>>>> even shut the alarm on the board off!
    >>>>>
    >>>>>
    >>>> Its sysutils/linux-megacli.
    >>>>
    >>>>
    >>> Nope. It builds and installs but doesn't work (already tried this one.)
    >>>
    >>> Attempting to ask it for the number of boards returns "0".
    >>>
    >>> This may have something to do with changes made to the device structure(s)
    >>> in FreeBSD-7 (I don't have a 6 machine to test on any more) but it
    >>> definitely does not function.
    >>>
    >>>

    >> It works fine for me on very recent 7-STABLE.
    >>
    >> megacli -CfgDsply -a0
    >>
    >> ================================================== ============================
    >> Adapter: 0
    >> Product Name: PERC 5/i Integrated
    >> Memory: 256MB
    >> BBU: Present
    >> Serial No: 12345
    >> ================================================== ============================
    >> -cut-
    >>
    >> megacli -AdpAllInfo -aALL also works without a problem.
    >>
    >> 7.0-STABLE-200805 FreeBSD 7.0-STABLE-200805 #0: Sun May 11 11:55:15 UTC
    >> 2008 root@driscoll.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64
    >>
    >> Did you follow all steps from ports/sysutils/linux-megacli/pkg-message ?
    >> Though I think there is a small bug in the port (maintainer CCed)
    >> It wants compat.linux.osrelease=2.6.12, but I think it should be
    >> compat.linux.osrelease=2.6.16 ?
    >> Just for reference I installed it with OVERRIDE_LINUX_BASE_PORT=f8
    >> I think you are confusing mega-cli with megamgr (which is the one that
    >> work with amr driver)
    >>

    >
    > dbms# megacli -CfgDsply -a0
    >
    > Failed to get ControllerId List.
    > Failed to get CpController object.
    > dbms#
    >

    Sorry if you already answer this and I missed it, but do you have
    mfi_linux, linprocfs, linsysfs loaded as modules (or build in kernel)?
    Also do you have mounted:
    linprocfs on /usr/compat/linux/proc (linprocfs, local)
    linsysfs on /usr/compat/linux/sys (linsysfs, local)
    If this is done I'm really out of ideas.
    > But the card is definitely there; this is what it last logged when I manually
    > yanked one of the mirrored drives, then shut down and told it to rebuild
    > from the GUI before boot and let it go on its way:
    >
    > mfi0: 501 - PD 10(e252/s1) progress 99% seconds 7017s: Rebuild progress on
    > PD 0a(e0xfc/s1) is 98.94%(7017s)
    > mfi0: 502 - PD 10(e252/s1) progress 100% seconds 7158s: Rebuild progress on
    > PD 0a(e0xfc/s1) is 99.94%(7158s)
    > mfi0: 503 - PD 10(e252/s1) event: Rebuild complete on PD 0a(e0xfc/s1)
    > mfi0: 504 - VD 00/0 state prior 2 new 3: State change on VD 00/0 from
    > DEGRADED(2) to OPTIMAL(3)
    > mfi0: 505 - VD 00/0 event: VD 00/0 is now OPTIMAL
    > mfi0: 506 - PD 10(e252/s1) state prior 20 new 24: State change on PD
    > 0a(e0xfc/s1) from REBUILD(14) to ONLINE(18)
    >
    > Oook!
    >
    > So we have a card that the driver is perfectly happy with, but the
    > management interface can't find.
    >
    > Note that this is the Intel board; I grabbed the Intel version of the
    > software for Linux (which appears to be essentially the same code but under
    > Intel's name), set the branding so the Linux emulator would play nice, and
    > got the same thing:
    >
    > dbms# ./CmdTool2 -Cfgdsply -a0
    >
    > Failed to get ControllerId List.
    > Failed to get CpController object.
    > dbms#
    >
    > Hmmmm... this is on a new machine that I loaded 7-RELEASE on and then
    > grabbed the CVS tree and pulled a -rRELENG_7 on and rebuilt.
    >
    > Let me dig around a bit..... I wonder if I got a "funny" in here...
    > looking at kernel versions, it appears I might.
    >
    > If so, that's going to be an interesting problem (as in "how did that happen?")
    > to figure out.
    >
    > uname -v is returning:
    > FreeBSD 7.0-CURRENT-200608 #0: Mon Jun 16 20:54:25 UTC 2008 karl@dbms.denninger.net:/usr/obj/usr/src/sys/GENERIC
    >
    > How'd that happen with the tag "RELENG_7" in the tag? (and yes, it IS
    > in there in the CVS directory in the "Tag" file....)
    >
    > -- Karl Denninger
    > karl@denninger.net
    >
    >
    > _______________________________________________
    > freebsd-stable@freebsd.org mailing list
    > http://lists.freebsd.org/mailman/lis...freebsd-stable
    > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"
    >


    --

    Best Wishes,
    Stefan Lambrev
    ICQ# 24134177

    _______________________________________________
    freebsd-stable@freebsd.org mailing list
    http://lists.freebsd.org/mailman/lis...freebsd-stable
    To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"


  11. Re: Management interface for cards powered by the "mfi" driver?


    On 18 Jun 2008, at 16:07, Stefan Lambrev wrote:

    >> Last time I checked it was still 2.6.12. it still is set to that
    >> value on our 2950's (running 6.2 and linux_base-fc-4_9)

    > I never saw 2.6.12 in documentation, but you may be right. Tough in
    > ports/UPDATING only 2.6.16 is mention.
    > http://blogs.freebsdish.org/netchild...d/linuxolator/
    > speaks also only for 2.6.16


    I got this from reading the commit message of r1.5 of http://www.freebsd.org/cgi/cvsweb.cg.../dev/mfi/mfi.c

    I don't know what the change set between 2.6.12 and 2.6.16 is
    regarding the linuxolator. As I see it the difference should only in
    more syscalls being supported. So if megacli has enough to get going
    on by setting it to just 2.6.12 it should not have a requirement for
    2.6.16. Correct me if I am wrong (and I'll fix the instructions in the
    port)

    Of course, if you think mfi(4) should support your board csup up to
    the latest stable and see whether it works. Also run through /usr/
    local/sbin/megacli instead of /usr/local/libexec/MegaCli as it will
    perform some sanity checks.

    >> People should only realise that they are running a linux binary
    >> under emulation in order to manage their mission critical servers.
    >> It is a thing you might not like.

    > It has been always this way with lsi ? Actually the difference now
    > is that we are forced to use tools under beta/experimental
    > linuxolator
    >>


    Sad but true. But I managed with ease to replace a hot spare not that
    long ago.

    You might want to have a peek at the cheat sheet hosted at http://tools.rapidsoft.de/perc/

    Cheers,
    Ruben


    -----BEGIN PGP SIGNATURE-----
    Version: GnuPG v1.4.6 (Darwin)

    iD8DBQFIWR0NZ88+mcQxRw0RArF1AJ9ZcYZOgdH4cpAIN6IpgE fHHvr2LQCfebRr
    YBBjQQBTjs/H4zVauWC5uLA=
    =XCN3
    -----END PGP SIGNATURE-----


  12. Re: Management interface for cards powered by the "mfi" driver?

    Hi Ruben,

    Ruben van Staveren wrote:
    >
    > On 18 Jun 2008, at 16:07, Stefan Lambrev wrote:
    >
    >>> Last time I checked it was still 2.6.12. it still is set to that
    >>> value on our 2950's (running 6.2 and linux_base-fc-4_9)

    >> I never saw 2.6.12 in documentation, but you may be right. Tough in
    >> ports/UPDATING only 2.6.16 is mention.
    >> http://blogs.freebsdish.org/netchild...d/linuxolator/
    >> speaks also only for 2.6.16

    >
    > I got this from reading the commit message of r1.5 of
    > http://www.freebsd.org/cgi/cvsweb.cg.../dev/mfi/mfi.c
    >
    > I don't know what the change set between 2.6.12 and 2.6.16 is
    > regarding the linuxolator. As I see it the difference should only in
    > more syscalls being supported. So if megacli has enough to get going
    > on by setting it to just 2.6.12 it should not have a requirement for
    > 2.6.16. Correct me if I am wrong (and I'll fix the instructions in the
    > port)

    Yes that's correct, so the sanity check should be >= 2.6.12, not just
    equal to 2.6.12?
    >
    > Of course, if you think mfi(4) should support your board csup up to
    > the latest stable and see whether it works. Also run through
    > /usr/local/sbin/megacli instead of /usr/local/libexec/MegaCli as it
    > will perform some sanity checks.

    I do not have problems with mfi and megacli It's the OP who have them.
    >
    >>> People should only realise that they are running a linux binary
    >>> under emulation in order to manage their mission critical servers.
    >>> It is a thing you might not like.

    >> It has been always this way with lsi ? Actually the difference now is
    >> that we are forced to use tools under beta/experimental linuxolator
    >>>

    >
    > Sad but true. But I managed with ease to replace a hot spare not that
    > long ago.
    >
    > You might want to have a peek at the cheat sheet hosted at
    > http://tools.rapidsoft.de/perc/

    nice site
    >
    > Cheers,
    > Ruben
    >


    --

    Best Wishes,
    Stefan Lambrev
    ICQ# 24134177

    _______________________________________________
    freebsd-stable@freebsd.org mailing list
    http://lists.freebsd.org/mailman/lis...freebsd-stable
    To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"


  13. Re: Management interface for cards powered by the "mfi" driver?

    > >
    > >dbms# ./CmdTool2 -Cfgdsply -a0
    > >
    > >Failed to get ControllerId List.
    > >Failed to get CpController object.
    > >dbms#
    > >
    > >Hmmmm... this is on a new machine that I loaded 7-RELEASE on and then
    > >grabbed the CVS tree and pulled a -rRELENG_7 on and rebuilt.
    > >
    > >Let me dig around a bit..... I wonder if I got a "funny" in here...
    > >looking at kernel versions, it appears I might.
    > >
    > >If so, that's going to be an interesting problem (as in "how did that
    > >happen?")
    > >to figure out.
    > >
    > >uname -v is returning:
    > >FreeBSD 7.0-CURRENT-200608 #0: Mon Jun 16 20:54:25 UTC 2008
    > >karl@dbms.denninger.net:/usr/obj/usr/src/sys/GENERIC
    > >How'd that happen with the tag "RELENG_7" in the tag? (and yes, it IS
    > >in there in the CVS directory in the "Tag" file....)
    > >
    > >-- Karl Denninger
    > >karl@denninger.net


    Ok, wiped the src tree, re-cvs'd out the RELENG_7, rebuild world and kernel
    and reinstalled (nice fast machine eh?)

    Anyway, no change:

    dbms# uname -v
    FreeBSD 7.0-STABLE #1: Wed Jun 18 14:43:29 CDT 2008 karl@dbms.denninger.net:/usr/obj/usr/src/sys/GENERIC

    dbms# megacli -adpCount


    Controller Count: 0.

    dbms# megacli -Cfgdsply -a0

    Failed to get ControllerId List.
    Failed to get CpController object.

    Still no joy

    dbms# kldstat
    Id Refs Address Size Name
    1 17 0xc0400000 943140 kernel
    2 1 0xc0d44000 6a2c4 acpi.ko
    3 1 0xc5534000 7000 linprocfs.ko
    4 3 0xc553b000 22000 linux.ko
    5 1 0xc5585000 3000 linsysfs.ko
    6 1 0xc7a34000 3000 daemon_saver.ko
    7 1 0xc7c2d000 2000 mfi_linux.ko

    Says I got the proper KLDs loaded.

    dbms# mount
    /dev/mfid0s1a on / (ufs, local, soft-updates)
    devfs on /dev (devfs, local)
    /dev/mfid0s1e on /dbms (ufs, local, soft-updates)
    /dev/mfid0s1d on /usr (ufs, local, soft-updates)
    linprocfs on /usr/compat/linux/proc (linprocfs, local)
    linsysfs on /usr/compat/linux/sys (linsysfs, local)

    The two linux "look-sees" are there.

    So it looks like all the pre-reqs are there, but it still doesn't work.

    Here's the ID on the card and volume:

    mfi0: 524 (267116948s/0x0020/0) - Adapter ticks 267116948 elapsed 61s: Time established as 06/18/08 15:09:08; (61 seconds since power on)
    mfid0: on mfi0
    mfid0: 237464MB (486326272 sectors) RAID volume '' is optimal

    What am I missing?

    -- Karl Denninger
    karl@denninger.net


    _______________________________________________
    freebsd-stable@freebsd.org mailing list
    http://lists.freebsd.org/mailman/lis...freebsd-stable
    To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"


  14. Re: Management interface for cards powered by the "mfi" driver?

    Karl Denninger writes:
    [snip]
    | Ok, wiped the src tree, re-cvs'd out the RELENG_7, rebuild world and kernel
    | and reinstalled (nice fast machine eh?)

    Not needed since FreeBSD 6.2 if I recall right. Forget if I got it in
    6.1.

    | Anyway, no change:
    |
    | dbms# uname -v
    | FreeBSD 7.0-STABLE #1: Wed Jun 18 14:43:29 CDT 2008 karl@dbms.denninger.net:/usr/obj/usr/src/sys/GENERIC
    |
    | dbms# megacli -adpCount
    |
    | Controller Count: 0.
    |
    | dbms# megacli -Cfgdsply -a0
    |
    | Failed to get ControllerId List.
    | Failed to get CpController object.
    |
    | Still no joy
    |
    | dbms# kldstat
    | Id Refs Address Size Name
    | 1 17 0xc0400000 943140 kernel
    | 2 1 0xc0d44000 6a2c4 acpi.ko
    | 3 1 0xc5534000 7000 linprocfs.ko
    | 4 3 0xc553b000 22000 linux.ko
    | 5 1 0xc5585000 3000 linsysfs.ko
    | 6 1 0xc7a34000 3000 daemon_saver.ko
    | 7 1 0xc7c2d000 2000 mfi_linux.ko
    |
    | Says I got the proper KLDs loaded.
    |
    | dbms# mount
    | /dev/mfid0s1a on / (ufs, local, soft-updates)
    | devfs on /dev (devfs, local)
    | /dev/mfid0s1e on /dbms (ufs, local, soft-updates)
    | /dev/mfid0s1d on /usr (ufs, local, soft-updates)
    | linprocfs on /usr/compat/linux/proc (linprocfs, local)
    | linsysfs on /usr/compat/linux/sys (linsysfs, local)
    |
    | The two linux "look-sees" are there.
    |
    | So it looks like all the pre-reqs are there, but it still doesn't work.
    |
    | Here's the ID on the card and volume:
    |
    | mfi0: 524 (267116948s/0x0020/0) - Adapter ticks 267116948 elapsed 61s: Time established as 06/18/08 15:09:08; (61 seconds since power on)
    | mfid0: on mfi0
    | mfid0: 237464MB (486326272 sectors) RAID volume '' is optimal
    |
    | What am I missing?

    The linux version sysctl is? Also I think you need to make sure
    mfi_linux.ko is loaded before linuxsys.ko mounts so you get the emulation
    hooks. Verify that via:
    head /compat/linux/sys/class/scsi_host/*/proc_name
    results in one saying:
    megaraid_sas
    or it won't think it is there.

    The count is good to see if your file system & linux version sysctl
    stuff is in the right state. Once it detects it, then the ioctl should
    work. 6-stable, 7-stable and -current all have the latest stuff to
    support all of the ioctl stuff as Linux does for MegaCli. MegaCli
    does various things to try to find the card in Linux that is really
    strange IMHO. For FreeBSD it doesn't have to be that complicated.
    They unfortunately, have not released a FreeBSD MegaCli which they
    could ...

    Doug A.
    _______________________________________________
    freebsd-stable@freebsd.org mailing list
    http://lists.freebsd.org/mailman/lis...freebsd-stable
    To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"


  15. Re: Management interface for cards powered by the "mfi" driver?

    On Wed, Jun 18, 2008 at 01:42:23PM -0700, Doug Ambrisko wrote:
    > Karl Denninger writes:
    > [snip]
    > | Ok, wiped the src tree, re-cvs'd out the RELENG_7, rebuild world and kernel
    > | and reinstalled (nice fast machine eh?)
    >
    > Not needed since FreeBSD 6.2 if I recall right. Forget if I got it in
    > 6.1.
    >
    > | Anyway, no change:
    > |
    > | dbms# uname -v
    > | FreeBSD 7.0-STABLE #1: Wed Jun 18 14:43:29 CDT 2008 karl@dbms.denninger.net:/usr/obj/usr/src/sys/GENERIC
    > |
    > | dbms# megacli -adpCount
    > |
    > | Controller Count: 0.
    > |
    > | dbms# megacli -Cfgdsply -a0
    > |
    > | Failed to get ControllerId List.
    > | Failed to get CpController object.
    > |
    > | Still no joy
    > |
    > | dbms# kldstat
    > | Id Refs Address Size Name
    > | 1 17 0xc0400000 943140 kernel
    > | 2 1 0xc0d44000 6a2c4 acpi.ko
    > | 3 1 0xc5534000 7000 linprocfs.ko
    > | 4 3 0xc553b000 22000 linux.ko
    > | 5 1 0xc5585000 3000 linsysfs.ko
    > | 6 1 0xc7a34000 3000 daemon_saver.ko
    > | 7 1 0xc7c2d000 2000 mfi_linux.ko
    > |
    > | Says I got the proper KLDs loaded.
    > |
    > | dbms# mount
    > | /dev/mfid0s1a on / (ufs, local, soft-updates)
    > | devfs on /dev (devfs, local)
    > | /dev/mfid0s1e on /dbms (ufs, local, soft-updates)
    > | /dev/mfid0s1d on /usr (ufs, local, soft-updates)
    > | linprocfs on /usr/compat/linux/proc (linprocfs, local)
    > | linsysfs on /usr/compat/linux/sys (linsysfs, local)
    > |
    > | The two linux "look-sees" are there.
    > |
    > | So it looks like all the pre-reqs are there, but it still doesn't work.
    > |
    > | Here's the ID on the card and volume:
    > |
    > | mfi0: 524 (267116948s/0x0020/0) - Adapter ticks 267116948 elapsed 61s: Time established as 06/18/08 15:09:08; (61 seconds since power on)
    > | mfid0: on mfi0
    > | mfid0: 237464MB (486326272 sectors) RAID volume '' is optimal
    > |
    > | What am I missing?
    >
    > The linux version sysctl is? Also I think you need to make sure
    > mfi_linux.ko is loaded before linuxsys.ko mounts so you get the emulation
    > hooks. Verify that via:
    > head /compat/linux/sys/class/scsi_host/*/proc_name
    > results in one saying:
    > megaraid_sas
    > or it won't think it is there.


    BINGO! That did it.

    > The count is good to see if your file system & linux version sysctl
    > stuff is in the right state. Once it detects it, then the ioctl should
    > work. 6-stable, 7-stable and -current all have the latest stuff to
    > support all of the ioctl stuff as Linux does for MegaCli. MegaCli
    > does various things to try to find the card in Linux that is really
    > strange IMHO. For FreeBSD it doesn't have to be that complicated.
    > They unfortunately, have not released a FreeBSD MegaCli which they
    > could ...


    It looks like I need to explicitly set those to load in the
    /boot/loader.conf instead of letting them autoload on demand...

    -- Karl Denninger
    karl@denninger.net


    _______________________________________________
    freebsd-stable@freebsd.org mailing list
    http://lists.freebsd.org/mailman/lis...freebsd-stable
    To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"


  16. Re: Management interface for cards powered by the "mfi" driver?


    On 20 Jun 2008, at 6:37, Karl Denninger wrote:
    >>
    >> The linux version sysctl is? Also I think you need to make sure
    >> mfi_linux.ko is loaded before linuxsys.ko mounts so you get the
    >> emulation
    >> hooks. Verify that via:
    >> head /compat/linux/sys/class/scsi_host/*/proc_name
    >> results in one saying:
    >> megaraid_sas
    >> or it won't think it is there.

    >
    > BINGO! That did it.
    >
    >> The count is good to see if your file system & linux version sysctl
    >> stuff is in the right state. Once it detects it, then the ioctl
    >> should
    >> work. 6-stable, 7-stable and -current all have the latest stuff to
    >> support all of the ioctl stuff as Linux does for MegaCli. MegaCli
    >> does various things to try to find the card in Linux that is really
    >> strange IMHO. For FreeBSD it doesn't have to be that complicated.
    >> They unfortunately, have not released a FreeBSD MegaCli which they
    >> could ...

    >
    > It looks like I need to explicitly set those to load in the
    > /boot/loader.conf instead of letting them autoload on demand...


    I'll make some changes in the port, for stressing out the load order
    and to let the wrapper work when compat.linux.osrelease is higher than
    2.6.12

    > -- Karl Denninger
    > karl@denninger.net



    Regards,
    Ruben

    -----BEGIN PGP SIGNATURE-----
    Version: GnuPG v1.4.6 (Darwin)

    iD8DBQFIW123Z88+mcQxRw0RAiTCAJ9gn0QZ51NhQ3TPecJ6Z3 0Lp485+wCggSZg
    al5/7o16S90k71IF5GNb06g=
    =eyGZ
    -----END PGP SIGNATURE-----


  17. Re: Management interface for cards powered by the "mfi" driver?


    On Jun 18, 2008, at 12:15 AM, Karl Denninger wrote:

    > No management tool = el-sucko, because you can't rebuild a failed
    > disk or
    > even shut the alarm on the board off!


    This is precisely the reason I have dropped using Adaptec
    controllers. The most recent ones cannot be managed with the FreeBSD
    tools.

    What I've ended up using is LSI Fibre Channel 4Gb cards (PCI-e for
    newer servers, but I have two with PCI-x) attached to an external RAID
    box. My external RAIDs are custom built by Partners Data Systems and
    use an Areca controller. All configuration and monitoring is done via
    web interface over ethernet so no proprietary software is needed.
    They fully support FreeBSD too.

    _______________________________________________
    freebsd-stable@freebsd.org mailing list
    http://lists.freebsd.org/mailman/lis...freebsd-stable
    To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"


+ Reply to Thread