Re: Best-performing disk I/O options for a DBMS server on 7-STABLE - FreeBSD

On Thu, May 08, 2008 at 10:30:01AM +0100, Pete French wrote:
> > I assume SCSI is the best path forward (either SA/SCSI or traditional) but
> > have been out of the loop on the card(s) that work properly for a good long
> > while.
>
> HP P400 cards are PCI express and SAS - they work very well under FreeBSD.
> I've also used the cheaper E200 and that appears to be fine too, though I
> havent run it for as long as the 400's.
>
> http://h18004.www1.hp.com/products/s...400/index.html
>
> -pete.
>

Following upon my own stuff, I just laid hands on a Quad-Xeon machine with an
Intel SRCSAS18E card in it, and well, the only way I can describe it is "oh
my GOD!"

Off a couple of older 250GB SATA drives, mirrored, it sustained 70MB/sec over
the entire disk's volume space on reads and had ZERO visible impact on CPU
OR another process doing mixed I/O at the same time (!).

That's impressive.

If its stable.

The cards are not cheap, however.

The only "gotcha" is that all configuration of the drive(s) appears to be
done through the controller. I assume I COULD use something like gmirror,
but see little reason to do so.

-- Karl Denninger
karl@denninger.net


_______________________________________________
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/lis...freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"

On Mon, Jun 16, 2008 at 08:20:53PM -0500, Karl Denninger wrote:
> On Thu, May 08, 2008 at 10:30:01AM +0100, Pete French wrote:
> > > I assume SCSI is the best path forward (either SA/SCSI or traditional) but
> > > have been out of the loop on the card(s) that work properly for a good long
> > > while.
> >
> > HP P400 cards are PCI express and SAS - they work very well under FreeBSD.
> > I've also used the cheaper E200 and that appears to be fine too, though I
> > havent run it for as long as the 400's.
> >
> > http://h18004.www1.hp.com/products/s...400/index.html
> >
> > -pete.
> >
>
> Following upon my own stuff, I just laid hands on a Quad-Xeon machine with an
> Intel SRCSAS18E card in it, and well, the only way I can describe it is "oh
> my GOD!"
>
> Off a couple of older 250GB SATA drives, mirrored, it sustained 70MB/sec over
> the entire disk's volume space on reads and had ZERO visible impact on CPU
> OR another process doing mixed I/O at the same time (!).
>
> That's impressive.
>
> If its stable.
>
> The cards are not cheap, however.
>
> The only "gotcha" is that all configuration of the drive(s) appears to be
> done through the controller. I assume I COULD use something like gmirror,
> but see little reason to do so.
>
> -- Karl Denninger
> karl@denninger.net

Ok, another followup!

Is there a mangement interface program for the "mfi" driver somewhere? I
rooted around in "ports" but didn't find one, nor on the base system.

The Linux ones want to talk to the "amr" driver which is the older LSI Logic
MegaRAID boards, not the newer ones that the mfi driver talks to.

No management tool = el-sucko, because you can't rebuild a failed disk or
even shut the alarm on the board off!

-- Karl Denninger
karl@denninger.net


_______________________________________________
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/lis...freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"

On Tue, Jun 17, 2008 at 11:15:30PM -0500, Karl Denninger wrote:
> On Mon, Jun 16, 2008 at 08:20:53PM -0500, Karl Denninger wrote:
> > On Thu, May 08, 2008 at 10:30:01AM +0100, Pete French wrote:
> > > > I assume SCSI is the best path forward (either SA/SCSI or traditional) but
> > > > have been out of the loop on the card(s) that work properly for a good long
> > > > while.
> > >
> > > HP P400 cards are PCI express and SAS - they work very well under FreeBSD.
> > > I've also used the cheaper E200 and that appears to be fine too, though I
> > > havent run it for as long as the 400's.
> > >
> > > http://h18004.www1.hp.com/products/s...400/index.html
> > >
> > > -pete.
> > >
> >
> > Following upon my own stuff, I just laid hands on a Quad-Xeon machine with an
> > Intel SRCSAS18E card in it, and well, the only way I can describe it is "oh
> > my GOD!"
>
> Ok, another followup!
>
> Is there a mangement interface program for the "mfi" driver somewhere? I
> rooted around in "ports" but didn't find one, nor on the base system.
>
> The Linux ones want to talk to the "amr" driver which is the older LSI Logic
> MegaRAID boards, not the newer ones that the mfi driver talks to.
>
> No management tool = el-sucko, because you can't rebuild a failed disk or
> even shut the alarm on the board off!

Its sysutils/linux-megacli.


Andrew
_______________________________________________
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/lis...freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"

On Tue, Jun 17, 2008 at 11:15:30PM -0500, Karl Denninger wrote:
> On Mon, Jun 16, 2008 at 08:20:53PM -0500, Karl Denninger wrote:
> > On Thu, May 08, 2008 at 10:30:01AM +0100, Pete French wrote:
> > > > I assume SCSI is the best path forward (either SA/SCSI or traditional) but
> > > > have been out of the loop on the card(s) that work properly for a good long
> > > > while.
> > >
> > > HP P400 cards are PCI express and SAS - they work very well under FreeBSD.
> > > I've also used the cheaper E200 and that appears to be fine too, though I
> > > havent run it for as long as the 400's.
> > >
> > > http://h18004.www1.hp.com/products/s...400/index.html
> > >
> > > -pete.
> > >
> >
> > Following upon my own stuff, I just laid hands on a Quad-Xeon machine with an
> > Intel SRCSAS18E card in it, and well, the only way I can describe it is "oh
> > my GOD!"
> >
> > Off a couple of older 250GB SATA drives, mirrored, it sustained 70MB/sec over
> > the entire disk's volume space on reads and had ZERO visible impact on CPU
> > OR another process doing mixed I/O at the same time (!).
> >
> > That's impressive.
> >
> > If its stable.
> >
> > The cards are not cheap, however.
> >
> > The only "gotcha" is that all configuration of the drive(s) appears to be
> > done through the controller. I assume I COULD use something like gmirror,
> > but see little reason to do so.
> >
> > -- Karl Denninger
> > karl@denninger.net
>
> Ok, another followup!
>
> Is there a mangement interface program for the "mfi" driver somewhere? I
> rooted around in "ports" but didn't find one, nor on the base system.
>
> The Linux ones want to talk to the "amr" driver which is the older LSI Logic
> MegaRAID boards, not the newer ones that the mfi driver talks to.
>
> No management tool = el-sucko, because you can't rebuild a failed disk or
> even shut the alarm on the board off!


The sysutil/linux-megacli port looks like it might do the trick.



--

Erik Trulsson
ertr1013@student.uu.se
_______________________________________________
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/lis...freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"

On Tue, Jun 17, 2008 at 10:05:31PM -0700, Andrew Thompson wrote:
> On Tue, Jun 17, 2008 at 11:15:30PM -0500, Karl Denninger wrote:
> > On Mon, Jun 16, 2008 at 08:20:53PM -0500, Karl Denninger wrote:
> > > On Thu, May 08, 2008 at 10:30:01AM +0100, Pete French wrote:
> > > > > I assume SCSI is the best path forward (either SA/SCSI or traditional) but
> > > > > have been out of the loop on the card(s) that work properly for a good long
> > > > > while.
> > > >
> > > > HP P400 cards are PCI express and SAS - they work very well under FreeBSD.
> > > > I've also used the cheaper E200 and that appears to be fine too, though I
> > > > havent run it for as long as the 400's.
> > > >
> > > > http://h18004.www1.hp.com/products/s...400/index.html
> > > >
> > > > -pete.
> > > >
> > >
> > > Following upon my own stuff, I just laid hands on a Quad-Xeon machine with an
> > > Intel SRCSAS18E card in it, and well, the only way I can describe it is "oh
> > > my GOD!"
> >
> > Ok, another followup!
> >
> > Is there a mangement interface program for the "mfi" driver somewhere? I
> > rooted around in "ports" but didn't find one, nor on the base system.
> >
> > The Linux ones want to talk to the "amr" driver which is the older LSI Logic
> > MegaRAID boards, not the newer ones that the mfi driver talks to.
> >
> > No management tool = el-sucko, because you can't rebuild a failed disk or
> > even shut the alarm on the board off!
>
> Its sysutils/linux-megacli.

Nope. It builds and installs but doesn't work (already tried this one.)

Attempting to ask it for the number of boards returns "0".

This may have something to do with changes made to the device structure(s)
in FreeBSD-7 (I don't have a 6 machine to test on any more) but it
definitely does not function.

-- Karl Denninger
karl@denninger.net


_______________________________________________
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/lis...freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"

Karl Denninger wrote:
> On Tue, Jun 17, 2008 at 10:05:31PM -0700, Andrew Thompson wrote:
>
>> On Tue, Jun 17, 2008 at 11:15:30PM -0500, Karl Denninger wrote:
>>
>>> On Mon, Jun 16, 2008 at 08:20:53PM -0500, Karl Denninger wrote:
>>>
>>>> On Thu, May 08, 2008 at 10:30:01AM +0100, Pete French wrote:
>>>>
>>>>>> I assume SCSI is the best path forward (either SA/SCSI or traditional) but
>>>>>> have been out of the loop on the card(s) that work properly for a good long
>>>>>> while.
>>>>>>
>>>>> HP P400 cards are PCI express and SAS - they work very well under FreeBSD.
>>>>> I've also used the cheaper E200 and that appears to be fine too, though I
>>>>> havent run it for as long as the 400's.
>>>>>
>>>>> http://h18004.www1.hp.com/products/s...400/index.html
>>>>>
>>>>> -pete.
>>>>>
>>>>>
>>>> Following upon my own stuff, I just laid hands on a Quad-Xeon machine with an
>>>> Intel SRCSAS18E card in it, and well, the only way I can describe it is "oh
>>>> my GOD!"
>>>>
>>> Ok, another followup!
>>>
>>> Is there a mangement interface program for the "mfi" driver somewhere? I
>>> rooted around in "ports" but didn't find one, nor on the base system.
>>>
>>> The Linux ones want to talk to the "amr" driver which is the older LSI Logic
>>> MegaRAID boards, not the newer ones that the mfi driver talks to.
>>>
>>> No management tool = el-sucko, because you can't rebuild a failed disk or
>>> even shut the alarm on the board off!
>>>
>> Its sysutils/linux-megacli.
>>
>
> Nope. It builds and installs but doesn't work (already tried this one.)
>
> Attempting to ask it for the number of boards returns "0".
>
> This may have something to do with changes made to the device structure(s)
> in FreeBSD-7 (I don't have a 6 machine to test on any more) but it
> definitely does not function.
>
It works fine for me on very recent 7-STABLE.

megacli -CfgDsply -a0

================================================== ============================
Adapter: 0
Product Name: PERC 5/i Integrated
Memory: 256MB
BBU: Present
Serial No: 12345
================================================== ============================
-cut-

megacli -AdpAllInfo -aALL also works without a problem.

7.0-STABLE-200805 FreeBSD 7.0-STABLE-200805 #0: Sun May 11 11:55:15 UTC
2008 root@driscoll.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64

Did you follow all steps from ports/sysutils/linux-megacli/pkg-message ?
Though I think there is a small bug in the port (maintainer CCed)
It wants compat.linux.osrelease=2.6.12, but I think it should be
compat.linux.osrelease=2.6.16 ?
Just for reference I installed it with OVERRIDE_LINUX_BASE_PORT=f8
I think you are confusing mega-cli with megamgr (which is the one that
work with amr driver)

--

Best Wishes,
Stefan Lambrev
ICQ# 24134177

_______________________________________________
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/lis...freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"

Hi,

On 18 Jun 2008, at 12:29, Stefan Lambrev wrote:

> Did you follow all steps from ports/sysutils/linux-megacli/pkg-
> message ?
> Though I think there is a small bug in the port (maintainer CCed)
> It wants compat.linux.osrelease=2.6.12, but I think it should be
> compat.linux.osrelease=2.6.16 ?

Last time I checked it was still 2.6.12. it still is set to that value
on our 2950's (running 6.2 and linux_base-fc-4_9)

> Just for reference I installed it with OVERRIDE_LINUX_BASE_PORT=f8
> I think you are confusing mega-cli with megamgr (which is the one
> that work with amr driver)

People should only realise that they are running a linux binary under
emulation in order to manage their mission critical servers. It is a
thing you might not like.

Regards,
Ruben
_______________________________________________
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/lis...freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"

On Wed, Jun 18, 2008 at 01:29:56PM +0300, Stefan Lambrev wrote:
> Karl Denninger wrote:
> >On Tue, Jun 17, 2008 at 10:05:31PM -0700, Andrew Thompson wrote:
> >
> >>On Tue, Jun 17, 2008 at 11:15:30PM -0500, Karl Denninger wrote:
> >>>>Intel SRCSAS18E card in it, and well, the only way I can describe it is
> >>>>"oh
> >>>>my GOD!"
> >>>>
> >>>Ok, another followup!
> >>>
> >>>Is there a mangement interface program for the "mfi" driver somewhere? I
> >>>rooted around in "ports" but didn't find one, nor on the base system.
> >>>
> >>>The Linux ones want to talk to the "amr" driver which is the older LSI
> >>>Logic
> >>>MegaRAID boards, not the newer ones that the mfi driver talks to.
> >>>
> >>>No management tool = el-sucko, because you can't rebuild a failed disk or
> >>>even shut the alarm on the board off!
> >>>
> >>Its sysutils/linux-megacli.
> >>
> >
> >Nope. It builds and installs but doesn't work (already tried this one.)
> >
> >Attempting to ask it for the number of boards returns "0".
> >
> >This may have something to do with changes made to the device structure(s)
> >in FreeBSD-7 (I don't have a 6 machine to test on any more) but it
> >definitely does not function.
> >
> It works fine for me on very recent 7-STABLE.
>
> megacli -CfgDsply -a0
>
> ================================================== ============================
> Adapter: 0
> Product Name: PERC 5/i Integrated
> Memory: 256MB
> BBU: Present
> Serial No: 12345
> ================================================== ============================
> -cut-
>
> megacli -AdpAllInfo -aALL also works without a problem.
>
> 7.0-STABLE-200805 FreeBSD 7.0-STABLE-200805 #0: Sun May 11 11:55:15 UTC
> 2008 root@driscoll.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64
>
> Did you follow all steps from ports/sysutils/linux-megacli/pkg-message ?
> Though I think there is a small bug in the port (maintainer CCed)
> It wants compat.linux.osrelease=2.6.12, but I think it should be
> compat.linux.osrelease=2.6.16 ?
> Just for reference I installed it with OVERRIDE_LINUX_BASE_PORT=f8
> I think you are confusing mega-cli with megamgr (which is the one that
> work with amr driver)

dbms# megacli -CfgDsply -a0

Failed to get ControllerId List.
Failed to get CpController object.
dbms#

But the card is definitely there; this is what it last logged when I manually
yanked one of the mirrored drives, then shut down and told it to rebuild
from the GUI before boot and let it go on its way:

mfi0: 501 - PD 10(e252/s1) progress 99% seconds 7017s: Rebuild progress on
PD 0a(e0xfc/s1) is 98.94%(7017s)
mfi0: 502 - PD 10(e252/s1) progress 100% seconds 7158s: Rebuild progress on
PD 0a(e0xfc/s1) is 99.94%(7158s)
mfi0: 503 - PD 10(e252/s1) event: Rebuild complete on PD 0a(e0xfc/s1)
mfi0: 504 - VD 00/0 state prior 2 new 3: State change on VD 00/0 from
DEGRADED(2) to OPTIMAL(3)
mfi0: 505 - VD 00/0 event: VD 00/0 is now OPTIMAL
mfi0: 506 - PD 10(e252/s1) state prior 20 new 24: State change on PD
0a(e0xfc/s1) from REBUILD(14) to ONLINE(18)

Oook!

So we have a card that the driver is perfectly happy with, but the
management interface can't find.

Note that this is the Intel board; I grabbed the Intel version of the
software for Linux (which appears to be essentially the same code but under
Intel's name), set the branding so the Linux emulator would play nice, and
got the same thing:

dbms# ./CmdTool2 -Cfgdsply -a0

Failed to get ControllerId List.
Failed to get CpController object.
dbms#

Hmmmm... this is on a new machine that I loaded 7-RELEASE on and then
grabbed the CVS tree and pulled a -rRELENG_7 on and rebuilt.

Let me dig around a bit..... I wonder if I got a "funny" in here...
looking at kernel versions, it appears I might.

If so, that's going to be an interesting problem (as in "how did that happen?")
to figure out.

uname -v is returning:
FreeBSD 7.0-CURRENT-200608 #0: Mon Jun 16 20:54:25 UTC 2008 karl@dbms.denninger.net:/usr/obj/usr/src/sys/GENERIC

How'd that happen with the tag "RELENG_7" in the tag? (and yes, it IS
in there in the CVS directory in the "Tag" file....)

-- Karl Denninger
karl@denninger.net


_______________________________________________
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/lis...freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"

Hi,

Ruben van Staveren wrote:
>
> Hi,
>
> On 18 Jun 2008, at 12:29, Stefan Lambrev wrote:
>
>> Did you follow all steps from ports/sysutils/linux-megacli/pkg-message ?
>> Though I think there is a small bug in the port (maintainer CCed)
>> It wants compat.linux.osrelease=2.6.12, but I think it should be
>> compat.linux.osrelease=2.6.16 ?
>
> Last time I checked it was still 2.6.12. it still is set to that value
> on our 2950's (running 6.2 and linux_base-fc-4_9)
I never saw 2.6.12 in documentation, but you may be right. Tough in
ports/UPDATING only 2.6.16 is mention.
http://blogs.freebsdish.org/netchild...d/linuxolator/
speaks also only for 2.6.16
>
>> Just for reference I installed it with OVERRIDE_LINUX_BASE_PORT=f8
>> I think you are confusing mega-cli with megamgr (which is the one
>> that work with amr driver)
>
> People should only realise that they are running a linux binary under
> emulation in order to manage their mission critical servers. It is a
> thing you might not like.
It has been always this way with lsi ? Actually the difference now is
that we are forced to use tools under beta/experimental linuxolator
>
> Regards,
> Ruben

--

Best Wishes,
Stefan Lambrev
ICQ# 24134177

_______________________________________________
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/lis...freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"

Hi Karl,

Karl Denninger wrote:
> On Wed, Jun 18, 2008 at 01:29:56PM +0300, Stefan Lambrev wrote:
>
>> Karl Denninger wrote:
>>
>>> On Tue, Jun 17, 2008 at 10:05:31PM -0700, Andrew Thompson wrote:
>>>
>>>
>>>> On Tue, Jun 17, 2008 at 11:15:30PM -0500, Karl Denninger wrote:
>>>>
>>>>>> Intel SRCSAS18E card in it, and well, the only way I can describe it is
>>>>>> "oh
>>>>>> my GOD!"
>>>>>>
>>>>>>
>>>>> Ok, another followup!
>>>>>
>>>>> Is there a mangement interface program for the "mfi" driver somewhere? I
>>>>> rooted around in "ports" but didn't find one, nor on the base system.
>>>>>
>>>>> The Linux ones want to talk to the "amr" driver which is the older LSI
>>>>> Logic
>>>>> MegaRAID boards, not the newer ones that the mfi driver talks to.
>>>>>
>>>>> No management tool = el-sucko, because you can't rebuild a failed disk or
>>>>> even shut the alarm on the board off!
>>>>>
>>>>>
>>>> Its sysutils/linux-megacli.
>>>>
>>>>
>>> Nope. It builds and installs but doesn't work (already tried this one.)
>>>
>>> Attempting to ask it for the number of boards returns "0".
>>>
>>> This may have something to do with changes made to the device structure(s)
>>> in FreeBSD-7 (I don't have a 6 machine to test on any more) but it
>>> definitely does not function.
>>>
>>>
>> It works fine for me on very recent 7-STABLE.
>>
>> megacli -CfgDsply -a0
>>
>> ================================================== ============================
>> Adapter: 0
>> Product Name: PERC 5/i Integrated
>> Memory: 256MB
>> BBU: Present
>> Serial No: 12345
>> ================================================== ============================
>> -cut-
>>
>> megacli -AdpAllInfo -aALL also works without a problem.
>>
>> 7.0-STABLE-200805 FreeBSD 7.0-STABLE-200805 #0: Sun May 11 11:55:15 UTC
>> 2008 root@driscoll.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64
>>
>> Did you follow all steps from ports/sysutils/linux-megacli/pkg-message ?
>> Though I think there is a small bug in the port (maintainer CCed)
>> It wants compat.linux.osrelease=2.6.12, but I think it should be
>> compat.linux.osrelease=2.6.16 ?
>> Just for reference I installed it with OVERRIDE_LINUX_BASE_PORT=f8
>> I think you are confusing mega-cli with megamgr (which is the one that
>> work with amr driver)
>>
>
> dbms# megacli -CfgDsply -a0
>
> Failed to get ControllerId List.
> Failed to get CpController object.
> dbms#
>
Sorry if you already answer this and I missed it, but do you have
mfi_linux, linprocfs, linsysfs loaded as modules (or build in kernel)?
Also do you have mounted:
linprocfs on /usr/compat/linux/proc (linprocfs, local)
linsysfs on /usr/compat/linux/sys (linsysfs, local)
If this is done I'm really out of ideas.
> But the card is definitely there; this is what it last logged when I manually
> yanked one of the mirrored drives, then shut down and told it to rebuild
> from the GUI before boot and let it go on its way:
>
> mfi0: 501 - PD 10(e252/s1) progress 99% seconds 7017s: Rebuild progress on
> PD 0a(e0xfc/s1) is 98.94%(7017s)
> mfi0: 502 - PD 10(e252/s1) progress 100% seconds 7158s: Rebuild progress on
> PD 0a(e0xfc/s1) is 99.94%(7158s)
> mfi0: 503 - PD 10(e252/s1) event: Rebuild complete on PD 0a(e0xfc/s1)
> mfi0: 504 - VD 00/0 state prior 2 new 3: State change on VD 00/0 from
> DEGRADED(2) to OPTIMAL(3)
> mfi0: 505 - VD 00/0 event: VD 00/0 is now OPTIMAL
> mfi0: 506 - PD 10(e252/s1) state prior 20 new 24: State change on PD
> 0a(e0xfc/s1) from REBUILD(14) to ONLINE(18)
>
> Oook!
>
> So we have a card that the driver is perfectly happy with, but the
> management interface can't find.
>
> Note that this is the Intel board; I grabbed the Intel version of the
> software for Linux (which appears to be essentially the same code but under
> Intel's name), set the branding so the Linux emulator would play nice, and
> got the same thing:
>
> dbms# ./CmdTool2 -Cfgdsply -a0
>
> Failed to get ControllerId List.
> Failed to get CpController object.
> dbms#
>
> Hmmmm... this is on a new machine that I loaded 7-RELEASE on and then
> grabbed the CVS tree and pulled a -rRELENG_7 on and rebuilt.
>
> Let me dig around a bit..... I wonder if I got a "funny" in here...
> looking at kernel versions, it appears I might.
>
> If so, that's going to be an interesting problem (as in "how did that happen?")
> to figure out.
>
> uname -v is returning:
> FreeBSD 7.0-CURRENT-200608 #0: Mon Jun 16 20:54:25 UTC 2008 karl@dbms.denninger.net:/usr/obj/usr/src/sys/GENERIC
>
> How'd that happen with the tag "RELENG_7" in the tag? (and yes, it IS
> in there in the CVS directory in the "Tag" file....)
>
> -- Karl Denninger
> karl@denninger.net
>
>
> _______________________________________________
> freebsd-stable@freebsd.org mailing list
> http://lists.freebsd.org/mailman/lis...freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"
>

--

Best Wishes,
Stefan Lambrev
ICQ# 24134177

_______________________________________________
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/lis...freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"

On 18 Jun 2008, at 16:07, Stefan Lambrev wrote:

>> Last time I checked it was still 2.6.12. it still is set to that
>> value on our 2950's (running 6.2 and linux_base-fc-4_9)
> I never saw 2.6.12 in documentation, but you may be right. Tough in
> ports/UPDATING only 2.6.16 is mention.
> http://blogs.freebsdish.org/netchild...d/linuxolator/
> speaks also only for 2.6.16

I got this from reading the commit message of r1.5 of http://www.freebsd.org/cgi/cvsweb.cg.../dev/mfi/mfi.c

I don't know what the change set between 2.6.12 and 2.6.16 is
regarding the linuxolator. As I see it the difference should only in
more syscalls being supported. So if megacli has enough to get going
on by setting it to just 2.6.12 it should not have a requirement for
2.6.16. Correct me if I am wrong (and I'll fix the instructions in the
port)

Of course, if you think mfi(4) should support your board csup up to
the latest stable and see whether it works. Also run through /usr/
local/sbin/megacli instead of /usr/local/libexec/MegaCli as it will
perform some sanity checks.

>> People should only realise that they are running a linux binary
>> under emulation in order to manage their mission critical servers.
>> It is a thing you might not like.
> It has been always this way with lsi ? Actually the difference now
> is that we are forced to use tools under beta/experimental
> linuxolator
>>

Sad but true. But I managed with ease to replace a hot spare not that
long ago.

You might want to have a peek at the cheat sheet hosted at http://tools.rapidsoft.de/perc/

Cheers,
Ruben


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (Darwin)

iD8DBQFIWR0NZ88+mcQxRw0RArF1AJ9ZcYZOgdH4cpAIN6IpgE fHHvr2LQCfebRr
YBBjQQBTjs/H4zVauWC5uLA=
=XCN3
-----END PGP SIGNATURE-----

Hi Ruben,

Ruben van Staveren wrote:
>
> On 18 Jun 2008, at 16:07, Stefan Lambrev wrote:
>
>>> Last time I checked it was still 2.6.12. it still is set to that
>>> value on our 2950's (running 6.2 and linux_base-fc-4_9)
>> I never saw 2.6.12 in documentation, but you may be right. Tough in
>> ports/UPDATING only 2.6.16 is mention.
>> http://blogs.freebsdish.org/netchild...d/linuxolator/
>> speaks also only for 2.6.16
>
> I got this from reading the commit message of r1.5 of
> http://www.freebsd.org/cgi/cvsweb.cg.../dev/mfi/mfi.c
>
> I don't know what the change set between 2.6.12 and 2.6.16 is
> regarding the linuxolator. As I see it the difference should only in
> more syscalls being supported. So if megacli has enough to get going
> on by setting it to just 2.6.12 it should not have a requirement for
> 2.6.16. Correct me if I am wrong (and I'll fix the instructions in the
> port)
Yes that's correct, so the sanity check should be >= 2.6.12, not just
equal to 2.6.12?
>
> Of course, if you think mfi(4) should support your board csup up to
> the latest stable and see whether it works. Also run through
> /usr/local/sbin/megacli instead of /usr/local/libexec/MegaCli as it
> will perform some sanity checks.
I do not have problems with mfi and megacli It's the OP who have them.
>
>>> People should only realise that they are running a linux binary
>>> under emulation in order to manage their mission critical servers.
>>> It is a thing you might not like.
>> It has been always this way with lsi ? Actually the difference now is
>> that we are forced to use tools under beta/experimental linuxolator
>>>
>
> Sad but true. But I managed with ease to replace a hot spare not that
> long ago.
>
> You might want to have a peek at the cheat sheet hosted at
> http://tools.rapidsoft.de/perc/
nice site
>
> Cheers,
> Ruben
>

--

Best Wishes,
Stefan Lambrev
ICQ# 24134177

_______________________________________________
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/lis...freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"

> >
> >dbms# ./CmdTool2 -Cfgdsply -a0
> >
> >Failed to get ControllerId List.
> >Failed to get CpController object.
> >dbms#
> >
> >Hmmmm... this is on a new machine that I loaded 7-RELEASE on and then
> >grabbed the CVS tree and pulled a -rRELENG_7 on and rebuilt.
> >
> >Let me dig around a bit..... I wonder if I got a "funny" in here...
> >looking at kernel versions, it appears I might.
> >
> >If so, that's going to be an interesting problem (as in "how did that
> >happen?")
> >to figure out.
> >
> >uname -v is returning:
> >FreeBSD 7.0-CURRENT-200608 #0: Mon Jun 16 20:54:25 UTC 2008
> >karl@dbms.denninger.net:/usr/obj/usr/src/sys/GENERIC
> >How'd that happen with the tag "RELENG_7" in the tag? (and yes, it IS
> >in there in the CVS directory in the "Tag" file....)
> >
> >-- Karl Denninger
> >karl@denninger.net

Ok, wiped the src tree, re-cvs'd out the RELENG_7, rebuild world and kernel
and reinstalled (nice fast machine eh?)

Anyway, no change:

dbms# uname -v
FreeBSD 7.0-STABLE #1: Wed Jun 18 14:43:29 CDT 2008 karl@dbms.denninger.net:/usr/obj/usr/src/sys/GENERIC

dbms# megacli -adpCount


Controller Count: 0.

dbms# megacli -Cfgdsply -a0

Failed to get ControllerId List.
Failed to get CpController object.

Still no joy

dbms# kldstat
Id Refs Address Size Name
1 17 0xc0400000 943140 kernel
2 1 0xc0d44000 6a2c4 acpi.ko
3 1 0xc5534000 7000 linprocfs.ko
4 3 0xc553b000 22000 linux.ko
5 1 0xc5585000 3000 linsysfs.ko
6 1 0xc7a34000 3000 daemon_saver.ko
7 1 0xc7c2d000 2000 mfi_linux.ko

Says I got the proper KLDs loaded.

dbms# mount
/dev/mfid0s1a on / (ufs, local, soft-updates)
devfs on /dev (devfs, local)
/dev/mfid0s1e on /dbms (ufs, local, soft-updates)
/dev/mfid0s1d on /usr (ufs, local, soft-updates)
linprocfs on /usr/compat/linux/proc (linprocfs, local)
linsysfs on /usr/compat/linux/sys (linsysfs, local)

The two linux "look-sees" are there.

So it looks like all the pre-reqs are there, but it still doesn't work.

Here's the ID on the card and volume:

mfi0: 524 (267116948s/0x0020/0) - Adapter ticks 267116948 elapsed 61s: Time established as 06/18/08 15:09:08; (61 seconds since power on)
mfid0: on mfi0
mfid0: 237464MB (486326272 sectors) RAID volume '' is optimal

What am I missing?

-- Karl Denninger
karl@denninger.net


_______________________________________________
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/lis...freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"

Karl Denninger writes:
[snip]
| Ok, wiped the src tree, re-cvs'd out the RELENG_7, rebuild world and kernel
| and reinstalled (nice fast machine eh?)

Not needed since FreeBSD 6.2 if I recall right. Forget if I got it in
6.1.

| Anyway, no change:
|
| dbms# uname -v
| FreeBSD 7.0-STABLE #1: Wed Jun 18 14:43:29 CDT 2008 karl@dbms.denninger.net:/usr/obj/usr/src/sys/GENERIC
|
| dbms# megacli -adpCount
|
| Controller Count: 0.
|
| dbms# megacli -Cfgdsply -a0
|
| Failed to get ControllerId List.
| Failed to get CpController object.
|
| Still no joy
|
| dbms# kldstat
| Id Refs Address Size Name
| 1 17 0xc0400000 943140 kernel
| 2 1 0xc0d44000 6a2c4 acpi.ko
| 3 1 0xc5534000 7000 linprocfs.ko
| 4 3 0xc553b000 22000 linux.ko
| 5 1 0xc5585000 3000 linsysfs.ko
| 6 1 0xc7a34000 3000 daemon_saver.ko
| 7 1 0xc7c2d000 2000 mfi_linux.ko
|
| Says I got the proper KLDs loaded.
|
| dbms# mount
| /dev/mfid0s1a on / (ufs, local, soft-updates)
| devfs on /dev (devfs, local)
| /dev/mfid0s1e on /dbms (ufs, local, soft-updates)
| /dev/mfid0s1d on /usr (ufs, local, soft-updates)
| linprocfs on /usr/compat/linux/proc (linprocfs, local)
| linsysfs on /usr/compat/linux/sys (linsysfs, local)
|
| The two linux "look-sees" are there.
|
| So it looks like all the pre-reqs are there, but it still doesn't work.
|
| Here's the ID on the card and volume:
|
| mfi0: 524 (267116948s/0x0020/0) - Adapter ticks 267116948 elapsed 61s: Time established as 06/18/08 15:09:08; (61 seconds since power on)
| mfid0: on mfi0
| mfid0: 237464MB (486326272 sectors) RAID volume '' is optimal
|
| What am I missing?

The linux version sysctl is? Also I think you need to make sure
mfi_linux.ko is loaded before linuxsys.ko mounts so you get the emulation
hooks. Verify that via:
head /compat/linux/sys/class/scsi_host/*/proc_name
results in one saying:
megaraid_sas
or it won't think it is there.

The count is good to see if your file system & linux version sysctl
stuff is in the right state. Once it detects it, then the ioctl should
work. 6-stable, 7-stable and -current all have the latest stuff to
support all of the ioctl stuff as Linux does for MegaCli. MegaCli
does various things to try to find the card in Linux that is really
strange IMHO. For FreeBSD it doesn't have to be that complicated.
They unfortunately, have not released a FreeBSD MegaCli which they
could ...

Doug A.
_______________________________________________
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/lis...freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"

On Wed, Jun 18, 2008 at 01:42:23PM -0700, Doug Ambrisko wrote:
> Karl Denninger writes:
> [snip]
> | Ok, wiped the src tree, re-cvs'd out the RELENG_7, rebuild world and kernel
> | and reinstalled (nice fast machine eh?)
>
> Not needed since FreeBSD 6.2 if I recall right. Forget if I got it in
> 6.1.
>
> | Anyway, no change:
> |
> | dbms# uname -v
> | FreeBSD 7.0-STABLE #1: Wed Jun 18 14:43:29 CDT 2008 karl@dbms.denninger.net:/usr/obj/usr/src/sys/GENERIC
> |
> | dbms# megacli -adpCount
> |
> | Controller Count: 0.
> |
> | dbms# megacli -Cfgdsply -a0
> |
> | Failed to get ControllerId List.
> | Failed to get CpController object.
> |
> | Still no joy
> |
> | dbms# kldstat
> | Id Refs Address Size Name
> | 1 17 0xc0400000 943140 kernel
> | 2 1 0xc0d44000 6a2c4 acpi.ko
> | 3 1 0xc5534000 7000 linprocfs.ko
> | 4 3 0xc553b000 22000 linux.ko
> | 5 1 0xc5585000 3000 linsysfs.ko
> | 6 1 0xc7a34000 3000 daemon_saver.ko
> | 7 1 0xc7c2d000 2000 mfi_linux.ko
> |
> | Says I got the proper KLDs loaded.
> |
> | dbms# mount
> | /dev/mfid0s1a on / (ufs, local, soft-updates)
> | devfs on /dev (devfs, local)
> | /dev/mfid0s1e on /dbms (ufs, local, soft-updates)
> | /dev/mfid0s1d on /usr (ufs, local, soft-updates)
> | linprocfs on /usr/compat/linux/proc (linprocfs, local)
> | linsysfs on /usr/compat/linux/sys (linsysfs, local)
> |
> | The two linux "look-sees" are there.
> |
> | So it looks like all the pre-reqs are there, but it still doesn't work.
> |
> | Here's the ID on the card and volume:
> |
> | mfi0: 524 (267116948s/0x0020/0) - Adapter ticks 267116948 elapsed 61s: Time established as 06/18/08 15:09:08; (61 seconds since power on)
> | mfid0: on mfi0
> | mfid0: 237464MB (486326272 sectors) RAID volume '' is optimal
> |
> | What am I missing?
>
> The linux version sysctl is? Also I think you need to make sure
> mfi_linux.ko is loaded before linuxsys.ko mounts so you get the emulation
> hooks. Verify that via:
> head /compat/linux/sys/class/scsi_host/*/proc_name
> results in one saying:
> megaraid_sas
> or it won't think it is there.

BINGO! That did it.

> The count is good to see if your file system & linux version sysctl
> stuff is in the right state. Once it detects it, then the ioctl should
> work. 6-stable, 7-stable and -current all have the latest stuff to
> support all of the ioctl stuff as Linux does for MegaCli. MegaCli
> does various things to try to find the card in Linux that is really
> strange IMHO. For FreeBSD it doesn't have to be that complicated.
> They unfortunately, have not released a FreeBSD MegaCli which they
> could ...

It looks like I need to explicitly set those to load in the
/boot/loader.conf instead of letting them autoload on demand...

-- Karl Denninger
karl@denninger.net


_______________________________________________
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/lis...freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"

On 20 Jun 2008, at 6:37, Karl Denninger wrote:
>>
>> The linux version sysctl is? Also I think you need to make sure
>> mfi_linux.ko is loaded before linuxsys.ko mounts so you get the
>> emulation
>> hooks. Verify that via:
>> head /compat/linux/sys/class/scsi_host/*/proc_name
>> results in one saying:
>> megaraid_sas
>> or it won't think it is there.
>
> BINGO! That did it.
>
>> The count is good to see if your file system & linux version sysctl
>> stuff is in the right state. Once it detects it, then the ioctl
>> should
>> work. 6-stable, 7-stable and -current all have the latest stuff to
>> support all of the ioctl stuff as Linux does for MegaCli. MegaCli
>> does various things to try to find the card in Linux that is really
>> strange IMHO. For FreeBSD it doesn't have to be that complicated.
>> They unfortunately, have not released a FreeBSD MegaCli which they
>> could ...
>
> It looks like I need to explicitly set those to load in the
> /boot/loader.conf instead of letting them autoload on demand...

I'll make some changes in the port, for stressing out the load order
and to let the wrapper work when compat.linux.osrelease is higher than
2.6.12

> -- Karl Denninger
> karl@denninger.net


Regards,
Ruben

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (Darwin)

iD8DBQFIW123Z88+mcQxRw0RAiTCAJ9gn0QZ51NhQ3TPecJ6Z3 0Lp485+wCggSZg
al5/7o16S90k71IF5GNb06g=
=eyGZ
-----END PGP SIGNATURE-----

On Jun 18, 2008, at 12:15 AM, Karl Denninger wrote:

> No management tool = el-sucko, because you can't rebuild a failed
> disk or
> even shut the alarm on the board off!

This is precisely the reason I have dropped using Adaptec
controllers. The most recent ones cannot be managed with the FreeBSD
tools.

What I've ended up using is LSI Fibre Channel 4Gb cards (PCI-e for
newer servers, but I have two with PCI-x) attached to an external RAID
box. My external RAIDs are custom built by Partners Data Systems and
use an Areca controller. All configuration and monitoring is done via
web interface over ethernet so no proprietary software is needed.
They fully support FreeBSD too.

_______________________________________________
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/lis...freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"