Solaris x86 11/06 , fragmentation & I/O problem, HP DL380 g5, Help !! Thank you - Solaris

Dear all,

Allthough, we successfully installed solaris 10 x86 ver 11/06 , on a
HP 380 g5 with p400 smartarray card and SAS 72Gb disks.
The problem is that from the beginning fsck displays a file
fragmentation much higher than usual ( 1.8 % and up ), doing Oracle
import is very slow ( much more slower than Solaris 9 + DL 380g3 ) and
the worse is that our database after few days become corrupted. We try
to find some clue about what's going on without success. Do you also
have the same fragmentation and corruption issue with solaris 10x86 ??
Any advice ???

Thanks

Steven

On Jun 29, 2:04 pm, and...@cucumber.demon.co.uk (Andrew Gabriel)
wrote:
> In article <1183087822.982005.276...@i38g2000prf.googlegroups. com>,
> fdouts...@gmail.com writes:
>
> > Dear all,
>
> > Allthough, we successfully installed solaris 10 x86 ver 11/06 , on a
> > HP 380 g5 with p400 smartarray card and SAS 72Gb disks.
> > The problem is that from the beginning fsck displays a file
> > fragmentation much higher than usual ( 1.8 % and up ), doing Oracle
> > import is very slow ( much more slower than Solaris 9 + DL 380g3 ) and
> > the worse is that our database after few days become corrupted. We try
> > to find some clue about what's going on without success. Do you also
> > have the same fragmentation and corruption issue with solaris 10x86 ??
>
> The fragmentation reported by fsck is an absolutely normal feature
> of ufs. It is an indication of the number of files which are sharing
> a fragmented ufs block in order to avoid wasting a whole ufs block
> when they don't need that much disk space. It does not have anything
> to do with fragmented allocation layout.
>
> When/why is fcsk being run?
>
> --
> Andrew Gabriel
> [email address is not usable -- followup in the newsgroup]

Hi andrew,

Thanks a lot for your update, the point is that even we applied latest
patch from HP and SUN ( we have 3 servers in the same bad condition ),
the IO are very slow ( compare to solaris9 + dl380 g3 ) and day by
day hard disks conditions are deteriorating ( our database is
corrupted with bad blocks every 2/3 days, corrupted tables seems the
one with the largest volume of Data and the largest IO ).
I am quite puzzled because i cannot find a lot of post about these
issues and i guess i am not the only one using HP G5 with solaris 10 !
( we did the default installation ).
We tried to create a file partition using ZFS , IO seems faster ( but
not exceptionnal ), so i am very wondering that an incompatibilty
issue still exists between HP and Solaris 10 , or pehaps it is the SAS
hard disk format that is not well managed by solaris....still very
very wondering what's going on ! :-)

Steven

fdoutside@gmail.com wrote:
> Thanks a lot for your update, the point is that even we applied latest
> patch from HP and SUN ( we have 3 servers in the same bad condition ),
> the IO are very slow ( compare to solaris9 + dl380 g3 ) and day by

What does iostat tell you? Is Oracle configured to do use directio?

Have you run any benchmarks using a tool such as Bonnie++?

I suspect he wanted to know when you're running fsck in order to establish
whether or not you're running fsck against an in-use filesystem, which might
account for data corruption.

--
Brandon Hume - hume -> BOFH.Ca, http://WWW.BOFH.Ca/

I had a problem quite similar to yours.
Please mount the file system (ufs) without caching. I had similar
issues with EMC storages, they do not work good with FS cahcing
enabled, Maybe this is also the case here.
Hope it helps.
Kind regards.

On Jun 29, 6:30 am, fdouts...@gmail.com wrote:
> Dear all,
>
> Allthough, we successfully installed solaris 10 x86 ver 11/06 , on a
> HP 380 g5 with p400 smartarray card and SAS 72Gb disks.
> The problem is that from the beginning fsck displays a file
> fragmentation much higher than usual ( 1.8 % and up ), doing Oracle
> import is very slow ( much more slower than Solaris 9 + DL 380g3 ) and
> the worse is that our database after few days become corrupted. We try
> to find some clue about what's going on without success. Do you also
> have the same fragmentation and corruption issue with solaris 10x86 ??
> Any advice ???
>
> Thanks
>
> Steven

On Jun 30, 7:28 am, pavelj wrote:
> I had a problem quite similar to yours.
> Please mount the file system (ufs) without caching. I had similar
> issues with EMC storages, they do not work good with FS cahcing
> enabled, Maybe this is also the case here.
> Hope it helps.
> Kind regards.
>
> On Jun 29, 6:30 am, fdouts...@gmail.com wrote:
>
> > Dear all,
>
> > Allthough, we successfully installed solaris 10 x86 ver 11/06 , on a
> > HP 380 g5 with p400 smartarray card and SAS 72Gb disks.
> > The problem is that from the beginning fsck displays a file
> > fragmentation much higher than usual ( 1.8 % and up ), doing Oracle
> > import is very slow ( much more slower than Solaris 9 + DL 380g3 ) and
> > the worse is that our database after few days become corrupted. We try
> > to find some clue about what's going on without success. Do you also
> > have the same fragmentation and corruption issue with solaris 10x86 ??
> > Any advice ???
>
> > Thanks
>
> > Steven

Fragmentation can occur on the datablocks in the oracle database.
Oracle has it's own tools for dealing with this. The fragmentation on
oracle is totally separate from the OS level. Please get a good
oracle dba or read the oracle documentation on your version db to see
how to use their tools. Solaris is also slow compared to linux for
performance. Version 10 is supposed to be better, but we gave up on
solaris due to our database vendor (progress) no longer supporting
their database on solaris x86 (this was after sun declared that they
would NOT support x86 a few years ago). Sun hurt their customers a
lot by doing that.

Good luck,

Oskar

On 2 juil, 02:56, "pheoni...@gmail.com" wrote:
> On Jun 30, 7:28 am, pavelj wrote:
>
>
>
>
>
> > I had a problem quite similar to yours.
> > Please mount the file system (ufs) without caching. I had similar
> > issues with EMC storages, they do not work good with FS cahcing
> > enabled, Maybe this is also the case here.
> > Hope it helps.
> > Kind regards.
>
> > On Jun 29, 6:30 am, fdouts...@gmail.com wrote:
>
> > > Dear all,
>
> > > Allthough, we successfully installed solaris 10 x86 ver 11/06 , on a
> > > HP 380 g5 with p400 smartarray card and SAS 72Gb disks.
> > > The problem is that from the beginning fsck displays a file
> > > fragmentation much higher than usual ( 1.8 % and up ), doing Oracle
> > > import is very slow ( much more slower than Solaris 9 + DL 380g3 ) and
> > > the worse is that our database after few days become corrupted. We try
> > > to find some clue about what's going on without success. Do you also
> > > have the same fragmentation and corruption issue with solaris 10x86 ??
> > > Any advice ???
>
> > > Thanks
>
> > > Steven
>
> Fragmentation can occur on the datablocks in the oracle database.
> Oracle has it's own tools for dealing with this. The fragmentation on
> oracle is totally separate from the OS level. Please get a good
> oracle dba or read the oracle documentation on your version db to see
> how to use their tools. Solaris is also slow compared to linux for
> performance. Version 10 is supposed to be better, but we gave up on
> solaris due to our database vendor (progress) no longer supporting
> their database on solaris x86 (this was after sun declared that they
> would NOT support x86 a few years ago). Sun hurt their customers a
> lot by doing that.
>
> Good luck,
>
> Oskar- Masquer le texte des messages précédents -
>
> - Afficher le texte des messages précédents -

Dear All,

Thanks a lot for all your post , i am just back from few days break.
Yes, we indeed fsck on unmounted disks, the purpose of fsck was to try
to find a clue about our frag' problem. I will checl you suggestion on
caching and other Oracle tuning if any. Keep u guys update. Thanks
again to the community, it rocks !!!

Dear All,

We switched from UFS file system to ZFS and things seem to improve a
bit. So far result of our tests :

- DL380 G5 / Smart array P400 or P800 + SAS disks = Not OK , ( ZFS or
UFS )
- DL380 G5 / Smart array P600 + SAS disks = Not OK with UFS , so far
seems OK with ZFS

I guess we are encountering so far some incompatibilty problems with
smart array P400 / P800 and solaris 10 x86 and also perhaos some
issues with SAS disk on solaris 10.

What still amaze me , is the lack of information about this on the web
( i cannot believe we are the only one using this particular
config ! )

Thanks

Steven

fdoutside@gmail.com wrote:
> We switched from UFS file system to ZFS and things seem to improve a
> bit. So far result of our tests :

I'd still be a bit worried... just because ZFS might be correcting/rebuilding
corruption on the fly doesn't mean it isn't happening. I'd also be nervous
about running Oracle on ZFS, which is just a bit too new (and I've heard of
plenty of performance problems with the combo, as well).

I'd really recommend allocating a chunk of disk space, and running some
filesystem performance tools like Bonnie++, and perhaps the analysis and
verification options in the format tool.

> What still amaze me , is the lack of information about this on the web
> ( i cannot believe we are the only one using this particular
> config ! )

Solaris x86 is still recovering from previous Sun neglect. You might also
be in the odd middle ground between "cheap" (Linux on x86) and "serious"
(Solaris on SPARC).

--
Brandon Hume - hume -> BOFH.Ca, http://WWW.BOFH.Ca/

On Jul 4, 8:43 am, hume.spamfil...@bofh.ca wrote:
> fdouts...@gmail.com wrote:
> > We switched from UFS file system to ZFS and things seem to improve a
> > bit. So far result of our tests :
>
> I'd still be a bit worried... just because ZFS might be correcting/rebuilding
> corruption on the fly doesn't mean it isn't happening. I'd also be nervous
> about running Oracle on ZFS, which is just a bit too new (and I've heard of
> plenty of performance problems with the combo, as well).
>
> I'd really recommend allocating a chunk of disk space, and running some
> filesystem performance tools like Bonnie++, and perhaps the analysis and
> verification options in the format tool.
>
> > What still amaze me , is the lack of information about this on the web
> > ( i cannot believe we are the only one using this particular
> > config ! )
>
> Solaris x86 is still recovering from previous Sun neglect. You might also
> be in the odd middle ground between "cheap" (Linux on x86) and "serious"
> (Solaris on SPARC).
>
> --
> Brandon Hume - hume -> BOFH.Ca,http://WWW.BOFH.Ca/

Like I said before, we got out of solaris due to lack of support
(progress database) and poor performance on x86. Linux is far
superior in performance on x86 boxes. since you have HP servers, Suse
or redhat work much better on HP servers. CentOS is a clone of redhat
enterprise that is free. We use suse a lot (great admin tools, far
better than solaris or redhat), suse runs awesome on HP servers and
has excellent driver support. Suse also has excellent support from
Novell. Speaking about vendor support, did you get the support
contract from Sun? This would be a great time to call them. They
also have very good support.

It looks like your company is spending serious $$$ on this database,
why risk running it on a flaky system? Solaris x86 has always had
poor support when it comes to hardware/drivers. In regards to admin
tools, Suse and Apple are far ahead of anyone else. Both make it easy
to work on unix/linux. When was the last time you saw that?

Good luck,

Oskar

On Wed, 04 Jul 2007 19:33:36 -0700, pheonix1t@gmail.com
wrote:
>
> Like I said before, we got out of solaris due to lack of support
> (progress database) and poor performance on x86. Linux is far
> superior in performance on x86 boxes. since you have HP servers, Suse

Please post some proof of this superior performance. Until you do, I
don't believe a word of this claim.

From what I've read, Solaris 10 performs well on multicore x64. I can
well imagine that on old 32bit hardware Linux performs better.

As for HP, my current experience says 'don't touch them with a
bargepole'.

> It looks like your company is spending serious $$$ on this database,
> why risk running it on a flaky system? Solaris x86 has always had
> poor support when it comes to hardware/drivers.

I've never had any problems with my hardware.

> In regards to admin
> tools, Suse and Apple are far ahead of anyone else. Both make it easy
> to work on unix/linux. When was the last time you saw that?

Eh? I'm writing this on a Mac, and other than the dialog boxes being a
bit prettier, it was no easier to set up.

A bientot
Paul

I've just had my second glass of wine.
Iv'e been an oracle DBA for 12 years.
Look you are all talking rubbish.
Solaris performance is really hot and linux is horrid to administer.
But the orginal post is about an import.
the import of data is not a real test. of performance
if your import is slow look at the commit rate and buffers size and the
striping of disks( spread the I/O load).
the operating system really shouldn't be getting in the way. it should
be a load to hardware thing. so you should be looking at the io rates.
remembering that the commit has to happen and write a header to all
datafiles.
so increase the number of rows processed before the commit. ( its a
classic problem)
Again HP gear is beautfully built as is sun just get the db processing
right.
Cheers Hope That helps


Paul Floyd wrote:
> On Wed, 04 Jul 2007 19:33:36 -0700, pheonix1t@gmail.com
> wrote:
>
>> Like I said before, we got out of solaris due to lack of support
>> (progress database) and poor performance on x86. Linux is far
>> superior in performance on x86 boxes. since you have HP servers, Suse
>>
>
> Please post some proof of this superior performance. Until you do, I
> don't believe a word of this claim.
>
> From what I've read, Solaris 10 performs well on multicore x64. I can
> well imagine that on old 32bit hardware Linux performs better.
>
> As for HP, my current experience says 'don't touch them with a
> bargepole'.
>
>
>> It looks like your company is spending serious $$$ on this database,
>> why risk running it on a flaky system? Solaris x86 has always had
>> poor support when it comes to hardware/drivers.
>>
>
> I've never had any problems with my hardware.
>
>
>> In regards to admin
>> tools, Suse and Apple are far ahead of anyone else. Both make it easy
>> to work on unix/linux. When was the last time you saw that?
>>
>
> Eh? I'm writing this on a Mac, and other than the dialog boxes being a
> bit prettier, it was no easier to set up.
>
> A bientot
> Paul
>
>

On Thu, 05 Jul 2007 20:16:47 GMT, Mike Duffy wrote:
> Again HP gear is beautfully built as is sun just get the db processing
> right.

It may be so, but sadly here their service is so bad that they seem to
be incapable of delivering what I ordered.

Honestly, I wish I'd ordered a Dell.

A bientot
Paul

On Jul 5, 3:31 pm, Paul Floyd wrote:
> On Thu, 05 Jul 2007 20:16:47 GMT, Mike Duffy wrote:
> > Again HP gear is beautfully built as is sun just get the db processing
> > right.
>
> It may be so, but sadly here their service is so bad that they seem to
> be incapable of delivering what I ordered.
>
> Honestly, I wish I'd ordered a Dell.
>
> A bientot
> Paul

Mr DBA (previous post), thank you. If you look at MY previous posts,
I suggested that the poster talk with or get an oracle dba to look at
this.
As for Paul, I'd love to post performance tests but public ones are
rare to come by since vendors make it illegal to make performance
tests public without their consent.

I've worked in IT for 16 years (hpux was my first unix at work, then
solaris). If you've never seen suse, please don't say anything about
it, it only make you look foolish. The admin tools from suse are much
better than just about anything else on the market. Sun hasn't
improved the management tools in YEARS.
When I mention apple I'm talking about their servers. Again, if
you've never worked on them please don't dismiss what I'm saying.

solaris on x86 has always been poor in performance. It got better for
version 10, but not as fast as linux. I've run suse and solaris 10 on
same hardware, suse is faster by a long shot on simple things (users
doing queries on database, samba file transfers, apache web sites
working faster). If it does this better, the hard stuff would only be
better too (keep in mind, this is on same hardware).

I like sun, but their support on x86 has not been consistent and it
shows. The place I work at just bought $15k HP server that was a
better server than anything sun had for same price. $15k isn't
enterprise class server, but for small/medium business this is a hefty
sum. i did my research. We also (due to progress) had to look at
other choices.

Dell has been a bad experience everywhere I've had to work with it.
One place (in 2004) bough 650 workstations and 12 servers. All 12
servers had recall due to main boards burning up and smoking heavily!
Imagine sites across country starting to call within 6 weeks saying
all the servers are burning!

workstations weren't much better, in less than 8 months more than half
of the workstations had cdroms, disks or mainboards that went bad.

I was amazed by it. But they are the cheapest....this showed me you
get what you pay for.

Anyway, mr DBA is the one who we should be paying attention to!

cheers mr DBA

On Thu, 05 Jul 2007 19:03:33 -0700, pheonix1t@gmail.com
wrote:
> On Jul 5, 3:31 pm, Paul Floyd wrote:
>> On Thu, 05 Jul 2007 20:16:47 GMT, Mike Duffy wrote:
>> > Again HP gear is beautfully built as is sun just get the db processing
>> > right.
>>
>> It may be so, but sadly here their service is so bad that they seem to
>> be incapable of delivering what I ordered.
>>
>> Honestly, I wish I'd ordered a Dell.
>
> Mr DBA (previous post), thank you. If you look at MY previous posts,
> I suggested that the poster talk with or get an oracle dba to look at
> this.
> As for Paul, I'd love to post performance tests but public ones are
> rare to come by since vendors make it illegal to make performance
> tests public without their consent.
>
> I've worked in IT for 16 years (hpux was my first unix at work, then
> solaris). If you've never seen suse, please don't say anything about
> it, it only make you look foolish. The admin tools from suse are much

I have used SUSE, though not recently. I didn't find it significantly
easier to install than Solaris. The trickiest thing on Solaris, for me,
was getting a networked PCL printer to work. That could have been
easier, I must admit.

> better than just about anything else on the market. Sun hasn't
> improved the management tools in YEARS.

There were significant changes in Solaris 10. In particular to the
services.

> When I mention apple I'm talking about their servers. Again, if
> you've never worked on them please don't dismiss what I'm saying.

Never used an Apple server, not even seen one. I have a Mac portable.
Don't imagine that we'll ever have anything from Apple at work, ever.

> solaris on x86 has always been poor in performance. It got better for
> version 10, but not as fast as linux. I've run suse and solaris 10 on
> same hardware, suse is faster by a long shot on simple things (users
> doing queries on database, samba file transfers, apache web sites
> working faster). If it does this better, the hard stuff would only be
> better too (keep in mind, this is on same hardware).

I'm not involved with databases, I'm a software developer. Perhaps Linux
is handicapped by the slowness of GCC and GNU binutils, but on the same
hardware, compiling/linking on Solaris is generally faster than on
Linux.

[Dell horror stories]

We have perhaps 50 Dell workstations at work. Nothing much to report
about them. My beef with HP is that they seem to choose the cheapest
subcontractors for their distribution, and they don't do a very good
job. But they finally seem to be getting their act together.

A bientot
Paul

I couldn't help digging this one up.

gcc uses /tmp which is ram on Solaris, disk in Linux.