New 64-Bit Server (Centos 4.3, Dual CPU) HORRIBLY SLOW! - Setup

This is a discussion on New 64-Bit Server (Centos 4.3, Dual CPU) HORRIBLY SLOW! - Setup ; I've been building Linux boxes since 1995 and have never seen anything quite like this. I'm out of ideas. I recently built a dual-processor Linux server, specifications as given below. Imagine my dismay when the boot process took nearly 25 ...

+ Reply to Thread
Results 1 to 10 of 10

Thread: New 64-Bit Server (Centos 4.3, Dual CPU) HORRIBLY SLOW!

  1. New 64-Bit Server (Centos 4.3, Dual CPU) HORRIBLY SLOW!


    I've been building Linux boxes since 1995 and have never seen anything
    quite like this. I'm out of ideas.

    I recently built a dual-processor Linux server, specifications as
    given below. Imagine my dismay when the boot process took nearly 25
    minutes! Shutdowns are almost as slow. X performance is pathetic, and
    when hot-keying to a console, the transition from X to terminal takes
    probably 10 seconds. Logons are equally slow. I've loaded Centos twice
    to rule out a bad install. I've turned off every unneeded option and
    disabled parallel and serial ports at the CMOS level.

    I'm finding no help from dmesg or system log files. Other than being
    slow, the server functions okay. We're primarily going to crunch
    firewall logs on this box, and we've run some jobs that terminated
    correctly after processing for more than 48 hours. Very stable.
    Drivers and updates to the OS seem to be in order. It seems not to
    matter whether I boot an SMP kernel or single-cpu. Output from the
    3ware/AMCC controllers suggests all drives are recognized and logical
    units are intact. I've tried running with and without write caching
    enabled. No noticeable difference.

    Anyone every seen anything like this? It's as if this thing is wading
    through pancake syrup.


    Specs:

    Mainboard: Tyan S2882-D (Thunder K8SD Pro)
    Dual Opteron 242 CPUs, matched stepping
    Tyan-recommended memory, 4 Gigs
    Two SATA Controllers: 3Ware 9400S 4-port
    Array 1: RAID5 3 disks + spare, Western Digital Raptors
    Array 2: Mirror + spare, Seagate Barracudas
    Centos 4.3 64-bit OS
    Video and Ethernet onboard (ATI Rage, Broadcom)


  2. Re: New 64-Bit Server (Centos 4.3, Dual CPU) HORRIBLY SLOW!

    I had similar experience with centos 4.4 on dual cpu server; the culprit was
    failed scsi drives in a raid array. Once I removed and replaced the drives,
    the system was fast as a siren. -Frank http://www.inwa.net/~frog/

    "T. Little" wrote in message
    news:fagsi2t7q3ms2j6hkghuea7fuug8dgrild@4ax.com...
    >
    > I've been building Linux boxes since 1995 and have never seen anything
    > quite like this. I'm out of ideas.
    >
    > I recently built a dual-processor Linux server, specifications as
    > given below. Imagine my dismay when the boot process took nearly 25
    > minutes! Shutdowns are almost as slow. X performance is pathetic, and
    > when hot-keying to a console, the transition from X to terminal takes
    > probably 10 seconds. Logons are equally slow. I've loaded Centos twice
    > to rule out a bad install. I've turned off every unneeded option and
    > disabled parallel and serial ports at the CMOS level.
    >
    > I'm finding no help from dmesg or system log files. Other than being
    > slow, the server functions okay. We're primarily going to crunch
    > firewall logs on this box, and we've run some jobs that terminated
    > correctly after processing for more than 48 hours. Very stable.
    > Drivers and updates to the OS seem to be in order. It seems not to
    > matter whether I boot an SMP kernel or single-cpu. Output from the
    > 3ware/AMCC controllers suggests all drives are recognized and logical
    > units are intact. I've tried running with and without write caching
    > enabled. No noticeable difference.
    >
    > Anyone every seen anything like this? It's as if this thing is wading
    > through pancake syrup.
    >
    >
    > Specs:
    >
    > Mainboard: Tyan S2882-D (Thunder K8SD Pro)
    > Dual Opteron 242 CPUs, matched stepping
    > Tyan-recommended memory, 4 Gigs
    > Two SATA Controllers: 3Ware 9400S 4-port
    > Array 1: RAID5 3 disks + spare, Western Digital Raptors
    > Array 2: Mirror + spare, Seagate Barracudas
    > Centos 4.3 64-bit OS
    > Video and Ethernet onboard (ATI Rage, Broadcom)
    >




  3. Re: New 64-Bit Server (Centos 4.3, Dual CPU) HORRIBLY SLOW!

    frank wrote:
    > I had similar experience with centos 4.4 on dual cpu server; the culprit was
    > failed scsi drives in a raid array. Once I removed and replaced the drives,
    > the system was fast as a siren. -Frank http://www.inwa.net/~frog/
    >


    Interesting and it bares out that using the old process of building a
    step at a time is a practical method toward qualifying components before
    moving on to the next step. IE, install to a single drive and shake the
    thing down before adding in the array.

  4. Re: New 64-Bit Server (Centos 4.3, Dual CPU) HORRIBLY SLOW!

    Frank -

    While I agree with you that it would be nice to do an incremental
    component-by-component test battery for every server one puts on the
    line, I've never heard of any IT department having the staff resources
    to do that kind thing for small systems. I've heard of that kind of
    detail being paid to clusters and high-availability systems, though.
    We don't build many white-box servers anymore, but, occasionally, we
    try to stretch our dollars using what's in inventory. Pulled
    components may get an extra life or two this way. We'll let a unit run
    a week or so and check the logs for trouble before putting it into
    production, but that's about all we can realistically hope to do.



    On Fri, 13 Oct 2006 05:37:58 GMT, Marshall wrote:

    >frank wrote:
    >> I had similar experience with centos 4.4 on dual cpu server; the culprit was
    >> failed scsi drives in a raid array. Once I removed and replaced the drives,
    >> the system was fast as a siren. -Frank http://www.inwa.net/~frog/
    >>

    >
    >Interesting and it bares out that using the old process of building a
    >step at a time is a practical method toward qualifying components before
    >moving on to the next step. IE, install to a single drive and shake the
    >thing down before adding in the array.



  5. Re: New 64-Bit Server (Centos 4.3, Dual CPU) HORRIBLY SLOW!

    In comp.os.linux.setup T. Little :
    [..]

    > I recently built a dual-processor Linux server, specifications as
    > given below. Imagine my dismay when the boot process took nearly 25
    > minutes! Shutdowns are almost as slow. X performance is pathetic, and
    > when hot-keying to a console, the transition from X to terminal takes
    > probably 10 seconds. Logons are equally slow. I've loaded Centos twice
    > to rule out a bad install. I've turned off every unneeded option and


    Never had to reinstall linux, what should it change?

    The question is where does the system stay during 25 minutes
    boot? At which point does it wait? What do onboard performance
    tools, sar/iostat/top and friends tell? Where does the system
    waste all the time?

    [..]

    --
    Michael Heiming (X-PGP-Sig > GPG-Key ID: EDD27B94)
    mail: echo zvpunry@urvzvat.qr | perl -pe 'y/a-z/n-za-m/'
    #bofh excuse 19: floating point processor overflow

  6. Re: New 64-Bit Server (Centos 4.3, Dual CPU) HORRIBLY SLOW!


    Michael Heiming wrote:
    > In comp.os.linux.setup T. Little :
    > [..]
    >
    > > I recently built a dual-processor Linux server, specifications as
    > > given below. Imagine my dismay when the boot process took nearly 25
    > > minutes! Shutdowns are almost as slow. X performance is pathetic, and
    > > when hot-keying to a console, the transition from X to terminal takes
    > > probably 10 seconds. Logons are equally slow. I've loaded Centos twice
    > > to rule out a bad install. I've turned off every unneeded option and

    >
    > Never had to reinstall linux, what should it change?
    >
    > The question is where does the system stay during 25 minutes
    > boot? At which point does it wait? What do onboard performance
    > tools, sar/iostat/top and friends tell? Where does the system
    > waste all the time?
    >
    >


    I had purchased a "Lindows" PC that was this way when the network was
    enabled...

    A dbl-click on an icon - go make a sandwich, come back, start eating,
    then the app would launch.

    Disable networking, and it screamed...

    It, too, would take about 20 minutes to boot with networking enabled, a
    couple minutes without...


  7. Re: New 64-Bit Server (Centos 4.3, Dual CPU) HORRIBLY SLOW!




    A reload did do the trick in this case, and I'll tell you why. But
    first, let me make clear that the problem I was having, which was
    severe, was not about the system choking on a certain process or
    during any sort of initialization during boot-up. EVERYTHING was mud
    slow. For instance, services that normally load in three seconds would
    take maybe a minute or longer. Once the system was fully booted,
    anything that required disk access was unbelievably slow. Logging in
    from a console was almost enough to make me cry.

    The problem? It was that, when I built the server originally, I wanted
    to include the very latest firmware and drivers from 3Ware. I flashed
    my controllers, which went fine, and made a driver disk to use during
    installation. I started the Redhat/Centos installation using the
    "Linux DD" (driver disk) option and the new driver was pulled in just
    fine - - or, at least, appeared to be.

    I had no way of knowing that, had I simply used the 3w-9xxx driver
    included with Centos, I'd have been in much better shape. I'm not
    enough of a kernel wizard to tell you what went wrong, but when I
    rebuilt the server using the stock 3Ware driver, I was rewarded with
    the 64-bit smoking performance I'd expected all along. This strikes me
    as being odd, but I know 3Ware SATA controllers are gaining in
    popularity, so this is something to keep in mind if you're considering
    using them.

    Regards,
    TL



    On Fri, 13 Oct 2006 23:37:02 +0200, Michael Heiming
    wrote:

    >In comp.os.linux.setup T. Little :
    >[..]
    >
    >> I recently built a dual-processor Linux server, specifications as
    >> given below. Imagine my dismay when the boot process took nearly 25
    >> minutes! Shutdowns are almost as slow. X performance is pathetic, and
    >> when hot-keying to a console, the transition from X to terminal takes
    >> probably 10 seconds. Logons are equally slow. I've loaded Centos twice
    >> to rule out a bad install. I've turned off every unneeded option and

    >
    >Never had to reinstall linux, what should it change?
    >
    >The question is where does the system stay during 25 minutes
    >boot? At which point does it wait? What do onboard performance
    >tools, sar/iostat/top and friends tell? Where does the system
    >waste all the time?
    >
    >[..]



  8. Re: New 64-Bit Server (Centos 4.3, Dual CPU) HORRIBLY SLOW!

    In comp.os.linux.setup T. LIttle :
    > On Fri, 13 Oct 2006 23:37:02 +0200, Michael Heiming
    > wrote:
    >>In comp.os.linux.setup T. Little :


    [ dog slow performance on CentOS (3ware controller) ]

    >>Never had to reinstall linux, what should it change?

    [..]

    > A reload did do the trick in this case, and I'll tell you why. But

    [..]

    > The problem? It was that, when I built the server originally, I wanted
    > to include the very latest firmware and drivers from 3Ware. I flashed
    > my controllers, which went fine, and made a driver disk to use during
    > installation. I started the Redhat/Centos installation using the
    > "Linux DD" (driver disk) option and the new driver was pulled in just
    > fine - - or, at least, appeared to be.


    I tell you, you could have just changed drivers you used with
    those delivered with CentOS in a few seconds, rebuild initrd and
    should have been fine after reboot, presuming / is on the 3ware
    controller. No re-installation needed, this isn't doze...;-)

    Good luck

    [..]

    --
    Michael Heiming (X-PGP-Sig > GPG-Key ID: EDD27B94)
    mail: echo zvpunry@urvzvat.qr | perl -pe 'y/a-z/n-za-m/'
    #bofh excuse 341: HTTPD Error 666 : BOFH was here

  9. Re: New 64-Bit Server (Centos 4.3, Dual CPU) HORRIBLY SLOW!

    Agreed. I considered compiling the driver from 3Ware source and
    building a custom kernel, but, given the speed of the machine in
    question, a reload and restore from tape gave me that warm, fuzzy
    feeling, and helped me rule out hardware issues . . . .

    >
    >I tell you, you could have just changed drivers you used with
    >those delivered with CentOS in a few seconds, rebuild initrd and
    >should have been fine after reboot, presuming / is on the 3ware
    >controller. No re-installation needed, this isn't doze...;-)
    >
    >Good luck
    >
    >[..]



  10. Re: New 64-Bit Server (Centos 4.3, Dual CPU) HORRIBLY SLOW!

    In article news:<1160780722.317887.30400@k70g2000cwa.googlegroups.c om>,
    Anoneds@netscape.net wrote:
    > I had purchased a "Lindows" PC that was this way when the network was
    > enabled...
    >
    > A dbl-click on an icon - go make a sandwich, come back, start eating,
    > then the app would launch.
    >
    > Disable networking, and it screamed...


    I've seen a similar thing on a (SuSE, IIRC) system that was trying to use
    IPv6 on a network that didn't support it. I disabled IPv6 support in linux
    and normal service was resumed.

    Cheers,
    Daniel.



+ Reply to Thread