Help with excessive I/O - Storage

This is a discussion on Help with excessive I/O - Storage ; (Please advise if this isn't the appropriate group for this post, and I will repost it elsewhere, thank you.) I'm trying to determine the cause of a sudden, drastic decrease in performance on my Windows 2003 Server box. Any action ...

+ Reply to Thread
Results 1 to 3 of 3

Thread: Help with excessive I/O

  1. Help with excessive I/O

    (Please advise if this isn't the appropriate group for this post, and
    I will repost it elsewhere, thank you.)

    I'm trying to determine the cause of a sudden, drastic decrease in
    performance on my Windows 2003 Server box.

    Any action at all, opening an application or an Explorer window,
    logging in, causes a huge amount of I/O to happen against the system
    drive (drive 0, the C: drive). For instance, when logging in, there
    is a greater than 3 minute delay between the time I OK the login
    dialog until Windows displays the desktop icons. During this long
    delay the drive activity LED is steadily on.

    At first I thought Windows might be thrashing due to low memory, but
    the physical memory usage seems to only be about 40%. I did not see
    any appreciable paging activity when running perfmon, either. Just to
    be sure, I increased the physical memory from 1 to 2GB in the machine,
    but that had no effect whatsoever.

    I checked the disk for errors via Windows Explorer - none were
    reported. I tried running Process Monitor to see the I/O being
    performed - there isn't one process that seems to be doing a majority
    of the I/O (although a process named lsass.exe seems to be doing quite
    a bit.) I did notice that all of the I/O's seem to have multiple
    sequential entries in PM's log - not sure if this is normal or there
    is heavy re-trying happening.

    I used some low-level disk checking utilities on the drive (Maxtor and
    SpinRite) - neither reported any errors on the disk.

    I set up a perfmon session to watch the following counters on Drive 0:
    Avg. Disk Queue Length, Avg Disk sec/Read, Avg Disk sec/Write.

    These all exhibit spikey, very high values. For instance the avg time
    per write on the disk repeatedly spikes up to 5-7 secs per write (yes,
    I'm sure of this value, I've tripled-checked the scaling and my
    interpretation.) The values for all three counters repeatedly
    oscillate from very low (normal) values to extremely high values over
    time, forming what almost looks like a sawtooth wave on perfmon's
    graph display. The period between these spikes hovers around 3-4
    secs.

    So, it seems that my drive does not have any errors on it, yet I/O to
    it has suddenly and mysteriously become abysmal.

    I'm running out of ideas - can anyone help?

    Thanks very much.


  2. Re: Help with excessive I/O

    Mr. Land wrote:
    > (Please advise if this isn't the appropriate group for this post, and
    > I will repost it elsewhere, thank you.)
    >
    > I'm trying to determine the cause of a sudden, drastic decrease in
    > performance on my Windows 2003 Server box.
    >
    > Any action at all, opening an application or an Explorer window,
    > logging in, causes a huge amount of I/O to happen against the system
    > drive (drive 0, the C: drive). For instance, when logging in, there
    > is a greater than 3 minute delay between the time I OK the login
    > dialog until Windows displays the desktop icons. During this long
    > delay the drive activity LED is steadily on.
    >
    > At first I thought Windows might be thrashing due to low memory, but
    > the physical memory usage seems to only be about 40%. I did not see
    > any appreciable paging activity when running perfmon, either. Just to
    > be sure, I increased the physical memory from 1 to 2GB in the machine,
    > but that had no effect whatsoever.
    >
    > I checked the disk for errors via Windows Explorer - none were
    > reported. I tried running Process Monitor to see the I/O being
    > performed - there isn't one process that seems to be doing a majority
    > of the I/O (although a process named lsass.exe seems to be doing quite
    > a bit.) I did notice that all of the I/O's seem to have multiple
    > sequential entries in PM's log - not sure if this is normal or there
    > is heavy re-trying happening.
    >
    > I used some low-level disk checking utilities on the drive (Maxtor and
    > SpinRite) - neither reported any errors on the disk.
    >
    > I set up a perfmon session to watch the following counters on Drive 0:
    > Avg. Disk Queue Length, Avg Disk sec/Read, Avg Disk sec/Write.
    >
    > These all exhibit spikey, very high values. For instance the avg time
    > per write on the disk repeatedly spikes up to 5-7 secs per write (yes,
    > I'm sure of this value, I've tripled-checked the scaling and my
    > interpretation.) The values for all three counters repeatedly
    > oscillate from very low (normal) values to extremely high values over
    > time, forming what almost looks like a sawtooth wave on perfmon's
    > graph display. The period between these spikes hovers around 3-4
    > secs.
    >
    > So, it seems that my drive does not have any errors on it, yet I/O to
    > it has suddenly and mysteriously become abysmal.
    >
    > I'm running out of ideas - can anyone help?


    Post the Everest SMART report.
    http://www.majorgeeks.com/download.php?det=4181

    One possibility is that you have some marginal sectors and the drive is trying on those
    and eventually getting the data successfully, but the retrys are slowing things down.



  3. Re: Help with excessive I/O

    Mr. Land wrote:

    > (Please advise if this isn't the appropriate group for this post, and
    > I will repost it elsewhere, thank you.)
    >
    > I'm trying to determine the cause of a sudden, drastic decrease in
    > performance on my Windows 2003 Server box.
    >
    > Any action at all, opening an application or an Explorer window,
    > logging in, causes a huge amount of I/O to happen against the system
    > drive (drive 0, the C: drive). For instance, when logging in, there
    > is a greater than 3 minute delay between the time I OK the login
    > dialog until Windows displays the desktop icons. During this long
    > delay the drive activity LED is steadily on.
    >
    > At first I thought Windows might be thrashing due to low memory, but
    > the physical memory usage seems to only be about 40%. I did not see
    > any appreciable paging activity when running perfmon, either. Just to
    > be sure, I increased the physical memory from 1 to 2GB in the machine,
    > but that had no effect whatsoever.
    >
    > I checked the disk for errors via Windows Explorer - none were
    > reported. I tried running Process Monitor to see the I/O being
    > performed - there isn't one process that seems to be doing a majority
    > of the I/O (although a process named lsass.exe seems to be doing quite
    > a bit.) I did notice that all of the I/O's seem to have multiple
    > sequential entries in PM's log - not sure if this is normal or there
    > is heavy re-trying happening.
    >
    > I used some low-level disk checking utilities on the drive (Maxtor and
    > SpinRite) - neither reported any errors on the disk.
    >
    > I set up a perfmon session to watch the following counters on Drive 0:
    > Avg. Disk Queue Length, Avg Disk sec/Read, Avg Disk sec/Write.
    >
    > These all exhibit spikey, very high values. For instance the avg time
    > per write on the disk repeatedly spikes up to 5-7 secs per write (yes,
    > I'm sure of this value, I've tripled-checked the scaling and my
    > interpretation.) The values for all three counters repeatedly
    > oscillate from very low (normal) values to extremely high values over
    > time, forming what almost looks like a sawtooth wave on perfmon's
    > graph display. The period between these spikes hovers around 3-4
    > secs.
    >
    > So, it seems that my drive does not have any errors on it, yet I/O to
    > it has suddenly and mysteriously become abysmal.
    >
    > I'm running out of ideas - can anyone help?
    >
    > Thanks very much.
    >


    1. Because it is free, not because it is terribly likely, reseat all of
    the cables (not to be insulting, but with the power off). While you're
    at it, examine the cables and connectors to see if one has been nicked.

    2. If you've added any hardware recently, remove it. It is possible that
    adding something has overloaded the power supply, causing sags during
    heavy I/O and retrys. And, if feasible, you might swap the PS with one
    that is known good and that delivers more juice, for the same reason.

    3. Run HDtach on the whole HD(s), to see if there are repeatable read spikes,
    indicating bad sectors. Or run HDtune's HealthCheck. Or try both.
    --
    Cheers, Bob

+ Reply to Thread