vmstat -s shows pending I/O waits increasing when vmstat -v doesn't show any wait - Aix

This is a discussion on vmstat -s shows pending I/O waits increasing when vmstat -v doesn't show any wait - Aix ; Hello everybody, I have something I don't understand (there are a lots !) and am sure one of you may have the answer. I am running a 6 CPUs partition with 24 GB RAM and AIX 5.2 ML4 (ML4 is ...

+ Reply to Thread
Results 1 to 4 of 4

Thread: vmstat -s shows pending I/O waits increasing when vmstat -v doesn't show any wait

  1. vmstat -s shows pending I/O waits increasing when vmstat -v doesn't show any wait

    Hello everybody,
    I have something I don't understand (there are a lots !) and am sure
    one of you may have the answer.
    I am running a 6 CPUs partition with 24 GB RAM and AIX 5.2 ML4 (ML4 is
    not the best AIX 5.2 level I could run, but have no choice for
    instance) for an Oracle DB server.
    I can see some I/O wait on topas.
    looking at vmstat -s, I see "pending I/O waits" counter increasing
    regularly.
    Looking at vmstat -v, the following counters are relatively small and
    don't increase at all, when "pending I/O waits" counter continues to
    increase in the same time.
    452 pending disk I/Os blocked with no pbuf
    0 paging space I/Os blocked with no psbuf
    1043 filesystem I/Os blocked with no fsbuf
    0 client filesystem I/Os blocked with no fsbuf
    0 external pager filesystem I/Os blocked with no
    fsbuf
    I would think about a possible lack of pbufs, because hd_pbuf_cnt
    seems to me relatively small (its value is set to 9088). But, if it is
    a lack of pbufs, why does vmstat -v counter "pending disk I/Os blocked
    with no pbuf" doesn't move and stays at a very low value when vmstat -
    s counter continues to increase rapidly in the same time ?
    Is it possible that this pending I/O waits come from an unsufficient
    aioservers tuning ?
    My aio setttings are the followings :
    minservers = 60
    maxservers = 120
    maxreqs = 8192
    Having 65 hdiskpowers attached on that box via 2 FC adapters.
    Looking at the number of aioservers, I can see that all of the 720
    aioservers are running...
    ..
    I really would appreciate any idea / suggestion on this case,
    especially if something has the explanation on the difference between
    vmstat -s and vmstat -v output.
    Thanks to all in advance for your help ;-)...
    Patrice.


  2. Re: vmstat -s shows pending I/O waits increasing when vmstat -v doesn't show any wait

    On Oct 1, 6:29 pm, patrice wrote:
    > Hello everybody,
    > I have something I don't understand (there are a lots !) and am sure
    > one of you may have the answer.
    > I am running a 6 CPUs partition with 24 GB RAM and AIX 5.2 ML4 (ML4 is
    > not the best AIX 5.2 level I could run, but have no choice for
    > instance) for an Oracle DB server.
    > I can see some I/O wait on topas.
    > looking at vmstat -s, I see "pending I/O waits" counter increasing
    > regularly.
    > Looking at vmstat -v, the following counters are relatively small and
    > don't increase at all, when "pending I/O waits" counter continues to
    > increase in the same time.
    > 452 pending disk I/Os blocked with no pbuf
    > 0 paging space I/Os blocked with no psbuf
    > 1043 filesystem I/Os blocked with no fsbuf
    > 0 client filesystem I/Os blocked with no fsbuf
    > 0 external pager filesystem I/Os blocked with no
    > fsbuf
    > I would think about a possible lack of pbufs, because hd_pbuf_cnt
    > seems to me relatively small (its value is set to 9088). But, if it is
    > a lack of pbufs, why does vmstat -v counter "pending disk I/Os blocked
    > with no pbuf" doesn't move and stays at a very low value when vmstat -
    > s counter continues to increase rapidly in the same time ?
    > Is it possible that this pending I/O waits come from an unsufficient
    > aioservers tuning ?
    > My aio setttings are the followings :
    > minservers = 60
    > maxservers = 120
    > maxreqs = 8192
    > Having 65 hdiskpowers attached on that box via 2 FC adapters.
    > Looking at the number of aioservers, I can see that all of the 720
    > aioservers are running...
    > .
    > I really would appreciate any idea / suggestion on this case,
    > especially if something has the explanation on the difference between
    > vmstat -s and vmstat -v output.
    > Thanks to all in advance for your help ;-)...
    > Patrice.


    As I understand it the lower number you're seeing is a subset of the
    larger. vmstat -s displays a running count of how many
    I/O have caused a wait under whatever circumstances. Mostly this is
    because I/O requests are coming in faster than the device can
    service them. pbufs are pinned memory structures used to keep track of
    i/o requests and the kernel has a certain number of them.
    Sometimes, but hopefully only rarely, things will get to the point
    where there are so many I/O requests in progress that the
    queue of pbufs is exhausted. You should expect to see, and are seeing,
    this vmstat -v number as a much smaller value that
    rarely changes because it doesn't often happen, and a good thing too.
    If you are having performance problems due to I/O contention then you
    should be considering more, and faster, disk devices,
    not more pbufs.

    HTH

    Jim Lane


  3. Re: vmstat -s shows pending I/O waits increasing when vmstat -v doesn't show any wait

    On Oct 2, 1:31 pm, Jim.L...@cibc.com wrote:
    > On Oct 1, 6:29 pm, patrice wrote:
    >
    >
    >
    >
    >
    > > Hello everybody,
    > > I have something I don't understand (there are a lots !) and am sure
    > > one of you may have the answer.
    > > I am running a 6 CPUs partition with 24 GB RAM and AIX 5.2 ML4 (ML4 is
    > > not the best AIX 5.2 level I could run, but have no choice for
    > > instance) for an Oracle DB server.
    > > I can see some I/O wait on topas.
    > > looking at vmstat -s, I see "pending I/O waits" counter increasing
    > > regularly.
    > > Looking at vmstat -v, the following counters are relatively small and
    > > don't increase at all, when "pending I/O waits" counter continues to
    > > increase in the same time.
    > > 452 pending disk I/Os blocked with no pbuf
    > > 0 paging space I/Os blocked with no psbuf
    > > 1043 filesystem I/Os blocked with no fsbuf
    > > 0 client filesystem I/Os blocked with no fsbuf
    > > 0 external pager filesystem I/Os blocked with no
    > > fsbuf
    > > I would think about a possible lack of pbufs, because hd_pbuf_cnt
    > > seems to me relatively small (its value is set to 9088). But, if it is
    > > a lack of pbufs, why does vmstat -v counter "pending disk I/Os blocked
    > > with no pbuf" doesn't move and stays at a very low value when vmstat -
    > > s counter continues to increase rapidly in the same time ?
    > > Is it possible that this pending I/O waits come from an unsufficient
    > > aioservers tuning ?
    > > My aio setttings are the followings :
    > > minservers = 60
    > > maxservers = 120
    > > maxreqs = 8192
    > > Having 65 hdiskpowers attached on that box via 2 FC adapters.
    > > Looking at the number of aioservers, I can see that all of the 720
    > > aioservers are running...
    > > .
    > > I really would appreciate any idea / suggestion on this case,
    > > especially if something has the explanation on the difference between
    > > vmstat -s and vmstat -v output.
    > > Thanks to all in advance for your help ;-)...
    > > Patrice.

    >
    > As I understand it the lower number you're seeing is a subset of the
    > larger. vmstat -s displays a running count of how many
    > I/O have caused a wait under whatever circumstances. Mostly this is
    > because I/O requests are coming in faster than the device can
    > service them. pbufs are pinned memory structures used to keep track of
    > i/o requests and the kernel has a certain number of them.
    > Sometimes, but hopefully only rarely, things will get to the point
    > where there are so many I/O requests in progress that the
    > queue of pbufs is exhausted. You should expect to see, and are seeing,
    > this vmstat -v number as a much smaller value that
    > rarely changes because it doesn't often happen, and a good thing too.
    > If you are having performance problems due to I/O contention then you
    > should be considering more, and faster, disk devices,
    > not more pbufs.
    >
    > HTH
    >
    > Jim Lane- Hide quoted text -
    >
    > - Show quoted text -


    Hi Jim,

    Many thanks for your advices.
    As it is quite difficult for me to rapidly change of disk technology
    or data layout, would it be a good idea for you, to increase aio
    maxservers to try to avoid I/O waits ?
    ps -kel | grep aio | wc -l showing the same number as maxservers x
    #CPUs, maybe increasing maxservers and/or maxrequests would help to
    handle more I/Os and decrease pending I/O waits ?
    In addition, what do you think about increasing hd_pbuf_cnt to have
    more VMM buffers also ?
    Thanks in advance for your advices.
    Patrice


  4. Re: vmstat -s shows pending I/O waits increasing when vmstat -v doesn't show any wait


    >
    > Hi Jim,
    >
    > Many thanks for your advices.
    > As it is quite difficult for me to rapidly change of disk technology
    > or data layout, would it be a good idea for you, to increase aio
    > maxservers to try to avoid I/O waits ?
    > ps -kel | grep aio | wc -l showing the same number as maxservers x
    > #CPUs, maybe increasing maxservers and/or maxrequests would help to
    > handle more I/Os and decrease pending I/O waits ?
    > In addition, what do you think about increasing hd_pbuf_cnt to have
    > more VMM buffers also ?
    > Thanks in advance for your advices.
    > Patrice


    As I understand it AIO is one of those things where the law of
    diminishing
    returns conspires against you. Going from no AIO to some AIO helps a
    lot.
    Going from a few servers to a lot of servers helps a fair bit. Beyond
    that going from
    a lot of servers to somewhat more than a lot really won't do much for
    you.
    You'll have to evaluate for yourself where you stand on that declining
    curve.
    I have no experience to share regarding hd_pbuf_cnt.
    The best thing I can do for you is to suggest a different way of
    thinking about
    your problem. The fastest I/O of all is the one you don't do. Have you
    considered
    whether the number of physical I/O's you're doing is reasonable for
    the workload
    in question? Has somebody looked at your Oracle setup? Is the SGA, and
    all that
    jazz, configured properly? Do you have indexes where you need them and
    not where
    you don't? Etc, etc.

    Regards,
    Jim Lane


+ Reply to Thread