vmstat -s shows pending I/O waits increasing when vmstat -v doesn't show any wait

Hello everybody,
I have something I don't understand (there are a lots !) and am sure
one of you may have the answer.
I am running a 6 CPUs partition with 24 GB RAM and AIX 5.2 ML4 (ML4 is
not the best AIX 5.2 level I could run, but have no choice for
instance) for an Oracle DB server.
I can see some I/O wait on topas.
looking at vmstat -s, I see "pending I/O waits" counter increasing
regularly.
Looking at vmstat -v, the following counters are relatively small and
don't increase at all, when "pending I/O waits" counter continues to
increase in the same time.
452 pending disk I/Os blocked with no pbuf
0 paging space I/Os blocked with no psbuf
1043 filesystem I/Os blocked with no fsbuf
0 client filesystem I/Os blocked with no fsbuf
0 external pager filesystem I/Os blocked with no
fsbuf
I would think about a possible lack of pbufs, because hd_pbuf_cnt
seems to me relatively small (its value is set to 9088). But, if it is
a lack of pbufs, why does vmstat -v counter "pending disk I/Os blocked
with no pbuf" doesn't move and stays at a very low value when vmstat -
s counter continues to increase rapidly in the same time ?
Is it possible that this pending I/O waits come from an unsufficient
aioservers tuning ?
My aio setttings are the followings :
minservers = 60
maxservers = 120
maxreqs = 8192
Having 65 hdiskpowers attached on that box via 2 FC adapters.
Looking at the number of aioservers, I can see that all of the 720
aioservers are running...
..
I really would appreciate any idea / suggestion on this case,
especially if something has the explanation on the difference between
vmstat -s and vmstat -v output.
Thanks to all in advance for your help ;-)...
Patrice.

On Oct 1, 6:29 pm, patrice <quet.patr...@neuf.fr> wrote:[color=blue]
> Hello everybody,
> I have something I don't understand (there are a lots !) and am sure
> one of you may have the answer.
> I am running a 6 CPUs partition with 24 GB RAM and AIX 5.2 ML4 (ML4 is
> not the best AIX 5.2 level I could run, but have no choice for
> instance) for an Oracle DB server.
> I can see some I/O wait on topas.
> looking at vmstat -s, I see "pending I/O waits" counter increasing
> regularly.
> Looking at vmstat -v, the following counters are relatively small and
> don't increase at all, when "pending I/O waits" counter continues to
> increase in the same time.
> 452 pending disk I/Os blocked with no pbuf
> 0 paging space I/Os blocked with no psbuf
> 1043 filesystem I/Os blocked with no fsbuf
> 0 client filesystem I/Os blocked with no fsbuf
> 0 external pager filesystem I/Os blocked with no
> fsbuf
> I would think about a possible lack of pbufs, because hd_pbuf_cnt
> seems to me relatively small (its value is set to 9088). But, if it is
> a lack of pbufs, why does vmstat -v counter "pending disk I/Os blocked
> with no pbuf" doesn't move and stays at a very low value when vmstat -
> s counter continues to increase rapidly in the same time ?
> Is it possible that this pending I/O waits come from an unsufficient
> aioservers tuning ?
> My aio setttings are the followings :
> minservers = 60
> maxservers = 120
> maxreqs = 8192
> Having 65 hdiskpowers attached on that box via 2 FC adapters.
> Looking at the number of aioservers, I can see that all of the 720
> aioservers are running...
> .
> I really would appreciate any idea / suggestion on this case,
> especially if something has the explanation on the difference between
> vmstat -s and vmstat -v output.
> Thanks to all in advance for your help ;-)...
> Patrice.[/color]

As I understand it the lower number you're seeing is a subset of the
larger. vmstat -s displays a running count of how many
I/O have caused a wait under whatever circumstances. Mostly this is
because I/O requests are coming in faster than the device can
service them. pbufs are pinned memory structures used to keep track of
i/o requests and the kernel has a certain number of them.
Sometimes, but hopefully only rarely, things will get to the point
where there are so many I/O requests in progress that the
queue of pbufs is exhausted. You should expect to see, and are seeing,
this vmstat -v number as a much smaller value that
rarely changes because it doesn't often happen, and a good thing too.
If you are having performance problems due to I/O contention then you
should be considering more, and faster, disk devices,
not more pbufs.

HTH

Jim Lane

On Oct 2, 1:31 pm, Jim.L...@cibc.com wrote:[color=blue]
> On Oct 1, 6:29 pm, patrice <quet.patr...@neuf.fr> wrote:
>
>
>
>
>[color=green]
> > Hello everybody,
> > I have something I don't understand (there are a lots !) and am sure
> > one of you may have the answer.
> > I am running a 6 CPUs partition with 24 GB RAM and AIX 5.2 ML4 (ML4 is
> > not the best AIX 5.2 level I could run, but have no choice for
> > instance) for an Oracle DB server.
> > I can see some I/O wait on topas.
> > looking at vmstat -s, I see "pending I/O waits" counter increasing
> > regularly.
> > Looking at vmstat -v, the following counters are relatively small and
> > don't increase at all, when "pending I/O waits" counter continues to
> > increase in the same time.
> > 452 pending disk I/Os blocked with no pbuf
> > 0 paging space I/Os blocked with no psbuf
> > 1043 filesystem I/Os blocked with no fsbuf
> > 0 client filesystem I/Os blocked with no fsbuf
> > 0 external pager filesystem I/Os blocked with no
> > fsbuf
> > I would think about a possible lack of pbufs, because hd_pbuf_cnt
> > seems to me relatively small (its value is set to 9088). But, if it is
> > a lack of pbufs, why does vmstat -v counter "pending disk I/Os blocked
> > with no pbuf" doesn't move and stays at a very low value when vmstat -
> > s counter continues to increase rapidly in the same time ?
> > Is it possible that this pending I/O waits come from an unsufficient
> > aioservers tuning ?
> > My aio setttings are the followings :
> > minservers = 60
> > maxservers = 120
> > maxreqs = 8192
> > Having 65 hdiskpowers attached on that box via 2 FC adapters.
> > Looking at the number of aioservers, I can see that all of the 720
> > aioservers are running...
> > .
> > I really would appreciate any idea / suggestion on this case,
> > especially if something has the explanation on the difference between
> > vmstat -s and vmstat -v output.
> > Thanks to all in advance for your help ;-)...
> > Patrice.[/color]
>
> As I understand it the lower number you're seeing is a subset of the
> larger. vmstat -s displays a running count of how many
> I/O have caused a wait under whatever circumstances. Mostly this is
> because I/O requests are coming in faster than the device can
> service them. pbufs are pinned memory structures used to keep track of
> i/o requests and the kernel has a certain number of them.
> Sometimes, but hopefully only rarely, things will get to the point
> where there are so many I/O requests in progress that the
> queue of pbufs is exhausted. You should expect to see, and are seeing,
> this vmstat -v number as a much smaller value that
> rarely changes because it doesn't often happen, and a good thing too.
> If you are having performance problems due to I/O contention then you
> should be considering more, and faster, disk devices,
> not more pbufs.
>
> HTH
>
> Jim Lane- Hide quoted text -
>
> - Show quoted text -[/color]

Hi Jim,

Many thanks for your advices.
As it is quite difficult for me to rapidly change of disk technology
or data layout, would it be a good idea for you, to increase aio
maxservers to try to avoid I/O waits ?
ps -kel | grep aio | wc -l showing the same number as maxservers x
#CPUs, maybe increasing maxservers and/or maxrequests would help to
handle more I/Os and decrease pending I/O waits ?
In addition, what do you think about increasing hd_pbuf_cnt to have
more VMM buffers also ?
Thanks in advance for your advices.
Patrice

[color=blue]
>
> Hi Jim,
>
> Many thanks for your advices.
> As it is quite difficult for me to rapidly change of disk technology
> or data layout, would it be a good idea for you, to increase aio
> maxservers to try to avoid I/O waits ?
> ps -kel | grep aio | wc -l showing the same number as maxservers x
> #CPUs, maybe increasing maxservers and/or maxrequests would help to
> handle more I/Os and decrease pending I/O waits ?
> In addition, what do you think about increasing hd_pbuf_cnt to have
> more VMM buffers also ?
> Thanks in advance for your advices.
> Patrice[/color]

As I understand it AIO is one of those things where the law of
diminishing
returns conspires against you. Going from no AIO to some AIO helps a
lot.
Going from a few servers to a lot of servers helps a fair bit. Beyond
that going from
a lot of servers to somewhat more than a lot really won't do much for
you.
You'll have to evaluate for yourself where you stand on that declining
curve.
I have no experience to share regarding hd_pbuf_cnt.
The best thing I can do for you is to suggest a different way of
thinking about
your problem. The fastest I/O of all is the one you don't do. Have you
considered
whether the number of physical I/O's you're doing is reasonable for
the workload
in question? Has somebody looked at your Oracle setup? Is the SGA, and
all that
jazz, configured properly? Do you have indexes where you need them and
not where
you don't? Etc, etc.

Regards,
Jim Lane