lsps shows high values but system appears to be fine - Aix

This is a discussion on lsps shows high values but system appears to be fine - Aix ; I have several systems that run AIX 5L and Informix databases on them. Most of these systems show values I would expect on a system. However, one seems to give me some weird values in the lsps command. Here are ...

+ Reply to Thread
Results 1 to 7 of 7

Thread: lsps shows high values but system appears to be fine

  1. lsps shows high values but system appears to be fine

    I have several systems that run AIX 5L and Informix databases on them.
    Most of these systems show values I would expect on a system. However,
    one seems to give me some weird values in the lsps command.

    Here are the details:

    System Model: IBM,9131-52A
    Number Of Processors: 4
    Processor Clock Speed: 1648 MHz
    CPU Type: 64-bit
    Kernel Type: 64-bit
    Memory Size: 7936 MB
    Good Memory Size: 7936 MB

    Informix Version 9.40 FC7

    # oslevel -s
    5200-10-00-0000

    # lsps -a
    Page Space Physical Volume Volume Group Size %Used Active
    Auto Type
    paging00 hdisk3 datavg 6400MB 37 yes
    yes lv
    hd6 hdisk5 rootvg 6656MB 100 yes
    yes lv

    # lsps -s
    Total Paging Space Percent Used
    13056MB 69%

    # vmstat 10 10
    System Configuration: lcpu=4 mem=7936MB
    kthr memory page faults cpu
    ----- ----------- ------------------------ ------------ -----------
    r b avm fre re pi po fr sr cy in sy cs us sy id wa
    1 1 2715508 79451 0 10 10 335 776 0 786 16640 5220 7 2 90
    2
    1 0 2715519 79440 0 0 0 0 0 0 601 11137 3835 7 1 92
    0
    1 0 2715520 79411 0 0 0 0 0 0 641 34843 12786 24 2 74
    0
    1 0 2715529 79401 0 0 0 0 0 0 606 4249 1631 2 0 97 0
    1 0 2715673 79242 0 0 0 0 0 0 648 17652 6941 11 1 88
    0
    1 0 2716015 78898 0 0 0 0 0 0 643 17268 6116 11 1 88
    0
    2 0 2716193 78742 0 0 0 0 0 0 697 22620 8057 14 1 85
    0
    2 0 2717113 77816 0 0 0 0 0 0 809 25793 9227 16 1 83
    0
    5 0 2716951 77872 0 1 0 0 0 0 777 26851 4535 12 3 84
    0
    0 0 2716921 77902 0 0 0 0 0 0 593 4828 1846 3 0 97 0

    # vmstat -v
    2031616 memory pages
    1952077 lruable pages
    73804 free pages
    1 memory pools
    333094 pinned pages
    80.1 maxpin percentage
    20.0 minperm percentage
    80.0 maxperm percentage
    57.1 numperm percentage
    1114800 file pages
    0.0 compressed percentage
    0 compressed pages
    57.7 numclient percentage
    80.0 maxclient percentage
    1127373 client pages
    0 remote pageouts scheduled
    5943302 pending disk I/Os blocked with no pbuf
    25321 paging space I/Os blocked with no psbuf
    2740 filesystem I/Os blocked with no fsbuf
    0 client filesystem I/Os blocked with no fsbuf
    1436111 external pager filesystem I/Os blocked with no
    fsbuf

    # lsattr -El aio0
    autoconfig available STATE to be configured at system restart True
    fastpath enable State of fast path True
    kprocprio 39 Server PRIORITY True
    maxreqs 4096 Maximum number of REQUESTS True
    maxservers 50 MAXIMUM number of servers per cpu True
    minservers 10 MINIMUM number of servers True


    The Informix databases are operating on raw LVs and so there should
    not be a double-caching issue that I have heard mentioned. I have
    other databases on other servers with less memory and those do not
    seem to be causing high paging activity.

    The users are not complaining about slow performance or any noticeable
    issues, so just wondering if anyone may have a suggestion as to
    whether this is a problem, and if there are any settings I should be
    looking at that could tell me why this system has decided to use a
    significant portion of its paging space for no apparent reason.

    Thanks in advance.

    Steve

  2. Re: lsps shows high values but system appears to be fine

    On Apr 23, 5:28 pm, "steven_nospam at Yahoo! Canada"
    wrote:
    > I have several systems that run AIX 5L and Informix databases on them.
    > Most of these systems show values I would expect on a system. However,
    > one seems to give me some weird values in the lsps command.
    >
    > Here are the details:
    >
    > System Model: IBM,9131-52A
    > Number Of Processors: 4
    > Processor Clock Speed: 1648 MHz
    > CPU Type: 64-bit
    > Kernel Type: 64-bit
    > Memory Size: 7936 MB
    > Good Memory Size: 7936 MB
    >
    > Informix Version 9.40 FC7
    >
    > # oslevel -s
    > 5200-10-00-0000
    >
    > # lsps -a
    > Page Space Physical Volume Volume Group Size %Used Active
    > Auto Type
    > paging00 hdisk3 datavg 6400MB 37 yes
    > yes lv
    > hd6 hdisk5 rootvg 6656MB 100 yes
    > yes lv
    >
    > # lsps -s
    > Total Paging Space Percent Used
    > 13056MB 69%
    >
    > # vmstat 10 10
    > System Configuration: lcpu=4 mem=7936MB
    > kthr memory page faults cpu
    > ----- ----------- ------------------------ ------------ -----------
    > r b avm fre re pi po fr sr cy in sy cs us sy id wa
    > 1 1 2715508 79451 0 10 10 335 776 0 786 16640 5220 7 2 90
    > 2
    > 1 0 2715519 79440 0 0 0 0 0 0 601 11137 3835 7 1 92
    > 0
    > 1 0 2715520 79411 0 0 0 0 0 0 641 34843 12786 24 2 74
    > 0
    > 1 0 2715529 79401 0 0 0 0 0 0 606 4249 1631 2 0 97 0
    > 1 0 2715673 79242 0 0 0 0 0 0 648 17652 6941 11 1 88
    > 0
    > 1 0 2716015 78898 0 0 0 0 0 0 643 17268 6116 11 1 88
    > 0
    > 2 0 2716193 78742 0 0 0 0 0 0 697 22620 8057 14 1 85
    > 0
    > 2 0 2717113 77816 0 0 0 0 0 0 809 25793 9227 16 1 83
    > 0
    > 5 0 2716951 77872 0 1 0 0 0 0 777 26851 4535 12 3 84
    > 0
    > 0 0 2716921 77902 0 0 0 0 0 0 593 4828 1846 3 0 97 0
    >
    > # vmstat -v
    > 2031616 memory pages
    > 1952077 lruable pages
    > 73804 free pages
    > 1 memory pools
    > 333094 pinned pages
    > 80.1 maxpin percentage
    > 20.0 minperm percentage
    > 80.0 maxperm percentage
    > 57.1 numperm percentage
    > 1114800 file pages
    > 0.0 compressed percentage
    > 0 compressed pages
    > 57.7 numclient percentage
    > 80.0 maxclient percentage
    > 1127373 client pages
    > 0 remote pageouts scheduled
    > 5943302 pending disk I/Os blocked with no pbuf
    > 25321 paging space I/Os blocked with no psbuf
    > 2740 filesystem I/Os blocked with no fsbuf
    > 0 client filesystem I/Os blocked with no fsbuf
    > 1436111 external pager filesystem I/Os blocked with no
    > fsbuf
    >
    > # lsattr -El aio0
    > autoconfig available STATE to be configured at system restart True
    > fastpath enable State of fast path True
    > kprocprio 39 Server PRIORITY True
    > maxreqs 4096 Maximum number of REQUESTS True
    > maxservers 50 MAXIMUM number of servers per cpu True
    > minservers 10 MINIMUM number of servers True
    >
    > The Informix databases are operating on raw LVs and so there should
    > not be a double-caching issue that I have heard mentioned. I have
    > other databases on other servers with less memory and those do not
    > seem to be causing high paging activity.
    >
    > The users are not complaining about slow performance or any noticeable
    > issues, so just wondering if anyone may have a suggestion as to
    > whether this is a problem, and if there are any settings I should be
    > looking at that could tell me why this system has decided to use a
    > significant portion of its paging space for no apparent reason.
    >
    > Thanks in advance.
    >
    > Steve


    1)
    From
    http://www.ibm.com/developerworks/vi...emory+settings

    Part 3

    Paging allocation type:
    - deferred page space allocation
    - late page space allocation
    - early page space allocation.

    The default policy of AIX is deferred page space allocation.

    2)
    For some reason your system is using pretty much file cache. Could be
    NFS, log files and what the heck i know but it looks like the
    computational pages get page out. So check for busy filesystem.

    In case the paging rate is low the system is working as designed ;-)

    hth
    Hajo

  3. Re: lsps shows high values but system appears to be fine

    On Apr 23, 10:52 pm, Hajo Ehlers wrote:
    > On Apr 23, 5:28 pm, "steven_nospam at Yahoo! Canada"
    >
    >
    >
    > wrote:
    > > I have several systems that run AIX 5L and Informix databases on them.
    > > Most of these systems show values I would expect on a system. However,
    > > one seems to give me some weird values in the lsps command.

    >
    > > Here are the details:

    >
    > > System Model: IBM,9131-52A
    > > Number Of Processors: 4
    > > Processor Clock Speed: 1648 MHz
    > > CPU Type: 64-bit
    > > Kernel Type: 64-bit
    > > Memory Size: 7936 MB
    > > Good Memory Size: 7936 MB

    >
    > > Informix Version 9.40 FC7

    >
    > > # oslevel -s
    > > 5200-10-00-0000

    >
    > > # lsps -a
    > > Page Space Physical Volume Volume Group Size %Used Active
    > > Auto Type
    > > paging00 hdisk3 datavg 6400MB 37 yes
    > > yes lv
    > > hd6 hdisk5 rootvg 6656MB 100 yes
    > > yes lv

    >
    > > # lsps -s
    > > Total Paging Space Percent Used
    > > 13056MB 69%

    >
    > > # vmstat 10 10
    > > System Configuration: lcpu=4 mem=7936MB
    > > kthr memory page faults cpu
    > > ----- ----------- ------------------------ ------------ -----------
    > > r b avm fre re pi po fr sr cy in sy cs us sy id wa
    > > 1 1 2715508 79451 0 10 10 335 776 0 786 16640 5220 7 2 90
    > > 2
    > > 1 0 2715519 79440 0 0 0 0 0 0 601 11137 3835 7 1 92
    > > 0
    > > 1 0 2715520 79411 0 0 0 0 0 0 641 34843 12786 24 2 74
    > > 0
    > > 1 0 2715529 79401 0 0 0 0 0 0 606 4249 1631 2 0 97 0
    > > 1 0 2715673 79242 0 0 0 0 0 0 648 17652 6941 11 1 88
    > > 0
    > > 1 0 2716015 78898 0 0 0 0 0 0 643 17268 6116 11 1 88
    > > 0
    > > 2 0 2716193 78742 0 0 0 0 0 0 697 22620 8057 14 1 85
    > > 0
    > > 2 0 2717113 77816 0 0 0 0 0 0 809 25793 9227 16 1 83
    > > 0
    > > 5 0 2716951 77872 0 1 0 0 0 0 777 26851 4535 12 3 84
    > > 0
    > > 0 0 2716921 77902 0 0 0 0 0 0 593 4828 1846 3 0 97 0

    >
    > > # vmstat -v
    > > 2031616 memory pages
    > > 1952077 lruable pages
    > > 73804 free pages
    > > 1 memory pools
    > > 333094 pinned pages
    > > 80.1 maxpin percentage
    > > 20.0 minperm percentage
    > > 80.0 maxperm percentage
    > > 57.1 numperm percentage
    > > 1114800 file pages
    > > 0.0 compressed percentage
    > > 0 compressed pages
    > > 57.7 numclient percentage
    > > 80.0 maxclient percentage
    > > 1127373 client pages
    > > 0 remote pageouts scheduled
    > > 5943302 pending disk I/Os blocked with no pbuf
    > > 25321 paging space I/Os blocked with no psbuf
    > > 2740 filesystem I/Os blocked with no fsbuf
    > > 0 client filesystem I/Os blocked with no fsbuf
    > > 1436111 external pager filesystem I/Os blocked with no
    > > fsbuf

    >
    > > # lsattr -El aio0
    > > autoconfig available STATE to be configured at system restart True
    > > fastpath enable State of fast path True
    > > kprocprio 39 Server PRIORITY True
    > > maxreqs 4096 Maximum number of REQUESTS True
    > > maxservers 50 MAXIMUM number of servers per cpu True
    > > minservers 10 MINIMUM number of servers True

    >
    > > The Informix databases are operating on raw LVs and so there should
    > > not be a double-caching issue that I have heard mentioned. I have
    > > other databases on other servers with less memory and those do not
    > > seem to be causing high paging activity.

    >
    > > The users are not complaining about slow performance or any noticeable
    > > issues, so just wondering if anyone may have a suggestion as to
    > > whether this is a problem, and if there are any settings I should be
    > > looking at that could tell me why this system has decided to use a
    > > significant portion of its paging space for no apparent reason.

    >
    > > Thanks in advance.

    >
    > > Steve

    >
    > 1)
    > Fromhttp://www.ibm.com/developerworks/views/aix/libraryview.jsp?search_by...
    >
    > Part 3
    >
    > Paging allocation type:
    > - deferred page space allocation
    > - late page space allocation
    > - early page space allocation.
    >
    > The default policy of AIX is deferred page space allocation.
    >
    > 2)
    > For some reason your system is using pretty much file cache. Could be
    > NFS, log files and what the heck i know but it looks like the
    > computational pages get page out. So check for busy filesystem.
    >
    > In case the paging rate is low the system is working as designed ;-)
    >
    > hth
    > Hajo


    A high % size of paging space in lsps could be partly due to the fact
    that at various points of time comp pages have been paged out to
    paging space. This is because about 57% of the memory is filled with
    file pages which have been protected due to their high repage rates
    and numperm being between minperm and maxperm.
    Now unless paging space garbage collection is enabled, space in the
    paging space is reserved (and shows up in lsps as used) even if the
    page is brought to RAM again unless the process stops and releases
    memory. (as per perf management guide)
    Secondly, does having almost double the paging space (of the amount of
    RAM) really helps?
    Also what is the lru_file_repage parameter set to?

    -Ashok Sangra

  4. Re: lsps shows high values but system appears to be fine

    Hi there,
    To find out what is using such paging space us the command svmon -Pg .
    It will show you a list of processes where you can find a column
    called Pgsp that indicates you how many 4k pages the process is using.
    I usually use the command on the following way:
    svmon -Pg |grep -p Pid|more

    Regards,

    Jackson

  5. Re: lsps shows high values but system appears to be fine


    > Secondly, does having almost double the paging space (of the amount of
    > RAM) really helps?
    > Also what is the lru_file_repage parameter set to?
    >
    > -Ashok Sangra-


    Hi Ashok,

    Thanks for the advice. I will review what you mentioned. As for the
    paging space sizes, our policy is that if the system has less that 4GB
    of memory, we provide double that size for paging space. Once you get
    to 4GB or more, we normally reduce this calculation to be equal to
    actual memory as the extra paging space is not of much benefit. These
    are calculations based on IBM recommendations and the redbooks we have
    reviewed for AIX 4 and AIX 5.

    The system in question has 8GB of memory, so I would normally set it
    to 8GB of paging space. However, one of our operators was nervous
    about the paging space going to 100% full and so increased the size to
    see if it would help.

    Steve



  6. Re: lsps shows high values but system appears to be fine

    On Apr 24, 8:25 am, Jack wrote:
    > Hi there,
    > To find out what is using such paging space us the command svmon -Pg .
    > It will show you a list of processes where you can find a column
    > called Pgsp that indicates you how many 4k pages the process is using.
    > I usually use the command on the following way:
    > svmon -Pg |grep -p Pid|more
    >
    > Regards,
    >
    > Jackson


    Hi Jackson,

    Thanks for the command. I tried it out and determined that the
    processes using up the most 4k pages were the Informix "oninit" pids.
    This is what I suspected, but at least now I can see which PIDs in
    particular are using the most, so I can try to fine tune a bit more
    and check for incorrect onconfig settings (incorrect as in different
    than our other servers that don't have this problem.)

    Here is what was displayed as the highest usage:

    -------------------------------------------------------------------------------
    Pid Command Inuse Pin Pgsp Virtual 64-bit
    Mthrd LPage
    975072 oninit 21579 5460 1359218 1369143 Y
    N N
    909510 oninit 21550 5460 1359205 1369105 Y
    N N
    925902 oninit 21472 5460 1359092 1369013 Y
    N N
    852150 oninit 21468 5460 1359088 1369015 Y
    N N
    856240 oninit 21455 5460 1359087 1369008 Y
    N N
    938044 oninit 21455 5460 1359085 1369005 Y
    N N
    700416 oninit 21455 5460 1359082 1369005 Y
    N N
    966882 oninit 21462 5460 1359075 1368996 Y
    N N
    1847378 oninit 134123 5460 496749 506238 Y
    N N
    901270 oninit 134074 5460 496694 506154 Y
    N N
    1728628 oninit 133980 5460 496582 506062 Y
    N N
    520402 oninit 133994 5460 496579 506066 Y
    N N
    1532078 oninit 133977 5460 496579 506058 Y
    N N
    860278 oninit 133977 5460 496578 506058 Y
    N N
    598160 oninit 133995 5460 496578 506065 Y
    N N
    548974 oninit 133977 5460 496578 506058 Y
    N N
    1327106 oninit 133995 5460 496578 506065 Y
    N N
    1822948 oninit 133978 5460 496576 506057 Y
    N N
    1544276 oninit 133978 5460 496575 506054 Y
    N N
    864346 oninit 133978 5460 496573 506054 Y
    N N
    676034 oninit 133978 5460 496572 506055 Y
    N N
    1237182 oninit 134010 5460 496532 506042 Y
    N N
    -------------------------------------------------------------------------------

    Thx.

    SteveN



  7. Re: lsps shows high values but system appears to be fine

    Upon further review of the issue, we found that the issue with the
    paging is because of what we believe is a bug in Informix 9.4 FC7. We
    have only three systems still running that version, and the symptom we
    see there is that the memory is not released after completing certain
    procedures in the Informix database.

    The paging space issue seems to be more noticeable on the one server,
    as that one does a lot of information warehouse data analysis, which
    use the "questionable" routines that were causing the problem.

    Temporary solution: Run "onmode -F" on the instance to free up memory
    segments that were reserved but not released properly.

    Permanent solution: Upgrade to Informix 10 or higher. We tested the
    same scenario on that platform and it worked fine. It does not chew up
    memory and start swapping it after some time.

    So it ends up it is not really an AIX issue, but hope this may help
    someone who is running Informix on AIX, or the AIX admin can advise
    their Infromix DBA that it is clearly a RDBMS issue causing the
    problem.

    SteveN

+ Reply to Thread