'Strange' paging behaviour - Aix

This is a discussion on 'Strange' paging behaviour - Aix ; Hello, I have a 570 LPAR configured with 2 CPUs, 64 bit kernel, 6GB of memory and 12GB of paging space, the (hopefully) relevant vmo output is shown below as is a short (but representative sample) of vmstat maxclient% = ...

+ Reply to Thread
Results 1 to 9 of 9

Thread: 'Strange' paging behaviour

  1. 'Strange' paging behaviour

    Hello,

    I have a 570 LPAR configured with 2 CPUs, 64 bit kernel, 6GB of memory
    and 12GB of paging space, the (hopefully) relevant vmo output is shown
    below as is a short (but representative sample) of vmstat

    maxclient% = 90
    maxfree = 1088
    maxperm = 1355915
    maxperm% = 90
    maxpin = 1269952
    maxpin% = 80
    minfree = 960
    minperm = 75328
    minperm% = 5

    vmstat 10

    System Configuration: lcpu=4 mem=6144MB

    kthr memory page faults cpu
    ----- ----------- ------------------------ ------------ -----------
    r b avm fre re pi po fr sr cy in sy cs us sy id wa
    5 1 1153747 17887 0 0 0 0 0 0 175 25885 3079 11 3 82
    4
    3 0 1155320 16268 0 0 149 0 0 0 192 75037 3135 55 10 35
    0
    4 1 1155607 15932 0 0 99 0 0 0 220 17316 3256 34 7 58
    1
    7 0 1155186 16370 0 0 157 0 0 0 189 4837 3039 25 3 72 0
    1 0 1156832 14639 0 0 0 0 0 0 105 23365 2965 27 7 66
    0
    4 0 1161039 10455 0 0 0 0 0 0 94 16985 2873 23 4 72
    0
    6 1 1162782 8646 0 0 232 0 0 0 352 83262 3472 42 11 46
    0
    10 1 1159752 11521 0 0 308 0 0 0 478 109845 3404 56 14 22
    8
    5 1 1158186 12857 0 0 157 0 0 0 357 26385 3261 21 6 68
    5
    4 0 1158173 12806 0 0 0 0 0 0 210 21841 3136 23 5 66
    6
    7 0 1156737 14237 0 0 149 0 0 0 219 31612 3154 28 5 66
    0
    3 0 1160070 10902 0 0 0 0 0 0 101 6989 2991 23 3 73 0
    6 0 1159279 11641 0 0 158 0 0 0 275 36641 3240 44 8 48
    0
    4 0 1158251 12728 0 0 113 0 0 0 170 58507 3028 32 6 61
    1
    1 1 1158085 12848 0 0 0 0 0 0 194 26118 3050 20 7 70
    3
    8 0 1156532 14349 0 0 154 0 0 0 186 22164 3094 20 6 73
    0
    2 0 1157827 13048 0 0 0 0 0 0 94 4320 2932 19 2 79 0
    2 0 1158108 12719 0 0 0 0 0 0 64 12954 2850 20 4 75
    0
    2 0 1155210 15606 0 0 287 0 0 0 214 42373 2953 23 6 70
    0
    2 0 1155863 14951 0 0 155 0 0 0 180 15686 3323 47 6 47
    0

    The vmstat output shows that the system is paging out but, as far as I
    can see isn't doing anything else of note paging wise.

    I'd appreciate it if someone could give me a clue as to why the system
    is behaving in this particular way, either pointers to explanations or
    explanations would be really helpful. At this moment all I'm trying to
    do is understand what is going on.

    Let me know if more information would help in my quest.

    Thanks in advance

    ken

  2. Re: 'Strange' paging behaviour

    vmo -a | egrep "lru|max"

    You probably want to set lru_file_repage=0 ..

    Rgds
    Mark Taylor


  3. Re: 'Strange' paging behaviour

    On Jul 3, 4:06*pm, Mark Taylor wrote:
    > vmo -a | egrep "lru|max"
    >
    > You probably want to set lru_file_repage=0 ..
    >
    > Rgds
    > Mark Taylor


    Thanks for the reply, I've attached the output of the vmo command
    below, the lru_file_repage is already and has been for a wee while,
    set to 0

    lru_file_repage = 0
    lru_poll_interval = 10
    lrubucket = 131072
    maxclient% = 90
    maxfree = 1088
    maxperm = 1355915
    maxperm% = 90
    maxpin = 1269952
    maxpin% = 80
    npsrpgmax = 196608
    npsscrubmax = 196608
    strict_maxclient = 1
    strict_maxperm = 0


    Any other pointers gratefully received

    Regards

    Ken

  4. Re: 'Strange' paging behaviour

    On 07/04/2008 11:46 AM, summeken@googlemail.com wrote:
    > On Jul 3, 4:06 pm, Mark Taylor wrote:
    >> vmo -a | egrep "lru|max"
    >>
    >> You probably want to set lru_file_repage=0 ..
    >>
    >> Rgds
    >> Mark Taylor

    >
    > Thanks for the reply, I've attached the output of the vmo command
    > below, the lru_file_repage is already and has been for a wee while,
    > set to 0


    What applications are running on the server?

    lru_file_repage=0 makes sense for a dedicated RDBMS server.

    You should also capture / post output of the following when you are
    seeing the paging problem:

    vmstat -v|grep perm
    svmon -G
    sar -q (if available)

    cheers
    Niel

  5. Re: 'Strange' paging behaviour

    On Jul 4, 1:15*pm, Niel Lambrechts wrote:
    > On 07/04/2008 11:46 AM, summe...@googlemail.com wrote:
    >
    > > On Jul 3, 4:06 pm, Mark Taylor wrote:
    > >> vmo -a | egrep "lru|max"

    >
    > >> You probably want to set lru_file_repage=0 ..

    >
    > >> Rgds
    > >> Mark Taylor

    >
    > > Thanks for the reply, I've attached the output of the vmo command
    > > below, the lru_file_repage is already and has been for a wee while,
    > > set to 0

    >
    > What applications are running on the server?
    >
    > lru_file_repage=0 makes sense for a dedicated RDBMS server.
    >
    > You should also capture / post output of the following when you are
    > seeing the paging problem:
    >
    > vmstat -v|grep perm
    > svmon -G
    > sar -q (if available)
    >
    > cheers
    > Niel


    Again, thanks for replying,

    The LPAR is hosting Websphere Application Server

    output as requested

    vmstat -v|grep perm
    5.0 minperm percentage
    90.0 maxperm percentage
    27.7 numperm percentage

    svmon -G

    size inuse free pin virtual
    memory 1572864 1558268 14596 232662 1114842
    pg space 3145728 3838

    work pers clnt
    pin 232662 0 0
    in use 1114842 429835 13591

    sar -q

    AIX rs6pap01_grn 3 5 00C8407B4C00 07/07/08

    System configuration: lcpu=4

    00:00:00 runq-sz %runocc swpq-sz %swpocc
    00:15:00 2.0 12 1.0 4
    00:30:00 2.4 12 1.0 3
    00:45:00 2.1 9 1.0 3
    01:00:00 2.3 9 1.0 4
    01:15:00 2.0 29 1.1 8
    01:30:00 2.2 12 1.0 3
    01:45:00 2.5 11 1.1 4
    02:00:00 2.1 9 1.0 3
    02:15:00 2.2 11 1.0 4
    02:30:00 2.7 7 1.0 3
    02:45:00 2.1 9 1.1 3
    03:00:00 1.7 28 1.2 11
    03:15:00 2.3 10 1.0 4
    03:30:00 1.9 10 1.0 3
    03:45:01 2.1 10 1.0 3
    04:00:01 1.8 9 1.1 3
    04:15:01 2.1 12 1.1 3
    04:30:01 2.5 11 1.0 3
    04:45:01 1.9 9 1.0 3
    05:00:00 2.5 11 1.0 3
    05:15:00 2.2 11 1.0 3
    05:30:00 2.2 10 1.1 3
    05:45:01 1.9 12 1.0 4
    06:00:00 2.1 11 1.0 3
    06:15:00 1.8 11 1.0 3
    06:30:00 3.0 9 1.1 3
    06:45:00 1.9 10 1.0 4
    07:00:00 2.3 9 1.0 4
    07:15:00 2.4 21 1.0 31
    07:30:00 2.2 22 1.6 20
    07:45:00 2.4 29 1.1 16
    08:00:00 2.8 19 1.1 4
    08:15:00 2.8 20 1.2 4
    08:30:00 3.1 21 1.0 4
    08:45:00 3.5 32 1.0 4
    09:00:00 3.2 38 1.0 4
    09:15:01 3.5 44 1.1 37
    09:30:01 3.3 44 1.5 19

    Average 2.5 16 1.1 7


    Anything else I can supply?

    As indicated earlier, all I really want to do is understand what is
    causing this (to me) strange paging activity, I can then go to our
    developer types and and capacity team to let them know. We might even
    justify the addition of more memory.

    Thanks in advance.

    Ken

  6. Re: 'Strange' paging behaviour

    On Jul 7, 10:45 am, summe...@googlemail.com wrote:
    > On Jul 4, 1:15 pm, Niel Lambrechts wrote:
    >
    >
    >
    > > On 07/04/2008 11:46 AM, summe...@googlemail.com wrote:

    >
    > > > On Jul 3, 4:06 pm, Mark Taylor wrote:
    > > >> vmo -a | egrep "lru|max"

    >
    > > >> You probably want to set lru_file_repage=0 ..

    >
    > > >> Rgds
    > > >> Mark Taylor

    >
    > > > Thanks for the reply, I've attached the output of the vmo command
    > > > below, the lru_file_repage is already and has been for a wee while,
    > > > set to 0

    >
    > > What applications are running on the server?

    >
    > > lru_file_repage=0 makes sense for a dedicated RDBMS server.

    >
    > > You should also capture / post output of the following when you are
    > > seeing the paging problem:

    >
    > > vmstat -v|grep perm
    > > svmon -G
    > > sar -q (if available)

    >
    > > cheers
    > > Niel

    >
    > Again, thanks for replying,
    >
    > The LPAR is hosting Websphere Application Server
    >
    > output as requested
    >
    > vmstat -v|grep perm
    > 5.0 minperm percentage
    > 90.0 maxperm percentage
    > 27.7 numperm percentage
    >
    > svmon -G
    >
    > size inuse free pin virtual
    > memory 1572864 1558268 14596 232662 1114842
    > pg space 3145728 3838
    >
    > work pers clnt
    > pin 232662 0 0
    > in use 1114842 429835 13591
    >
    > sar -q
    >
    > AIX rs6pap01_grn 3 5 00C8407B4C00 07/07/08
    >
    > System configuration: lcpu=4
    >
    > 00:00:00 runq-sz %runocc swpq-sz %swpocc
    > 00:15:00 2.0 12 1.0 4
    > 00:30:00 2.4 12 1.0 3
    > 00:45:00 2.1 9 1.0 3
    > 01:00:00 2.3 9 1.0 4
    > 01:15:00 2.0 29 1.1 8
    > 01:30:00 2.2 12 1.0 3
    > 01:45:00 2.5 11 1.1 4
    > 02:00:00 2.1 9 1.0 3
    > 02:15:00 2.2 11 1.0 4
    > 02:30:00 2.7 7 1.0 3
    > 02:45:00 2.1 9 1.1 3
    > 03:00:00 1.7 28 1.2 11
    > 03:15:00 2.3 10 1.0 4
    > 03:30:00 1.9 10 1.0 3
    > 03:45:01 2.1 10 1.0 3
    > 04:00:01 1.8 9 1.1 3
    > 04:15:01 2.1 12 1.1 3
    > 04:30:01 2.5 11 1.0 3
    > 04:45:01 1.9 9 1.0 3
    > 05:00:00 2.5 11 1.0 3
    > 05:15:00 2.2 11 1.0 3
    > 05:30:00 2.2 10 1.1 3
    > 05:45:01 1.9 12 1.0 4
    > 06:00:00 2.1 11 1.0 3
    > 06:15:00 1.8 11 1.0 3
    > 06:30:00 3.0 9 1.1 3
    > 06:45:00 1.9 10 1.0 4
    > 07:00:00 2.3 9 1.0 4
    > 07:15:00 2.4 21 1.0 31
    > 07:30:00 2.2 22 1.6 20
    > 07:45:00 2.4 29 1.1 16
    > 08:00:00 2.8 19 1.1 4
    > 08:15:00 2.8 20 1.2 4
    > 08:30:00 3.1 21 1.0 4
    > 08:45:00 3.5 32 1.0 4
    > 09:00:00 3.2 38 1.0 4
    > 09:15:01 3.5 44 1.1 37
    > 09:30:01 3.3 44 1.5 19
    >
    > Average 2.5 16 1.1 7
    >
    > Anything else I can supply?
    >
    > As indicated earlier, all I really want to do is understand what is
    > causing this (to me) strange paging activity, I can then go to our
    > developer types and and capacity team to let them know. We might even
    > justify the addition of more memory.
    >
    > Thanks in advance.
    >
    > Ken



    I would say that this is rather normal. AIX is discarding filecache
    pages from memory to be used for application usage.
    And since it's filecache it's not beeing paged to pageingspace and
    therefore no pagein.

    //Jesper

  7. Re: 'Strange' paging behaviour

    On Jul 11, 12:28*pm, JesperBergh wrote:
    > On Jul 7, 10:45 am, summe...@googlemail.com wrote:
    >
    >
    >
    >
    >
    > > On Jul 4, 1:15 pm, Niel Lambrechts wrote:

    >
    > > > On 07/04/2008 11:46 AM, summe...@googlemail.com wrote:

    >
    > > > > On Jul 3, 4:06 pm, Mark Taylor wrote:
    > > > >> vmo -a | egrep "lru|max"

    >
    > > > >> You probably want to set lru_file_repage=0 ..

    >
    > > > >> Rgds
    > > > >> Mark Taylor

    >
    > > > > Thanks for the reply, I've attached the output of the vmo command
    > > > > below, the lru_file_repage is already and has been for a wee while,
    > > > > set to 0

    >
    > > > What applications are running on the server?

    >
    > > > lru_file_repage=0 makes sense for a dedicated RDBMS server.

    >
    > > > You should also capture / post output of the following when you are
    > > > seeing the paging problem:

    >
    > > > vmstat -v|grep perm
    > > > svmon -G
    > > > sar -q (if available)

    >
    > > > cheers
    > > > Niel

    >
    > > Again, thanks for replying,

    >
    > > The LPAR is hosting Websphere Application Server

    >
    > > output as requested

    >
    > > *vmstat -v|grep perm
    > > *5.0 minperm percentage
    > > *90.0 maxperm percentage
    > > *27.7 numperm percentage

    >
    > > svmon -G

    >
    > > * * * * * * * *size * * *inuse * * * free * * * *pin * *virtual
    > > memory * * *1572864 * *1558268 * * *14596 * * 232662 * *1114842
    > > pg space * *3145728 * * * 3838

    >
    > > * * * * * * * *work * * * pers * * * clnt
    > > pin * * * * *232662 * * * * *0 * * * * *0
    > > in use * * *1114842 * * 429835 * * *13591

    >
    > > sar -q

    >
    > > AIX rs6pap01_grn 3 5 00C8407B4C00 * *07/07/08

    >
    > > System configuration: lcpu=4

    >
    > > 00:00:00 runq-sz %runocc swpq-sz %swpocc
    > > 00:15:00 * * 2.0 * * *12 * * 1.0 * * * 4
    > > 00:30:00 * * 2.4 * * *12 * * 1.0 * * * 3
    > > 00:45:00 * * 2.1 * * * 9 * * 1.0 * * * 3
    > > 01:00:00 * * 2.3 * * * 9 * * 1.0 * * * 4
    > > 01:15:00 * * 2.0 * * *29 * * 1.1 * * * 8
    > > 01:30:00 * * 2.2 * * *12 * * 1.0 * * * 3
    > > 01:45:00 * * 2.5 * * *11 * * 1.1 * * * 4
    > > 02:00:00 * * 2.1 * * * 9 * * 1.0 * * * 3
    > > 02:15:00 * * 2.2 * * *11 * * 1.0 * * * 4
    > > 02:30:00 * * 2.7 * * * 7 * * 1.0 * * * 3
    > > 02:45:00 * * 2.1 * * * 9 * * 1.1 * * * 3
    > > 03:00:00 * * 1.7 * * *28 * * 1.2 * * *11
    > > 03:15:00 * * 2.3 * * *10 * * 1.0 * * * 4
    > > 03:30:00 * * 1.9 * * *10 * * 1.0 * * * 3
    > > 03:45:01 * * 2.1 * * *10 * * 1.0 * * * 3
    > > 04:00:01 * * 1.8 * * * 9 * * 1.1 * * * 3
    > > 04:15:01 * * 2.1 * * *12 * * 1.1 * * * 3
    > > 04:30:01 * * 2.5 * * *11 * * 1.0 * * * 3
    > > 04:45:01 * * 1.9 * * * 9 * * 1.0 * * * 3
    > > 05:00:00 * * 2.5 * * *11 * * 1.0 * * * 3
    > > 05:15:00 * * 2.2 * * *11 * * 1.0 * * * 3
    > > 05:30:00 * * 2.2 * * *10 * * 1.1 * * * 3
    > > 05:45:01 * * 1.9 * * *12 * * 1.0 * * * 4
    > > 06:00:00 * * 2.1 * * *11 * * 1.0 * * * 3
    > > 06:15:00 * * 1.8 * * *11 * * 1.0 * * * 3
    > > 06:30:00 * * 3.0 * * * 9 * * 1.1 * * * 3
    > > 06:45:00 * * 1.9 * * *10 * * 1.0 * * * 4
    > > 07:00:00 * * 2.3 * * * 9 * * 1.0 * * * 4
    > > 07:15:00 * * 2.4 * * *21 * * 1.0 * * *31
    > > 07:30:00 * * 2.2 * * *22 * * 1.6 * * *20
    > > 07:45:00 * * 2.4 * * *29 * * 1.1 * * *16
    > > 08:00:00 * * 2.8 * * *19 * * 1.1 * * * 4
    > > 08:15:00 * * 2.8 * * *20 * * 1.2 * * * 4
    > > 08:30:00 * * 3.1 * * *21 * * 1.0 * * * 4
    > > 08:45:00 * * 3.5 * * *32 * * 1.0 * * * 4
    > > 09:00:00 * * 3.2 * * *38 * * 1.0 * * * 4
    > > 09:15:01 * * 3.5 * * *44 * * 1.1 * * *37
    > > 09:30:01 * * 3.3 * * *44 * * 1.5 * * *19

    >
    > > Average * * *2.5 * * *16 * * 1.1 * * * 7

    >
    > > Anything else I can supply?

    >
    > > As indicated earlier, all I really want to do is understand what is
    > > causing this (to me) strange paging activity, I can then go to our
    > > developer types and and capacity team to let them know. We might even
    > > justify the addition of more memory.

    >
    > > Thanks in advance.

    >
    > > Ken

    >
    > I would say that this is rather normal. AIX is discarding filecache
    > pages from memory to be used for application usage.
    > And since it's filecache it's not beeing paged to pageingspace and
    > therefore no pagein.
    >
    > //Jesper- Hide quoted text -
    >
    > - Show quoted text -



    Thanks for replying (apologies for the delay in response, other things
    and continuing to monitor).

    In our case we see this paging activity (using nmon) as paging to
    paging space not to file system, and we have never seen in subsequent
    page in activity.
    I should point out that if there is any performance implications they
    are not being reported, this was only spotted as part of our normal
    monitoring and we can't explain why there is all this page out (to
    paging space according to nmon) activity without any other paging
    activity.

    Thanks for taking the time to reply.

    Ken

  8. Re: 'Strange' paging behaviour

    its not fs cache .. po is pages paging out to paging space ..

    Your avm is 4.5GB'ish, so you should be ok with lru_file_repage=0 ..
    someone hasnt set defps=0 have they ? or set PSALLOC ? are you running
    bespoke Java apps ? sometimes Java devlopers set some weird stuff that
    causes paging .. ah, just seen WAS in your post above .. check with
    the devlopers to see what they have been doing with paging space
    allocation for thier apps ..

    Rgds
    Mark Taylor


  9. Re: 'Strange' paging behaviour

    On Jul 16, 10:27*am, Mark Taylor wrote:
    > its not fs cache .. po is pages paging out to paging space ..
    >
    > Your avm is 4.5GB'ish, so you should be ok with lru_file_repage=0 ..
    > someone hasnt set defps=0 have they ? or set PSALLOC ? are you running
    > bespoke Java apps ? sometimes Java devlopers set some weird stuff that
    > causes paging .. ah, just seen WAS in your post above .. check with
    > the devlopers to see what they have been doing with paging space
    > allocation for thier apps ..
    >
    > Rgds
    > Mark Taylor


    Again, thanks for replying
    defps=1, PSALLOC is not set (as far as I can discover)
    We are coming to the same conclusion, the 'problem' is something to do
    with Java, this is a WAS box, with lots of bespoke code, we're already
    talking to our developer types to see if they'll admit to anything.

    Thanks for taking the time to reply.

    Ken

+ Reply to Thread