tomiskra@vodatel.net wrote:
> Hi,
>
> I run some application on p570 ( 8 cpu, 16 GB RAM, one virtual disk on
> EMC CX500 storage
> connected via fibre channel ),and lately I have some perf problems.
> Load is low ( 3-4 ), and
> users have slow app ( telnet connection, and some terminal app.). here
> is a vmstat output.
>
> As you can see there are a lot of cpu wa and page fr and sr.
>
> Can I try to solve high wa with creating more than one virtual disk (
> in iostat that virtual disk is on 100% tm_act during high wa in vmstat
> ). Or should I search for most intense used files on that disk (
> filemon or someting else ) and separate those on internal disks?
>
> For page fr+sr, should I play with vmo parameters?
>
> This is aix 5.2 ML5.
>
> Regards
>
>
> 4 1 564594 371619 0 0 0 1622 3111 0 1870 142697 2357 38 6 44
> 11
> 3 1 564782 371507 0 0 0 1067 2035 0 1869 74414 1859 26 4 53
> 17
> 2 2 564758 373346 0 0 0 478 871 0 1678 72387 1444 26 4 48
> 22
> kthr memory page faults cpu
> ----- ----------- ------------------------ ------------ -----------
> r b avm fre re pi po fr sr cy in sy cs us sy id wa
> 2 3 564885 374567 0 0 0 728 1340 0 1981 57838 2392 20 4 40
> 35
> 2 3 564932 372040 0 0 0 0 0 0 2362 62634 2816 17 3 41
> 39
> 2 3 564954 371258 0 0 0 849 1681 0 2500 74852 3113 26 4 44
> 26
> 1 3 565326 370890 0 0 0 1217 2305 0 2288 57467 2987 17 3 47
> 32
> 1 4 565484 370728 0 0 0 1006 1907 0 2173 58679 2801 17 3 40
> 40
> 2 3 565360 370852 0 0 0 1104 2178 0 2114 53251 2623 15 3 40
> 41
> 1 3 565143 371071 0 0 0 1292 2357 0 2290 136778 3100 22 5 44
> 29
> 2 2 564978 371237 0 0 0 1199 2410 0 2371 73151 3171 18 4 51
> 27
> 1 3 564527 371685 0 0 0 1104 2250 0 2230 61044 2924 17 4 45
> 34
> 2 3 564530 371683 0 0 0 1356 2603 0 2454 93692 3328 18 5 37
> 41
> 2 2 564403 371810 0 0 0 1170 2168 0 2256 86606 3006 18 3 48
> 31
> 2 2 564609 371616 0 0 0 1011 2018 0 2492 195912 3756 23 11 41
> 25
> 2 2 564632 371585 0 0 0 1235 2534 0 2238 172757 2983 26 5 42
> 26
> 2 3 564530 371688 0 0 0 1290 2641 0 2105 105394 2694 22 5 40
> 34
> 2 3 564323 371894 0 0 0 1216 2457 0 2115 134377 2804 24 6 46
> 24
> 2 2 564641 371571 0 0 0 1036 2107 0 2066 143970 2571 22 6 50
> 22
> 2 2 564201 372017 0 0 0 1279 2799 0 2173 131716 2808 23 5 56
> 16
> 2 2 563687 372531 0 0 0 2096 3965 0 2151 32638 3110 28 3 48
> 21
> 2 2 563747 372879 0 0 0 1963 4036 0 2330 67790 3021 29 4 43
> 24
> 4 2 563571 373377 0 0 0 2806 5474 0 2354 113217 3230 34 6 37
> 23
> kthr memory page faults cpu
> ----- ----------- ------------------------ ------------ -----------
> r b avm fre re pi po fr sr cy in sy cs us sy id wa
> 2 2 563838 374988 0 0 0 153 307 0 2244 166267 2818 27 7 43
> 24
> 2 2 563949 372267 0 0 0 46 78 0 2270 145547 2849 23 5 48
> 23
> 2 2 564105 372114 0 0 0 2239 4403 0 2334 31453 3435 25 3 48
> 24
> 2 2 564028 374036 0 0 0 2625 5188 0 2416 95678 3411 29 5 43
> 23
> 3 1 563971 373478 0 0 0 2391 5110 0 2313 49128 3206 28 4 46
> 22
> 2 2 563292 376399 0 0 0 347 750 0 2116 85576 2412 23 5 50
> 22
> 2 2 563607 373725 0 0 0 0 0 0 2101 165341 2495 35 6 37
> 22
> 3 2 563485 372728 0 0 0 608 1271 0 2133 109498 2661 26 5 37
> 33
> 3 2 563921 372295 0 0 0 2041 4196 0 2062 133236 2933 38 6 34
> 21
> 2 2 564514 372594 0 0 0 2041 4378 0 2189 49298 2881 26 4 50
> 20
> 2 2 564629 372778 0 0 0 1980 3796 0 1707 145610 1798 30 7 51
> 13
> 2 2 564629 374675 0 0 0 75 148 0 1745 97392 1727 25 4 48
> 23
> 3 2 564643 372252 0 0 0 0 0 0 1821 138273 1913 34 4 37
> 25
> 2 3 564016 372201 0 0 0 941 1871 0 1910 107911 2577 27 6 39
> 28
> 4 2 563804 372412 0 0 0 1515 3065 0 2082 216388 2808 38 6 39
> 17
> 2 2 563913 372553 0 0 0 2849 5928 0 2204 68525 3184 25 4 51
> 19
> 2 2 563624 378446 0 0 0 554 1110 0 2118 103663 2314 23 6 50
> 21
> 4 1 563819 376519 0 0 0 0 0 0 1884 171237 2105 31 7 45
> 17
> 3 2 564773 373759 0 0 0 0 0 0 1898 174753 2173 33 6 37
> 24
> 3 2 564931 371986 0 0 0 0 0 0 1825 202174 1985 35 9 36
> 21
> kthr memory page faults cpu
> ----- ----------- ------------------------ ------------ -----------
> r b avm fre re pi po fr sr cy in sy cs us sy id wa
> 4 2 565843 370369 0 0 0 417 905 0 1797 251796 1920 38 8 37
> 17
> 2 3 565293 370924 0 0 0 1783 3861 0 2301 90474 3209 26 5 40
> 28
> 4 3 564923 371290 0 0 0 1991 4364 0 3006 81357 5007 35 10 23
> 33
> 3 4 563910 372312 0 0 0 2756 5716 0 2980 84322 4731 36 7 25
> 33
> 2 3 564630 373467 0 0 0 3698 7740 0 2937 60806 4364 29 5 24
> 41
> 3 3 564546 371666 0 0 0 558 1229 0 2812 122095 3775 33 5 21
> 40
> 3 3 564695 371524 0 0 0 2020 4383 0 2720 116506 3891 30 5 28
> 36
> 2 4 564697 371521 0 0 0 1907 4018 0 2468 123484 3506 26 4 32
> 37
> 2 3 564695 371524 0 0 0 2126 4314 0 2853 54902 4282 20 4 38
> 39
> 2 4 564708 371514 0 0 0 2157 5547 0 2729 36779 4066 18 3 38
> 41
> 1 3 564886 371331 0 0 0 1807 5647 0 2426 39444 3349 18 3 42
> 37
> 3 2 564756 371457 0 0 0 864 2247 0 1967 143322 2273 30 5 40
> 25
> 4 2 565043 371174 0 0 0 1011 2765 0 1692 212173 2088 36 6 36
> 23
> 3 1 565812 371433 0 0 0 1404 4078 0 1824 143599 2268 32 7 42
> 19
> 3 1 565935 370280 0 0 0 322 915 0 1937 112335 2254 27 4 49
> 20
> 2 2 566226 369986 0 0 0 689 2243 0 1740 104347 1916 26 5 51
> 19
> 2 2 565134 371083 0 0 0 772 2262 0 1789 146524 2157 31 6 42
> 21
> 3 2 565837 370377 0 0 0 531 1396 0 1524 116124 1513 31 5 39
> 25
> 5 1 565644 370576 0 0 0 609 1924 0 1576 186281 1576 34 6 37
> 23
> 3 2 566259 369957 0 0 0 1123 3198 0 1741 150147 2005 31 8 39
> 23
> kthr memory page faults cpu
> ----- ----------- ------------------------ ------------ -----------
> r b avm fre re pi po fr sr cy in sy cs us sy id wa
> 2 3 566218 369998 0 0 0 594 1655 0 1604 105701 1697 25 5 43
> 27
> 3 2 566269 369947 0 0 0 489 1504 0 1489 231652 1310 40 6 40
> 13
> 3 2 566435 369783 0 0 0 469 1280 0 1503 192042 1384 37 6 46
> 11
> 3 2 566599 369619 0 0 0 477 1629 0 1466 175614 1329 37 5 44
> 14
> 2 2 566420 369793 0 0 0 555 1838 0 1524 131176 1467 31 4 49
> 16
> 4 1 566786 369432 0 0 0 492 1423 0 1464 233682 1272 44 6 34
> 17
> 3 2 566568 369646 0 0 0 555 1962 0 1547 115006 1490 30 4 36
> 30
> 3 2 566405 369814 0 0 0 716 2127 0 1520 157555 1465 35 6 37
> 22
> 4 2 566884 370350 0 0 0 1684 6167 0 1622 47034 1741 26 4 37
> 33
> 3 2 565250 374890 0 0 0 998 3559 0 1430 118767 1277 37 7 42
> 14
> 2 1 565057 374568 0 0 0 0 0 0 1280 122006 1069 24 5 60
> 10
> 2 1 565216 373664 0 0 0 0 0 0 1421 57196 1120 21 2 57
> 20
> 3 1 565271 373026 0 0 0 0 0 0 1338 121278 1011 29 4 56
> 12
> 2 1 565270 372558 0 0 0 0 0 0 1936 38655 2465 25 5 57
> 14
> 2 1 565270 372115 0 0 0 0 0 0 1393 33789 1067 16 2 70
> 12
> 1 1 565616 371288 0 0 0 0 0 0 1320 74157 971 22 3 63 12
> 3 0 565625 371077 0 0 0 0 0 0 1097 101771 539 33 3 60 4



Might want to check your network card settings.
A mismatch with your lan switch can cause what you are seeing
1000Base-T should be auto-negotiate.

Outside of this, please make sure your single PV lun
presented to AIX on your CX500 is not pinned to a single
physical disk on your array.

Check the EMC compatability matrix for your equipment
(array,server,adapters.. etc) and make sure all required
paramaters are on your hba's.

My CX600's and CX700's with ODM kit 5.2.0.3 or higher
required these changes to hba:

num_cmd_elems=2048

and for fibre channel switched fabrics:

fc_error_recov=fast_fail

Yours may not, check the compat matrix.


>From an earlier post on nic's in this newsgroup:



When both sides of the link are set to autoneg, they will "negotiate"
the duplex setting and select full-duplex if both sides can do
full-duplex.
If one side is hardcoded and not using autoneg, the autoneg process
will "fail" and the side trying to autoneg is required by spec to use
half-duplex mode.
If one side is using half-duplex, and the other is using full-duplex,
sorrow and woe is the usual result.
So, the following table shows what will happen given various settings
on each side:
Auto Half Full

Auto Happiness Lucky Sorrow
Half Lucky Happiness Sorrow
Full Sorrow Sorrow Happiness


Happiness means that there is a good shot of everything going well.
Lucky means that things will likely go well, but not because you did
anything correctly Sorrow means that there _will_ be a duplex
mis-match.
When there is a duplex mismatch, on the side running half-duplex you
will see various errors and probably a number of _LATE_ collisions
("normal" collisions don't count here). On the side running
full-duplex you will see things like FCS errors. Note that those
errors are not necessarily conclusive, they are simply indicators.
Further, it is important to keep in mind that a "clean" ping (or the
like - eg "linkloop" or default netperf TCP_RR) test result is
inconclusive here - a duplex mismatch causes lost traffic _only_ when
both sides of the link try to speak at the same time. A typical ping
test, being synchronous, one at a time request/response, never tries
to have both sides talking at the same time.
Finally, when/if you migrate to 1000Base-T, everything has to be set
to auto-neg anyway.