I have spent the last few hours running a test and am not sure whether
I am looking at the right problem, hence I am looking for some help.

I have 2 identical DL585 servers running SLES 9 with EMC attached. As
my users are complaining that the one of the systems is slower, hence
I decided to run some dd tests. Basically I ran the following
time dd if=/dev/zero of=filesystems bs=1024k count=8000

Machine A (the slower machine) gave me the following
8000+0 records in
8000+0 records out

real 1m41.624s
user 0m0.004s
sys 0m0.306s

Machine B gave the following

8000+0 records in
8000+0 records out

real 0m59.653s
user 0m0.011s
sys 0m1.119s

Obviously, Machine A is slower, but what I do not understand is the
difference in sys time, as nothing was running on both systems, how
can I account for the difference in sys time? Why was Machine A spent
less time in the kernel mode (as defined in sys time) as both
machines were running the same job? Can I conclude that the disk is
slower in Machine A?

While running the test I also gathered the iostat. I noticed that the
faster machine (Machine B) showed 1000 is avgrq-sz while Machine A
showed 660. Why the difference? I assume that I am writing 1k block
size, I would assume the request size should be similar. Is there
anything I may have missed?

Machine A

Time: 09:00:15 PM
avg-cpu: %user %nice %sys %iowait %idle
1.20 0.02 0.22 0.09 98.48

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s
avgrq-sz avgqu-sz await svctm %util
cciss/c0d0 0.10 134.72 0.07 11.73 0.00 0.00 0.00
0.00 0.00 0.26 22.28 0.35 0.41
cciss/c0d1 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00
sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.02 0.00 0.27 0.00
0.13 16.00 0.00 3.50 3.50 0.01
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00
sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00
sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00
sdf 0.00 196.07 0.00 86.21 0.00 57398.31 0.00
28699.15 665.81 0.77 8.91 3.89 33.52
sdg 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00
sdh 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00
sdi 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00
sdj 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00
sdk 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00
sdl 0.00 193.10 0.00 85.11 0.00 56658.67 0.00
28329.33 665.72 0.76 8.88 3.88 33.01
fd0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00

Machine B

Time: 09:12:47 PM
avg-cpu: %user %nice %sys %iowait %idle
1.26 0.01 0.43 0.10 98.19

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s
avgrq-sz avgqu-sz await svctm %util
cciss/c0d0 0.00 158.82 0.00 12.34 0.00 1311.86 0.00
655.93 106.30 0.34 27.73 0.39 0.48
cciss/c0d1 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00
sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.01 0.00 0.13 0.00
0.07 16.00 0.00 1.00 1.00 0.00
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00
sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00
sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00
sdf 0.00 0.00 0.00 71.70 0.00 72293.97 0.00
36146.98 1008.25 0.50 6.97 4.62 33.12
sdg 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00
sdh 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00
sdi 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00
sdj 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00
sdk 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00
sdl 0.00 0.00 0.00 74.14 0.00 72278.25 0.00
36139.13 974.95 0.50 6.70 4.45 32.96
fd0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00


Can someone point me to the right direction?

thanks
BB