Samba performance tuning - Fedora Core 4 and Windows XP

Here is my setup:

Linux server:
- Supermicro X6DA8-G2 mainboard
- Dual Intel Xeon 3.6 GHz, 800 MHz FSB
- 8GB DDR2 PC3200 ECC memory
- 2 x 300GB 10.000RPM SCSI U320 disks (Software RAID-1 via mdadm tool)
- Gigabit LAN on mainboard
- Fedora Core 4, 2.6.11-1.1369_FC4smp kernel
- SAMBA server version 3.0.14a-2
- LAN MTU set to 1500 (reported by ifconfig eth0)

Windows XP client:
- Asus P4C800-E mainboard
- Intel P4 3.4GHz CPU
- 2GB DDR1 PC3200 memory
- 2 x WD Raptor 74GB 10.000RPM SATA disks (RAID-0 via ICH5R chip on
mainboard)
- Gigabit LAN on mainboard
- Windows XP SP2
- LAN MTU set to 1500 (via DRTCP utility)

smb.conf file:
# Samba config file created using SWAT
# from 192.168.123.100 (192.168.123.100)
# Date: 2005/12/28 13:12:24
# Global parameters
[global]
workgroup = TEST1
server string = Samba Server
username map = /etc/samba/smbusers
log file = /var/log/samba/%m.log
max log size = 50
max xmit = 65535
socket options = IPTOS_LOWDELAY TCP_NODELAY SO_RCVBUF=262140
SO_SNDBUF=262140
printcap name = /etc/printcap
dns proxy = No
ldap ssl = no
idmap uid = 16777216-33554431
idmap gid = 16777216-33554431
guest ok = Yes
cups options = raw
[public]
comment = public directory: R/O by all
path = /data/d0/disk0/public


When I FTP an about 5.2 GB test.tar.gz file from the Linux server to
the Windows XP client, ftp reports about 44MB/sec transfer speed.

How can I tune further the transfer speed?

My network switches don't support jumbo frames, so that is not an
option at the moment. Also I haven't changed any networking parameters
on the Windows XP side yet, except set MTU to 1500 with DRTCP utility.
What tuning would help on the client side?

Finally is there a way to FTP from the Linux server, but writing to
/dev/null on the Windows XP side in order to eliminate the overhead of
the disk-subsystem on the client side, sort of a dd over the LAN?

dd on the Linux server reports that the disk-subsystem is capable of
about 90MB/sec, so there is room to improve the LAN transfer rate (I
hope).

Any ideas welcome.

TIA
Jeff

[email]jeff000069@yahoo.com[/email] wrote:
....[color=blue]
> When I FTP an about 5.2 GB test.tar.gz file from the Linux server to
> the Windows XP client, ftp reports about 44MB/sec transfer speed.[/color]

FTP? Samba is not involved when you ftp.

What does top say on your linux box when you transfer
this file? What is especially interesting is the Cpu line
at the top, how much time is spent in "sy"? If this is high,
you might need a better NIC.

You could try and test your raw network speed with ttcp (you'll
have to ask google where you can find this for Windows).


....[color=blue]
> Finally is there a way to FTP from the Linux server, but writing to
> /dev/null on the Windows XP side in order to eliminate the overhead of
> the disk-subsystem on the client side, sort of a dd over the LAN?[/color]

If you actually mean ftp, you can do a:

get test.tar.gz nul:

in the ftp program on the Windows client.

Mogens

--
Mogens Kjær, Dataarkæolog
Email: [email]mk@datamuseum.dk[/email]
Homepage: [url]http://www.datamuseum.dk[/url]

Hej Mogens,

Thanks for replying.

I did as you suggested on the Windows XP: "get test.tar.gz nul:"
This gives a transfer rate of 90-95MB/sec, which sounds too good to be
true.
Is such a rate even possible on GigE without jumbo frames? (I will have
to google for maximum possible throughput of GigE).

While the above get was running on the client, top on the server shows
0.5%. I suppose my NIC is good enough?

I will also try with TTCP (once I find a version for XP).

Jeff.

[email]jeff000069@yahoo.com[/email] wrote:[color=blue]
> Hej Mogens,
>
> Thanks for replying.
>
> I did as you suggested on the Windows XP: "get test.tar.gz nul:"
> This gives a transfer rate of 90-95MB/sec, which sounds too good to be
> true.
> Is such a rate even possible on GigE without jumbo frames? (I will have
> to google for maximum possible throughput of GigE).[/color]

I get the same rate, also without jumbo frames.

ttcp is a bit higher, around 100Mbytes/sec.
[color=blue]
>
> While the above get was running on the client, top on the server shows
> 0.5%. I suppose my NIC is good enough?[/color]

I doubt that any NIC can feed a gigabit network
with a load of just 0.5 %. What exactly does "top" show
during the transfer?

Mogens

--
Mogens Kjær, Dataarkæolog
Email: [email]mk@datamuseum.dk[/email]
Homepage: [url]http://www.datamuseum.dk[/url]

I looked at output of top again on the server (while a xcopy on Windows
XP was copying the test.tar.gz file from the server). Typical values
are:
0.3%us 2.5%sy 0.0%ni 96.0%id 0.2%wa 0.0%hi 0.7%si

The numbers are of course the average of both Xeon processors (with
hyperthreading enabled). The linux server is not running anything else
at the moment, besides standard OS processes.

The 0.5% mentioned in my earlier post was the "sy" value while running
a FTP get on the Windows XP. I should probably rerun this just to be
sure, since it does sound a bit low.

Incidently I forget to mention the results of the CIFS throughput in my
initial post, here they are:
copy U:\test.tar.gz to C:\ gives 42MB/sec[color=blue]
>From GUI copy and paste to C:\ gives 35MB/sec[/color]

Keep in mind that FTP gave about 44MB/sec.
Using the Windows XP GUI (copy & paste) gave surprisingly low result of
35MB/sec.

Is there a way to cache files on the Windows XP side, to avoid reading
files over-the-wire all the time?

I will be storing all my digital photos on the Linux SAMBA server and
viewing these from Windows XP via applications such as Photoshop or
BreezeBrowser. Since I have 2GB memory on my XP client, caching the
read-only photos would be usefull.

Thanks again for all your help.

Jeff.

[email]jeff000069@yahoo.com[/email] wrote:[color=blue]
> I looked at output of top again on the server (while a xcopy on Windows
> XP was copying the test.tar.gz file from the server). Typical values
> are:
> 0.3%us 2.5%sy 0.0%ni 96.0%id 0.2%wa 0.0%hi 0.7%si
>
> The numbers are of course the average of both Xeon processors (with
> hyperthreading enabled). The linux server is not running anything else
> at the moment, besides standard OS processes.[/color]

On my server (2xXeon 2.4 GHz with HT) and an Intel server NIC,
I get 100% system CPU usage (i.e. one of the four CPU's is totally
busy) when I do a ttcp. What kind of NIC do you have on your system?

....[color=blue]
> Is there a way to cache files on the Windows XP side, to avoid reading
> files over-the-wire all the time?[/color]

Isn't this what oplocks are used for in samba?

- I always turn these off...

Mogens

--
Mogens Kjær, Dataarkæolog
Email: [email]mk@datamuseum.dk[/email]
Homepage: [url]http://www.datamuseum.dk[/url]

I redid the test to be sure about the CPU load.

While xcopy command on Windows XP was copying the test.tar.gz file from
the Samba server, I looked at the graphical output of
/usr/bin/gnome-system-monitor. This shows CPU4 at more or less 15%
constant load during the entire duration of the data transfer. The
other 3 CPUs are at virtually zero.

Clearly the top output is wrong, it only shows 1 line for all the CPUs.
I was wrongly assumming that this was the average of all 4 CPUs, but
seems more likely to be CPU1 only. Assumption is the mother of all
screw-ups, as the saying goes....I need surely to get a better version
of top than the included version in FC4.

My NIC is the onboard port of my Supermicro X6DA8-G2 mainboard, this is
based on the Intel 82546GB dual port Gigabit controller chip. It
appears that the FC4 drivers for this NIC are performing well.

Would moving to jumbo frames be usefull in your opinion? (I would have
to buy a new switch, of course).

Thanks for the pointer to oplocks in SAMBA, I will surely research
further. I am interested in caching since my SAMBA server is mainly
serving read-only content (mainly digital photos).

Other ideas surely welcome!

Jeff

[email]jeff000069@yahoo.com[/email] wrote:
....[color=blue]
> Clearly the top output is wrong, it only shows 1 line for all the CPUs.
> I was wrongly assumming that this was the average of all 4 CPUs, but
> seems more likely to be CPU1 only. Assumption is the mother of all
> screw-ups, as the saying goes....I need surely to get a better version
> of top than the included version in FC4.[/color]

You can press "1" while top is running, and you'll get a line
for each cpu.
[color=blue]
>
> My NIC is the onboard port of my Supermicro X6DA8-G2 mainboard, this is
> based on the Intel 82546GB dual port Gigabit controller chip. It
> appears that the FC4 drivers for this NIC are performing well.[/color]

I have a 82545GM controller. This is on a redhat 9 machine, with
a (more or less) vanilla 2.4.32 kernel.
[color=blue]
>
> Would moving to jumbo frames be usefull in your opinion? (I would have
> to buy a new switch, of course).[/color]

I have no experience with jumbo frames. None of my switches support them.

Mogens

--
Mogens Kjaer, Carlsberg A/S, Computer Department
Gamle Carlsberg Vej 10, DK-2500 Valby, Denmark
Phone: +45 33 27 53 25, Fax: +45 33 27 47 08
Email: [email]mk@crc.dk[/email] Homepage: [url]http://www.crc.dk[/url]

I try to summarise my results/obervations so far, after admittedly some
quick & dirty testing:

TEST1: From Client1 (Windows XP, Asus P4C800-ED, Intel P4 3.4GHz, 2GB
RAM, RAID-0 stripe on 2 x 74GB raptors):
run1: ftp U:\test.tar.gz C:\ gives 45-55MB/sec & CPU-util=85%
run2: ftp U:\test.tar.gz nul: gives 90-100MB/sec & CPU-util=100%
run3: copy U:\test.tar.gz C:\ gives 40-45MB/sec & CPU-util=55%
run4: copy U:\test.tar.gz NUL gives 55-60MB/sec & CPU-util=35%

TEST2: From Client2 (Windows XP, Asus A8N-SLI Premium, AMD Athlon 3500,
1GB RAM, single 74GB raptor):
run1: ftp U:\test.tar.gz C:\ gives 50-55MB/sec & CPU-util=55%
run2: ftp U:\test.tar.gz nul: gives 65-70MB/sec & CPU-util=90%
run3: copy U:\test.tar.gz C:\ gives 50-55/sec & CPU-util=60%
run4: copy U:\test.tar.gz NUL gives 75-80/sec & CPU-util=60%

Client1 has an integrated Intel Pro/1000 CT NIC and Client2 has an
integrated Nvidia nForce4-SLI NIC. Client1 network utilisation improved
significantly after upgrading to latest network drivers from Intel
website.

Each run was executed 4 times, since I noticed significant better
transfer rates on the 4th run; appears that file-system caching of the
5.2GB test file really kicked-in on the server side after a few runs.

On client1, going from ftp to copy (i.e. using the CIFS layer) incurred
quite a penalty. This was not the case on client2; I do not know why?

On client2, copy is faster than ftp, when destination is NUL. A bit
strange?

It is clear that approaching gigE wire speed requires close to 100%
CPU-utilisation, on the above 2 clients. (Pity I do not have a dual-cpu
or dual-core client I could run the test with). A thanks to Mogens who
pointed this out.

On the server only about 15% of CPU4 is required, according to
gnome-system-monitor. I wonder how accurate this is, but anyway there
is sufficient cpu power on the server so it doesn't really matter.
Running dd if=test.tar.gz of=/dev/zero bs=4K on the server gave
100MB/sec and even 200MB/sec once file-system caching kicked-in.
Increasing bs to 128K even gave 1.2GB/sec!!! Clearly the server is
powerfull enough to provide gigE wire speeds to clients without
choking.

The Intel NIC appears to have the advantage relative to the Nvidia NIC.
It was possible to reach 100MB/sec on client1, but never on client2
when executing run2. I am not sure if this is hardware or software
driver related, perhaps a combination of both?

Since gigE wire speed is being reached (on some tests), jumbo frames
probably won't help except for reducing cpu utilisation.

Tentative Conclusion:
1) My next client PC should have at least have a dual-core cpu (2 x
dual-cores would be even better for Photoshop, does Windows XP support
4 CPUs?).
2) Intel NICs are the preffered choice, but I should probably run more
tests to be really sure on this point.
3) The SCSI disk sub-system on the server is superior to the SATA disk
sub-systems on the clients. But this is probably due to the faster SCSI
disks, which have about half the seek time of the SATA raptor disks.

Hope above is of help to someone.

Jeff