ZFS/NFS speed problems, seeking tuning advice
807557Aug 16 2006 — edited Aug 16 2006Hi,
I've set up a Solaris 10 x86 server to act as a home directory server for a group of build/testing machines that we use to test our software product on multiple platforms. I'm seeing extremely poor write performance from clients mounting ZFS filesystems over NFS and am looking for guidance on how best to overcome it.
The server is a modest dual Xeon with 1GB of RAM. It is running the 6/06 release. The storage is a D1000 attached in a split-bus configuration via 2 differential SCSI channels. The ZFS pool contains two raidz vdevs with 5 disks each (36 GB 10K rpm) with each vdev containing disks from each channel. There are a total of 17 ZFS filesystems in the pool, each using compression and shared via NFS. The server is connected via gigabit ethernet, though most of the build machines only have 100 Mbit connections.
I've tested local write performance at around 30MB/s (based on zpool iostat, copying a large file from /tmp). That's less than I might expect, but I know I have at least that much to work with.
My first test was from an x2100 over GbE. The x2100 only has a single ATA spindle, but was otherwise idle, so I would expect plenty of throughput on the read side. When rsyncing /export/home to the NFS mount, the throughput was pretty lousy at 600 KB/s. For comparison, I thought I'd test a second box to make sure it wasn't an anomaly. The second client was a FreeBSD 4.x host with a single ATA spindle and 100 Mbit connectivity. The rsync throughput was again lousy, comparable to the x2100.
I suspect the problem has to do with writing many small files versus a few large ones. Single large file copies are faster by a factor of at least 10. The x2100 was also running Solaris 10, and so was using NFS v4. The FreeBSD client was NFS v3. In both cases the clients mounted with default NFS options.
For this NFS server to be usable for developer home directories, it has to have much better write performance. I'll be the first to admit that I have only a little experience with ZFS or NFS, so any advice on either ZFS setup or NFS options would be appreciated.