Java APIs

Greets,

I'm developping a huge application and got a latency/block problem using the write(ByteBuffer) method on a Socket from a socketchannel connection.

Running java 1.5 (diablo) on Freebsd 6 servers, 4Gb ram (2.2 allocated to jvm), with dual xeon dual-core (total 4 cores)

Here is the application schema :

- A thread accepting connexion on the socketchannel
- A thread selecting keys with data to process, enqueuing it after some basic checks on a command FIFO
- A thread getting commands from the FIFO and processing 'em, generating answers on 4 answer FIFOs
- 4 threads (1 per FIFO) to get answers and send 'em back to the socket.

The application usually runs with 4500-5000 simultaneous clients.

My problem is that the only write() method sometimes takes over 20ms to write a message, with a length smaller than 50 bytes.

As I got about 25000 answers to process each second, when some of 'em decide to be slow, the 4 threads runs slowly, and all the connected clients are suffering of that latency, for the few minutes needed to empty the FIFOs.
Every client socket get about 5 answers per second.

On about 1 hour running, there are about 3 'peaks' of slowness, that I cannot explain yet. That's why I'm in need of advices !

I monitored the application when such case happens. TOP indicates 40% cpu idle, JVM memory got >500Mb free, network runs @ about 1.2Mbps, where maximal transfer rate is >20Mbps. netstat -m told me no erros, and a large amount of free buffers available.

As the only slow process is the write() method that usually runs faster than 1ms each call, but in those case I got delays over 20ms.

freebsd tcp default sendbuffer size is 64k, receive buffer is 32k
Commands average received size is below 1k, Answers average sending size below 8k.

This application is running live, and as I cannot emulate 5000+ connections with a similar beahviour to test withour being sure that won't crash all.

What points could be responsible of such slow write() calls ? Seems it's not CPU, not RAM, not network itself...
I suppose it's the network buffers that are causing problems. But I don't really know if I have to fit 'em to a lower size, fitting my requirements, or to a larger size, to be sure there won't be full buffers blocking all ?

I need advices. Thanks for your ideas !

Bill