Java HotSpot Virtual Machine

Let me start off by giving some background.

We began experiencing this issue within the past month
in our production environment, but cannot reproduce
it in an isolated testing environment.

I have performed extensive tests in regards to Heap Sizes
and Thread Stack sizes in hopes of resolving this issue, but
none have worked.

Hardware/OS Settings
----------------------------

OS for platform is Linux
Distribution is RHAS 2.1
Kernel version is 2.4.9-e.30enterprise SMP
Box has 2 CPUs which are hyperthreaded

ULIMIT settings

core file size (blocks) 0
data seg size (kbytes) unlimited
file size (blocks) unlimited
max locked memory (kbytes) unlimited
max memory size (kbytes) unlimited
open files 1024
pipe size (512 bytes) 8
stack size (kbytes) 8192
cpu time (seconds) unlimited
max user processes 8191
virtual memory (kbytes) unlimited

/etc/sysctl.conf , contains (effectively overrides the ulimt setting)

fs.file-max = 65535

/proc/sys/kernel/threads-max

16383

glibc version is 2.2.4

Physical Mem : ~ 2 Gig
Swap : ~ 4 Gig

JVM Settings
----------------
java version "1.4.2_03"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.2_03-b02)
Java HotSpot(TM) Client VM (build 1.4.2_03-b02, mixed mode)

-server -Djava.awt.headless=true -XX:+DisableExplicitGC -Xms1228m -Xmx1228m -XX:NewSize=256m -XX:MaxNewSize=256m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+UseCMSCompactAtFullCollection -verbosegc -verbosegc -XX:+PrintGCDetails -Xloggc:/opt/logs/gc_log.txt


Tests
-------

The way i understand it, the memory available for thread creation is:

Physical Memory - "-Xmx" = ~ 800 MB (in our case)

Also, the default stack size on linux is 512k, so a base line (with no other
phyical memory usage than the JVM) is:

800 * 1000 / 512 = ~ 1562 threads

however, i when the error recurrs, i can do a :

ps -ef | grep java | wc -l (each java thread is represented by a process on linux)

and that # is way below 1562 (approx 90 or so)

additionally, via java code at the time of the exception you can get
the Mother Thread group:

        private static ThreadGroup getMotherThreadGroup()
	{        
		ThreadGroup mother = Thread.currentThread().getThreadGroup();
		while(mother.getParent() != null)
			mother = mother.getParent();
		return mother;
	}

and then get the # of threads in the group:

        Thread      currentThread = Thread.currentThread();
        ThreadGroup mother        = getMotherThreadGroup();        
	Thread[]    threads       = new Thread[mother.activeCount() + 5];		
        
        int threadCount;
        
        for (int count;;) 
        {
            count = mother.enumerate(threads,true);
            if (count < threads.length)
            {
                threadCount = count;
                break;
            }
            
            threads = new Thread[threads.length << 1];
        }  

once again this # is way below 1562 (matches rougly to the 'ps' output)

Additionally, I have modified the JVM parameters to indepdently reduce
-Xms by 100 m and reduce the stack size to -Xss256k and neither of
these resolve the issue.

Questions
-------------

What other OS level settings could be affecting the # of threads that can be created?
What JVM settings could be affecting the # of threads that can be created?
Are there ever Java threads present that will no be represented by the
code in this post, or a 'ps' process listing?