Java HotSpot Virtual Machine

We are getting Java Heap memory error for the application we are running on linux 64 bit machine (VM).
The OOM came when heap usage was 1.7gb though we have specified min as 1gb and max as 4gb. If I understand correctly then it should not have been thrown as we have specified max as 4gb. If address space was the problem then it should have thrown swap space error.
Also, there were no other processes running on this node.

Below are the specifics of linux node we are using:

linux kernel: 2.6.18-128.el5
Linux Version: Red Hat Enterprise Linux Server release 5.3 (Tikanga) 64 Bit

Ulimts

[ppoker@aquariusvir11 ~]$ ulimit -a
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 139264
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 100000
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 139264
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

Java Version
[ppoker@aquariusvir11 ~]$ java -version
java version "1.6.0_21"
Java(TM) SE Runtime Environment (build 1.6.0_21-b06)
Java HotSpot(TM) 64-Bit Server VM (build 17.0-b16, mixed mode)


Kernel Semaophores
[ppoker@aquariusvir11 ~]$ ipcs -l

------ Shared Memory Limits --------
max number of segments = 4096
max seg size (kbytes) = 67108864
max total shared memory (kbytes) = 17179869184
min seg size (bytes) = 1

------ Semaphore Limits --------
max number of arrays = 128
max semaphores per array = 250
max semaphores system wide = 32000
max ops per semop call = 32
semaphore max value = 32767

------ Messages: Limits --------
max queues system wide = 16
max size of message (bytes) = 65536
default max size of queue (bytes) = 65536


Please suggest what could be the reason for this error.

Thanks,
Ashish