Java HotSpot Virtual Machine

I work on a legacy system that has a VB6 app that needs to call Java code. The solution we use is to have the VB app call a C++ dll that uses JNI to call the Java code. A bit funky, but it's actually worked pretty well. However, I'm moving to a new dev box, and I've just run into a serious problem with this. The compiled VB app works fine on the new box, but when I try to run it from the VB dev environment, the dll fails to load the VM, getting a return code of -4 (JNI_ENOMEM) from JNI_CreateJavaVM. This all works just fine on my old dev box (and on all the other machines we've used this on).

Both the built app and VB are calling the exact same dll, and I've tried it with both Java 1.5 and 1.6. I've tried the suggestions here (redirecting stdout and stderr to files, adding a vfprint option, adding an -Xcheck:jni option), but to no avail. I can't seem to get any additional information out of the jvm. As far as I can tell, the new box is configured pretty much the same as the old one (installed software, Path, Classpath, etc.), and both are running the same release of Windows Server 2003. The new machine is an x64 box with more memory (4GB rather than 2GB), but it's running 32-bit Windows.

Originally, we were just using the default options (just passing a class path). I tried passing in various values for -Xmx and -Xms. Up to 52m, the dll call just caused VB to quit (with no error messages). From 53m up, I get the JNI_ENOMEM return code. There's certainly plenty of memory available, and unless VB is doing something really odd with memory allocation for dlls, there should be enough contiguous memory available for a JVM with a 53MB heap. I've tried rebasing the dll, rebooting the machine with PAE off, and a few other random things that I can't remember right now.

Any suggestions or ideas about what else to look into, or at least how to get more information out of the jvm about why it's failing? Rewriting the whole thing in a more sane way is not an option -- I need to find a way to have the dll get the jvm to load without thinking that it's out of memory. Any help would be much appreciated.