Java SE (Java Platform, Standard Edition)

Dear all,

Problem

what? 3rd party software communicates using the sendMessage function User32.dll with flag HWND_BROADCAST.
when? as soon as they are used in combination with a JavaFX application the 3rd party applications crash.

Our own investigation

We fear that JavaFX fails(?) to respond (correctly?) to this WinApi call and hence the sendMessage broadcast is failing.
Here is a stacktrace of such a 3rd party application (made with OllyDbg):

CPU Stack
Address   Value      ASCII Comments
0018F8CC  /0000FFFF  ÿÿ    ; |hWnd = 0000FFFF
0018F8D0  |0000C3AA  ªÃ    ; |Msg = 0C3AA
0018F8D4  |000003E8  è    ; |wParam = 3E8
0018F8D8  |0000C098  ˜À    ; \lParam = 0C098
0018F8DC  \004859B0  °YH   ; RETURN from thirdparty.004859D8 to thirdparty.004859B0
0018F8E0  /0018F900   ù   ; Pointer to next SEH record
0018F8E4  |004859C6  ÆYH   ; SE handler
0018F8E8  |0018F8F8  øø
0018F8EC  |023BA998  ˜©;
0018F8F0  |068BAC44  D¬‹  ; ASCII "C:\otherthirdparty"
0018F8F4  |0000C098  ˜À
0018F8F8  |0018F918  ù
0018F8FC  \007461BA  ºat   ; RETURN from thirdparty.00485964 to thirdparty.007461BA

As you can see the hWnd parameter has value 0000FFFF or -1 or HWND_BROADCAST.

CPU Disasm
Address   Hex dump          Command                                  Comments
004859D8  /$  52            PUSH EDX                                 ; /lParam => ARG.EDX
004859D9  |.  0FB7C0        MOVZX EAX,AX                             ; |
004859DC  |.  50            PUSH EAX                                 ; |wParam
004859DD  |.  0FB705 B06F7A MOVZX EAX,WORD PTR DS:[7A6FB0]           ; |
004859E4  |.  50            PUSH EAX                                 ; |Msg
004859E5  |.  68 FFFF0000   PUSH 0FFFF                               ; |hWnd = 0000FFFF
004859EA  |.  E8 4531F8FF   CALL <JMP.&user32.SendMessageA>          ; \USER32.SendMessageA

This is typically bad practise from ages ago, when interprocess communication was safe, back in the 16bit days.

The third party applications at our clients appear to only crash systematically when running JavaFX applications next to them.

JNA + Swing Code which should fix it

public class JnaWinEvents extends JFrame {

	private static final long serialVersionUID = -3985408320174213410L;
	public LONG_PTR prevWndProc; // so it won't get GC'ed
	public WndProc wndProcCallbackListener = null; // so it won't get GC'ed

	public JnaWinEvents() {
		this.add(new JLabel("Hello StackExchange!"));
		this.pack();
		this.setVisible(true);
		setupEventsListener();
	}

	public interface MyUser32 extends User32 {

		public static final MyUser32 MYINSTANCE = (MyUser32) Native.loadLibrary("user32", MyUser32.class, W32APIOptions.UNICODE_OPTIONS);

		/**
		 * Changes an attribute of the specified window
		 * 
		 * @param hWnd
		 *            A handle to the window
		 * @param nIndex
		 *            The zero-based offset to the value to be set.
		 * @param callback
		 *            The callback function for the value to be set.
		 */
		public LONG_PTR SetWindowLong(HWND hWnd, int nIndex, WndProc callback);
	}

	public class WndProc implements StdCallCallback {
		public LRESULT callback(HWND hWnd, int uMsg, WPARAM wParam, LPARAM lParam) {
			System.out.println("callback " + hWnd + "\t" + uMsg + "\t" + wParam.toString() + "\t" + lParam.toString());
			return new LRESULT(0l);
		}
	}

	private void setupEventsListener() {
		HWND hWnd = new HWND();
		hWnd.setPointer(Native.getWindowPointer(this));

		this.wndProcCallbackListener = new WndProc();

		// Set the WndProc function to use our callback listener instead of the
		// window's one.
		LONG_PTR result = MyUser32.MYINSTANCE.SetWindowLong(hWnd, com.sun.jna.platform.win32.WinUser.GWL_WNDPROC, wndProcCallbackListener);
		System.out.println("setting my window proc, result = " + result);
	}

	public static void main(String[] args) {
		SwingUtilities.invokeLater(new Runnable() {
			@Override
			public void run() {
				new JnaWinEvents();
			}
		});
	}
}

The code above should override the WndProc method of the view and hence fix the issue, obviously when using JavaFX instead of Swing.
The code above is able to print the sendMessage requests - please not that only HWND_BROADCAST will make the application fail.

IMPORTANT Swing does NOT crash like JavaFX - do NOT expect the code above to crash.

Question?

Is there any way to get the pointer from the JavaFX Window as we are doing now with the JFrame ?

References
http://msdn.microsoft.com/en-us/library/windows/desktop/ms644950(v=vs.85).aspx
http://blogs.msdn.com/b/oldnewthing/archive/2006/06/13/629451.aspx
http://msdn.microsoft.com/en-us/library/windows/desktop/dd744765(v=vs.85).aspx

Tools

We have simulated the behaviour by coding a simple application in C# which calls the sendMessage with HWND_BROADCAST

When needed we can provide the source code - I did not find a way to attach files to this forum.