It seems to be impossible now to prevent a modal dialog from closing using the beforeClose function of the dialog.
This used to work to prevent the dialog from closing:
But since we have upgraded to apex 21.2 the modal dialog is almost always closed but the overlay on the calling page remains making the page unusable until a manual refresh.
Also tested on 22.2 and same problem occurs.