APEX

I was recently a bit confused as to what sort of response is expected back from an AJAX Callback and how how to handle errors, so today I went diving into apex.server to figure out what was expected. I have this mostly sorted in my head and thought I'd share.
Background
An AJAX Callback is a bit of PL/SQL code that runs in the database that is invoked by an Apex page via AJAX. This means there is a round-trip to the database server and a response sent back to the page. This is mostly likely done via a Dynamic Action of type Execute JavaScript Code that calls apex.server.process().
The apex.server.process function is a wrapper around jQuery.ajax() with some Apex-specific functionality added on. Not to rehash the documentation, but there are two syntaxes for handling the response. They are mostly equivalent, except for error handling (see below).
The older, "options syntax", where callbacks are registered in the pOptions parameter:

apex.server.process( "MY_CALLBACK", { ... pData ... }, {
    ... other pOptions ...
    success: function( responseData ){
        // Handle a successful response
    },
    error: function( jqXHR, textStatus, errorThrown ){
        // Handle an error condition
    }
});

The other, "promise syntax", where done and fail callbacks are passed to the Promise object returned by process():

apex.server.process( "MY_CALLBACK", { ... pData ... }, {
    ... other pOptions ...
}).done( function( responseData ){
    // Handle a successful response
}).fail( function( jqXHR, textStatus, errorThrown ){
    // Handle an error condition
});

The pOptions parameter has a dataType member that defaults to "json", but can be a couple of other values, like "text".
Response from the AJAX Callback
The Success Message and Error Message options in the AJAX Callback configuration appear to be completely ignored.
Since the default dataType is "json", you will mostly likely want to write a PL/SQL block that responds with JSON:

begin
    htp.prn('{"foo": "bar"}');
end;

In this case, the responseData passed to your success or done method will be the equivalent JavaScript object.
You can also set dataType to "text" and respond with just a string value:

begin
    htp.prn('This is my response');
end;

In this case, responseData is that string.
Unhandled exceptions, where we start getting into undocumented behavior
If your AJAX Callback hits an unhandled exception, like in this example:

begin
    raise_application_error( -20001, 'Intentional error' );
end;

The response from the server is sqlerr:ORA-20001: Intentional error. If the expected response dataType is json, the default, your page will wind up logging something in the console like, "SyntaxError: JSON.parse: unexpected character at line 1 column 1 of the JSON data" because that is not valid JSON. This is what started me down this whole road.
If dataType is set to text, that error string is passed as a string to your success callback. The error/fail callback is not invoked!
It looks like an HTTP header of APEX-ERROR will also be used as the error text, but I couldn't find any way to get Apex to respond with that.
Handled exceptions
So I was a good little programmer and added an exception block to my AJAX Callback:

exception when others then
    htp.prn('{"error": "' || SQLERRM || '"}');
end;

Here's where things get different depending on what syntax you used to invoke apex.server.process. If the JSON response contains an "error" attribute:
If you used the "options" syntax, the error callback is invoked, text status is "APEX", and errorThrown is the contents of the error attribute.
If you used the "promise" syntax, Apex automatically invokes the apex.message.showErrors() function with the contents of the error attribute. That is, the little error window at the top of the page pops up, and only then is the fail callback invoked.
There is some mention in the code when doing this automatic error handling about it expecting a "page submission id" parameter as well, to avoid errors when actually submitting the page, but I couldn't create any test case where that's relevant. That seems to matter if the request was made with the "fullpage" submit option, but that isn't documented in pOptions anywhere. This may be some Apex internal feature or relevant to some other part of the apex.server API.
Conclusions
I don't use these kinds of callbacks too often, which is why I had to look into this to figure out what exactly was going on, but I think the pattern I'm going to follow from now on is:
Always have a WHEN OTHERS exception handler and catch any exceptions. (Should probably be doing this anyway.)
Avoid using "error" in my JSON response so as to not invoke any automatic error handling. Instead use something like "errorMessage". Leave the automatic handling for server or browser errors and such.
That means the success/done callback must properly handle errors passed to it from the PL/SQL program, whatever that means to your application and page.