Some of the output our app generates is dynamic; that is, we can't provide the content-length header ahead of time. We need to be able to stream this content, i.e. we want the client to start receiving it immediately.
This is working fine when we go to the app running under Glassfish directly:
When our API does not provide the content-length header, and if more than 1 buffer-worth of output (8K) is generated, Glassfish automatically goes into the "transfer-encoding: chunked" mode; and the client starts receiving the bytes immediately.
It is NOT working when the app is being accessed via Apache mode_proxy/mode_proxy_ajp. Namely, Glassfish, apparently tries to buffer the entire stream before it sends any bytes over to Apache; and if the size of the content is large enough it eventually runs out of heap space and gives 503 Service Unavailable.
Note that this is very similar to what you get when you disable chunking encoding on the Glassfish side; for example:
asadmin set server-config.network-config.protocols.protocol.http-listener-1.http.chunking-enabled="false"
- then you are seeing the same behavior when going to Glassfish directly. No streaming, trying to buffer everything before sending any content over.
This has been tested with Apache 2.2 and 2.4, with the same behavior observed.
We can't tell whether the reason for this is on the Glassfish or apache_mod_proxy side.
I.e., whether it is
a) something with the Glassfish implementation of AJP; or
b) something with how Apache mod_proxy_ajp communicates to Glassfish. As in, does it need to specifically communicate to Glassfish that it is willing to accept transfer-encoding=chunked? (I have tried explicitly adding the header "TE: chunked" to every proxied request; but it didn't do it).
We do know for the fact, however, that the issue is specific to AJP. Because streaming IS working when we're going through Apache, but proxying to Glassfish at http://localhost:... on a local port.
I keep thinking there must be an easy solution - like a flag "streaming=yes" or "use-chunking=true" that just needs to be set somewhere.
For reproducing the issue, we have created a miniature standalone app. You can get the sources here:
https://github.com/IQSS/apachetest
and the war file here:
https://github.com/IQSS/apachetest/releases/download/v0.1/apachetest-0.1...
The entire app is basically one REST API call:
api/download/streamBytes/{Number of Bytes}
It will try to send the requested number of bytes from /dev/urandom, using javax.ws.rs.core.StreamingOutput (the code fragment is below).
Testing it on a command line, with curl, going directly to the Glassfish port:
# curl -v -k http://localhost:8080/apachetest-0.1/api/download/streamBytes/1000000000 > /dev/null
- curl starts receiving content right away:
...
< X-Powered-By: Servlet/3.1 JSP/2.3 (GlassFish Server Open Source Edition 4.1 Java/Oracle Corporation/1.7)
< Content-Type: application/octet-stream
< Date: Mon, 22 Jun 2015 20:07:04 GMT
< Transfer-Encoding: chunked
<
{ [data not shown]
100 9912k 0 9912k 0 0 7293k 0 --:--:-- 0:00:01 --:--:-- 7325k^C
Going through Apache: When requesting 100MB, I get the bytes, but with a noticeable delay. When requesting 1GB, I don't get anything, and I see the size of Glassfish memory heap increasing rapidly, until it maxes out.
The REST API method:
@Path("streamBytes/{numBytes}")
@GET
@Produces({"application/octet-stream"})
public Response streamBytes(@PathParam("numBytes") Long numBytes) {
final long totalBytes = numBytes;
StreamingOutput stream = new StreamingOutput() {
@Override
public void write(OutputStream os) throws IOException,
WebApplicationException {
try (FileInputStream instream = new FileInputStream("/dev/urandom")) {
byte[] data = new byte[8192];
int i = 0;
long total = 0;
while (((i = instream.read(data))) > 0 && (total < totalBytes)) {
long numBytesOut = total + i < totalBytes ? i : totalBytes - total;
os.write(data, 0, (int) numBytesOut);
total += numBytesOut;
os.flush();
}
}
os.flush();
os.close();
}
};
return Response.ok(stream).build();
}
---
Thank you very much,
Any help/input is appreciated.
-Leonid Andreev
Dataverse Project