We recently applied Tomcat security patches in production that had already been applied in dev and test to linux ORDS servers. Suddenly application that currently works fine in dev, test and was working in Prod has started to throw 403 errors during authentication process.
Error seems to be occurring during callback to application authentication process.
Apex version 5.1.3 same on dev, test, prod
FACTS: What works; what didn’t
DEV/TEST urls all work OK with the same patches applied to Tomcat.
Other Prod Apex applications work as before the upgrade if not using remote_header SAML type authentication. Using Authenticated Okta Tomcat ORDS service to login into the Apex workspace and run SQL workshop works and running other Apex functionality;
Only executing an APEX Application ids seems to throw a 403 . Workspace logins do not. Application ids are executed by URL suffix invoking an APEX application: (/f?p=103)
Checked all privileges related to Apex and all seem to be matching dev, test.
We have not been able to resolve this with Oracle support. Below is some analysis our Tomcat engineer has done:
TCPdump showed the last response from APEX Pl/sql gateway was a HTTP-302 redirect location, not a 403. The way this works is that ORDS on the Tomcat server receives a response from APEX database on port 1521, then unpacks the Pl/sql gateway response (a 302 from PL/sql gateway) then ORDS sends this response to the Browser as a HTTP response.
When the browser tried to honor the 302 redirect location given by ORDS-Tomcat , this resulted in Tomcat throwing a 403 Forbidden.
The location given is an authenticated URL under the Tomcat Shib listener and the logs indicated Apache-Shib had indeed filled in REMOTE_USER for this pass to Tomcat AJP. The NetId shows up on the logs to indicate authentication working as configured.
This is where the weirdness starts. Logs show dev/test honoring this response (200 ok) while PROD tomcat throws a 403 forbidden, but only for a few SELECT URLs that happen to be authenticated callback URLs resulting from APEX database unpacked by ORDS response
Any help with this issue would be greatly appreciated.