We have a Java based application which connects to few servers over https.
The self signed certificates of these servers are added to client trust store to ensure https connections work.
Here is the scenario in question :
1. Server has a self signed certificate C1 with Public Key Pub1. This certificate is added to client trust store and connection works fine.
2. A new self signed certificate is generated on Server say C2 which has same public key as C1. i.e C2 has a different Serial Number, thumbprint, validaty dates but has the same public key as certificate C1.
3. Though C2 is NOT added to client trust store, the connection between client and server is working.
So it appears that X509TrustManager/X509ExtendedTrustManager checkServerTrusted implementation is only doing a public key match.
Same scenario tested with browser :
1. When server's self signed certificate C1 is NOT added to browser certificate store, Security exception is raised by browser. In case of Firefox(add server exception) and for Chrome(add the self signed server cert to trust store).
2. Browser does not throw any security exception now.
3. As explained before, when server changes to certificate C2 which has same public key as C1 (which is added to browser), browser still raises security exception.
So effectively there is a difference in Java v/s Browser trust behavior.