Java Programming

I'm trying to write a Server Socket that listens for connections. But I only want it to accept connections from known IP addresses. If I'm using the code below:

try {
serverSocket = new ServerSocket(myPort);
} catch (IOException e) {
...
}
for (;;) { //loop forever
Socket clientSocket = null;
try {
clientSocket = serverSocket.accept();
....
}

Is there a way that I can influence the serverSocket.accept() method to make it first check the IP address BEFORE the connection is made? If not, then is it a security vulnerability to do the following code (knownClientAddress is the only IP address I want to accept connections from):

try {
serverSocket = new ServerSocket(myPort);
} catch (IOException e) {
...
}
for (;;) { //loop forever
Socket clientSocket = null;
try {
clientSocket = serverSocket.accept();
if (clientSocket.getRemoteSocketAddress() != knownClientAddress)
clientSocket.close();
....
}