Java APIs

I've been experiencing some very weird behaviour regarding UDP packets. I understand the nature of the problem but I am confused as to what could be causing it. I know that there are a lot of intelligent people on this board so I'm hoping someone might have an idea.

To start with my server receives duplicate packets A LOT. Basically my server is transmitting live voice audio but in order to penetrate picky NAT networks I have each of the clients send a UDP handshake (a single packet) to the server in order. If they don't receive an acknowledgement from the server in a certain amount of time they resend the packet. The server compares the packet with any that it had previously received and ensures that it is from a new location (that it is not a duplicate). Ok so one would think that there would be no problems right?

Well I discovered a real oddity upon shutting down the server, closing all of the ports and closing all of the clients, then restarting it 1 hour later with only 1 client connecting. When the server is waiting for a handshake from the 1 and only client it gets an extra packet leftover from the previous session that took place over 1 hour ago. It is from a client that had been closed down and had all of its sockets closed and hadn't sent a packet to this port in over 1 hour (this much I am sure of). The server doesn't know that it is a duplicate packet because it only has listings of previously received packet addresses from the time that it has been started up, and this is a packet that had somehow been leftover from a session from over 1 hour ago. Why would a leftover UDP packet be lingering around the network and get received so long after it was sent?

I appreciate any input you can give, even if they're just half formed thoughts. I'm really stumped on this one.