Oracle Database Discussions

My listener is blocked. And it does not seem to be for the usual reasons which I have seen on the Web.

It is blocked directly after startup, when there are no user connections. So it is not a PROCESSES issue.

LSNRCTL> services LISTENER_POPB
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=212.165.120.13)(PORT=1521)))
Services Summary...
Service "hhme" has 2 instance(s).
Instance "hhme1", status READY, has 1 handler(s) for this service...
Handler(s):
"DEDICATED" established:0 refused:0 state:ready
REMOTE SERVER
(ADDRESS=(PROTOCOL=TCP)(HOST=PoPA)(PORT=1521))
Instance "hhme2", status READY, has 1 handler(s) for this service...
Handler(s):
"DEDICATED" established:0 refused:0 state:blocked
REMOTE SERVER
(ADDRESS=(PROTOCOL=TCP)(HOST=PoPB)(PORT=1521))
Service "hhmeXDB" has 2 instance(s).
Instance "hhme1", status READY, has 1 handler(s) for this service...
Handler(s):
"D000" established:0 refused:0 current:0 max:972 state:ready
DISPATCHER <machine: PoPA, pid: 5260>
(ADDRESS=(PROTOCOL=tcp)(HOST=PoPA.CleanLANA.bgan.xantic.net)(PORT=32797))
Instance "hhme2", status READY, has 1 handler(s) for this service...
Handler(s):
"D000" established:0 refused:0 current:0 max:1022 state:ready
DISPATCHER <machine: PoPB, pid: 6543>
(ADDRESS=(PROTOCOL=tcp)(HOST=POPB)(PORT=32855))
The command completed successfully

For a user connection, in the listener log, I get

[oracle@PoPB log]$ tail -f listener_popb.log
23-AUG-2008 11:24:14 * (CONNECT_DATA=(SERVICE_NAME=HHME)(INSTANCE_NAME=HHME2)(CID=(PROGRAM=C:\Program Files\Quest Software\SQL Navigator 5.5\SQLNav5.exe)(HOST=JOHN_LAPTOP)(USER=John?Bourke))) * (ADDRESS=(PROTOCOL=tcp)(HOST=84.9.3.187)(PORT=2659)) * establish * HHME * 12516
TNS-12516: TNS:listener could not find available handler with matching protocol stack

Which corresponds to the following on the user client

ORA-12516: TNS:listener could not find available handler with matching protocol stack

I have tried cold and warm starts, with the same results.
The A node in the cluster does not have this problem.

On startup, there are no errors in the alert log or listener log.

TNSPINGs work or and crs_stat -t shows everything online, and VIP is in place in the ifconfig.

I have checked the listener log and service registrations and service updates are happening

23-AUG-2008 11:04:10 * service_register * hhme1 * 0
23-AUG-2008 11:04:49 * service_register * hhme2 * 0

23-AUG-2008 19:08:57 * service_update * hhme1 * 0
23-AUG-2008 19:09:05 * service_update * hhme2 * 0

I have checked the oracle documentation for configuration and debugging of the listener

Oracle® Database Net Services Administrator's Guide
10g Release 1 (10.1)

There is nothing in this document which I have configured incorrectly, and there is no test or debugging information which I have not tried.

In order to do a gap analysis, I have a working node to compare with, and also I have an identically configured cluster to compare.

The "blocked" server has been running for years without problems. I have searched all the configurations to see if anything was changed recently, but I cannot find any changes. I am also cross checking with the working server configs and cannot find any differences or changes.

The only things which have changed recently

1. We had a SCSI controller hitch, which corrupted some log files
2. We are dismantling a standby server configuration, so there may be some issues there

LSNRCTL services working node shows me

Instance "hhme1", status READY, has 2 handler(s) for this service...
Handler(s):
"DEDICATED" established:5655 refused:0 state:ready
LOCAL SERVER
"DEDICATED" established:0 refused:0 state:ready
REMOTE SERVER
(ADDRESS=(PROTOCOL=TCP)(HOST=PoPA)(PORT=1521))

LSNRCTL services blocked node shows me

Instance "hhme2", status READY, has 1 handler(s) for this service...
Handler(s):
"DEDICATED" established:0 refused:0 state:blocked
REMOTE SERVER
(ADDRESS=(PROTOCOL=TCP)(HOST=PoPB)(PORT=1521))

On the blocked node I am missing

"DEDICATED" established:5655 refused:0 state:ready
LOCAL SERVER

On the identical working cluster, both nodes have a LOCAL SERVER entry in the LSNRCTL services output.

So if registration/updates are ok, should I be chasing this difference ??


john