I have shut down one of the managed servers in my three-member OSB 10.3.1 cluster to load test the reduced configuration, and that has broken the "service health" tab on the OSB console dashboard.
The problem is that with the mgd1 server shut down (the one the Singleton marker ear is deployed to, if that matters), in the OSB console's Dashboard everything in the "Service Health" tab is broken. I get the warning message "Unable to obtain metrics data from the server" at the top of the page whenever I go to the dashboard. I can't even look at the displays for the individual (mgd2, mgd3) servers that are running. I get the following error in the log when trying to access the service health for the services in the cluster, or even if I try to get the health of services in either of the two remaining cluster members individually:
+####<Apr 8, 2010 11:00:02 PM EDT> <Error> <AlertLog> <nj09mhm0033.edmz.mcgraw-hill.com> <SS_ESB_Admin> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1270782002573> <BEA-000000> <WLDFAccessRuntimeMBean is not available for server SS_ESB_mgd1>+
Also, as part of the load test procedure, I need to collect performance stats from the proxy service pipeline and the business service. To do this, I've written JMX code called out of soapUI scripts to reset the stats on these services before the run, then collect the stats after the run ends; the code connects to the admin server. This too is broken when the mgd1 server is shut down.
I get the error below when trying to collect the cluster stats from the Admin server from the JMX code. This error is fatal, and no statistics come back from the rest of the cluster either. (you can ignore all the groovy stuff in the stack trace, it's the nested Exceptions that are more interesting).
Does anyone have an idea about what's going on, and how I fix this? I would have thought the DomainService Runtime MBean would be able to catch and handle an Exception like this if one of the managed servers were down, and collect the stats from the available part of the cluster, but it looks like that's not the case.
TIA for any help.
/Steve Elkind
java.lang.reflect.UndeclaredThrowableException
at $Proxy6.getBusinessServiceStatistics(Unknown Source)
at com.mgh.im.esb.jmxutils.ServiceStatsRetriever.getAndResetStatsForMonitoredBizService(ServiceStatsRetriever.java:193)
at com.mgh.im.esb.jmxutils.ServiceStatsRetriever.getStatsForMonitoredBizService(ServiceStatsRetriever.java:142)
at com.mgh.im.esb.jmxutils.ServiceStatsRetriever$GetAndResetStatisticsTask.run(ServiceStatsRetriever.java:652)
at com.mgh.im.esb.jmxutils.ServiceStatsRetriever$GetAndResetStatisticsTask.getStats(ServiceStatsRetriever.java:619)
at com.mgh.im.esb.jmxutils.ServiceStatsRetriever.getStats(ServiceStatsRetriever.java:779)
at com.mgh.im.esb.jmxutils.ServiceStatsRetriever$getStats.call(Unknown Source)
at Script1.sleepGetStats(Script1.groovy:75)
at Script1$sleepGetStats.callCurrent(Unknown Source)
at Script1.runBenchMarkTest(Script1.groovy:104)
at Script1$runBenchMarkTest.callCurrent(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:44)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:143)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:163)
at Script1.run(Script1.groovy:152)
at com.eviware.soapui.support.scripting.groovy.SoapUIGroovyScriptEngine.run(SoapUIGroovyScriptEngine.java:93)
at com.eviware.soapui.impl.wsdl.loadtest.WsdlLoadTest.runSetupScript(WsdlLoadTest.java:915)
at com.eviware.soapui.impl.wsdl.loadtest.WsdlLoadTestRunner.start(WsdlLoadTestRunner.java:102)
at com.eviware.soapui.impl.wsdl.loadtest.WsdlLoadTest.run(WsdlLoadTest.java:418)
at com.eviware.soapui.impl.wsdl.panels.loadtest.WsdlLoadTestDesktopPanel$RunLoadTestAction.actionPerformed(WsdlLoadTestDesktopPanel.java:424)
at javax.swing.AbstractButton.fireActionPerformed(AbstractButton.java:1995)
at javax.swing.AbstractButton$Handler.actionPerformed(AbstractButton.java:2318)
at javax.swing.DefaultButtonModel.fireActionPerformed(DefaultButtonModel.java:387)
at javax.swing.DefaultButtonModel.setPressed(DefaultButtonModel.java:242)
at javax.swing.plaf.basic.BasicButtonListener.mouseReleased(BasicButtonListener.java:236)
at java.awt.AWTEventMulticaster.mouseReleased(AWTEventMulticaster.java:272)
at java.awt.Component.processMouseEvent(Component.java:6263)
at javax.swing.JComponent.processMouseEvent(JComponent.java:3267)
at java.awt.Component.processEvent(Component.java:6028)
at java.awt.Container.processEvent(Container.java:2041)
at java.awt.Component.dispatchEventImpl(Component.java:4630)
at java.awt.Container.dispatchEventImpl(Container.java:2099)
at java.awt.Component.dispatchEvent(Component.java:4460)
at java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4574)
at java.awt.LightweightDispatcher.processMouseEvent(Container.java:4238)
at java.awt.LightweightDispatcher.dispatchEvent(Container.java:4168)
at java.awt.Container.dispatchEventImpl(Container.java:2085)
at java.awt.Window.dispatchEventImpl(Window.java:2478)
at java.awt.Component.dispatchEvent(Component.java:4460)
at java.awt.EventQueue.dispatchEvent(EventQueue.java:599)
at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:269)
at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:184)
at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:174)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:169)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:161)
at java.awt.EventDispatchThread.run(EventDispatchThread.java:122)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.mgh.im.esb.jmxutils.ServiceStatsRetriever$ServiceDomainMBeanInvocationHandler.invoke(ServiceStatsRetriever.java:567)
... 46 more
Caused by: com.bea.wli.monitoring.MonitoringException: [Monitoring Runtime:473057]Failed to get statistics for services due to java.rmi.UnknownHostException: Could not discover administration URL for server 'SS_ESB_mgd1'
at com.bea.wli.sb.management.monitoring.ServiceDomainMBeanImpl.getStatisticsForServices(ServiceDomainMBeanImpl.java:967)
at com.bea.wli.sb.management.monitoring.ServiceDomainMBeanImpl.getServiceStatistics(ServiceDomainMBeanImpl.java:802)
at com.bea.wli.sb.management.monitoring.ServiceDomainMBeanImpl.getBusinessServiceStatistics(ServiceDomainMBeanImpl.java:366)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at weblogic.management.jmx.modelmbean.WLSModelMBean.invoke(WLSModelMBean.java:437)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
at weblogic.management.mbeanservers.domainruntime.internal.FederatedMBeanServerInterceptor.invoke(FederatedMBeanServerInterceptor.java:255)
at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$16.run(WLSMBeanServerInterceptorBase.java:447)
at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.invoke(WLSMBeanServerInterceptorBase.java:445)
at weblogic.management.mbeanservers.internal.SecurityMBeanMgmtOpsInterceptor.invoke(SecurityMBeanMgmtOpsInterceptor.java:65)
at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$16.run(WLSMBeanServerInterceptorBase.java:447)
at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.invoke(WLSMBeanServerInterceptorBase.java:445)
at weblogic.management.mbeanservers.internal.SecurityInterceptor.invoke(SecurityInterceptor.java:443)
at weblogic.management.mbeanservers.internal.AuthenticatedSubjectInterceptor$10$1.run(AuthenticatedSubjectInterceptor.java:582)
at weblogic.management.mbeanservers.internal.AuthenticatedSubjectInterceptor$10.run(AuthenticatedSubjectInterceptor.java:580)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
at weblogic.management.mbeanservers.internal.AuthenticatedSubjectInterceptor.invoke(AuthenticatedSubjectInterceptor.java:573)
at weblogic.management.jmx.mbeanserver.WLSMBeanServer.invoke(WLSMBeanServer.java:307)
at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1426)
at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1264)
at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1366)
at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788)
at javax.management.remote.rmi.RMIConnectionImpl_WLSkel.invoke(Unknown Source)
at weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:589)
at weblogic.rmi.internal.BasicServerRef$1.run(BasicServerRef.java:477)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
at weblogic.security.service.SecurityManager.runAs(Unknown Source)
at weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:473)
at weblogic.rmi.internal.wls.WLSExecuteRequest.run(WLSExecuteRequest.java:118)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
Caused by: java.rmi.UnknownHostException: Could not discover administration URL for server 'SS_ESB_mgd1'
at weblogic.protocol.URLManager.findAdministrationURL(URLManager.java:175)
at com.bea.wli.sb.util.server.AggregatorUtils.getAggregatorServerAdminURL(AggregatorUtils.java:105)
at com.bea.wli.sb.management.monitoring.ServiceDomainMBeanImpl.getStatisticsRetrieverMBean(ServiceDomainMBeanImpl.java:2036)
at com.bea.wli.sb.management.monitoring.ServiceDomainMBeanImpl.getStatisticsForServices(ServiceDomainMBeanImpl.java:933)
... 35 more
java.lang.reflect.UndeclaredThrowableException
at $Proxy6.getBusinessServiceStatistics(Unknown Source)
at com.mgh.im.esb.jmxutils.ServiceStatsRetriever.getAndResetStatsForMonitoredBizService(ServiceStatsRetriever.java:193)
at com.mgh.im.esb.jmxutils.ServiceStatsRetriever.getStatsForMonitoredBizService(ServiceStatsRetriever.java:142)
at com.mgh.im.esb.jmxutils.ServiceStatsRetriever$GetAndResetStatisticsTask.run(ServiceStatsRetriever.java:652)
at com.mgh.im.esb.jmxutils.ServiceStatsRetriever$GetAndResetStatisticsTask.getStats(ServiceStatsRetriever.java:619)
at com.mgh.im.esb.jmxutils.ServiceStatsRetriever.getStats(ServiceStatsRetriever.java:779)
at com.mgh.im.esb.jmxutils.ServiceStatsRetriever$getStats.call(Unknown Source)
at Script1.sleepGetStats(Script1.groovy:75)
at Script1$sleepGetStats.callCurrent(Unknown Source)
at Script1.runBenchMarkTest(Script1.groovy:104)
at Script1$runBenchMarkTest.callCurrent(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:44)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:143)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:163)
at Script1.run(Script1.groovy:152)
at com.eviware.soapui.support.scripting.groovy.SoapUIGroovyScriptEngine.run(SoapUIGroovyScriptEngine.java:93)
at com.eviware.soapui.impl.wsdl.loadtest.WsdlLoadTest.runSetupScript(WsdlLoadTest.java:915)
at com.eviware.soapui.impl.wsdl.loadtest.WsdlLoadTestRunner.start(WsdlLoadTestRunner.java:102)
at com.eviware.soapui.impl.wsdl.loadtest.WsdlLoadTest.run(WsdlLoadTest.java:418)
at com.eviware.soapui.impl.wsdl.panels.loadtest.WsdlLoadTestDesktopPanel$RunLoadTestAction.actionPerformed(WsdlLoadTestDesktopPanel.java:424)
at javax.swing.AbstractButton.fireActionPerformed(AbstractButton.java:1995)
at javax.swing.AbstractButton$Handler.actionPerformed(AbstractButton.java:2318)
at javax.swing.DefaultButtonModel.fireActionPerformed(DefaultButtonModel.java:387)
at javax.swing.DefaultButtonModel.setPressed(DefaultButtonModel.java:242)
at javax.swing.plaf.basic.BasicButtonListener.mouseReleased(BasicButtonListener.java:236)
at java.awt.AWTEventMulticaster.mouseReleased(AWTEventMulticaster.java:272)
at java.awt.Component.processMouseEvent(Component.java:6263)
at javax.swing.JComponent.processMouseEvent(JComponent.java:3267)
at java.awt.Component.processEvent(Component.java:6028)
at java.awt.Container.processEvent(Container.java:2041)
at java.awt.Component.dispatchEventImpl(Component.java:4630)
at java.awt.Container.dispatchEventImpl(Container.java:2099)
at java.awt.Component.dispatchEvent(Component.java:4460)
at java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4574)
at java.awt.LightweightDispatcher.processMouseEvent(Container.java:4238)
at java.awt.LightweightDispatcher.dispatchEvent(Container.java:4168)
at java.awt.Container.dispatchEventImpl(Container.java:2085)
at java.awt.Window.dispatchEventImpl(Window.java:2478)
at java.awt.Component.dispatchEvent(Component.java:4460)
at java.awt.EventQueue.dispatchEvent(EventQueue.java:599)
at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:269)
at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:184)
at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:174)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:169)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:161)
at java.awt.EventDispatchThread.run(EventDispatchThread.java:122)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.mgh.im.esb.jmxutils.ServiceStatsRetriever$ServiceDomainMBeanInvocationHandler.invoke(ServiceStatsRetriever.java:567)
... 46 more
Caused by: com.bea.wli.monitoring.MonitoringException: [Monitoring Runtime:473057]Failed to get statistics for services due to java.rmi.UnknownHostException: Could not discover administration URL for server 'SS_ESB_mgd1'
at com.bea.wli.sb.management.monitoring.ServiceDomainMBeanImpl.getStatisticsForServices(ServiceDomainMBeanImpl.java:967)
at com.bea.wli.sb.management.monitoring.ServiceDomainMBeanImpl.getServiceStatistics(ServiceDomainMBeanImpl.java:802)
at com.bea.wli.sb.management.monitoring.ServiceDomainMBeanImpl.getBusinessServiceStatistics(ServiceDomainMBeanImpl.java:366)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at weblogic.management.jmx.modelmbean.WLSModelMBean.invoke(WLSModelMBean.java:437)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
at weblogic.management.mbeanservers.domainruntime.internal.FederatedMBeanServerInterceptor.invoke(FederatedMBeanServerInterceptor.java:255)
at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$16.run(WLSMBeanServerInterceptorBase.java:447)
at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.invoke(WLSMBeanServerInterceptorBase.java:445)
at weblogic.management.mbeanservers.internal.SecurityMBeanMgmtOpsInterceptor.invoke(SecurityMBeanMgmtOpsInterceptor.java:65)
at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$16.run(WLSMBeanServerInterceptorBase.java:447)
at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.invoke(WLSMBeanServerInterceptorBase.java:445)
at weblogic.management.mbeanservers.internal.SecurityInterceptor.invoke(SecurityInterceptor.java:443)
at weblogic.management.mbeanservers.internal.AuthenticatedSubjectInterceptor$10$1.run(AuthenticatedSubjectInterceptor.java:582)
at weblogic.management.mbeanservers.internal.AuthenticatedSubjectInterceptor$10.run(AuthenticatedSubjectInterceptor.java:580)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
at weblogic.management.mbeanservers.internal.AuthenticatedSubjectInterceptor.invoke(AuthenticatedSubjectInterceptor.java:573)
at weblogic.management.jmx.mbeanserver.WLSMBeanServer.invoke(WLSMBeanServer.java:307)
at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1426)
at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1264)
at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1366)
at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788)
at javax.management.remote.rmi.RMIConnectionImpl_WLSkel.invoke(Unknown Source)
at weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:589)
at weblogic.rmi.internal.BasicServerRef$1.run(BasicServerRef.java:477)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
at weblogic.security.service.SecurityManager.runAs(Unknown Source)
at weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:473)
at weblogic.rmi.internal.wls.WLSExecuteRequest.run(WLSExecuteRequest.java:118)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
Caused by: java.rmi.UnknownHostException: Could not discover administration URL for server 'SS_ESB_mgd1'
at weblogic.protocol.URLManager.findAdministrationURL(URLManager.java:175)
at com.bea.wli.sb.util.server.AggregatorUtils.getAggregatorServerAdminURL(AggregatorUtils.java:105)
at com.bea.wli.sb.management.monitoring.ServiceDomainMBeanImpl.getStatisticsRetrieverMBean(ServiceDomainMBeanImpl.java:2036)
at com.bea.wli.sb.management.monitoring.ServiceDomainMBeanImpl.getStatisticsForServices(ServiceDomainMBeanImpl.java:933)
... 35 more