Hi,
We are using Weblogic 10.3.1.0 version.
Totaly 4 Managed Severs
1 Admin Server
with Nodemanger.
In Admin Console When ever we going to Stop the Manged Servers.
1. Environment ----- > Servers ---- > Control ----------- > Mananged Server 1 -----------> Force shutdown
2. Then click refresh button ---- showing "FAILED_NOT_RESTARTABLE"
3. after that when ever we going to start the Managed Server in Admin console
First its Showing ----- Starting
and next its showing ----- FAILED_NOT_RESTARTABLE
4. its not coming to Running State.
In Logs
-----------------------
Domain log
-----------> <1273141345460> <BEA-000450> <Socket 8 internal data record unavailable (probable closure due idle timeout), event received 17>
####<06-May-2010 11:54:40 o'clock BST> <Alert> <WebLogicServer> <ukirt156> <twpserver1> <[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'> <weblogic> <> <> <1273143280738> <BEA-000396> <Server shutdown has been requested by weblogic>
####<06-May-2010 11:54:40 o'clock BST> <Notice> <WebLogicServer> <ukirt156> <twpserver1> <[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'> <weblogic> <> <> <1273143280771> <BEA-000365> <Server state changed to FORCE_SUSPENDING>
####<06-May-2010 11:54:40 o'clock BST> <Notice> <Cluster> <ukirt156> <twpserver1> <[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'> <weblogic> <> <> <1273143280844> <BEA-000163> <Stopping "async" replication service>
####<06-May-2010 11:54:41 o'clock BST> <Notice> <WebLogicServer> <ukirt156> <twpserver1> <[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'> <weblogic> <> <> <1273143281061> <BEA-000365> <Server state changed to ADMIN>
####<06-May-2010 11:54:41 o'clock BST> <Notice> <WebLogicServer> <ukirt156> <twpserver1> <[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'> <weblogic> <> <> <1273143281064> <BEA-000365> <Server state changed to FORCE_SHUTTING_DOWN>
####<06-May-2010 11:54:41 o'clock BST> <Notice> <Server> <ukirt156> <twpserver1> <DynamicListenThread[Default]> <<WLS Kernel>> <> <> <1273143281113> <BEA-002607> <Channel "Default" listening on 172.21.149.168:7003 was shutdown.>
####<06-May-2010 11:56:30 o'clock BST> <Error> <NodeManager> <ukirt156> <AdminServer> <[ACTIVE] ExecuteThread: '45' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1273143390016> <BEA-300048> <Unable to start the server twpserver1 : Exception while starting server 'twpserver1': java.io.IOException: Server failed to start up. See server output log for more details.>
AdminServer log
------------------
<06-May-2010 11:56:30 o'clock BST> <Error> <NodeManager> <BEA-300048> <Unable to start the server twpserver1 : Exception while starting server 'twpserver1': java.io.IOException: Server failed to start up. See server output log for more details.>
<06-May-2010 11:57:33 o'clock BST> <Error> <NodeManager> <BEA-300048> <Unable to start the server twpserver1 : Exception while starting server 'twpserver1': java.io.IOException: Server failed to start up. See server output log for more details.>
<06-May-2010 12:07:02 o'clock BST> <Error> <NodeManager> <BEA-300048> <Unable to start the server twpserver1 : Exception while starting server 'twpserver1': java.io.IOException: Server failed to start up. See server output log for more details.>
ManagedServer Log
-----------------
<06-May-2010 11:54:40 o'clock BST> <Alert> <WebLogicServer> <BEA-000396> <Server shutdown has been requested by weblogic>
06-May-2010 11:54:41 org.quartz.core.QuartzScheduler shutdown
INFO: Scheduler DefaultQuartzScheduler_$_NON_CLUSTERED shutting down.
06-May-2010 11:54:41 org.quartz.core.QuartzScheduler pause
INFO: Scheduler DefaultQuartzScheduler_$_NON_CLUSTERED paused.
06-May-2010 11:54:41 org.quartz.core.QuartzScheduler shutdown
INFO: Scheduler DefaultQuartzScheduler_$_NON_CLUSTERED shutdown complete.
<06-May-2010 11:54:54> <Debug> <NodeManager> <Waiting for the process to die: 19316>
<06-May-2010 11:54:54> <Info> <NodeManager> <Server was shut down normally>
<06-May-2010 11:54:54> <Debug> <NodeManager> <runMonitor returned, setting finished=true and notifying waiters>
NodeManager Log
---------------
<06-May-2010 11:54:54> <Info> <twp_domain> <twpserver1> <Server was shut down normally>
<06-May-2010 11:56:21> <Info> <twp_domain> <twpserver1> <Boot identity properties saved to "/appl/weblogic/scripts/servers/twpserver1/data/nodemanager/boot.properties">
<06-May-2010 11:56:21> <Info> <twp_domain> <twpserver1> <Startup configuration properties saved to "/appl/weblogic/scripts/servers/twpserver1/data/nodemanager/startup.properties">
<06-May-2010 11:56:21> <Info> <twp_domain> <twpserver1> <Rotated server output log to "/appl/weblogic/scripts/servers/twpserver1/logs/twpserver1.out00020">
<06-May-2010 11:56:21> <Info> <twp_domain> <twpserver1> <Server error log also redirected to server log>
<06-May-2010 11:56:21> <Info> <twp_domain> <twpserver1> <Starting WebLogic server with command line: /appl/weblogic/oracle/middleware/jrockit_160_05_R27.6.2-20/jre/bin/java -Dweblogic.Name=twpserver1 -Djava.security.policy=/appl/weblogic/oracle/middleware/wlserver_10.3/server/lib/weblogic.policy -Dweblogic.management.server=http://ukirt156:7001 -Djava.library.path="/appl/weblogic/oracle/middleware/jrockit_160_05_R27.6.2-20/jre/lib/i386/jrockit:/appl/weblogic/oracle/middleware/jrockit_160_05_R27.6.2-20/jre/lib/i386:/appl/weblogic/oracle/middleware/jrockit_160_05_R27.6.2-20/jre/../lib/i386::/appl/nsm/CA/UnicenterNSM/lib:/appl/nsm/CA/SharedComponents/JRE/1.4.2_09/lib/i386/client:/appl/nsm/CA/SharedComponents/JRE/1.4.2_09/lib/i386:/appl/nsm/CA/SharedComponents/ccs/dia/dna/lib:/appl/nsm/CA/SharedComponents/ccs/dia/lib:/usr/lib:/opt/CA/CAlib:/appl/nsm/CA/SharedComponents/lib:/appl/nsm/CA/SharedComponents/Csam/SockAdapter/lib:/appl/weblogic/oracle/middleware/wlserver_10.3/server/native/linux/i686:/appl/weblogic/oracle/middleware/wlserver_10.3/server/native/linux/i686/oci920_8" -Djava.class.path=/data/share/package/twp/WEB-INF/lib/ojdbc14.jar:/appl/weblogic/oracle/middleware/wlserver_10.3/server/lib/weblogic_sp.jar:/appl/weblogic/oracle/middleware/wlserver_10.3/server/lib/weblogic.jar -Dweblogic.system.BootIdentityFile=/appl/weblogic/scripts/servers/twpserver1/data/nodemanager/boot.properties -Dweblogic.nodemanager.ServiceEnabled=true -Dweblogic.security.SSL.ignoreHostnameVerification=false -Dweblogic.ReverseDNSAllowed=false -Xmanagement:port=7091,ssl=false,authenticate=false -Xms=1200m -Xmx=1200m -Xverbose:gc,gcpause -Dreflexis.scheduler=true -Dreflexis.server=twpserver1 -Djava.awt.headless=true -Dclient.encoding.override=UTF-8 -Doracle.jdbc.V8Compatible=true weblogic.Server >
<06-May-2010 11:56:21> <Info> <twp_domain> <twpserver1> <Working directory is "/appl/weblogic/scripts">
<06-May-2010 11:56:21> <Info> <twp_domain> <twpserver1> <Rotated server output log to "/appl/weblogic/scripts/servers/twpserver1/logs/twpserver1.out00021">
<06-May-2010 11:56:21> <Info> <twp_domain> <twpserver1> <Server error log also redirected to server log>
<06-May-2010 11:56:21> <Info> <twp_domain> <twpserver1> <Server output log file is "/appl/weblogic/scripts/servers/twpserver1/logs/twpserver1.out">
<06-May-2010 11:56:29> <Info> <twp_domain> <twpserver1> <Server failed during startup so will not be restarted>
<06-May-2010 11:56:29> <Warning> <Exception while starting server 'twpserver1': java.io.IOException: Server failed to start up. See server output log for more details.>
java.io.IOException: Server failed to start up. See server output log for more details.
at weblogic.nodemanager.server.ServerManager.start(ServerManager.java:332)
at weblogic.nodemanager.server.Handler.handleStart(Handler.java:542)
at weblogic.nodemanager.server.Handler.handleCommand(Handler.java:119)
at weblogic.nodemanager.server.Handler.run(Handler.java:66)
at java.lang.Thread.run(Thread.java:619)
Server logs
------------------
<06-May-2010 12:06:57> <Info> <NodeManager> <Server output log file is "/appl/weblogic/scripts/servers/twpserver1/logs/twpserver1.out">
[INFO ][memory ] GC mode: Garbage collection optimized for throughput, initial strategy: Generational Parallel Mark & Sweep
[INFO ][memory ] heap size: 1228800K, maximal heap size: 1228800K, nursery size: 614400K
[INFO ][memory ] <s>-<end>: GC <before>K-><after>K (<heap>K), <pause> ms
[INFO ][memory ] <s/start> - start time of collection (seconds since jvm start)
[INFO ][memory ] <end> - end time of collection (seconds since jvm start)
[INFO ][memory ] <before> - memory used by objects before collection (KB)
[INFO ][memory ] <after> - memory used by objects after collection (KB)
[INFO ][memory ] <heap> - size of heap after collection (KB)
[INFO ][memory ] <pause> - total sum of pauses during collection (milliseconds)
[INFO ][memory ] run with -Xverbose:gcpause to see individual pauses
[JRockit] Management server started on port 7091, ssl=false, authenticate=false.
<06-May-2010 12:06:59 o'clock BST> <Info> <WebLogicServer> <BEA-000377> <Starting WebLogic Server with BEA JRockit(R) Version R27.6.2-20_o-108500-1.6.0_05-20090120-1115-linux-ia32 from BEA Systems, Inc.>
<06-May-2010 12:07:00 o'clock BST> <Critical> <Security> <BEA-090518> <Could not decrypt the username attribute value of {AES}xvFgPysVi5b89pYwaAppoqPXQ5wFVW13yoFhzhKmuQo= from the file /appl/weblogic/scripts/servers/twpserver1/data/nodemanager/boot.properties. If you have copied an encrypted attribute from boot.properties from another domain into /appl/weblogic/scripts/servers/twpserver1/data/nodemanager/boot.properties, change the encrypted attribute to its cleartext value then reboot the server. The attribute will be re-encrypted. Otherwise, change all encrypted attributes to their cleartext values, then reboot the server. All encryptable attributes will be re-encrypted. The decryption failed with the exception weblogic.security.internal.encryption.EncryptionServiceException: com.rsa.jsafe.JSAFE_PaddingException: Could not perform unpadding: invalid pad byte..>
<06-May-2010 12:07:00 o'clock BST> <Critical> <Security> <BEA-090518> <Could not decrypt the password attribute value of {AES}WSsjiXk6ppStq4WFajxaXkqCwpcBwaMJni7vk/pqXvk= from the file /appl/weblogic/scripts/servers/twpserver1/data/nodemanager/boot.properties. If you have copied an encrypted attribute from boot.properties from another domain into /appl/weblogic/scripts/servers/twpserver1/data/nodemanager/boot.properties, change the encrypted attribute to its cleartext value then reboot the server. The attribute will be re-encrypted. Otherwise, change all encrypted attributes to their cleartext values, then reboot the server. All encryptable attributes will be re-encrypted. The decryption failed with the exception weblogic.security.internal.encryption.EncryptionServiceException: com.rsa.jsafe.JSAFE_PaddingException: Could not perform unpadding: invalid pad byte..>
Enter username to boot WebLogic server:<06-May-2010 12:07:00 o'clock BST> <Info> <Management> <BEA-141107> <Version: WebLogic Server 10.3.1.0 Thu Jun 11 00:26:56 EDT 2009 1227385 >
<06-May-2010 12:07:01 o'clock BST> <Info> <Management> <BEA-141223> <The server name twpserver1 specified with -Dweblogic.Name does not exist. The configuration includes the following servers {AdminServer}.>
<06-May-2010 12:07:01 o'clock BST> <Critical> <WebLogicServer> <BEA-000362> <Server failed. Reason:
There are 1 nested errors:
weblogic.management.ManagementException: [Management:141223]The server name twpserver1 specified with -Dweblogic.Name does not exist. The configuration includes the following servers {AdminServer}.
at weblogic.management.provider.internal.RuntimeAccessImpl.<init>(RuntimeAccessImpl.java:149)
at weblogic.management.provider.internal.RuntimeAccessService.start(RuntimeAccessService.java:41)
at weblogic.t3.srvr.ServerServicesManager.startService(ServerServicesManager.java:461)
at weblogic.t3.srvr.ServerServicesManager.startInStandbyState(ServerServicesManager.java:166)
at weblogic.t3.srvr.T3Srvr.initializeStandby(T3Srvr.java:749)
at weblogic.t3.srvr.T3Srvr.startup(T3Srvr.java:488)
at weblogic.t3.srvr.T3Srvr.run(T3Srvr.java:446)
at weblogic.Server.main(Server.java:67)
>
<06-May-2010 12:07:01 o'clock BST> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FAILED>
<06-May-2010 12:07:01 o'clock BST> <Error> <WebLogicServer> <BEA-000383> <A critical service failed. The server will shut itself down>
<06-May-2010 12:07:01 o'clock BST> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FORCE_SHUTTING_DOWN>
<06-May-2010 12:07:02> <Debug> <NodeManager> <Waiting for the process to die: 19443>
<06-May-2010 12:07:02> <Info> <NodeManager> <Server failed during startup so will not be restarted>
<06-May-2010 12:07:02> <Debug> <NodeManager> <runMonitor returned, setting finished=true and notifying waiters>
Note: if i start the manged servers through WLST scripts its working fine.
[twpuser@ukirt156 /appl/weblogic/scripts]$ java weblogic.WLST
Initializing WebLogic Scripting Tool (WLST) ...
Welcome to WebLogic Server Administration Scripting Shell
Type help() for help on available commands
wls:/offline> nmConnect('weblogic', 'w3bl0g1c', 'ukirt156', '5556', 'twp_domain', '/appl/weblogic/oracle/middleware/user_projects/domains/twp_domain', 'plain')
Connecting to Node Manager ...
Successfully Connected to Node Manager.
wls:/nm/twp_domain> nmStart('twpserver1')
Starting server twpserver1 ...
Successfully started server twpserver1 ...
Please provide the solution, how to solve this issue.
Thanks
SRK