Cannot add a server to a server pool
I did an unusual thing, but I think it should still work:
I installed Oracle VM 3.0.3, with two servers in a server pool, in my "lab". It was working just fine.
For various reasons unrelated to Oracle VM, I removed "oravm2" from the pool, because I need to temporarily use that machine for something else.
I tried to delete that server from the manager, but it wouldn't let me. I understand, from other postings, that this is a bug in 3.0.3. The error message was:
OVMRU_002041E Cannot delete/remove pool file system: Pool filesystem for pool1. Server pool: pool1, still has servers in it
But that's not what I'm posting about.
I installed another operating system on that server, did some things, and then tried to make an Oracle VM server again, by reinstalling Oracle VM on it, using the same IP address.
First, since the server was still defined in the manager (I couldn't remove it!), I tried to simply add that back into the pool. That failed, with messages that I'll show later.
When that failed, I renamed the server "old-oravm2", and did a "discover server". This step did not fail, but it apparently matched the IP address to that of "old-oravm2", and so I didn't get a new "oravm2" discovered.
I optimistically tried to add "old-oravm2" (really a re-discovered oravm2) to the pool. It failed, just like the first time. The detailed messages are below.
My question is: How can I get my 2-node server pool going again?
Here is the detailed error stuff:
Job Construction Phase
----------------------
begin()
Appended operation 'Server Role Update' to object 'c0:dd:bd:b8:fe:7a:b6:01:13:31:00:1a:64:db:74:a4 (old-oravm2.acbl.net)'.
Appended operation 'Server Join Server Pool' to object 'c0:dd:bd:b8:fe:7a:b6:01:13:31:00:1a:64:db:74:a4 (old-oravm2.acbl.net)'.
Appended operation 'Server Pool Member Update' to object '0004fb0000020000c02eeebb9de3a8e7 (pool1)'.
Appended operation 'Server Cluster Configuration Update' to object 'c0:e3:8b:d3:47:61:b6:01:1a:48:00:1a:64:36:28:6c (oravm1.acbl.net)'.
Appended operation 'Server Cluster Configure' to object 'c0:dd:bd:b8:fe:7a:b6:01:13:31:00:1a:64:db:74:a4 (old-oravm2.acbl.net)'.
Appended operation 'Server Cluster Join' to object 'c0:dd:bd:b8:fe:7a:b6:01:13:31:00:1a:64:db:74:a4 (old-oravm2.acbl.net)'.
commit()
Completed Step: COMMIT
Objects and Operations
----------------------
Object (IN_USE): [Server] c0:dd:bd:b8:fe:7a:b6:01:13:31:00:1a:64:db:74:a4 (old-oravm2.acbl.net)
Operation: Server Role Update
Operation: Server Join Server Pool
Operation: Server Cluster Configure
Operation: Server Cluster Join
Object (IN_USE): [Server] c0:e3:8b:d3:47:61:b6:01:1a:48:00:1a:64:36:28:6c (oravm1.acbl.net)
Operation: Server Cluster Configuration Update
Object (IN_USE): [Cluster] c02eeebb9de3a8e7
Object (IN_USE): [ServerPool] 0004fb0000020000c02eeebb9de3a8e7 (pool1)
Operation: Server Pool Member Update
Job Running Phase at 14:11 on Fri, Mar 16, 2012
----------------------------------------------
Job Participants: []
Actioner
--------
Starting operation 'Server Role Update' on object 'c0:dd:bd:b8:fe:7a:b6:01:13:31:00:1a:64:db:74:a4 (old-oravm2.acbl.net)'
Completed operation 'Server Role Update' completed with direction ==> DONE
Starting operation 'Server Join Server Pool' on object 'c0:dd:bd:b8:fe:7a:b6:01:13:31:00:1a:64:db:74:a4 (old-oravm2.acbl.net)'
Completed operation 'Server Join Server Pool' completed with direction ==> LATER
Starting operation 'Server Cluster Configuration Update' on object 'c0:e3:8b:d3:47:61:b6:01:1a:48:00:1a:64:36:28:6c (oravm1.acbl.net)'
Completed operation 'Server Cluster Configuration Update' completed with direction ==> LATER
Starting operation 'Server Pool Member Update' on object '0004fb0000020000c02eeebb9de3a8e7 (pool1)'
Completed operation 'Server Pool Member Update' completed with direction ==> DONE
Starting operation 'Server Cluster Configure' on object 'c0:dd:bd:b8:fe:7a:b6:01:13:31:00:1a:64:db:74:a4 (old-oravm2.acbl.net)'
Completed operation 'Server Cluster Configure' completed with direction ==> LATER
Starting operation 'Server Cluster Configuration Update' on object 'c0:e3:8b:d3:47:61:b6:01:1a:48:00:1a:64:36:28:6c (oravm1.acbl.net)'
Completed operation 'Server Cluster Configuration Update' completed with direction ==> LATER
Starting operation 'Server Cluster Join' on object 'c0:dd:bd:b8:fe:7a:b6:01:13:31:00:1a:64:db:74:a4 (old-oravm2.acbl.net)'
Completed operation 'Server Cluster Join' completed with direction ==> LATER
Starting operation 'Server Cluster Configuration Update' on object 'c0:e3:8b:d3:47:61:b6:01:1a:48:00:1a:64:36:28:6c (oravm1.acbl.net)'
Completed operation 'Server Cluster Configuration Update' completed with direction ==> LATER
Starting operation 'Server Join Server Pool' on object 'c0:dd:bd:b8:fe:7a:b6:01:13:31:00:1a:64:db:74:a4 (old-oravm2.acbl.net)'
Completed operation 'Server Join Server Pool' completed with direction ==> DONE
Starting operation 'Server Cluster Configure' on object 'c0:dd:bd:b8:fe:7a:b6:01:13:31:00:1a:64:db:74:a4 (old-oravm2.acbl.net)'
Completed operation 'Server Cluster Configure' completed with direction ==> LATER
Starting operation 'Server Cluster Configuration Update' on object 'c0:e3:8b:d3:47:61:b6:01:1a:48:00:1a:64:36:28:6c (oravm1.acbl.net)'
Completed operation 'Server Cluster Configuration Update' completed with direction ==> DONE
Starting operation 'Server Cluster Join' on object 'c0:dd:bd:b8:fe:7a:b6:01:13:31:00:1a:64:db:74:a4 (old-oravm2.acbl.net)'
Completed operation 'Server Cluster Join' completed with direction ==> LATER
Starting operation 'Server Cluster Configure' on object 'c0:dd:bd:b8:fe:7a:b6:01:13:31:00:1a:64:db:74:a4 (old-oravm2.acbl.net)'
Job Internal Error (Operation)com.oracle.ovm.mgr.api.exception.FailedOperationException: OVMAPI_4010E Attempt to send command: dispatch to server: old-oravm2.acbl.net failed. OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@172.16.2.52:8899/api/1 configure_server_for_cluster cluster:
name = c02eeebb9de3a8e7
heartbeat_mode = global
node_count = 2
node:
cluster = c02eeebb9de3a8e7
name = oravm1.acbl.net
number = 0
ip_address = 172.16.2.51
ip_port = 7777
node:
cluster = c02eeebb9de3a8e7
name = oravm2.acbl.net
number = 1
ip_address = 172.16.2.52
ip_port = 7777
heartbeat:
cluster = c02eeebb9de3a8e7
region = 0004FB000005000076F67B41034868F5
lun /dev/mapper/20017380004470086 0004fb000005000076f67b41034868f5 , Status: org.apache.xmlrpc.XmlRpcException: exceptions.OSError:[Errno 2] No such file or directory
Fri Mar 16 14:11:18 EDT 2012
Fri Mar 16 14:11:18 EDT 2012
at com.oracle.ovm.mgr.action.ActionEngine.sendCommandToServer(ActionEngine.java:475)
at com.oracle.ovm.mgr.action.ActionEngine.sendDispatchedServerCommand(ActionEngine.java:415)
at com.oracle.ovm.mgr.action.ActionEngine.sendServerCommand(ActionEngine.java:362)
at com.oracle.ovm.mgr.action.ClusterAction.configureServerForCluster(ClusterAction.java:82)
at com.oracle.ovm.mgr.op.physical.ServerClusterConfigure.configureCluster(ServerClusterConfigure.java:137)
at com.oracle.ovm.mgr.op.physical.ServerClusterConfigure.action(ServerClusterConfigure.java:56)
at com.oracle.ovm.mgr.api.job.JobEngine.operationActioner(JobEngine.java:193)
at com.oracle.ovm.mgr.api.job.JobEngine.objectActioner(JobEngine.java:264)
at com.oracle.ovm.mgr.api.job.InternalJobDbImpl.objectCommitter(InternalJobDbImpl.java:1090)
at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:247)
at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:207)
at com.oracle.odof.core.BasicWork.invokeMethod(BasicWork.java:136)
at com.oracle.odof.command.InvokeMethodCommand.process(InvokeMethodCommand.java:100)
at com.oracle.odof.core.BasicWork.processCommand(BasicWork.java:81)
at com.oracle.odof.core.TransactionManager.processCommand(TransactionManager.java:751)
at com.oracle.odof.core.WorkflowManager.processCommand(WorkflowManager.java:401)
at com.oracle.odof.core.WorkflowManager.processWork(WorkflowManager.java:459)
at com.oracle.odof.io.AbstractClient.run(AbstractClient.java:42)
at java.lang.Thread.run(Thread.java:662)
Caused by: com.oracle.ovm.mgr.api.exception.IllegalOperationException: OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@172.16.2.52:8899/api/1 configure_server_for_cluster cluster:
name = c02eeebb9de3a8e7
heartbeat_mode = global
node_count = 2
node:
cluster = c02eeebb9de3a8e7
name = oravm1.acbl.net
number = 0
ip_address = 172.16.2.51
ip_port = 7777
node:
cluster = c02eeebb9de3a8e7
name = oravm2.acbl.net
number = 1
ip_address = 172.16.2.52
ip_port = 7777
heartbeat:
cluster = c02eeebb9de3a8e7
region = 0004FB000005000076F67B41034868F5
lun /dev/mapper/20017380004470086 0004fb000005000076f67b41034868f5 , Status: org.apache.xmlrpc.XmlRpcException: exceptions.OSError:[Errno 2] No such file or directory
Fri Mar 16 14:11:18 EDT 2012
at com.oracle.ovm.mgr.action.ActionEngine.sendAction(ActionEngine.java:753)
at com.oracle.ovm.mgr.action.ActionEngine.sendCommandToServer(ActionEngine.java:471)
... 21 more
FailedOperationCleanup
----------
Starting failed operation 'Server Cluster Configure' cleanup on object 'old-oravm2.acbl.net'
Complete rollback operation 'Server Cluster Configure' completed with direction=old-oravm2.acbl.net
Rollbacker
----------
Executing rollback operation 'Server Join Server Pool' on object 'c0:dd:bd:b8:fe:7a:b6:01:13:31:00:1a:64:db:74:a4 (old-oravm2.acbl.net)'
Complete rollback operation 'Server Join Server Pool' completed with direction=LATER
Executing rollback operation 'Server Cluster Configuration Update' on object 'c0:e3:8b:d3:47:61:b6:01:1a:48:00:1a:64:36:28:6c (oravm1.acbl.net)'
Complete rollback operation 'Server Cluster Configuration Update' completed with direction=DONE
Executing rollback operation 'Server Pool Member Update' on object '0004fb0000020000c02eeebb9de3a8e7 (pool1)'
Complete rollback operation 'Server Pool Member Update' completed with direction=LATER
Executing rollback operation 'Server Join Server Pool' on object 'c0:dd:bd:b8:fe:7a:b6:01:13:31:00:1a:64:db:74:a4 (old-oravm2.acbl.net)'
Complete rollback operation 'Server Join Server Pool' completed with direction=DONE
Executing rollback operation 'Server Role Update' on object 'c0:dd:bd:b8:fe:7a:b6:01:13:31:00:1a:64:db:74:a4 (old-oravm2.acbl.net)'
Complete rollback operation 'Server Role Update' completed with direction=DONE
Executing rollback operation 'Server Pool Member Update' on object '0004fb0000020000c02eeebb9de3a8e7 (pool1)'
Complete rollback operation 'Server Pool Member Update' completed with direction=DONE
Objects To Be Rolled Back
-------------------------
Object (IN_USE): [Server] c0:dd:bd:b8:fe:7a:b6:01:13:31:00:1a:64:db:74:a4 (old-oravm2.acbl.net)
Object (IN_USE): [Server] c0:e3:8b:d3:47:61:b6:01:1a:48:00:1a:64:36:28:6c (oravm1.acbl.net)
Object (IN_USE): [Cluster] c02eeebb9de3a8e7
Object (IN_USE): [ServerPool] 0004fb0000020000c02eeebb9de3a8e7 (pool1)
Write Methods Invoked
-------------------
Class=InternalJobDbImpl vessel_id=25335 method=addTransactionIdentifier accessLevel=6
Class=ServerPoolDbImpl vessel_id=2977 method=addServer accessLevel=6
Class=ServerDbImpl vessel_id=2460 method=lock accessLevel=6
Class=ServerDbImpl vessel_id=2460 method=addServerRole accessLevel=6
Class=ServerDbImpl vessel_id=2460 method=addServerRole accessLevel=6
Class=ServerDbImpl vessel_id=2460 method=addServerRole accessLevel=6
Class=ServerPoolDbImpl vessel_id=2977 method=addServerInternal accessLevel=6
Class=ServerDbImpl vessel_id=2460 method=setServerPool accessLevel=6
Class=ClusterDbImpl vessel_id=2983 method=allocateSlotForServer accessLevel=6
Class=ClusterDbImpl vessel_id=2983 method=addServer accessLevel=6
Class=ServerDbImpl vessel_id=2460 method=lock accessLevel=6
Class=ServerDbImpl vessel_id=2175 method=lock accessLevel=6
Class=InternalJobDbImpl vessel_id=25335 method=setAsset accessLevel=6
Class=ServerDbImpl vessel_id=2175 method=reconfigureCluster accessLevel=6
Class=ServerDbImpl vessel_id=2460 method=setCluster accessLevel=6
Class=InternalJobDbImpl vessel_id=25335 method=setAsset accessLevel=6
Class=InternalJobDbImpl vessel_id=25335 method=setAsset accessLevel=6
Class=InternalJobDbImpl vessel_id=25335 method=setCompletedStep accessLevel=6
Class=InternalJobDbImpl vessel_id=25335 method=setAssociatedHandles accessLevel=6
Class=ServerDbImpl vessel_id=2460 method=nextJobOperation accessLevel=6
Class=InternalJobDbImpl vessel_id=25335 method=setTuringMachineFlag accessLevel=6
Class=ServerDbImpl vessel_id=2460 method=setCurrentOperationToLater accessLevel=6
Class=InternalJobDbImpl vessel_id=25335 method=setTuringMachineFlag accessLevel=6
Class=InternalJobDbImpl vessel_id=25335 method=setTuringMachineFlag accessLevel=6
Class=ServerDbImpl vessel_id=2175 method=setCurrentOperationToLater accessLevel=6
Class=InternalJobDbImpl vessel_id=25335 method=setTuringMachineFlag accessLevel=6
Class=ServerPoolDbImpl vessel_id=2977 method=nextJobOperation accessLevel=6
Class=InternalJobDbImpl vessel_id=25335 method=setTuringMachineFlag accessLevel=6
Class=ServerDbImpl vessel_id=2460 method=setCurrentOperationToLater accessLevel=6
Class=InternalJobDbImpl vessel_id=25335 method=setTuringMachineFlag accessLevel=6
Class=InternalJobDbImpl vessel_id=25335 method=setTuringMachineFlag accessLevel=6
Class=ServerDbImpl vessel_id=2175 method=setCurrentOperationToLater accessLevel=6
Class=InternalJobDbImpl vessel_id=25335 method=setTuringMachineFlag accessLevel=6
Class=InternalJobDbImpl vessel_id=25335 method=setTuringMachineFlag accessLevel=6
Class=ServerDbImpl vessel_id=2460 method=setCurrentOperationToLater accessLevel=6
Class=InternalJobDbImpl vessel_id=25335 method=setTuringMachineFlag accessLevel=6
Class=InternalJobDbImpl vessel_id=25335 method=setTuringMachineFlag accessLevel=6
Class=ServerDbImpl vessel_id=2175 method=setCurrentOperationToLater accessLevel=6
Class=InternalJobDbImpl vessel_id=25335 method=setTuringMachineFlag accessLevel=6
Class=ServerDbImpl vessel_id=2460 method=nextJobOperation accessLevel=6
Class=InternalJobDbImpl vessel_id=25335 method=setTuringMachineFlag accessLevel=6
Class=ServerDbImpl vessel_id=2460 method=setCurrentOperationToLater accessLevel=6
Class=InternalJobDbImpl vessel_id=25335 method=setTuringMachineFlag accessLevel=6
Class=InternalJobDbImpl vessel_id=25335 method=setProgressMessage accessLevel=6
Class=ServerDbImpl vessel_id=2175 method=nextJobOperation accessLevel=6
Class=InternalJobDbImpl vessel_id=25335 method=setTuringMachineFlag accessLevel=6
Class=ServerDbImpl vessel_id=2460 method=setCurrentOperationToLater accessLevel=6
Class=InternalJobDbImpl vessel_id=25335 method=setTuringMachineFlag accessLevel=6
Class=InternalJobDbImpl vessel_id=25335 method=setFailedOperation accessLevel=6
Class=ServerDbImpl vessel_id=2460 method=nextJobOperation accessLevel=6
Class=ServerDbImpl vessel_id=2175 method=nextJobOperation accessLevel=6
Class=ClusterDbImpl vessel_id=2983 method=nextJobOperation accessLevel=6
Class=ServerPoolDbImpl vessel_id=2977 method=nextJobOperation accessLevel=6
Class=InternalJobDbImpl vessel_id=25335 method=setTuringMachineFlag accessLevel=6
Class=InternalJobDbImpl vessel_id=25335 method=setTuringMachineFlag accessLevel=6
Class=ServerDbImpl vessel_id=2175 method=nextJobOperation accessLevel=6
Class=InternalJobDbImpl vessel_id=25335 method=setTuringMachineFlag accessLevel=6
Class=InternalJobDbImpl vessel_id=25335 method=setTuringMachineFlag accessLevel=6
Class=ServerDbImpl vessel_id=2460 method=nextJobOperation accessLevel=6
Class=ServerDbImpl vessel_id=2460 method=nextJobOperation accessLevel=6
Class=ServerPoolDbImpl vessel_id=2977 method=nextJobOperation accessLevel=6
Completed Step: ROLLBACK
Job failed commit (internal) due to OVMAPI_4010E Attempt to send command: dispatch to server: old-oravm2.acbl.net failed. OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@172.16.2.52:8899/api/1 configure_server_for_cluster cluster:
name = c02eeebb9de3a8e7
heartbeat_mode = global
node_count = 2
node:
cluster = c02eeebb9de3a8e7
name = oravm1.acbl.net
number = 0
ip_address = 172.16.2.51
ip_port = 7777
node:
cluster = c02eeebb9de3a8e7
name = oravm2.acbl.net
number = 1
ip_address = 172.16.2.52
ip_port = 7777
heartbeat:
cluster = c02eeebb9de3a8e7
region = 0004FB000005000076F67B41034868F5
lun /dev/mapper/20017380004470086 0004fb000005000076f67b41034868f5 , Status: org.apache.xmlrpc.XmlRpcException: exceptions.OSError:[Errno 2] No such file or directory
Fri Mar 16 14:11:18 EDT 2012
Fri Mar 16 14:11:18 EDT 2012
com.oracle.ovm.mgr.api.exception.FailedOperationException: OVMAPI_4010E Attempt to send command: dispatch to server: old-oravm2.acbl.net failed. OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@172.16.2.52:8899/api/1 configure_server_for_cluster cluster:
name = c02eeebb9de3a8e7
heartbeat_mode = global
node_count = 2
node:
cluster = c02eeebb9de3a8e7
name = oravm1.acbl.net
number = 0
ip_address = 172.16.2.51
ip_port = 7777
node:
cluster = c02eeebb9de3a8e7
name = oravm2.acbl.net
number = 1
ip_address = 172.16.2.52
ip_port = 7777
heartbeat:
cluster = c02eeebb9de3a8e7
region = 0004FB000005000076F67B41034868F5
lun /dev/mapper/20017380004470086 0004fb000005000076f67b41034868f5 , Status: org.apache.xmlrpc.XmlRpcException: exceptions.OSError:[Errno 2] No such file or directory
Fri Mar 16 14:11:18 EDT 2012
Fri Mar 16 14:11:18 EDT 2012
at com.oracle.ovm.mgr.action.ActionEngine.sendCommandToServer(ActionEngine.java:475)
at com.oracle.ovm.mgr.action.ActionEngine.sendDispatchedServerCommand(ActionEngine.java:415)
at com.oracle.ovm.mgr.action.ActionEngine.sendServerCommand(ActionEngine.java:362)
at com.oracle.ovm.mgr.action.ClusterAction.configureServerForCluster(ClusterAction.java:82)
at com.oracle.ovm.mgr.op.physical.ServerClusterConfigure.configureCluster(ServerClusterConfigure.java:137)
at com.oracle.ovm.mgr.op.physical.ServerClusterConfigure.action(ServerClusterConfigure.java:56)
at com.oracle.ovm.mgr.api.job.JobEngine.operationActioner(JobEngine.java:193)
at com.oracle.ovm.mgr.api.job.JobEngine.objectActioner(JobEngine.java:264)
at com.oracle.ovm.mgr.api.job.InternalJobDbImpl.objectCommitter(InternalJobDbImpl.java:1090)
at sun.reflect.GeneratedMethodAccessor1008.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:247)
at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:207)
at com.oracle.odof.core.BasicWork.invokeMethod(BasicWork.java:136)
at com.oracle.odof.command.InvokeMethodCommand.process(InvokeMethodCommand.java:100)
at com.oracle.odof.core.BasicWork.processCommand(BasicWork.java:81)
at com.oracle.odof.core.TransactionManager.processCommand(TransactionManager.java:751)
at com.oracle.odof.core.WorkflowManager.processCommand(WorkflowManager.java:401)
at com.oracle.odof.core.WorkflowManager.processWork(WorkflowManager.java:459)
at com.oracle.odof.io.AbstractClient.run(AbstractClient.java:42)
at java.lang.Thread.run(Thread.java:662)
Caused by: com.oracle.ovm.mgr.api.exception.IllegalOperationException: OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@172.16.2.52:8899/api/1 configure_server_for_cluster cluster:
name = c02eeebb9de3a8e7
heartbeat_mode = global
node_count = 2
node:
cluster = c02eeebb9de3a8e7
name = oravm1.acbl.net
number = 0
ip_address = 172.16.2.51
ip_port = 7777
node:
cluster = c02eeebb9de3a8e7
name = oravm2.acbl.net
number = 1
ip_address = 172.16.2.52
ip_port = 7777
heartbeat:
cluster = c02eeebb9de3a8e7
region = 0004FB000005000076F67B41034868F5
lun /dev/mapper/20017380004470086 0004fb000005000076f67b41034868f5 , Status: org.apache.xmlrpc.XmlRpcException: exceptions.OSError:[Errno 2] No such file or directory
Fri Mar 16 14:11:18 EDT 2012
at com.oracle.ovm.mgr.action.ActionEngine.sendAction(ActionEngine.java:753)
at com.oracle.ovm.mgr.action.ActionEngine.sendCommandToServer(ActionEngine.java:471)
... 21 more
----------
End of Job
----------