I'm running Oracle VM 3.2.4. I have added and removed physical disks from VMs many times without any problem. Today I cannot remove a physical disk.
No doubt it is related to what I did just before this: I resized the LUN. I do NOT have any Oracle VM storage plugins. My SAN is an IBM XIV, if it matters to you. Here's what I think I did:
Unmounted the filesystem in the VM.
Removed the partition on the LUN, since I was happy to completely wipe out the disk.
From the SAN, I resized the LUN from 16 to 32 GB.
I did a 'rescan physical disks' on each Oracle VM server, in the fond hope that he would see the new LUN size, and things would be okay. (But I've NEVER had the 'rescan physical disks' from OVMM do anything. So, I did what I always do when I add or remove a LUN...)
On each Oracle VM server, I did a "rescan-scsi-bus.sh". This has always caused the OVMM to update his list of physical disks. But this time, nothing happened.
Okay, I thought, I'll remove the physical disk from the VM, remove it from OVMM, un-present it from the SAN, and add it all back again.
I edit the VM, and removed the physical disk, like I've done many times, and hit OK. It thinks for many seconds, and this FAILs. I'll paste in the details, after I'm finished writing here.
So, I can't remove the LUN from the VM, and I can't remove the LUN from the "FibreChannel Volume Group", because it's mapped into a VM.
I have restarted OVMM, but that didn't do anything.
Any ideas??
Okay, here are the error details:
Job Construction Phase
----------------------
Job ID: 1392820419550
begin()
Appended operation 'Virtual Machine Cfg File Configure' to object 'cfgFile_0004fb0000060000d69fd57f4fdd0830'.
commit()
Completed Step: COMMIT
Objects and Operations
----------------------
Object (IN_USE): [VirtualMachine] 0004fb0000060000d69fd57f4fdd0830 (jidldb01)
Object (IN_USE): [StorageElement] 0004fb0000180000290d6bf2dac7f445 (jidldb01_tldev)
Object (IN_USE): [Vnic] 0004fb0000070000105a685f23193f8f (00:21:f6:00:00:00)
Object (DELETING): [VmDiskMapping] 0004fb0000130000449523548475f2e0
Object (IN_USE): [VirtualMachineCfgFile] cfgFile_0004fb0000060000d69fd57f4fdd0830
Operation: Virtual Machine Cfg File Configure
Job Running Phase at 2014-02-19 09:33:39,550
----------------------------------------------
Job Participants: [26:bc:94:d8:86:91:e1:11:81:c8:e4:1f:13:eb:d8:d2 (oravm1.acbl.net)]
Actioner
--------
09:33:40,092: Starting operation 'Virtual Machine Cfg File Configure' on object 'cfgFile_0004fb0000060000d69fd57f4fdd0830'
Job Internal Error (Operation)com.oracle.ovm.mgr.api.exception.FailedOperationException: OVMAPI_4010E Attempt to send command: dispatch to server: oravm1.acbl.net failed. OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@172.16.2.51:8899/api/3 configure_vm 0004fb0000030000053541e3be4465ef 0004fb0000060000d69fd57f4fdd0830, Status: org.apache.xmlrpc.XmlRpcException: exceptions.RuntimeError:disk:setup_error:Command: ['xm', 'block-detach', '0004fb0000060000d69fd57f4fdd0830', '51776'] failed (1): stderr: Error: Device 51776 (vbd) could not be disconnected.
stdout: Usage: xm block-detach [-f|--force]
Destroy a domain's virtual block device.
[Wed Feb 19 09:35:20 EST 2014] [Wed Feb 19 09:35:20 EST 2014]
at com.oracle.ovm.mgr.action.ActionEngine.sendCommandToServer(ActionEngine.java:512)
at com.oracle.ovm.mgr.action.ActionEngine.sendDispatchedServerCommand(ActionEngine.java:449)
at com.oracle.ovm.mgr.action.ActionEngine.sendServerCommand(ActionEngine.java:383)
at com.oracle.ovm.mgr.action.VirtualMachineCfgFileAction.configureVirtualMachineCfgFile(VirtualMachineCfgFileAction.java:152)
at com.oracle.ovm.mgr.op.virtual.VirtualMachineCfgFileConfigure.configureVirtualMachineCfgFile(VirtualMachineCfgFileConfigure.java:82)
at com.oracle.ovm.mgr.op.virtual.VirtualMachineCfgFileConfigure.action(VirtualMachineCfgFileConfigure.java:55)
at com.oracle.ovm.mgr.api.collectable.ManagedObjectDbImpl.executeCurrentJobOperationAction(ManagedObjectDbImpl.java:1156)
at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:356)
at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:333)
at com.oracle.odof.core.storage.Transaction.invokeMethod(Transaction.java:865)
at com.oracle.odof.core.Exchange.invokeMethod(Exchange.java:244)
at com.oracle.ovm.mgr.api.virtual.VirtualMachineCfgFileProxy.executeCurrentJobOperationAction(Unknown Source)
at com.oracle.ovm.mgr.api.job.JobEngine.operationActioner(JobEngine.java:230)
at com.oracle.ovm.mgr.api.job.JobEngine.objectActioner(JobEngine.java:322)
at com.oracle.ovm.mgr.api.job.InternalJobDbImpl.objectCommitter(InternalJobDbImpl.java:1357)
at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:356)
at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:333)
at com.oracle.odof.core.BasicWork.invokeMethod(BasicWork.java:106)
at com.oracle.odof.command.InvokeMethodCommand.process(InvokeMethodCommand.java:92)
at com.oracle.odof.core.BasicWork.processCommand(BasicWork.java:81)
at com.oracle.odof.core.TransactionManager.processCommand(TransactionManager.java:752)
at com.oracle.odof.core.WorkflowManager.processCommand(WorkflowManager.java:467)
at com.oracle.odof.core.WorkflowManager.processWork(WorkflowManager.java:525)
at com.oracle.odof.io.AbstractClient.run(AbstractClient.java:42)
at java.lang.Thread.run(Thread.java:662)
Caused by: com.oracle.ovm.mgr.api.exception.IllegalOperationException: OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@172.16.2.51:8899/api/3 configure_vm 0004fb0000030000053541e3be4465ef 0004fb0000060000d69fd57f4fdd0830, Status: org.apache.xmlrpc.XmlRpcException: exceptions.RuntimeError:disk:setup_error:Command: ['xm', 'block-detach', '0004fb0000060000d69fd57f4fdd0830', '51776'] failed (1): stderr: Error: Device 51776 (vbd) could not be disconnected.
stdout: Usage: xm block-detach [-f|--force]
Destroy a domain's virtual block device.
[Wed Feb 19 09:35:20 EST 2014]
at com.oracle.ovm.mgr.action.ActionEngine.sendAction(ActionEngine.java:803)
at com.oracle.ovm.mgr.action.ActionEngine.sendCommandToServer(ActionEngine.java:508)
... 32 more
FailedOperationCleanup
----------
Starting failed operation 'Virtual Machine Cfg File Configure' cleanup on object 'cfgFile_0004fb0000060000d69fd57f4fdd0830'
Complete rollback operation 'Virtual Machine Cfg File Configure' cleanup on object 'cfgFile_0004fb0000060000d69fd57f4fdd0830'
Rollbacker
----------
09:35:20,799: Starting rollbacker...
Executing rollback operation 'Virtual Machine Cfg File Configure' on object 'cfgFile_0004fb0000060000d69fd57f4fdd0830'
Complete rollback operation 'Virtual Machine Cfg File Configure' completed with direction=DONE
09:35:20,817: Rollbacker completed...
Objects To Be Rolled Back
-------------------------
Object (IN_USE): [StorageElement] 0004fb0000180000290d6bf2dac7f445 (jidldb01_tldev)
Object (IN_USE): [VirtualMachine] 0004fb0000060000d69fd57f4fdd0830 (jidldb01)
Object (IN_USE): [Vnic] 0004fb0000070000105a685f23193f8f (00:21:f6:00:00:00)
Object (DELETING): [VmDiskMapping] 0004fb0000130000449523548475f2e0
Object (IN_USE): [VirtualMachineCfgFile] cfgFile_0004fb0000060000d69fd57f4fdd0830
Write Methods Invoked
-------------------
09:33:39,762 class="InternalJobDbImpl" vessel_id=204501 method=addTransactionIdentifier accessLevel=6 owningTx=1392820419762
09:33:39,766 class="VnicDbImpl" vessel_id=1656 method=setEthernetNetwork accessLevel=6 owningTx=1392820419762
09:33:39,767 class="VirtualMachineDbImpl" vessel_id=98707 method=recordCfgChange accessLevel=6 owningTx=1392820419762
09:33:39,767 class="VirtualMachineCfgFileDbImpl" vessel_id=149655 method=setAsset accessLevel=6 owningTx=1392820419762
09:33:39,768 class="VirtualMachineCfgFileDbImpl" vessel_id=149655 method=configure accessLevel=6 owningTx=1392820419762
09:33:39,788 class="VirtualMachineDbImpl" vessel_id=98707 method=removeVmDiskMapping accessLevel=6 owningTx=1392820419762
09:33:39,789 class="StorageElementDbImpl" vessel_id=203407 method=removeVmDiskMapping accessLevel=6 owningTx=1392820419762
09:33:39,789 class="VmDiskMappingDbImpl" vessel_id=203762 method=setVirtualMachineDisk accessLevel=6 owningTx=1392820419762
09:33:39,790 class="VirtualMachineDbImpl" vessel_id=98707 method=removeVmDiskMapping accessLevel=6 owningTx=1392820419762
09:33:39,790 class="VirtualMachineCfgFileDbImpl" vessel_id=149655 method=setAsset accessLevel=6 owningTx=1392820419762
09:33:39,790 class="VirtualMachineCfgFileDbImpl" vessel_id=149655 method=configure accessLevel=6 owningTx=1392820419762
09:33:39,791 class="VmDiskMappingDbImpl" vessel_id=203762 method=setVirtualMachine accessLevel=6 owningTx=1392820419762
09:33:39,808 class="ModelManagerDbImpl" vessel_id=9 method=deleteObject accessLevel=5 owningTx=-1
09:33:39,809 class="VmDiskMappingDbImpl" vessel_id=203762 method=setLifecycleState accessLevel=6 owningTx=1392820419762
09:33:39,809 class="VmDiskMappingDbImpl" vessel_id=203762 method=setRollbackLifecycleState accessLevel=6 owningTx=1392820419762
09:33:39,809 class="VmDiskMappingDbImpl" vessel_id=203762 method=onPersistableClean accessLevel=6 owningTx=1392820419762
09:33:39,809 class="VirtualMachineCfgFileDbImpl" vessel_id=149655 method=setAsset accessLevel=6 owningTx=1392820419762
09:33:39,809 class="VirtualMachineCfgFileDbImpl" vessel_id=149655 method=configure accessLevel=6 owningTx=1392820419762
09:33:39,940 class="InternalJobDbImpl" vessel_id=204501 method=setCompletedStep accessLevel=6 owningTx=1392820419762
09:33:39,945 class="InternalJobDbImpl" vessel_id=204501 method=setAssociatedHandles accessLevel=6 owningTx=1392820419762
09:33:40,139 class="VirtualMachineCfgFileDbImpl" vessel_id=149655 method=setChecksum accessLevel=6 owningTx=1392820419762
09:35:20,654 class="VirtualMachineCfgFileDbImpl" vessel_id=149655 method=nextJobOperation accessLevel=6 owningTx=1392820419762
09:35:20,654 class="InternalJobDbImpl" vessel_id=204501 method=setFailedOperation accessLevel=6 owningTx=1392820419762
09:35:20,803 class="VirtualMachineDbImpl" vessel_id=98707 method=nextJobOperation accessLevel=6 owningTx=1392820419762
09:35:20,803 class="StorageElementDbImpl" vessel_id=203407 method=nextJobOperation accessLevel=6 owningTx=1392820419762
09:35:20,804 class="VnicDbImpl" vessel_id=1656 method=nextJobOperation accessLevel=6 owningTx=1392820419762
09:35:20,804 class="VmDiskMappingDbImpl" vessel_id=203762 method=nextJobOperation accessLevel=6 owningTx=1392820419762
09:35:20,804 class="VirtualMachineCfgFileDbImpl" vessel_id=149655 method=nextJobOperation accessLevel=6 owningTx=1392820419762
09:35:20,817 class="VirtualMachineCfgFileDbImpl" vessel_id=149655 method=nextJobOperation accessLevel=6 owningTx=1392820419762
Completed Step: ROLLBACK
Job failed commit (internal) due to OVMAPI_4010E Attempt to send command: dispatch to server: oravm1.acbl.net failed. OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@172.16.2.51:8899/api/3 configure_vm 0004fb0000030000053541e3be4465ef 0004fb0000060000d69fd57f4fdd0830, Status: org.apache.xmlrpc.XmlRpcException: exceptions.RuntimeError:disk:setup_error:Command: ['xm', 'block-detach', '0004fb0000060000d69fd57f4fdd0830', '51776'] failed (1): stderr: Error: Device 51776 (vbd) could not be disconnected.
stdout: Usage: xm block-detach [-f|--force]
Destroy a domain's virtual block device.
[Wed Feb 19 09:35:20 EST 2014] [Wed Feb 19 09:35:20 EST 2014]
com.oracle.ovm.mgr.api.exception.FailedOperationException: OVMAPI_4010E Attempt to send command: dispatch to server: oravm1.acbl.net failed. OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@172.16.2.51:8899/api/3 configure_vm 0004fb0000030000053541e3be4465ef 0004fb0000060000d69fd57f4fdd0830, Status: org.apache.xmlrpc.XmlRpcException: exceptions.RuntimeError:disk:setup_error:Command: ['xm', 'block-detach', '0004fb0000060000d69fd57f4fdd0830', '51776'] failed (1): stderr: Error: Device 51776 (vbd) could not be disconnected.
stdout: Usage: xm block-detach [-f|--force]
Destroy a domain's virtual block device.
[Wed Feb 19 09:35:20 EST 2014] [Wed Feb 19 09:35:20 EST 2014]
at com.oracle.ovm.mgr.action.ActionEngine.sendCommandToServer(ActionEngine.java:512)
at com.oracle.ovm.mgr.action.ActionEngine.sendDispatchedServerCommand(ActionEngine.java:449)
at com.oracle.ovm.mgr.action.ActionEngine.sendServerCommand(ActionEngine.java:383)
at com.oracle.ovm.mgr.action.VirtualMachineCfgFileAction.configureVirtualMachineCfgFile(VirtualMachineCfgFileAction.java:152)
at com.oracle.ovm.mgr.op.virtual.VirtualMachineCfgFileConfigure.configureVirtualMachineCfgFile(VirtualMachineCfgFileConfigure.java:82)
at com.oracle.ovm.mgr.op.virtual.VirtualMachineCfgFileConfigure.action(VirtualMachineCfgFileConfigure.java:55)
at com.oracle.ovm.mgr.api.collectable.ManagedObjectDbImpl.executeCurrentJobOperationAction(ManagedObjectDbImpl.java:1156)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:356)
at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:333)
at com.oracle.odof.core.storage.Transaction.invokeMethod(Transaction.java:865)
at com.oracle.odof.core.Exchange.invokeMethod(Exchange.java:244)
at com.oracle.ovm.mgr.api.virtual.VirtualMachineCfgFileProxy.executeCurrentJobOperationAction(Unknown Source)
at com.oracle.ovm.mgr.api.job.JobEngine.operationActioner(JobEngine.java:230)
at com.oracle.ovm.mgr.api.job.JobEngine.objectActioner(JobEngine.java:322)
at com.oracle.ovm.mgr.api.job.InternalJobDbImpl.objectCommitter(InternalJobDbImpl.java:1357)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:356)
at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:333)
at com.oracle.odof.core.BasicWork.invokeMethod(BasicWork.java:106)
at com.oracle.odof.command.InvokeMethodCommand.process(InvokeMethodCommand.java:92)
at com.oracle.odof.core.BasicWork.processCommand(BasicWork.java:81)
at com.oracle.odof.core.TransactionManager.processCommand(TransactionManager.java:752)
at com.oracle.odof.core.WorkflowManager.processCommand(WorkflowManager.java:467)
at com.oracle.odof.core.WorkflowManager.processWork(WorkflowManager.java:525)
at com.oracle.odof.io.AbstractClient.run(AbstractClient.java:42)
at java.lang.Thread.run(Thread.java:662)
Caused by: com.oracle.ovm.mgr.api.exception.IllegalOperationException: OVMAPI_4004E Server Failed Command: dispatch https://?uname?:?pwd?@172.16.2.51:8899/api/3 configure_vm 0004fb0000030000053541e3be4465ef 0004fb0000060000d69fd57f4fdd0830, Status: org.apache.xmlrpc.XmlRpcException: exceptions.RuntimeError:disk:setup_error:Command: ['xm', 'block-detach', '0004fb0000060000d69fd57f4fdd0830', '51776'] failed (1): stderr: Error: Device 51776 (vbd) could not be disconnected.
stdout: Usage: xm block-detach [-f|--force]
Destroy a domain's virtual block device.
[Wed Feb 19 09:35:20 EST 2014]
at com.oracle.ovm.mgr.action.ActionEngine.sendAction(ActionEngine.java:803)
at com.oracle.ovm.mgr.action.ActionEngine.sendCommandToServer(ActionEngine.java:508)
... 32 more
----------
End of Job
----------