I'm finding the root complex configuration different on the S7-2 than what I've had on previous T5's. My question has to do with setting up a primary and alternate service domains on the S7-2 / Solaris 11.3
It appears that the S7-2 configuration does not support (or maybe it doesn’t require) a primary and alternate service domain as outlined in Oracle’s Best Practices guide (http://www.oracle.com/technetwork/server-storage/vm/ovmsparc-best-practices-2334546.pdf) I also read in that document that direct I/O is not supported on the S7 systems, so perhaps that's why.
I have not found a Best Practices guide that describes setting up multiple service domains on the S7-2, other than it is still best practice to use them. The problem is the PCI buses that I see and it does not appear that I can separate them from the primary control/service Ldom. The Primary's Internal disks use pci_2 and all 4 network interfaces use pci_0
All I see are the pci_0 & pci_2 buses
# ldm list-io
NAME TYPE BUS DOMAIN STATUS
---- ---- --- ------ ------
pci_0 BUS pci_0 primary IOV
pci_2 BUS pci_2 primary IOV
<snipped>
All of the internal disks use the same initiator on the pci@302 (pci_2) bus when I follow their initiator back to the device name
AVAILABLE DISK SELECTIONS:
0. c0t5000CCA0803D0B2Cd0 <HGST-H101812SFSUN1.2T-A990-1.09TB>
/scsi_vhci/disk@g5000cca0803d0b2c
/dev/chassis/SYS/HDD0/disk
1. c0t5000CCA0803D0E50d0 <HGST-H101812SFSUN1.2T-A990-1.09TB> solaris
/scsi_vhci/disk@g5000cca0803d0e50
/dev/chassis/SYS/HDD1/disk
2. c0t5000CCA0803C71ECd0 <HGST-H101812SFSUN1.2T-A990-1.09TB> solaris
/scsi_vhci/disk@g5000cca0803c71ec
/dev/chassis/SYS/HDD2/disk
3. c0t5000CCA0803D0C40d0 <HGST-H101812SFSUN1.2T-A990-1.09TB> solaris
/scsi_vhci/disk@g5000cca0803d0c40
/dev/chassis/SYS/HDD3/disk
4. c1t0d0 <MICRON-eUSB DISK-1112 cyl 246 alt 0 hd 255 sec 63>
/pci@300/pci@1/pci@0/pci@2/usb@0/storage@1/disk@0,0
/dev/chassis/SYS/MB/EUSB-DISK/disk
root@morap2:~# mpathadm show lu /dev/rdsk/c0t5000CCA0803D0B2Cd0
Paths:
Initiator Port Name: w500605b00c305a70
Target Port Name: w5000cca0803d0b2d
root@morap2:~# mpathadm show lu /dev/rdsk/c0t5000CCA0803D0E50d0
Paths:
Initiator Port Name: w500605b00c305a70
Target Port Name: w5000cca0803d0e51
root@morap2:~# mpathadm show lu /dev/rdsk/c0t5000CCA0803C71ECd0
Paths:
Initiator Port Name: w500605b00c305a70
Target Port Name: w5000cca0803c71ed
root@morap2:~# mpathadm show lu /dev/rdsk/c0t5000CCA0803D0C40d0
Paths:
Initiator Port Name: w500605b00c305a70
Target Port Name: w5000cca0803d0c41
root@~# mpathadm show initiator-port w500605b00c305990
Initiator Port: w500605b00c305990
Transport Type: unknown
OS Device File: /devices/pci@302/pci@2/pci@0/pci@14/LSI,sas@0/iport@1
Initiator Port: w500605b00c305990
Transport Type: unknown
OS Device File: /devices/pci@302/pci@2/pci@0/pci@14/LSI,sas@0/iport@2
Initiator Port: w500605b00c305990
Transport Type: unknown
OS Device File: /devices/pci@302/pci@2/pci@0/pci@14/LSI,sas@0/iport@4
Initiator Port: w500605b00c305990
Transport Type: unknown
OS Device File: /devices/pci@302/pci@2/pci@0/pci@14/LSI,sas@0/iport@8
In addition, all of the network interfaces are on the same PCI Bus (pci_0):
root@morap2:~# ls -l /dev/i40e0
lrwxrwxrwx 1 root root 52 Apr 2 19:59 /dev/i40e0 -> ../devices/pci@300/pci@1/pci@0/pci@1/network@0:i40e0
root@morap2:~# ls -l /dev/i40e1
lrwxrwxrwx 1 root root 54 Apr 2 19:59 /dev/i40e1 -> ../devices/pci@300/pci@1/pci@0/pci@1/network@0,1:i40e1
root@morap2:~# ls -l /dev/i40e2
lrwxrwxrwx 1 root root 54 Apr 2 19:59 /dev/i40e2 -> ../devices/pci@300/pci@1/pci@0/pci@1/network@0,2:i40e2
root@morap2:~# ls -l /dev/i40e3
lrwxrwxrwx 1 root root 54 Apr 2 19:59 /dev/i40e3 -> ../devices/pci@300/pci@1/pci@0/pci@1/network@0,3:i40e3
I don’t see how to remove either the pci_0 or pci_2 bus from the primary ldom for use in an alternate service domain without affecting either the internal disks or the network interfaces on the Primary domain. I even tried to reduce the Primary service domain by removing pci_0 and also tried for pci_2 and was unsuccessful. Can anyone share their experience or documentation referring to best practice for setting up the service domains on the S7-2?
Are multiple service domains not supported or not required on the S7-2 platform? I have always setup my servers with multiple service domains (when the hardware supports it) so that I can update and perform maintenance on each service domain without taking the guest Ldoms down.