Hardware

Dear All,

I am having problem with my Sun Fire V890. The system had rebooted more than 2 times after having some panic in the system. I had checked all the log file and collected information and just want to know what problem it is having.
I am sending only the selected logs, and please feel free to ask if you want some more information regarding this.
Here with prtdiag �v we can see Brd B is having CPU 1 and 17. When first time the system was rebooted it showed panic with CPU=11 (17) as can see below from file act.0 in /var/crash/hostname. And when the system rebooted second time it showed problem with CPU=1 (1) as can see from act.1 in /var/crash/hostname. And after seeing message file in /var/adm we can see that it is showing error with PCI Bus. Please advise what might be the problem as the system is very critical and is in running state. The system is working fine except sudden reboot time to time.

I will be very thankful if I get you reply. And thanks in advance from my side.

Best regards,
Bikash

# prtdiag �v

Run E$ CPU CPU
Brd CPU MHz MB Impl. Mask
--- ----- ---- ---- ------- ----
A 0, 16 1350 16.0 US-IV 3.1
B 1, 17 1350 16.0 US-IV 3.1
A 2, 18 1350 16.0 US-IV 3.1
B 3, 19 1350 16.0 US-IV 3.1
�.
Output Truncated

For First time rebooted
# cat /var/crash/host name of the server/act.0

###
CPU 0x11 is RUNNING READY CONFIGURED ENABLED ### CPU=11 (17) ###
###

PANIC occurred on this CPU

cpu addr 0x3000111c000
running thread addr 0x2a100167cc0
pause thread addr 0x2a1005f7cc0
dispatched thread addr 0x2a100167cc0

interrupt stack is 0x2a1005fff50
interrupt thread list starts at 0x2a100607cc0
UltraSPARC-IV (portid 1 impl 0x18 ver 0x31 clock 1350 MHz)
in_prom 0


PANIC occurred on this thread

***
Kernel thread:

thread addr 0x2a100167cc0, proc addr 0x1835c00, lwp addr 0x0
Thread bound to cpu id 0x11

t_state is 0x4 - TS_ONPROC

Scheduling info:
t_pri is 0x63, t_epri is 0x0, t_cid is 0x0
scheduling class is: SYS
t_disp_time: is 0x15538cd, 0t22362317
last switched: 36 secs ago, 8 ticks before panic on cpu 0x11

pc is 0x105b8b8, sp is 0x1842630, t_stk 0x2a100167ad0

Stack trace is:

unix: setjmp ()
unix: panicsys+0x48 (0x11f14e8,0x2a100166d38,0x0,0x0,0xaf717fffffffffff,0x508e800000000000)
unix: vpanic ()
genunix: fm_panic+0x30 (0x11f14e8,0x2a100166d60)
SUNW,UltraSPARC-IV: cpu_deferred_error+0x56c (0x2a100167800)
unix: priv_rtt
INTERRUPT FRAME: 2a100167800

pc=0x103f954 (unix: i_ddi_swap_get32+4), npc=0x13e35d8 (mpt: mpt_watch+8c)

unix: i_ddi_swap_get32 (0x60018580f40,0x29ebfa74000)
mpt: mpt_watch+0x84 (?,?,0x13e354c,?,0x3000111c000)
genunix: callout_execute+0x98 (0x6000184a000)
genunix: taskq_thread+0x1a4 (0x6000180d298,0x0)
unix: thread_start+0x4 (0x6000180d298,0x0,0x0,0x0,0x0,0x0)

idlethread is 0x2a100597cc0

For Second time rebooted
# cat /var/crash/hostname of the Server/ act.1

###
CPU 0x1 is RUNNING READY CONFIGURED ENABLED ### CPU=1 (1) ###
###

PANIC occurred on this CPU

cpu addr 0x30000ec8000
running thread addr 0x2a1003ddcc0
pause thread addr 0x2a100459cc0
dispatched thread addr 0x2a1003ddcc0

interrupt stack is 0x2a1004698d0
interrupt thread list starts at 0x2a100423cc0
interrupt levels active is 0x0, at time of panic 0x4000
UltraSPARC-IV (portid 1 impl 0x18 ver 0x31 clock 1350 MHz)
in_prom 0


PANIC occurred on this thread

***
Kernel thread:

thread addr 0x2a1003ddcc0, proc addr 0x1835c00, lwp addr 0x0
Thread bound to cpu id 0x1

t_state is 0x4 - TS_ONPROC

Scheduling info:
t_pri is 0xffffffffffffffff, t_epri is 0x0, t_cid is 0x0
scheduling class is: SYS
t_disp_time: is 0x2407afc, 0t37780220
last switched: 34 secs ago, 2 ticks before panic on cpu 0x1

pc is 0x105b8b8, sp is 0x1842630, t_stk 0x2a1003ddad0

Stack trace is:

unix: setjmp ()
unix: panicsys+0x48 (0x12ec148,0x2a100461ef8,0x0,0x2fb,0x368,0x137)
unix: vpanic ()
genunix: fm_panic+0x30 (0x12ec148,0x30000159780,0x3)
pcisch: pbm_error_intr+0x158 (0x600018901c0)
unix: current_thread+0x140 (0x0,0x0,0x0,0xffffffffffffffff,0x3000026fab8,0x10)
unix: splhi ()
unix: disp_getwork+0x38 (0x30000ec8000)
unix: idle+0xd4 (0x0,0x0)
unix: thread_start+0x4 (0x0,0x0,0x0,0x0,0x0,0x0)

idlethread is 0x2a1003ddcc0
CPU is running IDLE thread


# cat /var/adm/messages

Feb 19 09:53:33 abs-master genunix: [ID 843051 kern.info] NOTICE: SUNW-MSG-ID: SUNOS-8000-0G, TYPE: Error, VER: 1, SEVERITY: Major
Feb 19 09:53:34 abs-master unix: [ID 836849 kern.notice]
Feb 19 09:53:34 abs-master ^Mpanic[cpu1]/thread=2a1003ddcc0:
Feb 19 09:53:34 abs-master unix: [ID 870354 kern.notice] pcisch-3: Fatal PCI bus error(s)
Feb 19 09:53:34 abs-master unix: [ID 100000 kern.notice]
Feb 19 09:53:34 abs-master genunix: [ID 723222 kern.notice] 000002a100461e70 pcisch:pbm_error_intr+158 (600018901c0, 12ec000, 30000135188, 30000135188, 0, 60001933a40)
Feb 19 09:53:34 abs-master genunix: [ID 179002 kern.notice] %l0-3: 000003000026fab8 0000000000000000 00000000018d2c00 00000000018d2c00
Feb 19 09:53:34 abs-master %l4-7: 0000000000000001 00000000018d2c00 0000030000159780 0000000000000003
Feb 19 09:53:35 abs-master genunix: [ID 723222 kern.notice] 000002a100461f50 unix:current_thread+140 (0, 0, 0, ffffffffffffffff, 3000026fab8, 10)
Feb 19 09:53:35 abs-master genunix: [ID 179002 kern.notice] %l0-3: 00000000010076b4 000002a1003dd021 000000000000000e 0000000000000272
Feb 19 09:53:35 abs-master %l4-7: 0000030000ec8510 0000036fbc9e7ff1 000000000000000b 000002a1003dd8d0
Feb 19 09:53:35 abs-master genunix: [ID 723222 kern.notice] 000002a1003dd970 unix:disp_getwork+38 (30000ec8000, 1842ce0, 3000111c000, 1b, 0, 0)
Feb 19 09:53:35 abs-master genunix: [ID 179002 kern.notice] %l0-3: 000003000026fab8 ffffffffffffffff 000003000026fa80 000003000026fb80
Feb 19 09:53:35 abs-master %l4-7: 000003000026fb80 0000000000000000 000003000026fb80 0000000000000000
Feb 19 09:53:35 abs-master genunix: [ID 723222 kern.notice] 000002a1003dda20 unix:idle+d4 (1813000, 0, 30000ec8000, ffffffffffffffff, 3, 1812000)
Feb 19 09:53:35 abs-master genunix: [ID 179002 kern.notice] %l0-3: 000006000194e798 000000000000001b 0000000000000000 ffffffffffffffff
Feb 19 09:53:35 abs-master %l4-7: 000006000194e798 ffffffffffffffff 0000000001842ce0 00000000010532c4
Feb 19 09:53:35 abs-master unix: [ID 100000 kern.notice]
Feb 19 09:53:35 abs-master genunix: [ID 672855 kern.notice] syncing file systems...
Feb 19 09:53:36 abs-master genunix: [ID 733762 kern.notice] 1
Feb 19 09:53:38 abs-master genunix: [ID 904073 kern.notice] done
Feb 19 09:53:39 abs-master genunix: [ID 111219 kern.notice] dumping to /dev/dsk/c1t0d0s1, offset 429326336, content: kernel
Feb 19 09:54:08 abs-master genunix: [ID 409368 kern.notice] ^M100% done: 121756 pages dumped, compression ratio 2.49,
Feb 19 09:54:08 abs-master genunix: [ID 851671 kern.notice] dump succeeded
Feb 19 09:55:21 abs-master genunix: [ID 540533 kern.notice] ^MSunOS Release 5.10 Version Generic_118822-26 64-bit
Feb 19 09:55:21 abs-master genunix: [ID 943906 kern.notice] Copyright 1983-2005 Sun Microsystems, Inc. All rights reserved.
Feb 19 09:55:21 abs-master Use is subject to license terms.
Feb 19 09:55:21 abs-master genunix: [ID 678236 kern.info] Ethernet address = 0:3:ba:ce:85:c1
Feb 19 09:55:21 abs-master unix: [ID 673563 kern.info] NOTICE: Kernel Cage is ENABLED
Feb 19 09:55:21 abs-master unix: [ID 389951 kern.info] mem = 16777216K (0x400000000)
Feb 19 09:55:21 abs-master unix: [ID 930857 kern.info] avail mem = 16808648704
Feb 19 09:55:21 abs-master rootnex: [ID 466748 kern.info] root nexus = Sun Fire V890
Feb 19 09:55:21 abs-master rootnex: [ID 349649 kern.info] pseudo0 at root
Feb 19 09:55:21 abs-master genunix: [ID 936769 kern.info] pseudo0 is /pseudo
Feb 19 09:55:21 abs-master rootnex: [ID 349649 kern.info] scsi_vhci0 at root
Feb 19 09:55:21 abs-master genunix: [ID 936769 kern.info] scsi_vhci0 is /scsi_vhci
Feb 19 09:55:21 abs-master rootnex: [ID 349649 kern.info] pcisch1 at root: SAFARI 0x8 0x600000
Feb 19 09:55:21 abs-master genunix: [ID 936769 kern.info] pcisch1 is /pci@8,600000
Feb 19 09:55:21 abs-master qlc: [ID 308162 kern.info] Qlogic qlc FCA Driver v20051013-2.08 (0)
Feb 19 09:55:21 abs-master qlc: [ID 630585 kern.info] NOTICE: Qlogic qlc(0): Loop ONLINE
Feb 19 09:55:21 abs-master qlc: [ID 147575 kern.info] NOTICE: ql_power(0): qlc is powered ON
Feb 19 09:55:21 abs-master qlc: [ID 694252 kern.info] NOTICE: qlc(0): Firmware version 2.1.143
Feb 19 09:55:21 abs-master pcisch: [ID 370704 kern.info] PCI-device: SUNW,qlc@2, qlc0
Feb 19 09:55:21 abs-master genunix: [ID 936769 kern.info] qlc0 is /pci@8,600000/SUNW,qlc@2
Feb 19 09:55:21 abs-master genunix: [ID 936769 kern.info] fp0 is /pci@8,600000/SUNW,qlc@2/fp@0,0
Feb 19 09:55:22 abs-master qlc: [ID 630585 kern.info] NOTICE: Qlogic qlc(0): Loop OFFLINE
Feb 19 09:55:22 abs-master qlc: [ID 630585 kern.info] NOTICE: Qlogic qlc(0): Loop ONLINE
Feb 19 09:55:27 abs-master scsi: [ID 799468 kern.info] ssd3 at fp0: name w500000e01144e181,0, bus address e8
Feb 19 09:55:27 abs-master genunix: [ID 936769 kern.info] ssd3 is /pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w500000e01144e181,0
Feb 19 09:55:27 abs-master genunix: [ID 408114 kern.info] /pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w500000e01144e181,0 (ssd3) online
Feb 19 09:55:27 abs-master scsi: [ID 799468 kern.info] ssd0 at fp0: name w500000e0114514d1,0, bus address e2
Feb 19 09:55:27 abs-master genunix: [ID 936769 kern.info] ssd0 is /pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w500000e0114514d1,0
Feb 19 09:55:27 abs-master genunix: [ID 408114 kern.info] /pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w500000e0114514d1,0 (ssd0) online
Feb 19 09:55:27 abs-master scsi: [ID 799468 kern.info] ssd1 at fp0: name w500000e01144c581,0, bus address e4
Feb 19 09:55:27 abs-master genunix: [ID 936769 kern.info] ssd1 is /pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w500000e01144c581,0
Feb 19 09:55:27 abs-master genunix: [ID 408114 kern.info] /pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w500000e01144c581,0 (ssd1) online
Feb 19 09:55:27 abs-master scsi: [ID 799468 kern.info] ssd2 at fp0: name w500000e01144a6c1,0, bus address e1
Feb 19 09:55:27 abs-master genunix: [ID 936769 kern.info] ssd2 is /pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w500000e01144a6c1,0
Feb 19 09:55:27 abs-master genunix: [ID 408114 kern.info] /pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w500000e01144a6c1,0 (ssd2) online
Feb 19 09:55:28 abs-master scsi: [ID 799468 kern.info] ssd4 at fp0: name w500000e01144c511,0, bus address e0
Feb 19 09:55:28 abs-master genunix: [ID 936769 kern.info] ssd4 is /pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w500000e01144c511,0
Feb 19 09:55:28 abs-master genunix: [ID 408114 kern.info] /pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w500000e01144c511,0 (ssd4) online
Feb 19 09:55:28 abs-master scsi: [ID 799468 kern.info] ssd5 at fp0: name w500000e01144c5f1,0, bus address ef
Feb 19 09:55:28 abs-master genunix: [ID 936769 kern.info] ssd5 is /pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w500000e01144c5f1,0
Feb 19 09:55:28 abs-master genunix: [ID 408114 kern.info] /pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w500000e01144c5f1,0 (ssd5) online
Feb 19 09:55:28 abs-master scsi: [ID 243001 kern.info] /pci@8,600000/SUNW,qlc@2/fp@0,0 (fcp0):
Feb 19 09:55:28 abs-master ndi_devi_online: failed for scsiclass,0d: target=dc lun=0 ffffffff
Feb 19 09:55:29 abs-master swapgeneric: [ID 308332 kern.info] root on /pci@8,600000/SUNW,qlc@2/fp@0,0/disk@w500000e01144c5f1,0:a fstype ufs
Feb 19 09:55:29 abs-master rootnex: [ID 349649 kern.info] pcisch2 at root: SAFARI 0x9 0x700000
Feb 19 09:55:29 abs-master genunix: [ID 936769 kern.info] pcisch2 is /pci@9,700000
Feb 19 09:55:29 abs-master pcisch: [ID 370704 kern.info] PCI-device: ebus@1, ebus0
Feb 19 09:55:29 abs-master genunix: [ID 936769 kern.info] ebus0 is /pci@9,700000/ebus@1
Feb 19 09:55:30 abs-master ebus: [ID 521012 kern.info] todds12870 at ebus0: offset 1,300070
Feb 19 09:55:30 abs-master genunix: [ID 936769 kern.info] todds12870 is /pci@9,700000/ebus@1/rtc@1,300070
Feb 19 09:55:30 abs-master rootnex: [ID 349649 kern.info] mc-us30 at root: SAFARI 0x0 0x400000 ...
Feb 19 09:55:30 abs-master genunix: [ID 936769 kern.info] mc-us30 is /memory-controller@0,400000
Feb 19 09:55:30 abs-master rootnex: [ID 349649 kern.info] mc-us31 at root: SAFARI 0x1 0x400000 ...
Feb 19 09:55:30 abs-master genunix: [ID 936769 kern.info] mc-us31 is /memory-controller@1,400000
Feb 19 09:55:30 abs-master rootnex: [ID 349649 kern.info] mc-us32 at root: SAFARI 0x2 0x400000 ...
Feb 19 09:55:30 abs-master genunix: [ID 936769 kern.info] mc-us32 is /memory-controller@2,400000
Feb 19 09:55:30 abs-master rootnex: [ID 349649 kern.info] mc-us33 at root: SAFARI 0x3 0x400000 ...
Feb 19 09:55:30 abs-master genunix: [ID 936769 kern.info] mc-us33 is /memory-controller@3,400000
Feb 19 09:55:31 abs-master rootnex: [ID 349649 kern.info] pcisch0 at root: SAFARI 0x8 0x700000
Feb 19 09:55:31 abs-master genunix: [ID 936769 kern.info] pcisch0 is /pci@8,700000
Feb 19 09:55:31 abs-master rootnex: [ID 349649 kern.info] pcisch3 at root: SAFARI 0x9 0x600000
Feb 19 09:55:31 abs-master genunix: [ID 936769 kern.info] pcisch3 is /pci@9,600000
Feb 19 09:55:32 abs-master pseudo: [ID 129642 kern.info] pseudo-device: dld0
Feb 19 09:55:32 abs-master genunix: [ID 936769 kern.info] dld0 is /pseudo/dld@0
Feb 19 09:55:32 abs-master ebus: [ID 521012 kern.info] se0 at ebus0: offset 1,400000
Feb 19 09:55:32 abs-master genunix: [ID 936769 kern.info] se0 is /pci@9,700000/ebus@1/serial@1,400000
Feb 19 09:55:32 abs-master pcisch: [ID 370704 kern.info] PCI-device: usb@1,3, ohci0
Feb 19 09:55:32 abs-master genunix: [ID 936769 kern.info] ohci0 is /pci@9,700000/usb@1,3
Feb 19 09:55:32 abs-master unix: [ID 950921 kern.info] cpu3: UltraSPARC-IV (portid 3 impl 0x18 ver 0x31 clock 1350 MHz)
Feb 19 09:55:32 abs-master unix: [ID 950921 kern.info] cpu0: UltraSPARC-IV (portid 0 impl 0x18 ver 0x31 clock 1350 MHz)
Feb 19 09:55:32 abs-master unix: [ID 557827 kern.info] cpu0 initialization complete - online
Feb 19 09:55:32 abs-master unix: [ID 950921 kern.info] cpu1: UltraSPARC-IV (portid 1 impl 0x18 ver 0x31 clock 1350 MHz)
Feb 19 09:55:32 abs-master unix: [ID 557827 kern.info] cpu1 initialization complete - online
Feb 19 09:55:32 abs-master unix: [ID 950921 kern.info] cpu2: UltraSPARC-IV (portid 2 impl 0x18 ver 0x31 clock 1350 MHz)
Feb 19 09:55:32 abs-master unix: [ID 557827 kern.info] cpu2 initialization complete - online
Feb 19 09:55:32 abs-master unix: [ID 950921 kern.info] cpu16: UltraSPARC-IV (portid 0 impl 0x18 ver 0x31 clock 1350 MHz)
Feb 19 09:55:32 abs-master unix: [ID 557827 kern.info] cpu16 initialization complete - online
Feb 19 09:55:32 abs-master unix: [ID 950921 kern.info] cpu17: UltraSPARC-IV (portid 1 impl 0x18 ver 0x31 clock 1350 MHz)
Feb 19 09:55:32 abs-master unix: [ID 557827 kern.info] cpu17 initialization complete - online
Feb 19 09:55:32 abs-master unix: [ID 950921 kern.info] cpu18: UltraSPARC-IV (portid 2 impl 0x18 ver 0x31 clock 1350 MHz)
Feb 19 09:55:32 abs-master unix: [ID 557827 kern.info] cpu18 initialization complete - online
Feb 19 09:55:32 abs-master unix: [ID 950921 kern.info] cpu19: UltraSPARC-IV (portid 3 impl 0x18 ver 0x31 clock 1350 MHz)
Feb 19 09:55:32 abs-master unix: [ID 557827 kern.info] cpu19 initialization complete - online
Feb 19 09:55:33 abs-master qlc: [ID 308162 kern.info] Qlogic qlc FCA Driver v20051013-2.08 (1)
Feb 19 09:55:34 abs-master qlc: [ID 630585 kern.info] NOTICE: Qlogic qlc(1): Loop ONLINE
Feb 19 09:55:34 abs-master qlc: [ID 694252 kern.info] NOTICE: qlc(1): Firmware version 3.3.116
Feb 19 09:55:34 abs-master pcisch: [ID 370704 kern.info] PCI-device: SUNW,qlc@1, qlc1
Feb 19 09:55:34 abs-master genunix: [ID 936769 kern.info] qlc1 is /pci@9,600000/SUNW,qlc@1
Feb 19 09:55:34 abs-master qlc: [ID 308162 kern.info] Qlogic qlc FCA Driver v20051013-2.08 (2)
Feb 19 09:55:36 abs-master eri: [ID 517527 kern.info] SUNW,eri0 : Local Ethernet address = 0:3:ba:ce:85:c1
Feb 19 09:55:36 abs-master eri: [ID 517527 kern.info] SUNW,eri0 : Using local MAC address
Feb 19 09:55:36 abs-master pcisch: [ID 370704 kern.info] PCI-device: network@1,1, eri0
Feb 19 09:55:36 abs-master genunix: [ID 936769 kern.info] eri0 is /pci@9,700000/network@1,1
Feb 19 09:55:41 abs-master eri: [ID 786680 kern.notice] SUNW,eri0 : 100 Mbps full duplex link up
Feb 19 09:55:44 abs-master qlc: [ID 630585 kern.info] NOTICE: Qlogic qlc(2): Loop OFFLINE
Feb 19 09:55:44 abs-master qlc: [ID 694252 kern.info] NOTICE: qlc(2): Firmware version 3.3.116
Feb 19 09:55:44 abs-master pcisch: [ID 370704 kern.info] PCI-device: SUNW,qlc@1,1, qlc2
Feb 19 09:55:44 abs-master genunix: [ID 936769 kern.info] qlc2 is /pci@9,600000/SUNW,qlc@1,1
Feb 19 09:55:44 abs-master genunix: [ID 936769 kern.info] fp1 is /pci@9,600000/SUNW,qlc@1,1/fp@0,0
Feb 19 09:55:44 abs-master genunix: [ID 936769 kern.info] fp3 is /pci@9,600000/SUNW,qlc@1/fp@0,0
Feb 19 09:55:44 abs-master scsi: [ID 799468 kern.info] ssd10 at fp3: name w216000c0ff898c8f,4, bus address a7
Feb 19 09:55:44 abs-master genunix: [ID 936769 kern.info] ssd10 is /pci@9,600000/SUNW,qlc@1/fp@0,0/ssd@w216000c0ff898c8f,4
Feb 19 09:55:44 abs-master genunix: [ID 408114 kern.info] /pci@9,600000/SUNW,qlc@1/fp@0,0/ssd@w216000c0ff898c8f,4 (ssd10) online
Feb 19 09:55:44 abs-master scsi: [ID 799468 kern.info] ssd6 at fp3: name w216000c0ff898c8f,3, bus address a7
Feb 19 09:55:44 abs-master genunix: [ID 936769 kern.info] ssd6 is /pci@9,600000/SUNW,qlc@1/fp@0,0/ssd@w216000c0ff898c8f,3
Feb 19 09:55:44 abs-master genunix: [ID 408114 kern.info] /pci@9,600000/SUNW,qlc@1/fp@0,0/ssd@w216000c0ff898c8f,3 (ssd6) online
Feb 19 09:55:44 abs-master scsi: [ID 799468 kern.info] ssd7 at fp3: name w216000c0ff898c8f,2, bus address a7
Feb 19 09:55:44 abs-master genunix: [ID 936769 kern.info] ssd7 is /pci@9,600000/SUNW,qlc@1/fp@0,0/ssd@w216000c0ff898c8f,2
Feb 19 09:55:44 abs-master genunix: [ID 408114 kern.info] /pci@9,600000/SUNW,qlc@1/fp@0,0/ssd@w216000c0ff898c8f,2 (ssd7) online
Feb 19 09:55:44 abs-master scsi: [ID 799468 kern.info] ssd8 at fp3: name w216000c0ff898c8f,1, bus address a7
Feb 19 09:55:44 abs-master genunix: [ID 936769 kern.info] ssd8 is /pci@9,600000/SUNW,qlc@1/fp@0,0/ssd@w216000c0ff898c8f,1
Feb 19 09:55:44 abs-master genunix: [ID 408114 kern.info] /pci@9,600000/SUNW,qlc@1/fp@0,0/ssd@w216000c0ff898c8f,1 (ssd8) online
Feb 19 09:55:44 abs-master scsi: [ID 799468 kern.info] ssd9 at fp3: name w216000c0ff898c8f,0, bus address a7
Feb 19 09:55:44 abs-master genunix: [ID 936769 kern.info] ssd9 is /pci@9,600000/SUNW,qlc@1/fp@0,0/ssd@w216000c0ff898c8f,0
Feb 19 09:55:44 abs-master genunix: [ID 408114 kern.info] /pci@9,600000/SUNW,qlc@1/fp@0,0/ssd@w216000c0ff898c8f,0 (ssd9) online
Feb 19 09:55:44 abs-master rootnex: [ID 349649 kern.info] iscsi0 at root
Feb 19 09:55:44 abs-master genunix: [ID 936769 kern.info] iscsi0 is /iscsi
Feb 19 09:55:44 abs-master genunix: [ID 454863 kern.info] dump on /dev/dsk/c1t0d0s1 size 2047 MB
Feb 19 09:55:53 abs-master pseudo: [ID 129642 kern.info] pseudo-device: devinfo0
Feb 19 09:55:53 abs-master genunix: [ID 936769 kern.info] devinfo0 is /pseudo/devinfo@0
Feb 19 09:55:53 abs-master pseudo: [ID 129642 kern.info] pseudo-device: tod0
Feb 19 09:55:53 abs-master genunix: [ID 936769 kern.info] tod0 is /pseudo/tod@0
Feb 19 09:55:53 abs-master pseudo: [ID 129642 kern.info] pseudo-device: pm0
Feb 19 09:55:53 abs-master genunix: [ID 936769 kern.info] pm0 is /pseudo/pm@0
Feb 19 09:55:53 abs-master pcisch: [ID 370704 kern.info] PCI-device: SUNW,XVR-100@5, pfb0
Feb 19 09:55:53 abs-master genunix: [ID 936769 kern.info] pfb0 is /pci@8,700000/SUNW,XVR-100@5
Feb 19 09:55:53 abs-master pfb: [ID 604756 kern.info] pfb#0: 1152x900, rev 5159.0
Feb 19 09:55:54 abs-master pseudo: [ID 129642 kern.info] pseudo-device: fcode0
Feb 19 09:55:54 abs-master genunix: [ID 936769 kern.info] fcode0 is /pseudo/fcode@0
Feb 19 09:56:53 abs-master pseudo: [ID 129642 kern.info] pseudo-device: dtrace0
Feb 19 09:56:53 abs-master genunix: [ID 936769 kern.info] dtrace0 is /pseudo/dtrace@0
Feb 19 09:56:59 abs-master savecore: [ID 570001 auth.error] reboot after panic: pcisch-3: Fatal PCI bus error(s)
Feb 19 09:56:59 abs-master savecore: [ID 748169 auth.error] saving system crash dump in /var/crash/abs-master/*.1
Feb 19 09:57:33 abs-master pseudo: [ID 129642 kern.info] pseudo-device: pool0
Feb 19 09:57:33 abs-master genunix: [ID 936769 kern.info] pool0 is /pseudo/pool@0
Feb 19 09:57:45 abs-master fmd: [ID 441519 daemon.error] SUNW-MSG-ID: PCI-8000-42, TYPE: Fault, VER: 1, SEVERITY: Critical
Feb 19 09:57:45 abs-master EVENT-TIME: Mon Feb 19 09:57:45 NPT 2007
Feb 19 09:57:45 abs-master PLATFORM: SUNW,Sun-Fire-V890, CSN: -, HOSTNAME: abs-master
Feb 19 09:57:45 abs-master SOURCE: eft, REV: 1.13
Feb 19 09:57:45 abs-master EVENT-ID: d68e0f21-9ed8-e0eb-97ab-902e32fd4514
Feb 19 09:57:45 abs-master DESC: A problem was detected in the SUNOS subsystem or controlling software. Refer to http://sun.com/msg/PCI-8000-42 for more information.
Feb 19 09:57:45 abs-master AUTO-RESPONSE: One or more device instances may be disabled
Feb 19 09:57:45 abs-master IMPACT: Loss of services provided by the device instances associated with this fault
Feb 19 09:57:45 abs-master REC-ACTION: Ensure that the latest drivers and patches are installed, schedule a repair procedure to replace the affected device if necessary, or contact Sun for support.
Feb 19 09:57:47 abs-master fmd: [ID 441519 daemon.error] SUNW-MSG-ID: SUNOS-8000-1L, TYPE: Defect, VER: 1, SEVERITY: Minor
Feb 19 09:57:47 abs-master EVENT-TIME: Mon Feb 19 09:57:47 NPT 2007
Feb 19 09:57:47 abs-master PLATFORM: SUNW,Sun-Fire-V890, CSN: -, HOSTNAME: abs-master
Feb 19 09:57:47 abs-master SOURCE: eft, REV: 1.13
Feb 19 09:57:47 abs-master EVENT-ID: f0c604e5-a789-497a-a33d-d2d074a55d13
Feb 19 09:57:47 abs-master DESC: The EFT Diagnosis Engine encountered telemetry for which it is unable to produce a diagnosis. This may indicate an incomplete EFT file provided or a defect in the diagnosis algorithm. Refer to http://sun.com/msg/SUNOS-8000-1L for more information.
Feb 19 09:57:47 abs-master AUTO-RESPONSE: Error reports from the component will be logged for examination by Sun.
Feb 19 09:57:47 abs-master IMPACT: Automated diagnosis and response for these events will not occur.
Feb 19 09:57:47 abs-master REC-ACTION: Run pkgchk -n SUNWfmd to ensure that fault management software is installed properly. Contact Sun for support.
Feb 19 09:57:57 abs-master fmd: [ID 441519 daemon.error] SUNW-MSG-ID: PCI-8000-5X, TYPE: Fault, VER: 1, SEVERITY: Critical
Feb 19 09:57:57 abs-master EVENT-TIME: Mon Feb 19 09:57:57 NPT 2007
Feb 19 09:57:57 abs-master PLATFORM: SUNW,Sun-Fire-V890, CSN: -, HOSTNAME: abs-master
Feb 19 09:57:57 abs-master SOURCE: eft, REV: 1.13
Feb 19 09:57:57 abs-master EVENT-ID: 9c5cf863-ba5e-c4cb-a34d-dfa7c5e25e72
Feb 19 09:57:57 abs-master DESC: A problem was detected in the PCI subsystem. Refer to http://sun.com/msg/PCI-8000-5X for more information.
Feb 19 09:57:57 abs-master AUTO-RESPONSE: One or more device instances may be disabled
Feb 19 09:57:57 abs-master IMPACT: Loss of services provided by the device instances associated with this fault
Feb 19 09:57:57 abs-master REC-ACTION: Schedule a repair procedure to replace the affected device. Use fmdump -v -u <EVENT_ID> to identify the device or contact Sun for support.
Feb 19 09:58:16 abs-master pseudo: [ID 129642 kern.info] pseudo-device: vol0
Feb 19 09:58:16 abs-master genunix: [ID 936769 kern.info] vol0 is /pseudo/vol@0
Feb 19 09:58:17 abs-master pcisch: [ID 370704 kern.info] PCI-device: ide@1, uata0
Feb 19 09:58:17 abs-master genunix: [ID 936769 kern.info] uata0 is /pci@8,700000/ide@1
Feb 19 09:58:17 abs-master scsi: [ID 193665 kern.info] sd3 at uata0: target 0 lun 0
Feb 19 09:58:17 abs-master genunix: [ID 936769 kern.info] sd3 is /pci@8,700000/ide@1/sd@0,0
Feb 19 10:00:08 abs-master pseudo: [ID 129642 kern.info] pseudo-device: devinfo0
Feb 19 10:00:08 abs-master genunix: [ID 936769 kern.info] devinfo0 is /pseudo/devinfo@0
Feb 19 10:00:08 abs-master pseudo: [ID 129642 kern.info] pseudo-device: fcsm0
Feb 19 10:00:08 abs-master genunix: [ID 936769 kern.info] fcsm0 is /pseudo/fcsm@0
Feb 19 10:00:09 abs-master pseudo: [ID 129642 kern.info] pseudo-device: fcp0
Feb 19 10:00:09 abs-master genunix: [ID 936769 kern.info] fcp0 is /pseudo/fcp@0
Feb 19 10:00:09 abs-master scsi: [ID 799468 kern.info] ses0 at fp0: name w50800200001f6471,0, bus address dc
Feb 19 10:00:09 abs-master genunix: [ID 936769 kern.info] ses0 is /pci@8,600000/SUNW,qlc@2/fp@0,0/ses@w50800200001f6471,0
Feb 19 10:00:09 abs-master genunix: [ID 408114 kern.info] /pci@8,600000/SUNW,qlc@2/fp@0,0/ses@w50800200001f6471,0 (ses0) online
Feb 19 10:00:09 abs-master scsi: [ID 365881 kern.info] /pci@8,700000/scsi@2 (mpt0):
Feb 19 10:00:09 abs-master Rev. 8 LSI, Inc. 1030 found.
Feb 19 10:00:09 abs-master scsi: [ID 365881 kern.info] /pci@8,700000/scsi@2 (mpt0):
Feb 19 10:00:09 abs-master mpt0 supports power management.
Feb 19 10:00:09 abs-master scsi: [ID 365881 kern.info] /pci@8,700000/scsi@2 (mpt0):
Feb 19 10:00:09 abs-master mpt0 Firmware version v1.3.27.0 (IT)
Feb 19 10:00:09 abs-master scsi: [ID 365881 kern.info] /pci@8,700000/scsi@2 (mpt0):
Feb 19 10:00:09 abs-master mpt0: IOC Operational.
Feb 19 10:00:11 abs-master pcisch: [ID 370704 kern.info] PCI-device: scsi@2, mpt0
Feb 19 10:00:11 abs-master genunix: [ID 936769 kern.info] mpt0 is /pci@8,700000/scsi@2
Feb 19 10:00:26 abs-master genunix: [ID 408822 kern.info] NOTICE: mpt0: fault detected in device; service still available
Feb 19 10:00:26 abs-master genunix: [ID 611667 kern.info] NOTICE: mpt0: Connected command timeout for Target 5
Feb 19 10:00:26 abs-master scsi: [ID 107833 kern.warning] WARNING: /pci@8,700000/scsi@2 (mpt0):
Feb 19 10:00:26 abs-master Target 5 reducing sync. transfer rate
Feb 19 10:00:26 abs-master mpt: [ID 795936 kern.warning] WARNING: ID[SUNWpd.mpt.sync_wide_backoff.6014]
Feb 19 10:00:45 abs-master genunix: [ID 408822 kern.info] NOTICE: mpt0: fault detected in device; service still available
Feb 19 10:00:45 abs-master genunix: [ID 611667 kern.info] NOTICE: mpt0: Connected command timeout for Target 5
Feb 19 10:00:45 abs-master scsi: [ID 107833 kern.warning] WARNING: /pci@8,700000/scsi@2 (mpt0):
Feb 19 10:00:45 abs-master Target 5 reverting to async. mode
Feb 19 10:00:45 abs-master mpt: [ID 675377 kern.warning] WARNING: ID[SUNWpd.mpt.sync_wide_backoff.6013]
Feb 19 10:00:46 abs-master scsi: [ID 799468 kern.info] ssd3 at fp0: name w500000e01144e181,0, bus address e8
Feb 19 10:00:46 abs-master genunix: [ID 936769 kern.info] ssd3 is /pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w500000e01144e181,0
Feb 19 10:00:46 abs-master scsi: [ID 799468 kern.info] ssd0 at fp0: name w500000e0114514d1,0, bus address e2
Feb 19 10:00:46 abs-master genunix: [ID 936769 kern.info] ssd0 is /pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w500000e0114514d1,0
Feb 19 10:00:46 abs-master scsi: [ID 799468 kern.info] ssd1 at fp0: name w500000e01144c581,0, bus address e4
Feb 19 10:00:46 abs-master genunix: [ID 936769 kern.info] ssd1 is /pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w500000e01144c581,0
Feb 19 10:00:46 abs-master scsi: [ID 799468 kern.info] ssd2 at fp0: name w500000e01144a6c1,0, bus address e1
Feb 19 10:00:46 abs-master genunix: [ID 936769 kern.info] ssd2 is /pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w500000e01144a6c1,0
Feb 19 10:00:46 abs-master scsi: [ID 799468 kern.info] ssd4 at fp0: name w500000e01144c511,0, bus address e0
Feb 19 10:00:46 abs-master genunix: [ID 936769 kern.info] ssd4 is /pci@8,600000/SUNW,qlc@2/fp@0,0/ssd@w500000e01144c511,0
Feb 19 10:00:48 abs-master scsi: [ID 365881 kern.info] /pci@8,700000/scsi@2/st@5,0 (st5):
Feb 19 10:00:48 abs-master <HP Ultrium LTO 2>
Feb 19 10:00:48 abs-master scsi: [ID 193665 kern.info] st5 at mpt0: target 5 lun 0
Feb 19 10:00:48 abs-master genunix: [ID 936769 kern.info] st5 is /pci@8,700000/scsi@2/st@5,0