On Solaris server, we have nfs mounted storage for oracle database.
During network disconnect or storage migration, application gets stopped. Below errors report that time.
Root cause or any solution/workaround
. 13:39:33 --------- nfs: [ID 733954 kern.info] NOTICE: [NFS4][Server: ---------------][Mntpt: /u03]NFS server --------------- not responding; still trying
. 13:39:33 --------- nfs: [ID 733954 kern.info] NOTICE: [NFS4][Server: ---------------][Mntpt: /flash]NFS server --------------- not responding; still trying
. 13:39:33 --------- nfs: [ID 733954 kern.info] NOTICE: [NFS4][Server: ---------------][Mntpt: /u01]NFS server --------------- not responding; still trying
. 13:39:35 --------- nfs: [ID 559769 kern.info] NOTICE: [NFS4][Server: ---------------][Mntpt: /u01]NFS server --------------- ok
. 13:39:35 --------- nfs: [ID 733954 kern.info] NOTICE: [NFS4][Server: ---------------][Mntpt: /u01]NFS server --------------- not responding; still trying
. 13:39:35 --------- nfs: [ID 559769 kern.info] NOTICE: [NFS4][Server: ---------------][Mntpt: /u01]NFS server --------------- ok
. 13:39:36 --------- nfs: [ID 733954 kern.info] NOTICE: [NFS4][Server: ---------------][Mntpt: /u01]NFS server --------------- not responding; still trying
. 13:39:36 --------- nfs: [ID 559769 kern.info] NOTICE: [NFS4][Server: ---------------][Mntpt: /u01]NFS server --------------- ok
. 13:39:41 --------- nfs: [ID 733954 kern.info] NOTICE: [NFS4][Server: ---------------][Mntpt: /u01]NFS server --------------- not responding; still trying
. 13:39:41 --------- nfs: [ID 559769 kern.info] NOTICE: [NFS4][Server: ---------------][Mntpt: /u01]NFS server --------------- ok
. 13:39:43 --------- nfs: [ID 733954 kern.info] NOTICE: [NFS4][Server: ---------------][Mntpt: /u01]NFS server --------------- not responding; still trying
. 13:39:43 --------- nfs: [ID 559769 kern.info] NOTICE: [NFS4][Server: ---------------][Mntpt: /u01]NFS server --------------- ok
. 13:39:43 --------- nfs: [ID 559769 kern.info] NOTICE: [NFS4][Server: ---------------][Mntpt: /u03]NFS server --------------- ok
. 13:39:43 --------- nfs: [ID 559769 kern.info] NOTICE: [NFS4][Server: ---------------][Mntpt: /flash]NFS server --------------- ok
. 13:40:18 --------- nfs: [ID 273629 kern.info] NOTICE: [NFS4][Server: ---------------][Mntpt: /u03]NFS Recovery done for mount /u03 (mi 0x6006b80f000) on server ---------------, rnode_pt1 <null string> (0x0), rno
de_pt2 <null string> (0x0)
. 13:40:18 --------- nfs: [ID 236337 kern.info] NOTICE: [NFS4][Server: ---------------][Mntpt: /u03]NFS op OP_READ got error NFS4ERR_DELAY causing recovery action NR_DELAY.
. 13:40:18 --------- nfs: [ID 236337 kern.info] NOTICE: [NFS4][Server: ---------------][Mntpt: /u03]NFS op OP_WRITE got error NFS4ERR_DELAY causing recovery action NR_DELAY.
. 13:40:18 --------- nfs: [ID 236337 kern.info] NOTICE: [NFS4][Server: ---------------][Mntpt: /u03]NFS op OP_READ got error NFS4ERR_DELAY causing recovery action NR_DELAY.
. 13:40:18 --------- nfs: [ID 236337 kern.info] NOTICE: [NFS4][Server: ---------------][Mntpt: /u03]NFS op OP_WRITE got error NFS4ERR_DELAY causing recovery action NR_DELAY.
. 13:40:18 --------- nfs: [ID 236337 kern.info] NOTICE: [NFS4][Server: ---------------][Mntpt: /u03]NFS op OP_READ got error NFS4ERR_DELAY causing recovery action NR_DELAY.
. 13:40:18 --------- nfs: [ID 733954 kern.info] NOTICE: [NFS4][Server: ---------------][Mntpt: /u03]NFS server --------------- not responding; still trying
. 13:40:18 --------- nfs: [ID 559769 kern.info] NOTICE: [NFS4][Server: ---------------][Mntpt: /u03]NFS server --------------- ok
. 13:40:18 --------- nfs: [ID 236337 kern.info] NOTICE: [NFS4][Server: ---------------][Mntpt: /u03]NFS op OP_WRITE got error NFS4ERR_STALE_STATEID causing recovery action NR_CLIENTID. Client also suspects th
at the server rebooted, or experienced a network partition.
. 13:40:18 --------- nfs: [ID 581112 kern.info] NOTICE: [NFS4][Server: ---------------][Mntpt: /u03]NFS Starting recovery for mount /u03 (mi 0x6006b80f000 mi_recovflags [0x21]) on server ---------------, rnode_pt1
--------- (0x3006f60e880), rnode_pt2 <null string> (0x0)
. 13:40:18 --------- nfs: [ID 733954 kern.info] NOTICE: [NFS4][Server: ---------------][Mntpt: /u03]NFS server --------------- not responding; still trying
. 13:40:18 --------- nfs: [ID 559769 kern.info] NOTICE: [NFS4][Server: ---------------][Mntpt: /u03]NFS server --------------- ok
. 13:40:18 --------- nfs: [ID 236337 kern.info] NOTICE: [NFS4][Server: ---------------][Mntpt: /u03]NFS op OP_WRITE got error NFS4ERR_STALE_STATEID causing recovery action NR_CLIENTID. Client also suspects th
at the server rebooted, or experienced a network partition.
. 13:40:18 --------- nfs: [ID 733954 kern.info] NOTICE: [NFS4][Server: ---------------][Mntpt: /u03]NFS server --------------- not responding; still trying
. 13:40:18 --------- nfs: [ID 559769 kern.info] NOTICE: [NFS4][Server: ---------------][Mntpt: /u03]NFS server --------------- ok
. 13:40:18 --------- nfs: [ID 236337 kern.info] NOTICE: [NFS4][Server: ---------------][Mntpt: /u03]NFS op OP_WRITE got error NFS4ERR_STALE_STATEID causing recovery action NR_CLIENTID. Client also suspects th
at the server rebooted, or experienced a network partition.
. 13:40:18 --------- nfs: [ID 273629 kern.info] NOTICE: [NFS4][Server: ---------------][Mntpt: /u03]NFS Recovery done for mount /u03 (mi 0x6006b80f000) on server ---------------, rnode_pt1 -------- (0x3006f60e880), rnode_pt2 <null string> (0x0)
. 13:40:18 --------- nfs: [ID 236337 kern.info] NOTICE: [NFS4][Server: ---------------][Mntpt: /u03]NFS op OP_READ got error NFS4ERR_DELAY causing recovery action NR_DELAY.
. 13:40:18 --------- nfs: [ID 236337 kern.info] NOTICE: [NFS4][Server: ---------------][Mntpt: /u03]NFS op OP_WRITE got error NFS4ERR_DELAY causing recovery action NR_DELAY.
. 13:40:18 --------- nfs: [ID 733954 kern.info] NOTICE: [NFS4][Server: ---------------][Mntpt: /u03]NFS server --------------- not responding; still trying
. 13:40:18 --------- nfs: [ID 559769 kern.info] NOTICE: [NFS4][Server: ---------------][Mntpt: /u03]NFS server --------------- ok
. 13:40:18 --------- nfs: [ID 236337 kern.info] NOTICE: [NFS4][Server: ---------------][Mntpt: /u03]NFS op OP_WRITE got error NFS4ERR_STALE_STATEID causing recovery action NR_CLIENTID. Client also suspects th
at the server rebooted, or experienced a network partition.
. 13:40:18 --------- nfs: [ID 581112 kern.info] NOTICE: [NFS4][Server: ---------------][Mntpt: /u03]NFS Starting recovery for mount /u03 (mi 0x6006b80f000 mi_recovflags [0x21]) on server ---------------, rnode_pt1
------, rnode_pt2 <null string> (0x0)
. 13:40:18 --------- nfs: [ID 236337 kern.info] NOTICE: [NFS4][Server: ---------------][Mntpt: /u03]NFS op OP_READ got error NFS4ERR_STALE_STATEID causing recovery action NR_CLIENTID. Client also suspects tha
t the server rebooted, or experienced a network partition.
. 13:40:18 --------- nfs: [ID 733954 kern.info] NOTICE: [NFS4][Server: ---------------][Mntpt: /u03]NFS server --------------- not responding; still trying
. 13:40:18 --------- nfs: [ID 559769 kern.info] NOTICE: [NFS4][Server: ---------------][Mntpt: /u03]NFS server --------------- ok
. 13:40:18 --------- nfs: [ID 236337 kern.info] NOTICE: [NFS4][Server: ---------------][Mntpt: /u03]NFS op OP_READ got error NFS4ERR_STALE_STATEID causing recovery action NR_CLIENTID. Client also suspects tha
t the server rebooted, or experienced a network partition.
. 13:40:18 --------- nfs: [ID 733954 kern.info] NOTICE: [NFS4][Server: ---------------][Mntpt: /u03]NFS server --------------- not responding; still trying
. 13:40:18 --------- nfs: [ID 559769 kern.info] NOTICE: [NFS4][Server: ---------------][Mntpt: /u03]NFS server --------------- ok
. 13:40:18 --------- nfs: [ID 236337 kern.info] NOTICE: [NFS4][Server: ---------------][Mntpt: /u03]NFS op OP_WRITE got error NFS4ERR_STALE_STATEID causing recovery action NR_CLIENTID. Client also suspects th
at the server rebooted, or experienced a network partition.
. 13:40:18 --------- nfs: [ID 733954 kern.info] NOTICE: [NFS4][Server: ---------------][Mntpt: /u03]NFS server --------------- not responding; still trying
. 13:40:18 --------- nfs: [ID 559769 kern.info] NOTICE: [NFS4][Server: ---------------][Mntpt: /u03]NFS server --------------- ok
. 13:40:18 --------- nfs: [ID 236337 kern.info] NOTICE: [NFS4][Server: ---------------][Mntpt: /u03]NFS op OP_WRITE got error NFS4ERR_STALE_STATEID causing recovery action NR_CLIENTID. Client also suspects th
at the server rebooted, or experienced a network partition.
. 13:40:18 --------- nfs: [ID 733954 kern.info] NOTICE: [NFS4][Server: ---------------][Mntpt: /u03]NFS server --------------- not responding; still trying
. 13:40:18 --------- nfs: [ID 559769 kern.info] NOTICE: [NFS4][Server: ---------------][Mntpt: /u03]NFS server --------------- ok
. 13:40:18 --------- nfs: [ID 236337 kern.info] NOTICE: [NFS4][Server: ---------------][Mntpt: /u03]NFS op OP_WRITE got error NFS4ERR_STALE_STATEID causing recovery action NR_CLIENTID. Client also suspects that the server rebooted, or experienced a network partition.
. 13:40:18 --------- nfs: [ID 733954 kern.info] NOTICE: [NFS4][Server: ---------------][Mntpt: /u03]NFS server --------------- not responding; still trying
. 13:40:18 --------- nfs: [ID 559769 kern.info] NOTICE: [NFS4][Server: ---------------][Mntpt: /u03]NFS server --------------- ok
. 13:40:18 --------- nfs: [ID 236337 kern.info] NOTICE: [NFS4][Server: ---------------][Mntpt: /u03]NFS op OP_WRITE got error NFS4ERR_STALE_STATEID causing recovery action NR_CLIENTID. Client also suspects th
at the server rebooted, or experienced a network partition.
. 13:40:18 --------- nfs: [ID 273629 kern.info] NOTICE: [NFS4][Server: ---------------][Mntpt: /u03]NFS Recovery done for mount /u03 (mi 0x6006b80f000) on server ---------------, rnode_pt1 ----, rnode_pt2 <null string> (0x0)
. 13:40:18 --------- nfs: [ID 236337 kern.info] NOTICE: [NFS4][Server: ---------------][Mntpt: /u03]NFS op OP_READ got error NFS4ERR_DELAY causing recovery action NR_DELAY.
. 13:40:18 --------- nfs: [ID 733954 kern.info] NOTICE: [NFS4][Server: ---------------][Mntpt: /u03]NFS server --------------- not responding; still trying
. 13:40:18 --------- nfs: [ID 559769 kern.info] NOTICE: [NFS4][Server: ---------------][Mntpt: /u03]NFS server --------------- ok
. 13:40:18 --------- nfs: [ID 236337 kern.info] NOTICE: [NFS4][Server: ---------------][Mntpt: /u03]NFS op OP_WRITE got error NFS4ERR_STALE_STATEID causing recovery action NR_CLIENTID. Client also suspects th
at the server rebooted, or experienced a network partition.
. 13:40:18 --------- nfs: [ID 581112 kern.info] NOTICE: [NFS4][Server: ---------------][Mntpt: /u03]NFS Starting recovery for mount /u03 (mi 0x6006b80f000 mi_recovflags [0x21]) on server ---------------, rnode_pt1
------------, rnode_pt2 <null string> (0x0)
. 13:40:18 --------- nfs: [ID 273629 kern.info] NOTICE: [NFS4][Server: ---------------][Mntpt: /u03]NFS Recovery done for mount /u03 (mi 0x6006b80f000) on server ---------------, rnode_pt1 ------, rnode_pt2 <null string> (0x0)
. 13:40:18 --------- nfs: [ID 236337 kern.info] NOTICE: [NFS4][Server: ---------------][Mntpt: /u03]NFS op OP_READ got error NFS4ERR_DELAY causing recovery action NR_DELAY.
. 13:40:18 --------- nfs: [ID 581112 kern.info] NOTICE: [NFS4][Server: ---------------][Mntpt: /u03]NFS Starting recovery for mount /u03 (mi 0x6006b80f000 mi_recovflags [0x22]) on server ---------------, rnode_pt1
<null string> (0x0), rnode_pt2 <null string> (0x0)
. 13:40:18 --------- nfs: [ID 257235 kern.info] NOTICE: [NFS4][Server: ---------------][Mntpt: /u03]Couldn't reclaim lock for pid 424 for file ------- (rnode_pt 0x300710dc4e0) on (server ---------------): error 10010
. 13:40:18 --------- nfs: [ID 286389 kern.info] NOTICE: [NFS4][Server: ---------------][Mntpt: /u03]File -------------(rnode_pt: 300
710dc4e0) was closed due to NFS recovery error on server ---------------(Failed to recover lost lock NFS4ERR_DENIED)
. 13:40:18 --------- nfs: [ID 941083 kern.info] NOTICE: NFS4 FACT SHEET:
. 13:40:18 --------- Action: NR_DELAY
. 13:40:18 --------- NFS4 error: NFS4ERR_DELAY
. 13:40:18 --------- Suspected server reboot.