被管服务器与控制台频繁失联。然后业务处理变慢，控制台也无法打开，只能重启失联的节点才能恢复。但是过了一段时间后又会在不同的节点出现同样的问题。
日志里显示服务超时，同时也在做垃圾回收，线程快照里有大量的stuck线程，都在wait：0x0000000402d424d0；
"[ACTIVE] ExecuteThread: '932' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x00007ff0f48bd000 nid=0x95fc waiting for monitor entry [0x00007fec6e4a5000]
java.lang.Thread.State: BLOCKED (on object monitor)
at weblogic.diagnostics.archive.wlstore.PersistentStoreDataArchive.insertRecord(PersistentStoreDataArchive.java:841)
- waiting to lock <0x0000000402d424d0> (a weblogic.diagnostics.archive.wlstore.HarvestedPersistentStoreDataArchive)
at weblogic.diagnostics.archive.wlstore.PersistentStoreDataArchive.writeData(PersistentStoreDataArchive.java:1488)
at weblogic.diagnostics.harvester.internal.MetricArchiver.archive(MetricArchiver.java:581)
at weblogic.diagnostics.harvester.internal.MetricArchiver.access$000(MetricArchiver.java:54)
at weblogic.diagnostics.harvester.internal.MetricArchiver$1.run(MetricArchiver.java:612)
at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:548)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:311)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:263)
然后这把锁0x0000000402d424d0锁在了
"[ACTIVE] ExecuteThread: '958' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x00007ff0f48f3000 nid=0x9616 runnable [0x00007fec6ca8a000]
java.lang.Thread.State: RUNNABLE
at java.lang.System.arraycopy(Native Method)
at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:146)
at weblogic.utils.io.ByteBufferInputStream.read(ByteBufferInputStream.java:42)
at java.io.ObjectInputStream$PeekInputStream.read(ObjectInputStream.java:2310)
at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2323)
at java.io.ObjectInputStream$BlockDataInputStream.readInt(ObjectInputStream.java:2818)
at java.io.ObjectInputStream.readHandle(ObjectInputStream.java:1452)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1330)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
at weblogic.diagnostics.accessor.DataRecord.readExternal(DataRecord.java:58)
at java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:1837)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1796)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
at weblogic.diagnostics.archive.wlstore.Snapshot.readExternal(Snapshot.java:64)
at java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:1837)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1796)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
at weblogic.utils.io.ByteBufferObjectInputStream.readObject(ByteBufferObjectInputStream.java:38)
at weblogic.diagnostics.archive.CustomObjectHandler.readObject(CustomObjectHandler.java:20)
at weblogic.store.internal.PersistentStoreRecordImpl.initBody(PersistentStoreRecordImpl.java:99)
at weblogic.store.internal.PersistentStoreRecordImpl.getData(PersistentStoreRecordImpl.java:125)
at weblogic.diagnostics.archive.wlstore.PersistentStoreDataArchive.readRecord(PersistentStoreDataArchive.java:696)
- locked <0x0000000402d424d0> (a weblogic.diagnostics.archive.wlstore.HarvestedPersistentStoreDataArchive)
at weblogic.diagnostics.archive.wlstore.PersistentStoreDataArchive.readRecordObject(PersistentStoreDataArchive.java:674)
at weblogic.diagnostics.archive.wlstore.RecordWrapper.getDataObject(RecordWrapper.java:70)
at weblogic.diagnostics.archive.wlstore.PersistentStoreDataArchive.getWrapper(PersistentStoreDataArchive.java:1772)
at weblogic.diagnostics.archive.wlstore.PersistentStoreDataArchive.removeGarbageInPage(PersistentStoreDataArchive.java:1813)
- locked <0x0000000402d424d0> (a weblogic.diagnostics.archive.wlstore.HarvestedPersistentStoreDataArchive)
at weblogic.diagnostics.archive.wlstore.PersistentStoreDataArchive.cleanupPages(PersistentStoreDataArchive.java:1697)
at weblogic.diagnostics.archive.wlstore.PersistentStoreDataArchive.cleanup(PersistentStoreDataArchive.java:1645)
at weblogic.diagnostics.archive.wlstore.PersistentStoreDataArchive.deleteDataRecords(PersistentStoreDataArchive.java:1361)
at weblogic.diagnostics.archive.wlstore.PersistentStoreDataArchive.retireOldestRecords(PersistentStoreDataArchive.java:1211)
at weblogic.diagnostics.archive.DataRetirementByQuotaTaskImpl.performDataRetirement(DataRetirementByQuotaTaskImpl.java:92)
at weblogic.diagnostics.archive.DataRetirementByQuotaTaskImpl.run(DataRetirementByQuotaTaskImpl.java:49)
at weblogic.diagnostics.archive.DataRetirementTaskImpl.run(DataRetirementTaskImpl.java:261)
at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:548)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:311)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:263)