Infrastructure Software

Hello,

I have a 2 Node SUN 3.2 cluster running as an HA-NFS solution. The NFS data is stored on a ZFS volume created on a SAN disk.
Since a few days ago the NFS RG run on node1 and used a lot of memory for filesystem caching. During some maintenance at the network-infrastructure the NFS RG switched to node2 where it used nearly the whole memory for caching, too. So far so good.
After the RG-switch, our monitoring still shows much used memory on node1, although the ZFS volume had switched to node2.

When I look at the memory stats with:

echo "::memstat"|pfexec mdb -k
Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 1535815 11998 75%
Anon 425970 3327 21%
Exec and libs 4930 38 0%
Page cache 21454 167 1%
Free (cachelist) 12815 100 1%
Free (freelist) 56806 443 3%

Total 2057790 16076
Physical 2046692 15989

I see that most of the memory is used by the kernel, what made sense as long as the ZFS volume was mounted. But why is the cache not cleaned after the ZFS was exported?

kstat shows this:

kstat -m zfs
module: zfs instance: 0
name: arcstats class: misc
c 7329487504
c_max 15783673856
c_min 1972959232
crtime 469.179448849
deleted 112519909
demand_data_hits 335975255
demand_data_misses 44014727
demand_metadata_hits 3515997301
demand_metadata_misses 25579771
evict_skip 6691178
hash_chain_max 19
hash_chains 0
hash_collisions 303863107
hash_elements 0
hash_elements_max 1154250
hdr_size 80555664
hits 6686299542
l2_abort_lowmem 0
l2_cksum_bad 0
l2_evict_lock_retry 0
l2_evict_reading 0
l2_feeds 0
l2_free_on_write 0
l2_hdr_size 0
l2_hits 0
l2_io_error 0
l2_misses 0
l2_rw_clash 0
l2_size 0
l2_writes_done 0
l2_writes_error 0
l2_writes_hdr_miss 0
l2_writes_sent 0
memory_throttle_count 0
mfu_ghost_hits 35405248
mfu_hits 4550953522
misses 84976385
mru_ghost_hits 15286202
mru_hits 145588572
mutex_miss 423259
p 6977488645
prefetch_data_hits 5395965
prefetch_data_misses 5325909
prefetch_metadata_hits 2828931021
prefetch_metadata_misses 10055978
recycle_miss 7325174
size 0
snaptime 26525967.0528634

module: zfs instance: 0
name: vdev_cache_stats class: misc
crtime 469.179633713
delegations 1261718
hits 8753963
misses 24152930
snaptime 26525967.0600794


Has anyone an explanation for this and / or a hint how to get the memory clean again? (Reboot is no option :-))
Currently 97% of memory are used and our monitoring complains constantly.

Regards
Markus