Infrastructure Software

It is my understanding that when using AMM the buffer cache is placed inside of /dev/shm. Also that by default /dev/shm is created as 1/2 of physical memory. I did some tests w/ /dev/shm that appear a bit disturbing. My question, is there any guarantee that /dev/shm will always be in physical memory? My thinking is that /dev/shm is not allocated until its used - so if other processes are using all or most of physical memory, it could use swap. That seems really unattractive. Here are some tests that I did:

for the test I created a /dev/shm larger than actual RAM.

Here I can see the benefit of using /dev/shm. The first 2GB file I create I create inside /dev/shm - look at the time. Its fast - really fast. Look at the second 2GB file I create on an ext3 filesystem - its fast but really slow compared to /dev/shm. Weare talking memory compared to disk here...

[root@myserver ~]# time dd if=/dev/zero of=/dev/shm/2GB bs=1M count=2048
2048+0 records in
2048+0 records out

real 0m3.433s
user 0m0.000s
sys 0m3.432s

[root@myserver ~]# time dd if=/dev/zero of=/u01/app/oracle/2GB bs=1M count=2048
2048+0 records in
2048+0 records out

real 0m18.690s
user 0m0.008s
sys 0m5.894s

Here is where things get "wonky" for me. Here is /proc/meminfo - I have created a 9GB /dev/shm ( its not being used, so we dont see any allocation from the MemFree or the SwapFree

[root@myserver ~]# cat /proc/meminfo
MemTotal: 8312628 kB
MemFree: 6164080 kB
Buffers: 2504 kB
Cached: 2061896 kB
SwapCached: 0 kB
Active: 33072 kB
Inactive: 2049540 kB
HighTotal: 7470840 kB
HighFree: 5384128 kB
LowTotal: 841788 kB
LowFree: 779952 kB
SwapTotal: 8388600 kB
SwapFree: 8388600 kB
Dirty: 0 kB
Writeback: 0 kB
Mapped: 31276 kB
Slab: 47276 kB
CommitLimit: 12544912 kB
Committed_AS: 52536 kB
PageTables: 968 kB
VmallocTotal: 106488 kB
VmallocUsed: 6240 kB
VmallocChunk: 99696 kB
HugePages_Total: 0
HugePages_Free: 0
Hugepagesize: 2048 kB

Just to show that removing files from /dev/shm clears memory.

[root@myserver ~]# cat /proc/meminfo | egrep Mem\|Swap
MemTotal: 8312628 kB
MemFree: 1931632 kB
SwapTotal: 8388600 kB
SwapFree: 8388600 kB
[root@myserver ~]# rm -rf /dev/shm/*
[root@myserver ~]# cat /proc/meminfo | egrep Mem\|Swap
MemTotal: 8312628 kB
MemFree: 8235440 kB
SwapTotal: 8388600 kB
SwapFree: 8388600 kB

This is the size of the unused tmpfs
[root@myserver ~]# df -h /dev/shm
Filesystem Size Used Avail Use% Mounted on
none 9.0G 0 9.0G 0% /dev/shm

Creation of the first 2GB file - its fast...

[root@myserver ~]# time dd if=/dev/zero of=/dev/shm/2GB.mem1 bs=1M count=2048
2048+0 records in
2048+0 records out

real 0m3.425s
user 0m0.002s
sys 0m3.422s

Its using real physical memory.

[root@myserver ~]# egrep Mem\|Swap /proc/meminfo
MemTotal: 8312628 kB
MemFree: 6136048 kB
SwapCached: 0 kB
SwapTotal: 8388600 kB
SwapFree: 8388600 kB

The secon file - again, its fast...

[root@myserver ~]# time dd if=/dev/zero of=/dev/shm/2GB.mem2 bs=1M count=2048
2048+0 records in
2048+0 records out

real 0m3.428s
user 0m0.004s
sys 0m3.423s

b/c again its using memory

[root@myserver ~]# egrep Mem\|Swap /proc/meminfo
MemTotal: 8312628 kB
MemFree: 4034416 kB
SwapCached: 0 kB
SwapTotal: 8388600 kB
SwapFree: 8388600 kB

The third file

[root@myserver ~]# time dd if=/dev/zero of=/dev/shm/2GB.mem3 bs=1M count=2048
2048+0 records in
2048+0 records out

real 0m3.422s
user 0m0.002s
sys 0m3.419s

Still using physical memory, but we are running low...

[root@myserver ~]# egrep Mem\|Swap /proc/meminfo
MemTotal: 8312628 kB
MemFree: 1932912 kB
SwapCached: 0 kB
The fourth file - slows way down - slower than if I wrote this to disk...

[root@myserver ~]# time dd if=/dev/zero of=/dev/shm/2GB.mem_swap bs=1M count=2048
2048+0 records in
2048+0 records out

real 0m29.130s
user 0m0.000s
sys 0m3.050s

Now we are using the remaining memory plus swap - but where did all my swap go?This is NOT right - I know this
because nothing is running on this machine - could we really have used 7+GB of swap to create a 2GB file?
My guess is Ive exhausted low mem b/c this is 32 bit.

[root@myserver ~]# egrep Mem\|Swap /proc/meminfo
MemTotal: 8312628 kB
MemFree: 25904 kB
SwapCached: 8010156 kB
SwapTotal: 8388600 kB
SwapFree: 188008 kB

The fifth file... OUCH! Way worse than writing to disk over 2x worse???

[root@myserver ~]# time dd if=/dev/zero of=/dev/shm/2GB.swap bs=1M count=2048
dd: writing `/dev/shm/2GB.swap': No space left on device
1015+0 records in
1014+0 records out

real 0m40.557s
user 0m0.000s
sys 0m1.461s

And some nasty side effects - furthering my belief that I have exhausted LowMem on a 32 bit machine.

[root@myserver ~]# egrep Mem\|Swap /proc/meminfo Connection to myserver closed by remote host.
Connection to myserver closed.
tfiedler@rlinux3:~$ ssh root@myserver
Connection closed by xxx.xxx.xxx.xxx

Also to further my belief that lowmem is exhausted,I realize that the OOM kills could probably be dispatched of by using a HighMem kernel or possibly adjusting some vm settings...

[root@myserver ~]# grep "Killed process" /var/log/messages
Dec 2 08:57:32 myserver kernel: Out of Memory: Killed process 4426 (gdmgreeter).
Dec 2 08:57:38 myserver kernel: Out of Memory: Killed process 3387 (ntpd).
Dec 2 08:57:49 myserver kernel: Out of Memory: Killed process 3434 (xfs).
Dec 2 08:57:58 myserver kernel: Out of Memory: Killed process 3476 (dbus-daemon-1).
Dec 2 08:58:04 myserver kernel: Out of Memory: Killed process 18375 (gdm-binary).
Dec 2 08:58:04 myserver kernel: Out of Memory: Killed process 18260 (sshd).
Dec 2 08:58:11 myserver kernel: Out of Memory: Killed process 18265 (bash).
Dec 2 08:58:12 myserver kernel: Out of Memory: Killed process 18416 (sshd).
Dec 2 08:58:12 myserver kernel: Out of Memory: Killed process 3924 (gdm-binary).
Dec 2 08:58:20 myserver kernel: Out of Memory: Killed process 18421 (prefdm).
Dec 2 08:58:32 myserver kernel: Out of Memory: Killed process 18423 (gdm-binary).
Dec 2 08:58:33 myserver kernel: Out of Memory: Killed process 18453 (gdm-binary).
Dec 2 08:58:43 myserver kernel: Out of Memory: Killed process 18631 (gdm-binary).
Dec 2 08:58:45 myserver kernel: Out of Memory: Killed process 18681 (gdm-binary).
Dec 2 08:58:50 myserver kernel: Out of Memory: Killed process 18869 (gdm-binary).
Dec 2 08:58:51 myserver kernel: Out of Memory: Killed process 18911 (gdm-binary).