Hi DBA's and admin's,
I've got an Oracle EL6 which lately keeps running out of memory. Now I know the memory on Linux topic is quite complex and the physical free memory can be low on Linux without cause for concern, however when the free memory get's to about 150MB then the server gets really slow and Oracle gets the 'Cannot allocate memory' and 'ORA-27102: out of memory' error.
After a server reboot I have about 19GB free memory. This is a log from (free -m) every 30 minutes from that point onwards.
Mem:19613
Mem:15023
Mem:12663
Mem:12588
Mem:12559
Mem:12470
Mem:12300
Mem:11951
Mem:11850
Mem:11833
Mem:11804
Mem:11771
Mem:11607
Mem:11601
Mem:11594
Mem:11590
Mem:11569
Mem:11269
Mem:11165
Mem:11171
Mem:11160
Mem:11157
Mem:11153
Mem:11150
Mem:11141
Mem:6911
Mem:5674
Mem:3812
Mem:5293
Mem:129
Mem:127
Mem:200
Mem:203
Mem:167
Mem:169
Mem:235
Mem:130
Mem:143
Mem:167
Mem:357
Mem:349
Mem:315
Mem:167
Mem:209
Mem:197
Mem:233
Mem:231
Mem:155
Mem:166
Mem:188
Mem:163
Mem:143
Mem:148
Mem:147
At this point I get the memory allocation issue. There is plenty of cached memory available and nearly nothing is in the swap. This server was working for a long time without issues, so I'm lost on why this keeps happening.
There are 24GB of RAM, 2 instances (1.5GB and 15GB memory target) and both are 12.1.0.2
The big dips from 11GB to 7GB and 5GB to 129MB is when my backups (expdp) are running. The backup jobs finish without issues and the worker processes are all gone.
If I force a cache drop then I get about 7GB back on physical RAM.
Any ideas? 12c preinstall package/script is installed and ran, so all the kernel min/max are as required and tmpfs is 30GB and around 60% free when this happens.