Whis is rman caching all files to memory during backup
745782Apr 3 2012 — edited Apr 26 2012I've 3 RAC environments on Linux 5.5 with Oracle 11.2.0.2 and ASM; 1 RAC environment has Automatic Memory Management (AMM with 32G MEMORY_TARGET and MEMORY_MAX_TARGET) and 2 RAC environments have Automatic Shared Memory Management (ASMM with 16G SGA_TARGET, SGA_MAX_TARGET, PGA of 5G) Each environment has 2 servers with 132G memory each. I backup databases on the 1st server of each environment locally. The problem I'm facing is that whenever I run rman, all the rman files are cached to memory while backing up to disk; and the memory cached is never released later. It comes to a point that after a few days of rman run, there will be no free memory left out of 132G total and starts taking SWAP memory (18G). Once half the swap is taken, the db connections will be refused unless I drop the cache. This behavior is strange; oracle says it's OS issue; Linux says it's Oracle's issue. I'll give an example below:
1st time backup today:
Before backup:
top - 06:29:15 up 11 days, 16:59, 2 users, load average: 0.52, 0.78, 0.78
Mem: 132081600k total, 42181080k used, 89900520k free, 540752k buffers
Swap: 17825768k total, 372k used, 17825396k free, 37775732k cached
After backup:
top - 06:36:06 up 11 days, 17:06, 2 users, load average: 1.92, 1.87, 1.29
Mem: 132081600k total, 44605392k used, 87476208k free, 552200k buffers
Swap: 17825768k total, 372k used, 17825396k free, 40080044k cached
2nd time backup today:
Before backup:
top - 06:46:28 up 11 days, 17:16, 2 users, load average: 0.75, 0.98, 1.07
Mem: 132081600k total, 44551264k used, 87530336k free, 555228k buffers
Swap: 17825768k total, 372k used, 17825396k free, 40083352k cached
After backup:
top - 06:52:18 up 11 days, 17:22, 2 users, load average: 2.23, 1.69, 1.36
Mem: 132081600k total, 46813808k used, 85267792k free, 559936k buffers
Swap: 17825768k total, 372k used, 17825396k free, 42268728k cached
3rd time backup:
Before backup:
top - 07:13:03 up 11 days, 17:42, 2 users, load average: 0.45, 0.73, 0.88
Mem: 132081600k total, 46818456k used, 85263144k free, 565864k buffers
Swap: 17825768k total, 372k used, 17825396k free, 42275028k cached
After backup:
top - 07:19:43 up 11 days, 17:49, 2 users, load average: 1.10, 1.43, 1.17
Mem: 132081600k total, 49080992k used, 83000608k free, 571088k buffers
Swap: 17825768k total, 372k used, 17825396k free, 44462220k cached
One more interesting thing is, if I delete the old backups whether using rman or simply by 'rm -rf' command, the memory released from cache to free would be exactly equal to the size of the deleted rman files as below (free memory increased to 24G from 9G; cache memory decreased from 111G to 97G):
top - 07:06:55 up 188 days, 19:51, 2 users, load average: 0.77, 0.76, 0.65
Mem: 132081600k total, 122547932k used, 9533668k free, 2153420k buffers
Swap: 17825768k total, 0k used, 17825768k free, 111547704k cached
rm -rf df_l0_NGSQA01_1321_1
rm -rf df_l0_NGSQA01_1322_1
rm -rf al_NGSQA01_1323_1
rm -rf al_NGSQA01_1324_1
rm -rf al_NGSQA01_1325_1
rm -rf al_NGSQA01_1326_1
rm -rf al_NGSQA01_1327_1
rm -rf al_NGSQA01_1328_1
rm -rf al_NGSQA01_1329_1
rm -rf al_NGSQA01_1330_1
rm -rf al_NGSQA01_1331_1
rm -rf al_NGSQA01_1332_1
top - 07:07:53 up 188 days, 19:52, 2 users, load average: 0.54, 0.69, 0.63
Mem: 132081600k total, 108021332k used, 24060268k free, 2153432k buffers
Swap: 17825768k total, 0k used, 17825768k free, 97254512k cached
Can anybody please explain to me how to correct this issue?
Thanks
Satish
Edited by: bgs on Apr 3, 2012 11:48 AM