ASM shows space available, but gives exhausted message when I try to use it
RByrdJun 5 2013 — edited Jun 6 2013Background:
Physical standby database at a hands off colo 3000 Miles from here.
oracle11.2.0.2
GI 11.2.0.2
Linux vaxrddb01 2.6.32-300.21.1.el5uek #1 SMP x86_64 x86_64 x86_64 GNU/Linux
3 single disk disk groups:
va_gi - small ocr, etc.
vi_main - top tier storage
vi_archive lower tier storage.
ASMCMD [+] > lsdg
State Type Rebal Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Voting_files Name
MOUNTED EXTERN N 512 4096 1048576 2097152 52154 0 52154 0 N VA_ARCHIVE/
MOUNTED EXTERN N 512 4096 1048576 1907 1554 0 1554 0 Y VA_GI/
MOUNTED EXTERN N 512 4096 1048576 2095222 38369 0 38369 0 N VA_MAIN/
VA_ARCHIVE shows about 50G available.
1* select group_number, os_mb, total_mb, free_mb, state, path, name, failgroup from v$asm_disk order by 1
GROUP_NUMBER OS_MB TOTAL_MB FREE_MB STATE PATH NAME FAILGROUP
------------ ---------- ---------- ---------- -------- ----------------------------------- -------------------- --------------------
1 1907 1907 1554 NORMAL /dev/oracleasm/disks/VA_GI VA_GI_0000 VA_GI_0000
2 2095222 2095222 38369 NORMAL /dev/oracleasm/disks/VA_MAIN_1 VA_MAIN_1 VA_MAIN_1
3 2097152 2097152 52154 NORMAL /dev/oracleasm/disks/VA_ARCHIVE_1 VA_ARCHIVE_1 VA_ARCHIVE_1
When I try to copy a 46G file from Main to Archive, the command fails with 'space exhausted'.
1* select name, file#, bytes/1024/1024/1024 from v$datafile where file# = 167
NAME FILE# BYTES/1024/1024/1024
---------------------------------------------------------------------------------------------------- ---------- --------------------
+VA_MAIN/prdx/datafile/daily_statistics_index_011301.259.809226329 167 46.7099609
RMAN> copy datafile '+VA_MAIN/prdx/datafile/daily_statistics_index_011301.259.809226329' to '+VA_ARCHIVE';
Starting backup at 05-JUN-13
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=68 device type=DISK
channel ORA_DISK_1: starting datafile copy
input datafile file number=00167 name=+VA_MAIN/prdx/datafile/daily_statistics_index_011301.259.809226329
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of backup command on ORA_DISK_1 channel at 06/05/2013 15:47:00
ORA-19504: failed to create file "+VA_ARCHIVE"
ORA-17502: ksfdcre:4 Failed to create file +VA_ARCHIVE
ORA-15041: diskgroup "VA_ARCHIVE" space exhausted
RMAN>
Is there an underlying physical structure that prevents this operation?
Something about the LUN that I do not see?
An 'invisible' file that ASM does not see as 'non-free' space?
I would appreciate any clues that might help me solve this mystery.
I recognize we are low on space. I am bailing as fast as I can. I am not an SA, so system insight might be particularly useful.