|
Replies:
19
-
Pages:
2
[
1
2
| Next
]
-
Last Post:
Aug 19, 2009 3:08 PM
Last Post By: Taral Desai
|
|
|
Posts:
90
Registered:
08/26/07
|
|
|
|
db_writer_processes data sets
Posted:
Jun 3, 2009 12:19 PM
|
|
|
|
Hi All,
Can you please educate me on below things. Also, please correct me if i am wrong
1.
db_writer_processes controls number of db writers that are been started when instance is initialized. So, once DB is up lets say we have 6 dbwriter process then your buffer cache is splitted in 6 working data sets. Can you please explain what is this datasets are ? or may be if i am wrong .
2.
If so this is true how can we find that which dataset is used by which dbwrite processes. I mean how to find one to one mapping for say 0x address- to 0y address is been worked by dbwr1 process and so on. how to find this
3.
If we have multiple buffer cache like db_16k and db_32k then how do we map this.
Reason behind this we are using 12 dbwriter process for we have 12gb of sga and 10gb of pga. Some time we are getting buffer busy waits. So, need to find that we are using multiple buffer pools. So, if it is from there then i can suggest management to use only default block size and not multiple. I already know that multiple buffer pool have overhead of managing one more memory structure so need to investigate and learn from you.
My idea is to reduce this and then start adding processes if necessary
|
|
|
Posts:
8,145
Registered:
10/20/03
|
|
|
|
Re: db_writer_processes data sets
Posted:
Jun 3, 2009 1:13 PM
in response to: Taral Desai
|
|
|
|
I can not help you reverse engineer Oracle and you are taking an extraordinarily difficult path to solving a very simple solution.
Oak Table members and experts such as Jonathan Lewis and Tom Kyte have repeatedly written about the rationale behind there being
multiple block sizes and this isn't it.
Set up a test environment and demonstrate that the problem is resolved. Personally I'd tear the thing down immediately.
12 db writers? Did anyone do testing or did they just conclude bigger numbers are better?
|
|
|
Posts:
2,573
Registered:
06/27/99
|
|
|
|
Re: db_writer_processes data sets
Posted:
Jun 3, 2009 1:22 PM
in response to: Taral Desai
|
|
|
Some time we are getting buffer busy waits.
Not all waits can or should be eliminated.
What percentage of overall elapsed time is be spent on buffer busy waits (BBW)?
If BBW were reduced to ZERO, by how much would application performance be improved?
Please quantify. |
|
|
Posts:
1,781
Registered:
01/23/07
|
|
|
|
Re: db_writer_processes data sets
Posted:
Jun 3, 2009 2:06 PM
in response to: Taral Desai
|
 |
Helpful |
|
|
db_writer_processes controls number of db writers that are been started when instance is initialized. So, once DB is up lets say we have 6 dbwriter process then your buffer cache is splitted in 6 working data sets. Can you please explain what is this datasets are ? or may be if i am wrong .
There have been some significant variations in the way working data sets work since 8i - and you haven't said which version of Oracle you're running (although we can guess it's 9i or later). In 10g the number of database writers does NOT affect the number of working data sets (x$kcbwds).
The following notes are based on my most recent observations in 10g:
The number of CPUs controls the number of working data sets. Each buffer pool (and there are 8 of them) will be split into N working data sets where N is cpu_count / 2 (but I haven't tried checking what happens with an odd number of CPUs). Edit: a quick check suggests that the value is rounded down when cpu_count is odd
The number of database writer processes is cpu_count / 8 (possibly rounded up when the number of CPUs is not a multiple of 8). Edit: A quick test suggests that this assumotion about rounding up is correct.
If so this is true how can we find that which dataset is used by which dbwrite processes. I mean how to find one to one mapping for say 0x address- to 0y address is been worked by dbwr1 process and so on. how to find this
Since the number of working data sets per buffer pool is larger than (possibly 4 x) the number of database writers, I think we can assume one of two strategies - each database writer is responsible for M working data sets, or a database writer just uses a round-robin algorithm to pick the next working data set (or rather its checkpoint queue) to clear - using a nowait latch, perhaps, to bypass a working data set that is being cleared by another writer.
3.
If we have multiple buffer cache like db_16k and db_32k then how do we map this.
Question pre-empted by notes above.
Reason behind this we are using 12 dbwriter process for we have 12gb of sga and 10gb of pga. Some time we are getting buffer busy waits. So, need to find that we are using multiple buffer pools. So, if it is from there then i can suggest management to use only default block size and not multiple. I already know that multiple buffer pool have overhead of managing one more memory structure so need to investigate and learn from you.
Go and search at Kevin Closson's blog for comments on multiple DB writers - but unless you have 96 CPUs you probably don't need 12 database writers.
Buffer busy waits aren't an indicator of problems with database writers - although if you've got 12 of them competing for CPU then any buffer busy waits that have appeared may get exaggerated because of the extra competition for CPU.
If you look at v$buffer_pool_statistics you can break down the buffer busy wait by cache - and most of them are in the 16KB and 32KB cache it may be an indication that you've put data into a larger block size when an alternative solution would have been better. Nevertheless you need to work out why you're getting buffer busy waits before you expend any significant efforts to restructure the database.
Regards
Jonathan Lewis
http://jonathanlewis.wordpress.com
http://www.jlcomp.demon.co.uk
"Science is more than a body of knowledge; it is a way of thinking"
Carl Sagan
Edited by: Jonathan Lewis on Jun 4, 2009 6:48 AM
Added checks on rounding. |
|
|
Posts:
90
Registered:
08/26/07
|
|
|
|
Re: db_writer_processes data sets
Posted:
Jun 3, 2009 3:18 PM
in response to: Jonathan Lewis
|
|
|
My bad We are using 10.2.0.3 and solaris 10 sparc 64 bit environment
Thanks Dan & Jonathan For helping me out understanding things. Because i want to present it to my management and some DBA's disagree with this things so i am planning to have some test setup to show that 12 dbwriter is not so necessary.
Also, I don't know what kind of test to provide can you please make light on this.
I tried to explain some DBA's saying that we have default 16k block size and there is no need to 32k cache for faster things. It could have been worse but they disagree and here are the proof
SQL>
SQL> select
2 block_size, free_buffer_wait, buffer_busy_wait, dirty_buffers_inspected,
3 physical_reads, physical_writes
4 from
5 v$buffer_pool_statistics
6 ;
BLOCK_SIZE FREE_BUFFER_WAIT BUFFER_BUSY_WAIT DIRTY_BUFFERS_INSPECTED PHYSICAL_READS PHYSICAL_WRITES
---------- ---------------- ---------------- ----------------------- -------------- ---------------
16384 102982 58474 296163 5986981 11116073
32768 107978876 137272925 109161143 319435124 144205406
Also one more thing does cpu_count parameter affect dbwriter processes. As it seems from your reply in that formula if so then in our environment previous dba might had made change to this parameter as we have 4 CPU development box but value for cpu_count is set to 8.
Management want me to provide some test case for reducing dbwriter process and also blocksize to default will make performance benifit and i don't know where to start. |
|
|
Posts:
1,781
Registered:
01/23/07
|
|
|
|
Re: db_writer_processes data sets
Posted:
Jun 3, 2009 10:31 PM
in response to: Taral Desai
|
|
Helpful |
|
|
SQL>
SQL> select
2 block_size, free_buffer_wait, buffer_busy_wait, dirty_buffers_inspected,
3 physical_reads, physical_writes
4 from
5 v$buffer_pool_statistics
6 ;
BLOCK_SIZE FREE_BUFFER_WAIT BUFFER_BUSY_WAIT DIRTY_BUFFERS_INSPECTED PHYSICAL_READS PHYSICAL_WRITES
---------- ---------------- ---------------- ----------------------- -------------- ---------------
16384 102982 58474 296163 5986981 11116073
32768 107978876 137272925 109161143 319435124 144205406
Is that the whole of the output ?
If so, do you have a default size of 16KB with an alternate of 32KB, or is your default block size 32KB ?
The output of statspack or AWR during a busy period would be more useful that this query, since the view shows statistics since database startup, so the numbers (without a time scale) don't really tell us anything.
Load profile, Top 5, Time model, OS stats (basically the first couple of pages) plus Instance Statistics, and in your case the buffer cache section (whatever it's called) would be a good start.
Addendum:
In answer to your question, the cpu_count does affect the number of working data sets and database writers directly - your development box will have 2 working data sets per cache size and your production box will have 4 working data sets per cache size if my comments are correct. And if you have only the two cache areas you've got more database writers (you said 12) than working data sets (total 8) - which is counter-productive.
Regards
Jonathan Lewis
http://jonathanlewis.wordpress.com
http://www.jlcomp.demon.co.uk
"Science is more than a body of knowledge; it is a way of thinking"
Carl Sagan |
|
|
Posts:
186
Registered:
12/19/07
|
|
|
|
Re: db_writer_processes data sets
Posted:
Jun 4, 2009 1:59 AM
in response to: Taral Desai
|
|
|
Also one more thing does cpu_count parameter affect dbwriter processes?
I think what Jonathan was referring to was how db_writer_processes gets set by default when you do not explicitly assign it a value in your parameter file. So by default it would be cpu_count / 8 rounded up, as documented in the Reference Manual. If you set db_writer_processes explicitly, then the cpu_count calculation is no longer relevant - the value of db_writer_processes is whatever you have set it to.
At least, I think that's what Jonathan meant. I hope I haven't got it wrong.
Oracle now uses asynchronous I/O by default. So a single Database Writer is able to schedule many I/O writes one after the other, with no delay, and later deal with the acknowledgements that they have completed. So one Database Writer is only really limited by the speed of the CPU and the internal I/O scheduling mechanism of the operating system - it is not limited by the actual disk speed at all. Faster CPUs should have no affect on how many Database Writer processes you need.
Which means that rather than explicitly setting the db_writer_processes value, you can let it default on both systems to the lower value. You could always test it on your test system - does setting db_writer_processes to 1 on the test system have any impact on your application performance or on these buffer busy waits you are seeing? If there is no difference, then your problem is nothing to do with the number of database writers.
The only major circumstance I can think of for having multiple Database Writer processes is when you do not have asynchronous I/O, and so it is using synchronous I/O instead. In this case the Database Writer has to wait for each I/O to complete, and can only issue one I/O at a time. But most systems now fully support asynchronous I/O.
Sometimes this happens when storing the data files as ordinary files in a file system. But Solaris does have asynchronous I/O by default, and does support it on its file systems (using threads inside light weight processes). Of course if you are using a different file system, such as Veritas, then this is not necessarily true. Either way you should have disk_asynch_io as true (which it is by default) and filesystemio_options as "setall", which supports both async I/O and Direct I/O on Solaris.
John |
|
|
Posts:
340
Registered:
10/17/06
|
|
|
|
Re: db_writer_processes data sets
Posted:
Jun 4, 2009 5:35 AM
in response to: John Brady
|
|
|
The only major circumstance I can think of for having multiple Database Writer processes is when you do not have asynchronous I/O, and so it is using synchronous I/O instead. In this case the Database Writer has to wait for each I/O to complete, and can only issue one I/O at a time. But most systems now fully support asynchronous I/O.
If this is the case then setting DBWR_IO_SLAVES may be a better option, although most systems do support async IO I still come accross the odd one or two that doesn't have the capability. |
|
|
Posts:
90
Registered:
08/26/07
|
|
|
|
Re: db_writer_processes data sets
Posted:
Jun 4, 2009 6:59 AM
in response to: Jonathan Lewis
|
|
|
|
Hi Jonathan,
Thank you so very much for the reply and explanination.
16k block size is our default blocksize and 32k is alternative.
"Addendum:
In answer to your question, the cpu_count does affect the number of working data sets and database writers directly - your development box will have 2 working data sets per cache size and your production box will have 4 working data sets per cache size if my comments are correct. And if you have only the two cache areas you've got more database writers (you said 12) than working data sets (total 8) - which is counter-productive.
"
Can you please explain above in detail. Like lets say we have 4 cpu box with exact same cpu_count=4 and we have 12 dbwriter processes.
Edited by: Taral Desai on Jun 4, 2009 7:08 AM
Edited by: Taral Desai on Jun 4, 2009 7:09 AM
|
|
|
Posts:
1,781
Registered:
01/23/07
|
|
|
|
Re: db_writer_processes data sets
Posted:
Jun 4, 2009 9:43 AM
in response to: Taral Desai
|
|
Helpful |
|
|
Can you please explain above in detail. Like lets say we have 4 cpu box with exact same cpu_count=4 and we have 12 dbwriter processes.
I've just engineered a system into this state - and I'm wrong about the effect of db_writer_processes.
My assumption was that if you have 4 CPUs (cpu_count = 4) then Oracle will allow 2 working data sets per cache - for a possible maximum of 16 working data sets. But you've only got 2 caches active, the 16KB and the 32KB, so you would have 4 working data sets, and with db_writer_processes = 12 you would have 8 database writers with "nothing to do".
What I actually see when I start a database with cpu_count = 4, db_writer_processes = 12, and two caches is:
Each cache gets split into 12 working data sets,
each database writer acquires one working data set from each cache
I wrote a quick script to show this - joining x$kcbwds (working data sets) with x$kcbwbpd (buffer pools) - but you have to be connected as SYS to run it:
break on bp_blksz skip 1 -
on bp_name skip 1 -
on bp_size
select
bfp.bp_blksz,
bfp.bp_name,
bfp.bp_size,
wds.indx wds_index,
wds.dbwr_num,
wds.set_latch
from
x$kcbwbpd bfp,
x$kcbwds wds
where
bfp.bp_size != 0
and wds.set_id between bfp.bp_lo_sid and bfp.bp_hi_sid
order by
bfp.bp_blksz,
bfp.bp_name,
wds.indx
;
Here's a sample of the output when I set up a small keep cache:
BP_BLKSZ BP_NAME BP_SIZE WDS_INDEX DBWR_NUM SET_LATC
---------- -------------------- ---------- ---------- ---------- --------
8192 DEFAULT 7984 24 0 1CB8CF20
25 1 1CB8D2D0
26 2 1CB8D680
27 3 1CB8DA30
28 4 1CB8DDE0
29 5 1CB8E190
30 6 1CB8E540
31 7 1CB8E8F0
32 8 1CB8ECA0
33 9 1CB8F050
34 10 1CB8F400
35 11 1CB8F7B0
KEEP 1996 0 0 1CB876A0
1 1 1CB87A50
2 2 1CB87E00
3 3 1CB881B0
4 4 1CB88560
5 5 1CB88910
6 6 1CB88CC0
7 7 1CB89070
8 8 1CB89420
9 9 1CB897D0
10 10 1CB89B80
11 11 1CB89F30
The set_latch is the addr column from v$latch_children for the "cache buffers lru chains" latch for that working data set. There are lots more columns you could report from x$kcbwds - including the breakdown by working data set of free buffer waits (FBWAIT) and buffer busy waits (BBWAIT) were appearing.
Regards
Jonathan Lewis
http://jonathanlewis.wordpress.com
http://www.jlcomp.demon.co.uk
"Science is more than a body of knowledge; it is a way of thinking"
Carl Sagan |
|
|
Posts:
1,781
Registered:
01/23/07
|
|
|
|
Re: db_writer_processes data sets
Posted:
Jun 4, 2009 9:53 AM
in response to: chris_c
|
|
Helpful |
|
|
If this is the case then setting DBWR_IO_SLAVES may be a better option, although most systems do support async IO I still come accross the odd one or two that doesn't have the capability.
Chris,
In the absence of asynch_io I think there's an argument to favour multiple db writer processes over IO slaves: if you have multiple dbwN processes you have multiple processes clearing blocks from multiple checkpoint chains, and spread the contention on the LRU and CKPT latches.
On the other hand, if you have a problem clearing the log buffer down to the redo log (assuming multiplexed files) then enabling I/O slaves automatically allows the log writer to hand off to I/O slaves as well - and this may allow "log file parallel writes" to operate more quickly.
Of course, Oracle does not allow multiple db writer processes AND I/O slaves at the same time. (But there are separate parameters for different I/O slaves for different functions, although I've never tried to play silly games with them).
So, as ever, it's swings and roundabouts and you try to address the most important bottleneck.
Regards
Jonathan Lewis
http://jonathanlewis.wordpress.com
http://www.jlcomp.demon.co.uk
"For every expert there is an equal and opposite expert."
Arthur C. Clarke
|
|
|
Posts:
90
Registered:
08/26/07
|
|
|
|
Re: db_writer_processes data sets
Posted:
Jun 4, 2009 11:39 AM
in response to: Jonathan Lewis
|
|
|
Hi Jonathan,
Thank you very much so here are the things which we can conclude from this. Please correct me if i am wrong as i am new to this whole thing.
1.
Excessive db writer processes can be overhead due to each working set requires it's own db lru latches.
2.
Excessive db writer processes can slow down I/O due to they need to acquire CPU and then run. But one question here is does all db writer process simultinouses compete for cpu or it waits for one to complete and then go to other cpu. As, i can understand cpu uses serialize mechanism but lets say we have 12 dbwriter among one is writing using one cpu and other cpu is free. So, other process can move to that or db writer process is also serial ?
3.
Also, this can lead to wrong buffer waits situation as we can have free buffer in other sets and in one set it won't and process acquire that dbwriter so that it will post this wait event but actually we have other free buffers available. How to find this as how many buffers are available, dirty or in other state for each working set ?
4.
NAME TYPE VALUE
-----------
db_32k_cache_size big integer 32M
So, it means that for 32MB scanning of memory we need 12 data working sets or this 32k block size dbwriter data sets can be used by 16k also ?
And here is the output from our environment
SQL> select bfp.bp_blksz,
bfp.bp_name,
bfp.bp_size,
wds.indx wds_index,
wds.dbwr_num,
wds.FBWAIT,
wds.BBWAIT,
wds.set_latch,
l.gets,
l.MISSES,
l.SLEEPS
from x$kcbwbpd bfp, x$kcbwds wds, v$latch_children l
where bfp.bp_size != 0
and wds.set_id between bfp.bp_lo_sid and bfp.bp_hi_sid
and l.ADDR = wds.set_latch
order by bfp.bp_blksz, bfp.bp_name, wds.indx;
BP_BLKSZ BP_NAME BP_SIZE WDS_INDEX DBWR_NUM FBWAIT BBWAIT SET_LATCH GETS MISSES SLEEPS
---------- -------------------- ---------- ---------- ---------- ---------- ---------- ---------------- ---------- ---------- ----------
16384 DEFAULT 370845 24 0 8753 6822 000000053D44E478 2088984 724 39
16384 DEFAULT 370845 25 1 8721 7027 000000053D44E9D0 2093594 962 29
16384 DEFAULT 370845 26 2 8543 4382 000000053D44EF28 2491595 962 29
16384 DEFAULT 370845 27 3 7907 4836 000000053D44F480 2155756 1042 43
16384 DEFAULT 370845 28 4 7655 5722 000000053D44F9D8 2131806 978 30
16384 DEFAULT 370845 29 5 7270 4440 000000053D44FF30 2117001 1074 34
16384 DEFAULT 370845 30 6 8350 2804 000000053D450488 2087645 930 35
16384 DEFAULT 370845 31 7 7841 6987 000000053D4509E0 2145802 908 37
16384 DEFAULT 370845 32 8 8699 4890 000000053D450F38 2119788 928 35
16384 DEFAULT 370845 33 9 8074 4679 000000053D451490 2167632 925 29
16384 DEFAULT 370845 34 10 8666 5068 000000053D4519E8 2103990 902 27
16384 DEFAULT 370845 35 11 12503 7753 000000053D451F40 2150953 991 38
BP_BLKSZ BP_NAME BP_SIZE WDS_INDEX DBWR_NUM FBWAIT BBWAIT SET_LATCH GETS MISSES SLEEPS
---------- -------------------- ---------- ---------- ---------- ---------- ---------- ---------------- ---------- ---------- ----------
32768 DEFAULT 1012 84 0 11205160 11401610 000000053D462518 37611275 129971 1188
32768 DEFAULT 1012 85 1 9643066 11547834 000000053D462A70 36582915 106640 1157
32768 DEFAULT 1012 86 2 10446003 11583563 000000053D462FC8 37427372 102028 1088
32768 DEFAULT 1012 87 3 10573808 11563144 000000053D463520 37295902 104804 1058
32768 DEFAULT 1012 88 4 10408529 11452657 000000053D463A78 37088311 102178 1169
32768 DEFAULT 1012 89 5 10339793 11544617 000000053D463FD0 37020968 102712 1097
32768 DEFAULT 1012 90 6 10385856 11522059 000000053D464528 37197322 101601 1084
32768 DEFAULT 1012 91 7 10379007 11495563 000000053D464A80 37131610 99125 982
32768 DEFAULT 1012 92 8 10388032 11491618 000000053D464FD8 37119317 100206 1081
32768 DEFAULT 1012 93 9 10437875 11557976 000000053D465530 37188171 100933 1108
32768 DEFAULT 1012 94 10 10004858 11523490 000000053D465A88 36865058 89088 1075
32768 DEFAULT 1012 95 11 11481299 11509310 000000053D465FE0 38390757 127075 1088
24 rows selected.
Edited by: Taral Desai on Jun 4, 2009 1:25 PM |
|
|
Posts:
1,781
Registered:
01/23/07
|
|
|
|
Re: db_writer_processes data sets
Posted:
Jun 5, 2009 1:23 AM
in response to: Taral Desai
|
|
Helpful |
|
|
Taral,
I like what you've done with the query I posted.
Before answering your specific questions, I'd like to point out that the more important problem is the sizing (for its use) of the 32KB cache. It's too small - and as a simple, low impact change, it looks as if you could safely reduce the size of the 16KB cache to increase the size of the 32KB cache.
This is just taking the short term "urgent" response, based on minimal information. The fact that you have time lost on the 16KB cache is something you may need to address eventually, but I'm taking the view point that we should be able to add a small amount of contention in the 16KB cache to remove a large amount of contention in the 32KB cache. We still need to see some basic statspack/AWR information before we can offer any strategic advice.
I'd start by subtracting 32MB from the 16KB cache and adding it to the 32KB cache, and monitoring for a while, and repeat a few times. Use statspack to take snapshots to see how much impact you have each time you change the settings.
Addressing the free buffer waits - it is commonly the case that free buffer waits appear because the database writer is unable to write fast enough, but there are other reasons, and in your case because the buffer is small it is possible that one session can't get a free buffer because all the blocks in the buffer are still pinned by other users and therefore can't be written (or even discarded if they are clean). At a minimum, if you make a lot of use a the cache the size of the cache should be "a few" blocks per concurrent session - where "a few" means something in the range 4 to 12 blocks.
Coming back to your questions:
1) Yes, excess db writers can be the cause of overhead because of (a) the extra latch activity and (b) the concurrency running - the db writers don't know about each other, it's perfectly possible for all 12 of them to want to run at once and (c) they introduce comptetiion on the cache buffers chains latches as well ... (d) probably other but I can't think of them off the top of my head.
2) Answered in (1) I think
3) INtersting question - I may have seen (or written) some notes somewhere about how a process picks an LRU list when it wants a free buffer, but I don't remember how it's done. (Also, my memory is likely to be out of date given the mistake I made earlier on). I think there's likely to be an element of wasted work/time relating to the type of thing you're thinking of - but I'd guess that a process picks an LRU, finds no free buffers on it, posts the relevant DBWn and then jumps to another LRU and repeats.
Regards
Jonathan Lewis
http://jonathanlewis.wordpress.com
http://www.jlcomp.demon.co.uk
"For every expert there is an equal and opposite expert."
Arthur C. Clarke |
|
|
Posts:
90
Registered:
08/26/07
|
|
|
|
Re: db_writer_processes data sets
Posted:
Jun 8, 2009 1:34 PM
in response to: Jonathan Lewis
|
|
|
|
Hi Jonathan,
Correct me if i am wrong. In our case for 32MB of scanning for 32k block cache it had split to 12 working data sets. So, for 32MB cache we had many overheads for managing this 12 sets and also latchs.
Can't we instruct oracle to use only one working dataset here.
|
|
|
Posts:
1,781
Registered:
01/23/07
|
|
|
|
Re: db_writer_processes data sets
Posted:
Jun 9, 2009 3:48 PM
in response to: Taral Desai
|
|
|
Hi Jonathan,
Correct me if i am wrong. In our case for 32MB of scanning for 32k block cache it had split to 12 working data sets. So, for 32MB cache we had many overheads for managing this 12 sets and also latchs.
Can't we instruct oracle to use only one working dataset here.
Not that I know of.
But I pointed out in my last response that the more important issue is the size of the 32KB buffer - you need to make it larger, even if it means shrinking the 16KB buffer. The number of db writer processes is a minor detail relative to this.
Regards
Jonathan Lewis
http://jonathanlewis.wordpress.com
http://www.jlcomp.demon.co.uk
"Science is more than a body of knowledge; it is a way of thinking"
Carl Sagan |
|
|
|
Legend
|
|
Guru : 2500
- 1000000
pts
|
|
Expert : 1000
- 2499
pts
|
|
Pro : 500
- 999
pts
|
|
Journeyman : 200
- 499
pts
|
|
Newbie : 0
- 199
pts
|
|
Oracle ACE Director
|
|
Oracle ACE Member
|
|
Oracle Employee ACE
|
|
|
Helpful Answer
(5 pts)
|
|
Correct Answer
(10 pts)
|
|
