Oracle Database Discussions

Database version - 10.2.0.4
OS : Solaris 5.8
Storage: SAN
Application : PeopleSoft Financials
DB size : 450 gb
DB server : 12 CPU ( 900 Mghz each ), 36 GB RAM
ASMM - sga_target_size = 5 gb
Locally managed tablespaces - MANUAL
- db_file_multiblock_read_count - not set explicitly in spfile ( implicitly defaulted to 128 )

- other optimizer related parameters are set to default

- system_statistics - CPUSPEEDNW=456.282722513089, IOSEEKTIM =10, IOTFRSPEED=4096

- dictionary object system stats were last gather in Nov 09

- stats on schema objs are gathered every night using custom script, but I have to say there are some histograms on some tables which were gathered by PS admins
begin
dbms_stats.gather_schema_stats(
ownname=> 'SCHEMANM' ,
cascade=> DBMS_STATS.AUTO_CASCADE,
estimate_percent=> DBMS_STATS.AUTO_SAMPLE_SIZE,
degree=> 10,
no_invalidate=> DBMS_STATS.AUTO_INVALIDATE,
granularity=> 'AUTO',
method_opt=> 'FOR ALL COLUMNS SIZE 1',
options=> 'GATHER STALE');
end;

Details :
We are experiencing erratic database performance. It was upgraded from 9i to 10g along with PS software upgrade. A job that runs on one day in 12 hrs(that is high in itself) would take more than 24 hrs another day.

We have Test and Staging envs on other servers which do not have performance issues of production magnitude. The test, staging dbs are clones of prod, created by ftp'ing files from one location to another, not sure if that makes any difference but just pointing out.

On Prod db some symptoms which baffle me are :
1) sql executing for over 40 minutes, check the plan and it is using an "incorrect" index, idx1. Hint sql with "correct" index, idx2, and it runs in few seconds. result set <400 rows. Cost with hint with idx2 is HIGHER. This scenario is still understandable as CBO is likely to pick a lower cost.
- But why is there so much discrepancy in cost and response time?
- what is influencing the CBO to pick an index which is obviously not the fastest response time?

2) sql plan shows FTS , execution time , runs forever . But a hint with index in this case shows the cost is LOWER and response time is in seconds, so why is CBO not even evaluating this index path? Because as I understand the CBO, it will always pick the lower cost plan.

What is influencing the CBO. Is it system stats? or any other database parameter? or the hardware ? the large SGA?? histogram ?? Stats gathering?? is there is any known issue with DBMS_STATS.AUTO_SAMPLE_SIZE and cardinaility?
Where should I put my focus?
What am I looking for ?

I do have Jonathan Lewis's Cost-Based Oracle Fundamentals which I have just started so perhaps I will be enlightened but while I read it, it would be of immense help to get some advice from forum gurus.

At this time I am not posting any exec plan or code, but I can do so if required.

Thanks,
JC