Analytics Software

Announcement

For appeals, questions and feedback about Oracle Forums, please email oracle-forums-moderators_us@oracle.com. Technical questions should be asked in the appropriate category. Thank you!

Processing billion records in ODI

3933331Oct 22 2019 — edited Oct 28 2019

Hi All,

I have two requirment as below mentioned

i. Once source table having 0.1 billion records will be processed into 15 target tables.There are around 700 target columns to be populated.Each target column has different logic.

ii. Once source table having 0.6 billion records will be processed into 15 target tables.There are around 1000 target columns to be populated.Each target column has different logic.

I have built a single mapping which has processed 0.1 billion of data into 15 different target tables by router transformation.

I have specified CASE statement in each of the target columns as per the requirement.

In the mapping I used IKM-Incremental update(merge) with DETECTION_STRATEGY - MINUS.

While executed it took 8 hrs to process 0.1 billion of records into 15 different target tables.

Now I am planning to process 0.6 billion of records into 15 different target tables.

I expect it will take around 48 hrs to process.

Note:This is for Intial Load only.

Is their any way to optimize this mapping?

What is the average row processing time in ODI 12c?

Need your valuable inputs.

Thanks

Added on Oct 22 2019

#business-intelligence-applications, #data-integrator, #system-management-and-integration

6 comments

665 views