Delta Calculation and Updating multiple tables
710933Jul 8 2009 — edited Jul 10 2009We pull data from a System of Records table that contains the most up to date information. The information changes daily so we have a delta process to identify what new records were added, which records were deleted (records that are not found in the table as compared to yesterday) and which were updated. Delta process compares the already loaded data with the newly updated SOR data to find the differences.
Once the delta is established, either new records get added or existing records get updated or existing records are marked as inactive (Deletes). Additions and Updates generally happen across multiple destination tables.
Updates are identified by looking at different columns to see if any one column is changed. These columns end up in different tables.
Example
Source Delta Table, S1
ID COL1 COL2 COL3 ACTION
1 abc xyz pqr A
2 bcd lmn def U
S1.Col1 maps to Destination Table D1.Col23
S1.Col2 maps to Destination Table D2.Col45
S1.Col3 maps to Destination Table D3.Col11
Currently all tables are updated irrespective of whether the relevant data has changed or not (All 3 destination tables are updated).
I would like to know which of the Columns for a given row has changed values so that I can update only the relevant tables.
Thus if additional columns are available that act as flags
Source Delta Table, S1
ID COL1 COL2 COL3 ACTION COL1 COL2 COL3
1 abc xyz pqr A - - -
2 bcd lmn def U N Y N
3 kjh qwe iop U Y Y N
then for incoming ID=2, I just have to update Destination Table D2 and not D1 and D3
for incoming ID= 3, I have to update Destination Tables D1 and D2 but not D3.
How can I achieve that?
This is mainly to improve performance as the processing time is very short - Faster the delta processing, better will it be.
Thanks in advance.