Skip to Main Content

SQL & PL/SQL

Announcement

For appeals, questions and feedback about Oracle Forums, please email oracle-forums-moderators_us@oracle.com. Technical questions should be asked in the appropriate category. Thank you!

Interested in getting your voice heard by members of the Developer Marketing team at Oracle? Check out this post for AppDev or this post for AI focus group information.

Tuning migration of a column

Prokopios Poulimenos2 days ago — edited 2 days ago

Hello everyone,
I have two BIG tables with names MSG_EXC_REC and MOVEMENT, and the relation is that multiple records of MSG_EXC_REC create a row of MOVEMENT using key MRN, DOMAIN and ENV .

Both tables are partitioned.

MSG_EXC_REC is partitioned as below

 PARTITION BY RANGE ("M_MIN_DATE") INTERVAL (NUMTOYMINTERVAL(1, 'MONTH')
 PARTITION "BEFORE_2010"  VALUES LESS THAN (TIMESTAMP' 2010-01-01 00:00:00')

MOVEMENT is partitioned as below

 PARTITION BY RANGE ("MIN_DATE") INTERVAL (NUMTOYMINTERVAL(1, 'MONTH')) 
 PARTITION "BEFORE_2010"  VALUES LESS THAN (TIMESTAMP' 2010-01-01 00:00:00')

On table MSG_EXC_REC there is a column IS_IECA NUMBER(1,0) and for performance reason I want to migrate this column and add it to the table MOVEMENT.
The Movement.is_ieca column will be set to 1 if and only if exists a message for the same combination (domain,mrn,env) in MSG_EXC_REC with value is_ieca=1.
It's worth to notice that the MOVEMENT.IS_IECA has been initiated with 0 (default 0).

Below you can find my actions to do the migration with plsql resulting in 1915sec --> 00:31:55 (per semester)

CREATE TABLE MIGRATE_IECA_ERROR_LOG (
ERROR_ID      NUMBER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
ERROR_MESSAGE VARCHAR2(4000),
ERROR_DATE    TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

DECLARE
 -- Create date ranges of month
 CURSOR get_ranges_cur(p_start_date TIMESTAMP)
 IS
 select add_months ( p_start_date,  level - 1 ) as start_date
      , add_months ( p_start_date,  level ) as end_date
   from dual  
connect by level <= months_between ( TRUNC(SYSDATE), p_start_date) + 1
  order by start_date desc;
 -- Get the distinct triplets for a period
 CURSOR movement_triplets_cur(p_start_date TIMESTAMP, p_end_date TIMESTAMP)
 IS
 SELECT ID, MRN, DOMAIN, ENV
   FROM MOVEMENT
  WHERE DOMAIN IN ('NCTS', 'ECS')
    AND MIN_DATE >= p_start_date
    AND MIN_DATE < p_end_date;
 TYPE movements_t IS TABLE OF movement_triplets_cur%ROWTYPE INDEX BY PLS_INTEGER;
 l_movements_data   movements_t;
 v_limit        PLS_INTEGER := 2000;
 v_start_date   TIMESTAMP := TO_TIMESTAMP('01/01/2024 00:00:00', 'DD/MM/YYYY HH24:MI:SS');
 l_error_count  NUMBER := 0;
 -- --------------
 -- Logging errors
 PROCEDURE migrate_ieca_error_log(p_error_msg VARCHAR2)
 IS
   PRAGMA AUTONOMOUS_TRANSACTION;
 BEGIN
   INSERT INTO migrate_ieca_error_log (error_message, error_date)
   VALUES (p_error_msg, SYSDATE);
   COMMIT;
 END migrate_ieca_error_log;
 -- --------------
BEGIN
 <<RANGE_LOOP>>
 FOR REC IN get_ranges_cur(v_start_date)
 LOOP
   OPEN movement_triplets_cur(REC.start_date, REC.end_date);
   LOOP
     FETCH movement_triplets_cur
     BULK COLLECT INTO l_movements_data LIMIT v_limit;
     EXIT WHEN l_movements_data.COUNT = 0;
     BEGIN
       FORALL i IN l_movements_data.FIRST..l_movements_data.LAST SAVE EXCEPTIONS
         UPDATE MOVEMENT M
            SET M.IS_IECA = 1
          WHERE exists ( select 1
                           from msg_exc_rec R
                            WHERE R.MRN = l_movements_data(i).MRN
                             AND R.DOMAIN = l_movements_data(i).DOMAIN
                             AND R.ENV = l_movements_data(i).ENV
                             AND R.IS_IECA = 0)
            AND M.ID = l_movements_data(i).ID;
         COMMIT;
     EXCEPTION
       WHEN OTHERS THEN
         l_error_count := SQL%BULK_EXCEPTIONS.COUNT;
         migrate_ieca_error_log('Number of failures: ' || l_error_count);
         FOR i IN 1 .. l_error_count LOOP
           migrate_ieca_error_log('Error: ' || i ||
                                  ' Array Index: ' || SQL%BULK_EXCEPTIONS(i).ERROR_INDEX ||
                                  ' Message: ' || SQLERRM(-SQL%BULK_EXCEPTIONS(i).ERROR_CODE));
         END LOOP;
     END;
     EXIT WHEN l_movements_data.COUNT = 0;
   END LOOP;
   COMMIT;
   CLOSE movement_triplets_cur;
 END LOOP RANGE_LOOP;
 COMMIT;
EXCEPTION
 WHEN OTHERS THEN
   migrate_ieca_error_log('Unhandled Exception: ' || SQLERRM);
   RAISE;
END;
/

and the sql approach resulting in 3786sec→01:03:06

MERGE INTO MOVEMENT M
USING (
 SELECT MRN, DOMAIN, ENV
   FROM MSG_EXC_REC
  WHERE DOMAIN IN ('NCTS', 'ECS')
    AND M_MIN_DATE >= TO_TIMESTAMP('01/01/2024 00:00:00', 'DD/MM/YYYY HH24:MI:SS')
    AND M_MIN_DATE <= TRUNC(SYSDATE)
    AND IS_IECA = 1
  GROUP BY MRN, DOMAIN, ENV
) E
ON (M.MRN = E.MRN AND M.DOMAIN = E.DOMAIN AND M.ENV = E.ENV)
WHEN MATCHED THEN
 UPDATE SET M.IS_IECA = 1
 WHERE M.IS_IECA = 0;
 
commit;

I think that the difference is based on the fact that the plsql block uses the monthly partitions.
Is there any way to do it more efficient ?

I have also indexes on MRN, DOMAIN, ENV in both tables.

Thanks in advance

Comments
Post Details
Added 2 days ago
12 comments
113 views