I know its T sql but it would be really really helpful if someone can just give a suggestion
The steps involved in the current data loading
* The data comes from external sources gets cleaned and held in temporary tables now to load it into an holding database.
* Scripts like stored procedures in which a cursor is being used to
* Fetch all the data from the temporary table into the cursor
* Then fetch each row from the cursor and check for consistency of the data like is the column null...
* Then sending the data into holding database.
* This is taking a lot of hours to load the data currently.
* It is resource hungry.
My idea was to place a clustered index on the primary key of the importing table and to replace the cursor with a while loop where something like this happens
Create Procedure test
as
-- Declare Variables
declare @current_id int;
declare @next_row_id int;
declare @customer_no nvarchar(25);
declare @customer_name nvarchar(50);
declare @err_messg nvarchar(20);
declare @loop_c int;
BEGIN
--Initialize Variables
SELECT @loop_c=1;
SELECT MIN(rowid)
INTO @next_row_id
FROM table_temp
-- Checking if the data is existing
IF ISNULL(@next_row_id,0)=0
THEN
BEGIN
SELECT 'empty_row_id'
RETURN
END
-- Retrieve the first row.
SELECT @current_id = MIN(row_id)
@customer_no = customernumber
@customer_name = customername
FROM table_test
WHERE rowid=@next_row_id
--Loop
WHILE @loop_c=1
BEGIN --row by row fetch
SELECT @next_row_id=NULL -- reset variables
-- Get the next row id
SELECT @next_row_id = MIN(rowid)
@customer_no = customer_number,
@customer_name = customer_name,
from table_test
where rowid>@next_row_id
-- is it valid
IF ISNULL(@next_row_id,0)=0
BEGIN
BREAK
END
-- Get the next row from Table
SELECT @current_row_id=row_id
FROM table_test
WHERE row_id=@next_row_id
END
RETURN
Is this an effective solution.
This is just for an rough idea, there are no insert statement so kindly bear with me
Kindly suggest ways which you may think are better for improving performance
or if you think this will work fine than a cursor and decrease loading time.
Thank you for your valuable time