INSERT Slow after 500,000 Rows
452269Sep 3 2005 — edited Sep 11 2005I have a table that is fairly simple, but has a lot of overhead as it stores and uniquely identifies qualified names that may be any length.
I have multiple "agents" inserting into this database in batches of 500 objects. I just use a very simple insert statement. Initially the inserts are VERY quick, after around 500,000 rows have been inserted, the time to insert can grow to 8-10 times it's initial time. I push all the inserts into a single thread (since the agents post to the server, I sync on this table on the server, synchronizing the code around each batch of 500). Whether the table is nologging or not doesn't really seem to make a large impact. Is there anything other than optimizing Disk IO that I can do; is there a problem here that the insert performance degrades so much as the table grows?
Here is some info on the table:
The table has a LONG id (the primary key), a Unique MD5 (16 bytes) Column, a DisplayName column that is a 1024 NVARCHAR, and a CLOB field for anything that doesn't fit in the 1024 Varchar.
The MD5 column is just a digest of the complete name string and it allows quick lookups into this table which could have many millions of names in it. It also keeps the names unique. Sadly, we do joins on this table that sometimes require ORDER BY which is why as much of the name as possible is stored in the DisplayName column. If the name is longer than the DisplayName size, the rest is put into the CLOB.
Thanks!
Robert