Creating big amount of Sitecore Items causes memor

Question

I have a Sitecore application where I need to create a large number of Sitecore items in a loop from another system. Below is my testing purpose loop;

using (new Sitecore.SecurityModel.SecurityDisabler())
{
    for (int i = 0; i < 20000; i++)
    {
        Item newItem = MyTemplateItem.CreateItemFrom(DateTime.Now.Ticks.ToString(), myParentItem);

        newItem.Editing.BeginEdit();
        newItem.Fields["Title"].Value = Guid.NewGuid().ToString();
        newItem.Editing.EndEdit();
    }
}

When this loop is running and while looking in task manager, the process memory usage is getting increased with the time.

Sitecore Item class has not implemented IDisposable interface, so I cannot call the Finalizer of the created item.

How can I avoid this memory leak?

PS: I'm using a windows application to perform this operation to bypass the IIS process memory leak where Sitecore does its cache updating and index building which should be done at the end of this process.

Answer 1

If you set your cache limits to high values (i.e. too high) and do what you're doing, it would seem reasonable to get the symptoms you're reporting. Looking at the page /sitecore/admin/cache.aspx while your script is running will show if this is the case. I'd image the entries master[items]and master[data] will keep on climbing...

If this is the case, simply setting a lower cache limit will mean the caches are emptied as needed, which seems a healthier way to keep things under control.

Also, if you want to dramatically speed up programmatic item creation, perhaps try using

using (new BulkUpdateContext()) { code.. }

It's possible that BulkUpdateContext() also skips adding items to the cache - I haven't checked if this is the case, but if so, it too would likely 'fix' the memory issue you have.

(also see this SO answer). Note that it disables several pipelines.

Answer 2

You can temporarily disable the database cache during the item creation:

using (new Sitecore.Data.DatabaseCacheDisabler())

You can disable indexing during the creation using:

Sitecore.Configuration.Settings.Indexing.Enabled = false;

Try to add those two things and see if it helps.

UPDATE: Maybe this will work. Add this inside your loop after EndEdit():

newItem = null;

if (i % 100 == 0)
{
  GC.Collect();
  GC.WaitForPendingFinalizers();
}

It will force the garbage collector to try reclaim unused memory every time 100 items have been added.

If this works then you don't really have to call the GC yourself, it will happen automatically at some point, you just don't know when.

Answer 3

Ruud's answer seems to work better than it was. The growth of the memory was rapid due to the database cache building. By disabling drastically slow down the memory growth, but still the growth is there in very slowly.

I have used the following statements to achieve it;

using (new Sitecore.Data.DatabaseCacheDisabler())

newItem = null; // did not worked

// replaced above line with the below
Marshal.Release(Marshal.GetIUnknownForObject(newItem));

if (i % 100 == 0)
{
   GC.Collect();
   GC.WaitForPendingFinalizers();
}

Thanks for all the supporters!