I have a background job that processes a list of items. However, the larger the list, the time to complete it grows exponentially. Not just that, the further down the list, the app gradually eats up my server's 64gb ram and slows down the server to a crawl.
I use Hangfire as my background job manager, and the job takes a list of id and performs multiple database actions on them. I access the database through ef core, using the repository pattern provided by aspnetboilerplate framework.
I assume it is slow because of the growing db context, so here is what I have tried:
Split the list across multiple jobs.
var splitIdList = ListExtensions.ChunkBy(idList, 100); foreach (var split in splitIdList) { await _backgroundJobManager.EnqueueAsync<MyJob, MyJobArgs>(new MyJobArgs(...)) }
Split the job into multiple unit of work
var splitIdList = ListExtensions.ChunkBy(idList, 100); foreach (var split in splitIdList) { using (var uow = UnitOfWorkManager.Begin()) { foreach(var id in split) { //create related entity .... //create barcode .... //link entity to created barcode .... } uow.Complete(); } }
Split the job into multiple unit of work with new transactions
var splitIdList = ListExtensions.ChunkBy(idList, 100); foreach (var split in splitIdList) { using (var uow = UnitOfWorkManager.Begin(System.Transactions.TransactionScopeOption.RequiresNew)) { foreach(var id in split) { //create related entity .... //create barcode .... //link entity to created barcode .... } uow.Complete(); } }
None seems to solve the problem, any help would be appreciated, thank you.
EDIT Here is what I do to each entity in the job:
- I create another entity and link them together.
- I create a qrcode, which is a running number, and link it to the entity
Operations involved are create, get, insert, and update