Developing an MVC 5 Application which calls external Web Service to get data and save to my DB. The external web service have async methods so I want to use async methods on my service layer.
I upgraded my Solution to .NET 4.5.2 to take advantage of the added feature of HostingEnvironment.QueueBackgroundWorkItem
So my MVC controller has the following:
HostingEnvironment.QueueBackgroundWorkItem(async cancellationToken =>
{
await _myService.AddCars(cars);
});
}.
return RedirectToAction("Index", "Home");
I have been told by external Web Service Client company that if count of cars passed in is less than 1000 I can run pass as is - if more than 1000 it is best to split into packages of 1000 and run up to 5 in parallel.
My method AddCars which the MVC Controller hits is shown:
public async Task AddCars(List<Car> cars)
{
if (cars.Count > 1000)
{
const int packageSize = 1000;
var packages = SplitCars(cars, packageSize);
//await Task.WhenAll(packages.Select(RunCarExternalServiceAndWriteResponseToDb));
}
else
{
await RunCarExternalServiceAndWriteResponseToDb(cars);
}
}
I have commented out the await Task.WhenAll I think I may need something like this - what I need is that if there were 3000 cars passed in - this is split into 3 packages of 1000 and then Ran in parallel - if there was 7000 for e.g - this would be split into 7 packages of 1000 with 5 being run in parallel - once that 5 return the next 2 packages of 1000 run in parallel - and then if for e.g 12,000 cars were passed in - split into 12 packages of 1000 - first batch of 5*1000 ran in parallel, then 2nd batch of 5 *1000 in parallel and finally the 3rd batch of 2*1000 ran in parallel
Can anyone suggest a good way to achieve this?
Note - my RunCarExternalServiceAndWriteResponseToDb method is marked as an async method here and it calls the actual External WebService methods - 3 of them in total - WCF External Service Calls - in each one of them a new client is created if null and in the finally block the client is closed and set to null