What is the fastest way to run an exe command many

Question

My code (written in C#) runs an exe command many times (800 times average).

Currently I run the exe command as a Process in C#:

  var process1 = new Process()
  {
      StartInfo = new ProcessStartInfo()
      {
          FileName = "latex",
          Arguments = String.Format("-quiet -output-directory=\"{0}\" \"{1}\"", equationDirectory, equationTEX),
          WorkingDirectory = equationDirectory,
          CreateNoWindow = true,
          UseShellExecute = false,
          RedirectStandardError = true,
          RedirectStandardOutput = true
      }
  };
  process1.Start();

This is taking so much time, some of it is the Windows starting the shell process.

Question I was wondering if it is faster to embed the exe in my code and run it? What is the fastest way to run an executable many times (in a loop let say)?

Answer 1

I don't know how running processes in c# works at the low level, but I have to imagine its not very efficient. You should look into some abstractions for how to run your exe, especially for the amount you're running it. The obvious one that comes to mind is using threads instead.

Threads are born from inside of a process so they share the address space of the parent process. This reduces some of the overhead associated with memory allocation. They also are more lightweight with regards to creation and destruction, which will greatly benefit your runtime.

There are many resources to running code inside threads, take a look around and try that as an alternative to speed up your code.

Answer 2

The approach I've used before is to simply look at your Windows resources - its very likely not to be CPU bound, and probably not IO bound either. Keep increasing the number of parallel Processes until you see either of these values get saturated. My similar problem was with a Process (DirectX Effect Compiler) which took 10 seconds. During that 10 seconds most of my resources were idle (the application was procedurally written and therefore didn't take advantage of the many cores available). Creating a List of Tasks and WaitAll() on that List reduced total time-to-completion radically.

In my case the total time to complete a single task did not changed; but it was clear my computer was far from dedicated to running that one task - multi-tasking the calls to initiate these tasks made more use of the existing resources and therefore reduced the time-to-completion.

Answer 3

It may be that running them all at once forces them to compete for resources, for example - using up all your system memory, causing paging to the HDD.

Maybe using something like below, and testing the number that can run concurrently, you'll find a sweet-spot:

var processes = new List<Process>();

var process1 = new Process()
{
    StartInfo = new ProcessStartInfo()
    {
        FileName = "latex",
        Arguments = String.Format("-quiet -output-directory=\"{0}\" \"{1}\"", equationDirectory, equationTEX),
        WorkingDirectory = equationDirectory,
        CreateNoWindow = true,
        UseShellExecute = false,
        RedirectStandardError = true,
        RedirectStandardOutput = true
    }
};

//Add all of your processes to a list before actually running them
processes.Add(process1);

//This will run 5 in parallel
Parallel.ForEach(processes, new ParallelOptions { MaxDegreeOfParallelism = 5 }, p => { p.WaitForExit(); });