I use python 'multiprocessing' module to run single process on multiple cores but I want to run a couple of independent process in parallel. For ex - Process one parses large file, Process two find pattern in different file and process three does some calculation; can all three different processed that have different set of arguments be run in parallel?
def Process1(largefile):
Parse large file
runtime 2hrs
return parsed_file
def Process2(bigfile)
Find pattern in big file
runtime 2.5 hrs
return pattern
def Process3(integer)
Do astronomical calculation
Run time 2.25 hrs
return calculation_results
def FinalProcess(parsed,pattern,calc_results):
Do analysis
Runtime 10 min
return final_results
def main():
parsed = Process1(largefile)
pattern = Process2(bigfile)
calc_res = Process3(integer)
Final = FinalProcess(parsed,pattern,calc_res)
if __name__ == __main__:
main()
sys.exit()
In above pseudo code Process1, Process2 and Process3 are single core process i.e they can't be run on multiple processors. These processes are run sequentially and take 2+2.5+2.25hrs = 6.75 hrs. Is it possible to run these three process in parallel? So that they run at same time on different processors/cores and when most time taking (Process2)finishes than we move to Final Process.
I would really appreciate your help.
AK