Running mpiexec with different script parameters f

2019-08-16 16:02发布

Apologies for the basic question, but I have not been able to find a solution on Google. I want to run a script separately for each chromosome in a list (called CHROMS in the code below) using parallelization via MPI. The script I am calling (some_script.sh) takes a chromosome parameter which I want to change for each call and another parameter which I want to keep constant across all calls. Basically, I want what the code below does, but with mpiexec instead of background processes.

#Run the pipeline for each chromosome separately.
#run_chromosome_iteration.sh
SOME_OTHER_PARAM="blah blah"
for c in $CHROMS;
    do 
        $SCRIPTS/some_script.sh $c $SOME_OTHER_PARAM &
    done

Edit: I actually have two levels of parallelization going on in my project: I am running my model 100 times using 100 jobs, and I would like to parallelize each job across a set of chromosomes (11 in this case). Please see the code below:

#Submit all jobs.
for i in {1..100};
    do 
        qsub -v ITER=${i} run_chromosome_iteration.sh
    done 

So I could use job arrays at the lower level of parallelization, but it would result in 1100 independent jobs and would be less memory efficient than sharing the memory between parallel processes. I also cannot simply use background processes because the amount of memory I will use requires two compute nodes in my cluster. This is why I want to use MPI.

0条回答
登录 后发表回答