Setting SGE for running an executable with differe

问题:

I used to work with a cluster using SLURM scheduler, but now I am more or less forced to switch to a SGE-based cluster, and I'm trying to get a hang of it. The thing I was working on SLURM system involves running an executable using N input files, and set a SLURM configuration file in this fashion,

slurmConf.conf SLURM configuration file
    0   /path/to/exec /path/to/input1
    1   /path/to/exec /path/to/input2
    2   /path/to/exec /path/to/input3
    3   /path/to/exec /path/to/input4
    4   /path/to/exec /path/to/input5
    5   /path/to/exec /path/to/input6
    6   /path/to/exec /path/to/input7
    7   /path/to/exec /path/to/input8
    8   /path/to/exec /path/to/input9
    9   /path/to/exec /path/to/input10

And my working submission script in SLURM contains this line;

srun -n $SLURM_NNODES --multi-prog $slconf
$slconf refers to a path to that configuration file

This setup worked as I wanted - to run the executable with 10 different inputs at the same time with 10 nodes. Now that I just transitioned to SGE system, I want to do the same thing but I tried to read the manual and found nothing quite like SLURM. Could you please give me some light on how to achieve the same thing on SGE system?

Thank you very much!

回答1:

You could use the "job array" feature of the Grid Engine.

Create a shell script sge_job.sh

#!/bin/sh
#
# sge_job.sh -- SGE job description script
#
#$ -t 1-10
/path/to/exec /path/to/input$SGE_TASK_ID

And submit this script to SGE with qsub.

qsub sge_job.sh

回答2:

Dmitri Chubarov's answer is excellent, and the most robust way to proceed as it places less load on the submit node when submitting many jobs (>1000). Alternatively, you can wrap qsub in a for loop:

for i in {1..10}
do
    echo "/path/to/exec /path/to/input${i}" | qsub
done

I sometimes use the above when whatever varies as input is not easily captured as a range of integers.

Example:

for f in `ls /some/path/input*`
do
    echo "/path/to/exec ${f}" | qsub
done