How to launch multithreaded mpi processes in lsf?

Question

I want to use LSF to submit a job which:

• runs on 4 nodes, in parallel
• each node has a single mpi process
• each process has 12 threads

In the absence of LSF, I would simply launch with mpi on 4 nodes, like:

mpirun -hosts host1,host2,host3,host4 -np 4 ./myprocess --numthreads=12

However, in the presence of LSF, I can't see how to do this? I'm sure there's probably a very standard way to do it, but I'm quite new to LSF. I googled around, but the answer wasn't immediately obvious to me. I found Hybrid MPI/OpenMP in LSF , but it doesn't seem to be quite the same, it seems to only need a single host at a time.

Answer 1

Whilst I've accepted Hristo's answer as the correct one, here is what I did in the end, which avoids issues with using '-x', -W and -M, and is consistent with what my sysadmin told me:

#!/bin/bash

machinefile="$(mktemp)"
n=0
cmdstring="mpirun -machinefile ${machinefile} "
for host in $LSB_MCPU_HOSTS; do {
   if [[ $host != 12 ]]; then {
       echo $host>>$machinefile
       n=$(($n+1))
   } fi
} done

echo $n
echo $@

mpirun -machinefile ${machinefile} -np $n $@

You use this script like, let's say it's called 'lsfrun.sh':

bsub -n 48 -R "span[ptile=12]" ./lsfrun.sh ./myprocess --threads 12

Answer 2

The other question that you have linked to gives you exactly what you need, but you have to adapt it slightly, as it is written for OpenMP applications whose number of threads is controlled by the OMP_NUM_THREADS environment variable.

Here are the most important parts of the job script:

• #BSUB -n 4 - request 4 slots
• #BSUB -R "span[ptile=1]" - request that slots are distributed one per node; this option in combination with the previous one spans the job over 4 different nodes and instructs LSF to put one slot per host in the generated host file
• #BSUB -x - request exclusive access to the nodes

The above three options would instruct LSF to allocate 4 nodes and it will reserve one slot on each node. Since also exclusive access is being requested, no other jobs will share the same nodes with the job and you can start as many threads as you like per node. Then all you need is to call Open MPI's mpiexec and if LSF integration was compiled in your Open MPI setup, it will automatically pick up the host list from LSF and start one process per node.

A sample LSF job file would look like this:

#BSUB -n 4
#BSUB -R "span[ptile=1]"
#BSUB -x

mpiexec -np 4 ./myprocess --numthreads=12

Make sure you also request enough run time with the -W option and a sufficient amount of memory with the -M option. Memory in LSF (as well as in most other distributed resource managers) is requested per slot, therefore you should specify the maximum amount of memory that any instance of ./myprocess would consume.

If LSF integration is not compiled in your Open MPI distribution, the process is somewhat more involved as you would have to parse the LSF hosts file and create an Open MPI hosts file from the former.