numpy OpenBLAS set maximum number of threads

I am using numpy and my model involves intensive matrix-matrix multiplication. To speed up, I use OpenBLAS multi-threaded library to parallelize the numpy.dot function.

My setting is as follows,

OS : CentOS 6.2 server #CPUs = 12, #MEM = 96GB
python version: Python2.7.6
numpy : numpy 1.8.0
OpenBLAS + IntelMKL

$ OMP_NUM_THREADS=8 python test_mul.py

code, of which I took from https://gist.github.com/osdf/

test_mul.py :

import numpy
import sys
import timeit

try:
    import numpy.core._dotblas
    print 'FAST BLAS'
except ImportError:
    print 'slow blas'

print "version:", numpy.__version__
print "maxint:", sys.maxint
print

x = numpy.random.random((1000,1000))

setup = "import numpy; x = numpy.random.random((1000,1000))"
count = 5

t = timeit.Timer("numpy.dot(x, x.T)", setup=setup)
print "dot:", t.timeit(count)/count, "sec"

when I use OMP_NUM_THREADS=1 python test_mul.py, the result is

dot: 0.200172233582 sec

OMP_NUM_THREADS=2

dot: 0.103047609329 sec

OMP_NUM_THREADS=4

dot: 0.0533880233765 sec

things go well.

However, when I set OMP_NUM_THREADS=8.... the code starts to "occasionally works".

sometimes it works, sometimes it does not even run and and gives me core dumps.

when OMP_NUM_THREADS > 10. the code seems to break all the time.. I am wondering what is happening here ? Is there something like a MAXIMUM number threads that each process can use ? Can I raise that limit, given that I have 12 CPUs in my machine ?

Thanks

Firstly, I don't really understand what you mean by 'OpenBLAS + IntelMKL'. Both of those are BLAS libraries, and numpy should only link to one of them at runtime. You should probably check which of these two numpy is actually using. You can do this by calling:

$ ldd <path-to-site-packages>/numpy/core/_dotblas.so

Update: numpy/core/_dotblas.so was removed in numpy v1.10, but you can check the linkage of numpy/core/multiarray.so instead.

For example, I link against OpenBLAS:

...
libopenblas.so.0 => /opt/OpenBLAS/lib/libopenblas.so.0 (0x00007f788c934000)
...

If you are indeed linking against OpenBLAS, did you build it from source? If you did, you should see that in the Makefile.rule there is a commented option:

...
# You can define maximum number of threads. Basically it should be
# less than actual number of cores. If you don't specify one, it's
# automatically detected by the the script.
# NUM_THREADS = 24
...

By default OpenBLAS will try to set the maximum number of threads to use automatically, but you could try uncommenting and editing this line yourself if it is not detecting this correctly.

Also, bear in mind that you will probably see diminishing returns in terms of performance from using more threads. Unless your arrays are very large it is unlikely that using more than 6 threads will give much of a performance boost because of the increased overhead involved in thread creation and management.