Intel MKL FATAL ERROR: Cannot load libmkl_avx2.so

问题:

I am running a python script and I get this error:

Intel MKL FATAL ERROR: Cannot load libmkl_avx2.so or libmkl_def.so.

Both files are present in the anaconda2/lib directory. How can I fix this error? Thanks.

回答1:

If you use conda, try with these two commands:

conda install nomkl numpy scipy scikit-learn numexpr
conda remove mkl mkl-service

It should fix your problem.

回答2:

I ran into this problem after installing anaconda3 (vesion 4.2.0). The fix for me was simple, and I was able to keep using mkl. Just update to the latest numpy version.

conda update numpy

回答3:

Just wanted to note that Anaconda 4.0.0, shipped with mkl enabled by default, has this issue. The problem is indeed with Anaconda, as it can be reproduced with the simple python test below.

The actual issue is that Anaconda linked with mkl, but not with libmkl_core.so, thus it has a missing symbol, and can be seen by running:

$ LD_DEBUG=symbols python -c 'import sklearn.linear_model.tests.test_randomized_l1' 2>&1 | grep -i error
      2200:     /opt/anaconda/lib/python2.7/site-packages/scipy/special/../../../../libmkl_avx.so: error: symbol lookup error: undefined symbol: mkl_dft_fft_fix_twiddle_table_32f (fatal)

I didn't want to uninstall mkl, as I'd like to have the performance boost, so I found a workaround which worked for me - preload libmkl_core.so before execution.

$ python -c 'import sklearn.linear_model.tests.test_randomized_l1'
Intel MKL FATAL ERROR: Cannot load libmkl_avx.so or libmkl_def.so.
$
$ LD_PRELOAD=/opt/anaconda/lib/libmkl_core.so python -c 'import sklearn.linear_model.tests.test_randomized_l1'
$

回答4:

Got the same problem and solved by using the command:

conda install nomkl

I got the solution from this discussion https://github.com/BVLC/caffe/issues/3884

回答5:

I had this same issue using scikit-learn 0.19 and numpy 1.13.3 when running MLPRegressor (and also with a package called pyearth running an algorithm called MARS). I believe the root of the problem was that our python is part of an Anaconda install, but scikit-learn and numpy were installed via pip, and their expectations for mkl must not agree.

Unfortunately my framework is managed by some dedicated company admins, not by me, so I haven't gotten my guy to try recompiling numpy yet. But I was able to find a workaround based on this thread: Adding export LD_PRELOAD=/path/to/anaconda/lib/libmkl_def.so:/path/to/anaconda/lib/libmkl_avx.so:/path/to/anaconda/lib/libmkl_core.so:/path/to/anaconda/lib/libmkl_intel_lp64.so:/path/to/anaconda/lib/libmkl_intel_thread.so:/path/to/anaconda/lib/libiomp5.so to my ~/.bashrc causes the problem to disappear. It's super hacky, and I'd be lying if I said I knew exactly what it's doing (but this is helpful), so I'm hoping a recompile of numpy is a cleaner fix. But at least it works.

Note it is better performance-wise to have the versions of these packages that use mkl. Installing the nomkl versions is a workaround but not a true solution.

回答6:

Wanted to add on Valilutzik and Zaikun Xu's thread. I can't add comments since I don't have enough points yet.
conda install nomkl numpy scipy scikit-learn numexpr worked for me without having to remove mkl and mkl-service.

Adding an answer to Lee's question if nomkl will be slower: mkl is a Intel math kernel library and is hand-optimized for intel cpus. nomkl uses OpenBlas according to this: https://docs.continuum.io/mkl-optimizations/ It seems that mkl is quite faster on many matrix operations for intel cpus (see https://software.intel.com/en-us/articles/performance-comparison-of-openblas-and-intel-math-kernel-library-in-r)
I've seen someone saying nomkl is faster for AMD cpus (maybe because mkl doesn't work correctly in AMD?)

回答7:

All the solutions provided above did not work for me, but I found a good compromise:

for someone who has the same error and want's to keep Anaconda on his computer, and steel use mkl for (numpy and scipy) good process performance, the solution I propose:

Edit your .bashrc file.

Look for something like export PATH="/home/anaconda2/bin:$PATH" in the file.

Put a # at the beginning to comment it from the script: #export PATH="/home/anaconda2/bin:$PATH"

Open a new terminal and you should be running the base python installation.

It worked for me, hope it will be helpfull.

回答8:

In case anyone has a similar issue and gets errors that libmkl_p4m.so or libmkl_p4.so cannot be found (this occurred for me when calling certain numpy functions), I tried reinstalling / updating different python modules and reverting / updating to various versions of Anaconda, and neither worked. However I found that completely uninstalling anaconda and re-installing it (to version 4.4.10) solved the issue.

回答9:

The following worked for me

conda install  -f  numpy

回答10:

None of the above answers worked for me. In my case, @Nehal J Wani's comment below the OP's question led to the solution: As stated in the linked Intel support document, a single dynamic library can be used to link to the MKL.

After editing my project's Makefile to replace the following libraries -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core with -lmkl_rt and re-building, the error disappeared.

回答11:

try export LD_DEBUG=libs and execute your program again.

It will tell you which .so is causing the problem.

Then you can do something, e.g. upgrading the package.

For me, I find upgrade numexpr useful as it prints the following message just before the error:

calling init: xxxxx/python3.6/site-packages/numexpr/interpreter.cpython-36m-x86_64-linux-gnu.so