CuDNN library compatibility error after loading mo

Question

I am trying to load NSynth weights and I am using tf version 1.7.0

from magenta.models.nsynth import utils
from magenta.models.nsynth.wavenet import fastgen

def wavenet_encode(file_path):

 # Load the model weights.
 checkpoint_path = './wavenet-ckpt/model.ckpt-200000'

 # Load and downsample the audio.
 neural_sample_rate = 16000
 audio = utils.load_audio(file_path, 
                          sample_length=400000, 
                          sr=neural_sample_rate)

 encoding = fastgen.encode(audio, checkpoint_path, len(audio))

 # Reshape to a single sound.
 return encoding.reshape((-1, 16))

# An array of n * 16 frames. 
wavenet_z_data = wavenet_encode(file_path)

I get the following error:

tensorflow/stream_executor/cuda/cuda_dnn.cc:396] Loaded runtime CuDNN library: 7103 (compatibility version 7100) but source was compiled with 7005 (compatibility version 7000). If using a binary install, upgrade your CuDNN library to match. If building from sources, make sure the library loaded at runtime matches a compatible version specified during compile configuration.

What should I do and, which version of tf should I install, and exactly which CUDA version do I need?

Answer 1

I suggest installing the version of cudnn that tensorflow is compiled with:

sudo apt install libcudnn7-dev=7.0.5.15-1+cuda<x> libcudnn7=7.0.5.15-1+cuda<x>

<x> symbol must be replaced with cuda version you have e.g. for cuda version 9.0 replace it with 9.0.

Later freeze the version in apt that would not automatically update:

sudo apt-mark hold libcudnn7 libcudnn7-dev

Answer 2

At the env:

ubuntu16.04   
cuda9.0   
cudnn7.0  
tensorflow 1.11.0  
python 3.5

I try to train the object detection with tensorflow, I meet this problem:

2018-10-18 21:31:36.796017: E tensorflow/stream_executor/cuda/cuda_dnn.cc:343] Loaded runtime CuDNN library: 7.0.5 but source was compiled with: 7.2.1.  CuDNN library major and minor version needs to match or have higher minor version in case of CuDNN 7.0 or later version. If using a binary install, upgrade your CuDNN library.  If building from sources, make sure the library loaded at runtime is compatible with the version specified during compile configuration.
Segmentation fault (core dumped)

it's because the tensorflow version is higher;

I use pip3 install --upgrade --force-reinstall tensorflow-gpu==1.9.0 --user to fix the problem.

Answer 3

Installing Tensorflow Nightly solved the problem for me:

pip3 install tf-nightly-gpu

Answer 4

As the error says, the Tensorflow version you are using is compiled for CuDNN 7.0.5 while your system has CuDNN 7.1.3 installed.

As the error also suggests, you can solve this problem:

• Either by installing CuDNN 7.0.5 (follow instructions here: https://developer.nvidia.com/cudnn);
• Or by compiling Tensorflow yourself for your system (follow instructions here: https://www.tensorflow.org/install/install_sources).