Low GPU usage by Keras / Tensorflow?

问题:

I'm using keras with tensorflow backend on a computer with a nvidia Tesla K20c GPU. (CUDA 8)

I'm tranining a relatively simple Convolutional Neural Network, during training I run the terminal program nvidia-smi to check the GPU use. As you can see in the following output, the GPU utilization commonly shows around 7%-13%

My question is: during the CNN training shouldn't the GPU usage be higher? is this a sign of a bad GPU configuration or usage by keras/tensorflow?

nvidia-smi output

回答1:

Could be due to several reasons but most likely you're having a bottleneck when reading the training data. As your GPU has processed a batch it requires more data. Depending on your implementation this can cause the GPU to wait for the CPU to load more data resulting in a lower GPU usage and also a longer training time.

Try loading all data into memory if it fits or use a QueueRunner which will make an input pipeline reading data in the background. This will reduce the time that your GPU is waiting for more data.

The Reading Data Guide on the TensorFlow website contains more information.

回答2:

You should find the bottleneck:

On windows use Task-Manager> Performance to monitor how you are using your resources

On Linux use nmon, nvidia-smi, and htop to monitor your resources.

The most possible scenarios are:

• If you have a huge dataset, take a look at the disk read/write rates; if you are accessing your hard-disk frequently, most probably you need to change they way you are dealing with the dataset to reduce number of disk access

• Use the memory to pre-load everything as much as possible.

• If you are using a restful API or any similar services, make sure that you do not wait much for receiving what you need. For restful services, the number of requests per second might be limited (check your network usage via nmon/Task manager)

• Make sure you do not use swap space in any case!

• Reduce the overhead of preprocessing by any means (e.g. using cache, faster libraries, etc.)

• Play with the bach_size (however, it is said that higher values (>512) for batch size might have negative effects on accuracy)

回答3:

Measuring GPU performance and utilization is not as straightforward as CPU or Memory. GPU is an extreme parallel processing unit and there are many factors. The GPU utilization number shown by nvidia-smi means what percentage of the time at least one gpu multiprocessing group was active. If this number is 0, it is a sign that none of your GPU is being utilized but if this number is 100 does not mean that the GPU is being used at its full potential.

These two articles have lots of interesting information on this topic: https://www.imgtec.com/blog/a-quick-guide-to-writing-opencl-kernels-for-rogue/ https://www.imgtec.com/blog/measuring-gpu-compute-performance/