Resetting GPU and driver after CUDA error

Question

Sometimes, bugs in my CUDA programs cause the desktop graphics to break (in Windows). Typically, the screen remains somewhat readable, but when graphics change, such as when dragging a window, lots of semi-random colored pixels and small blocks appear.

I have tried to reset the GPU and driver by changing the desktop resolution, but that doesn't help. The only fix I have found is to reboot the computer.

Is there a program out there or some trick I can use to get the driver and GPU to reset without rebooting?

Background:

I have had 1.0, 1.1, 1.3 and 2.0 cards but I only have a 1.1 and 2.0 card now. I've seen the issue on 1.0 and 1.1. I'm pretty sure I've seen it on 1.3. I'm unsure about 2.0. Did memory protection get added some time around 1.3? I am almost sure it's not due to unstable hardware as the problems have seemed to be triggered by bugs in my code and have disappeared when the bugs were fixed. When running finished code, the cards have been stable. I wrote this question after seeing it on my 1.1 card, but it disappeared after I fixed a bug and now I don't have any code that reproduces it. Maybe I should try to write to random locations on the 1.1 card and see if anything happens...

Answer 1

To reset the graphics stack in Windows, press Win+Ctrl+Shift+B.

Answer 2

1. from "device manager", under Display adapters tab, find the driver
2. disable it
3. press win + ctrl +shift + B (monitor will blink)
4. enable the driver

there you go.

Answer 3

Edit:

If you are on Tesla hardware on Linux and can run nvidia-smi, then you can reset the GPU using

nvidia-smi -r

or

nvidia-smi --gpu-reset

Here is the man output for this switch:

Resets GPU state. Can be used to clear double bit ECC errors or recover hung GPU. Requires -i switch to target specific device. Available on Linux only.

Otherwise...

---

The way to truly reset the hardware is to reboot.

What you describe shouldn't happen. I recommend testing with different hardware and let us know if it still occurs.

Answer 4

Because the same problem occurs sometimes on unix and google forwarded me to this thread, I hope this helps somebody else..

On ubuntu unloading and reloading the nvidia kernel module solved the problem for me:

sudo rmmod nvidia_uvm
sudo modprobe nvidia_uvm

Answer 5

I have a GeForce GTX 260 over NVDIA GPU SDK 4.2 and I am experiencing the some problems. Sometimes developing I have bugs in the programs. This causes the screen to show the random colored pixels described in this post.

As stated here, if I change resolution they do not disappear. Moreover, if I only change the COLOUR DEPTH from 32 to 16 bits, the random colored pixels disappear, but going back to 32 bits (without rebooting) make them appear again. Last bug that caused this behaviour was using __constant__ memory but passing it as a pointer:

test<<<grid, threadsPerBlock>>>( cuda_malloc_data, cuda_constant_data );

If I do not pass cudb_constant_data, then there is no bug (and consequently, the random coloured pixels do not appear).