Multithreaded backpropagation

Question

I have written a back propagation class in VB.NET -it works well- and I'm using it in a C# artificial intelligence project.

But I have a AMD Phenom X3 at home and a Intel i5 at school. and my neural network is not multi-threaded.

How to convert that back propagation class to a multithreaded algorithm? or how to use GPGPU programming in it? or should I use any third party libraries that have a multithreaded back propagation neural network?

Answer 1

JeffHeaton has recommend that you use resilient propagation (RPROP) instead of backpropagation. There are examples on how to do multithreaded RPROP (MPROP):

• Article on C# multithreaded backpropagation (from Jeff heaton)
• Chapter 7.2.1- "Propagation and Multithreading" (p.94 of Introduction to Encog 2.5 for C#)

It's a difficult to discuss all of the details here, so I would recommend that you either read that article and take a look at the relevant chapters of the book I referenced. This, of course, is assuming you're familiar with concurrent programming.

Update:

Resilient propagation will typically outperform backpropagation by a considerable factor. Additionally, RPROP has no parameters that must be set. Backpropagation requires that a learning rate and momentum value be specified. Finding an optimal learning rate and momentum value for backpropagation can be difficult. This is not necessary with resilient propagation. (source: Encog Machine Learning)

Answer 2

I've tried implementing multiple threads for RPROP batch processing, but it seemed it was always slower than using a single thread. I've tried to implement separately at the loop level "#pragma omp parallel" and by calculating the errors, gradients and weights in separate threads. In my interpretation, it seems that the computing done in each thread is too small to outcome the computing done in switching the threads and synchronizing the results (mutex.) I'm wondering if I've done something wrong? My conclusion is that would be smarter to run RPROP single-threaded, while processing multiple neuronal networks at the same time in separate threads. Most of the implementations usually imply multiple interconnected NNs so that would make sense.