I'm trying to implement the softmax function for a neural network written in Numpy. Let h be the softmax value of a given signal i.
I've struggled to implement the softmax activation function's partial derivative.
I'm currently stuck at issue where all the partial derivatives approaches 0 as the training progresses. I've cross-referenced my math with this excellent answer, but my math does not seem to work out.
import numpy as np
def softmax_function( signal, derivative=False ):
# Calculate activation signal
e_x = np.exp( signal )
signal = e_x / np.sum( e_x, axis = 1, keepdims = True )
if derivative:
# Return the partial derivation of the activation function
return np.multiply( signal, 1 - signal ) + sum(
# handle the off-diagonal values
- signal * np.roll( signal, i, axis = 1 )
for i in xrange(1, signal.shape[1] )
)
else:
# Return the activation signal
return signal
#end activation function
The signal
parameter contains the input signal sent into the activation function and has the shape (n_samples, n_features).
# sample signal (3 samples, 3 features)
signal = [[0.3394572666491664, 0.3089068053925853, 0.3516359279582483], [0.33932706934615525, 0.3094755563319447, 0.3511973743219001], [0.3394407172182317, 0.30889042266755573, 0.35166886011421256]]
The following code snipped is a fully working activation function and is only included as a reference and proof (mostly for myself) that the conceptual idea actually work.
from scipy.special import expit
import numpy as np
def sigmoid_function( signal, derivative=False ):
# Prevent overflow.
signal = np.clip( signal, -500, 500 )
# Calculate activation signal
signal = expit( signal )
if derivative:
# Return the partial derivation of the activation function
return np.multiply(signal, 1 - signal)
else:
# Return the activation signal
return signal
#end activation function
Edit
- The problem intuitively persist with simple single layer networks. The softmax (and its derivative) is applied at the final layer.
This is an answer on how to calculate the derivative of the softmax function in a more vectorized numpy fashion. However, the fact that the partial derivatives approach to zero might not be a math issue, and just be a problem of the learning rate or the known dying weight issue with complex deep neural networks. Layers like ReLU help preventing the latter issue.
First, I've used the following signal (just duplicating your last entry) to make it
4 samples x 3 features
so is easier to see what is going on with the dimensions.Next, you want to compute the Jacobian matrix of your softmax function. According to the cited page it is defined as
-hi * hj
for the off-diagonal entries (majority of the matrix forn_features > 2
), so lets start there. In numpy, you can efficiently calculate that Jacobian matrix using broadcasting:The first
signal[..., None]
(equivalent tosignal[:, :, None]
) reshapes the signal to(4, 3, 1)
while the secondsignal[:, None, :]
reshapes the signal to(4, 1, 3)
. Then, the*
just multiplies both matrices element-wise. Numpy's internal broadcasting repeats both matrices to form then_features x n_features
matrix for every sample.Then, we need to fix the diagonal elements:
The above lines extract diagonal indices for
n_features x n_features
matrix. It is equivalent of doingiy = np.arange(n_features); ix = np.arange(n_features)
. Then, replaces the diagonal entries with your defitinionhi * (1 - hi)
.Last, according to the linked source, you need to sum across rows for each of the samples. That can be done as:
Find bellow a summarized version:
Comparison of the derivatives:
Yours:
Mine: