I copied the hinge loss function from here (also LossC and LossFunc upon which it's based. Then I included it in my gradient descent algorithm like so:
do
{
iteration++;
error = 0.0;
cost = 0.0;
//loop through all instances (complete one epoch)
for (p = 0; p < number_of_files__train; p++)
{
// 1. Calculate the hypothesis h = X * theta
hypothesis = calculateHypothesis( theta, feature_matrix__train, p, globo_dict_size );
// 2. Calculate the loss = h - y and maybe the squared cost (loss^2)/2m
//cost = hypothesis - outputs__train[p];
cost = HingeLoss.loss(hypothesis, outputs__train[p]);
System.out.println( "cost " + cost );
// 3. Calculate the gradient = X' * loss / m
gradient = calculateGradent( theta, feature_matrix__train, p, globo_dict_size, cost, number_of_files__train);
// 4. Update the parameters theta = theta - alpha * gradient
for (int i = 0; i < globo_dict_size; i++)
{
theta[i] = theta[i] - LEARNING_RATE * gradient[i];
}
}
//summation of squared error (error value for all instances)
error += (cost*cost);
/* Root Mean Squared Error */
//System.out.println("Iteration " + iteration + " : RMSE = " + Math.sqrt( error/number_of_files__train ) );
System.out.println("Iteration " + iteration + " : RMSE = " + Math.sqrt( error/number_of_files__train ) );
}
while( error != 0 );
But this doesnt work at all. Is that due to the loss function? Maybe how I added the loss function to my code?
I guess it's also possible that my implementation of gradient descent is faulty.
Here are my methods for calculating the gradient and the hypothesis, are these right?
static double calculateHypothesis( double[] theta, double[][] feature_matrix, int file_index, int globo_dict_size )
{
double hypothesis = 0.0;
for (int i = 0; i < globo_dict_size; i++)
{
hypothesis += ( theta[i] * feature_matrix[file_index][i] );
}
//bias
hypothesis += theta[ globo_dict_size ];
return hypothesis;
}
static double[] calculateGradent( double theta[], double[][] feature_matrix, int file_index, int globo_dict_size, double cost, int number_of_files__train)
{
double m = number_of_files__train;
double[] gradient = new double[ globo_dict_size];//one for bias?
for (int i = 0; i < gradient.length; i++)
{
gradient[i] = (1.0/m) * cost * feature_matrix[ file_index ][ i ] ;
}
return gradient;
}
The rest of the code is here if you're interested to take a look.
Below this sentence is what those loss functions look like. Should I use the loss
or deriv
, are these even correct?
/**
* Computes the HingeLoss loss
*
* @param pred the predicted value
* @param y the target value
* @return the HingeLoss loss
*/
public static double loss(double pred, double y)
{
return Math.max(0, 1 - y * pred);
}
/**
* Computes the first derivative of the HingeLoss loss
*
* @param pred the predicted value
* @param y the target value
* @return the first derivative of the HingeLoss loss
*/
public static double deriv(double pred, double y)
{
if (pred * y > 1)
return 0;
else
return -y;
}