How to understand the functional margin in SVM ?

Question

I'm reading Andrew NG's Machine Learning notes, but the functional margin definition confused me :

I can understand to geometric margin is the distance from x to its hyperplane, but how to understand functional margin ? And why they define its formula like that ?

Answer 1

Think of it like this: w^T.x_i +b is the model's prediction for the i-th data point. Y_i is its label. If the prediction and ground truth have the same sign, then gamma_i will be positive. The further "inside" the class boundary this instance is, the bigger gamma_i will be : this is better because, summed over all i, you will have greater separation between your classes. If the prediction and the label don't agree in sign, then this quantity will be negative (incorrect decision by the predictor), which will reduce your margin, and it will be reduced more the more incorrect you are (analogous to slack variables).

Answer 2

functional margin is used to scale.

geometric margin = functional margin / norm(w).

Or, when norm(w) = 1 then the margin is geometric margin

Answer 3

You can transfer functional margin to geometric margin based on the following two hypothesis:

1. ||w|| == 1, therefore (w^T)x+b == ((w^T)x+b)/||w||, which is the geometry distance from point x to the line y=(w^T)x+b.

2. There are only two categories for targets, where y_i can only be +1 and -1. Therefore, if the sign of y_i matches the side of the line where the point x lies in (y_i > 0 when (w^T)x+b > 0, y_i < 0 when (w^T)x+b < 0), multiplying y_i is simply equivalent to getting the absolute value of the distance (w^T)x+b.

Answer 4

For this question

And why they define its formula like that ?

Explanation: Functional margin doesn't tell us about the exact distance or measurement of different points to the separating plane/line.

For instance, just consider following lines they are same but functional margin would vary (a limitation of functional margin).

2*x  + 3*y  + 1 = 0 
4*x  + 6*y  + 2 = 0 
20*x + 30*y +10 = 0

Functional Margin just give an idea about the confidence of our classification, nothing concrete.

Please also read below reference for more details.

Referenced Andrew NG's lecture notes, please click here for more details

If y(i) = 1, then for the functional margin to be large (i.e., for our prediction to be confident and correct), we need wTx + b to be a large positive number. Conversely, if y(i) = −1, then for the functional margin to be large, we need wTx + b to be a large negative number. Moreover, if y(i)(wTx + b) > 0, then our prediction on this example is correct. (Check this yourself.) Hence, a large functional margin represents a confident and a correct prediction.

Answer 5

Functional Margin:

This gives the position of the point with respect to the plane, which does not depend on the magnitude.

Geometric Margin:

This gives the distance between the given training example and the given plane.