underflow and log values in java

Question

I have a question in regard to dealing with small probabilities values in machine learning models.

The standard way to avoid underflow problems which results from multiplying small floating-point numbers is to use log(x) instead of x

suppose that x=0.50 the log of which is log(x)=-0.301029996

to recover x later on the value of exp(log(x)) != x that is

0.740055574 != 0.50

So, how is using the logarithm is useful to deal with underflow??

Answer 1

(Not at all sure I remember correctly, so please correct me if I'm wrong.)

This is not really about overflow or underflow, but about floating point precision.

The idea is that if you have many very small numbers, multiplying them will produce an extremely small number. Say, you have ten probabilities of 1%, or 0.01 each. Multiply them, and the result is 1e-20. In those regions, floating point precision is not very good, which can introduce errors. In the worst case, the number could be 'rounded' to zero, which would break the entire calculation.

The trick wth logarithms is that after conversion to logarithms,

1. the values will generally be on a much smaller scale (in the sense of having a smaller exponent),
2. instead of multiplying the values, you just have to add them, so very small (or very big) numbers do not get even smaller (or bigger) as fast, and
3. once you've taken the log of all the candudate probabilities, all the other calculations are just additions, not multiplication, which should also be a bit faster.

Example (using Python, because I'm too lazy to fire up Eclipse, but the same works for Java):

>>> x,y,z = 0.01, 0.02, 0.03
>>> x*y*z
6.0000000000000002e-06
>>> log(x)+log(y)+log(z)
-12.023751088736219
>>> exp(log(x)+log(y)+log(z))
6.0000000000000002e-06

Also, as pointed out in the other answer, the problem with your particular calculation is that you seem to use a logarithm base-10 (log10 in Java), for which not exp(x) is the inverse function, but 10^x. Note, however, that in most languages / math libraries, log is in fact the natural logarithm.

Answer 2

This has nothing to do with the overflow. In the first log, you compute the log in base 10, instead of the natural logarithm. You can do this:

raise 10^log(x) to get back x, or use the natural logarithm.