Deep-Learning Nan loss reasons

问题:

Perhaps too general a question, but can anyone explain what would cause a Convolutional Neural Network to diverge?

Specifics:

I am using Tensorflow's iris_training model with some of my own data and keep getting

ERROR:tensorflow:Model diverged with loss = NaN.

Traceback...

tensorflow.contrib.learn.python.learn.monitors.NanLossDuringTrainingError: NaN loss during training.

Traceback originated with line:

 tf.contrib.learn.DNNClassifier(feature_columns=feature_columns,
                                        hidden_units=[300, 300, 300],
                                        #optimizer=tf.train.ProximalAdagradOptimizer(learning_rate=0.001, l1_regularization_strength=0.00001),                                                          
                                        n_classes=11,
                                        model_dir="/tmp/iris_model")

I've tried adjusting the optimizer, using a zero for learning rate, and using no optimizer. Any insights into network layers, data size, etc is appreciated.

回答1:

There are lots of things I have seen make a model diverge.

1. Too high of a learning rate. You can often tell if this is the case if the loss begins to increase and then diverges to infinity.

2. I am not to familiar with the DNNClassifier but I am guessing it uses the categorical cross entropy cost function. This involves taking the log of the prediction which diverges as the prediction approaches zero. That is why people usually add a small epsilon value to the prediction to prevent this divergence. I am guessing the DNNClassifier probably does this or uses the tensorflow opp for it. Probably not the issue.

3. Other numerical stability issues can exist such as division by zero where adding the epsilon can help. Another less obvious one if the square root who's derivative can diverge if not properly simplified when dealing with finite precision numbers. Yet again I doubt this is the issue in the case of the DNNClassifier.

4. You may have an issue with the input data. Try calling assert not np.any(np.isnan(x)) on the input data to make sure you are not introducing the nan. Also make sure all of the target values are valid. Finally, make sure the data is properly normalized. You probably want to have the pixels in the range [-1, 1] and not [0, 255].

5. The labels must be in the domain of the loss function, so if using a logarithmic-based loss function all labels must be non-negative (as noted by evan pu and the comments below).

回答2:

If you're training for cross entropy, you want to add a small number like 1e-8 to your output probability.

Because log(0) is negative infinity, when your model trained enough the output distribution will be very skewed, for instance say I'm doing a 4 class output, in the beginning my probability looks like

0.25 0.25 0.25 0.25

but toward the end the probability will probably look like

1.0 0 0 0

And you take a cross entropy of this distribution everything will explode. The fix is to artifitially add a small number to all the terms to prevent this.

回答3:

If using integers as targets, makes sure they aren't symmetrical at 0.

I.e., don't use classes -1, 0, 1. Use instead 0, 1, 2.

回答4:

If you'd like to gather more information on the error and if the error occurs in the first few iterations, I suggest you run the experiment in CPU-only mode (no GPUs). The error message will be much more specific.

Source: https://github.com/tensorflow/tensor2tensor/issues/574