How to fit/run the neural network multiple times i

Question

I'm using keras to construct a simple neural network as follows:

import keras
from keras.models import Sequential
from keras.layers import Dense
classifier = Sequential()
classifier.add(Dense(10, kernel_initializer='uniform', activation= 'relu', input_dim = 2))
...
classifier.compile(optimizer= 'adam',loss='binary_crossentropy', metrics=['accuracy'])
classifier.fit(X_train,y_train,batch_size=10,epochs=100)

The code works totally fine and get 90% accuracy when I first run it in jupyter notebook. But when I rerun it, its accuracy dramatically dropped to 50%, and the accuracy didn't improve during the training process. Also, if I construct another NN like this in the same notebook page, it also has this problem.

So what should I do if I want to get the right result when I rerun the code or run the another NN in the same notebook page?

PS: I'm using tensorflow backend.

Answer 1

Edit: Results are different mostly because of weights initialization and batches. But seed fixing is not enough for full reproducibility, see:

• https://keras.io/getting-started/faq/#how-can-i-obtain-reproducible-results-using-keras-during-development
• Why can't I get reproducible results in Keras even though I set the random seeds?

---

Previous answer:

Neural networks learning have random results due to

1. random weight initialization
2. random batch splitting/sorting in SGD algorithms such as Adam

For example, this code

import numpy as np
import keras 
from keras.models import Sequential
from keras.layers import Dense, Flatten

def run():
    classifier = Sequential()
    classifier.add(Flatten(input_shape=(28, 28)))
    classifier.add(Dense(10, kernel_initializer='uniform', activation= 'relu'))
    classifier.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
    X_train, y_train = keras.datasets.mnist.load_data()[0]
    X_train = X_train[:100] # for example
    y_train = keras.utils.to_categorical(y_train)[:100]
    classifier.fit(X_train, y_train, batch_size=10, epochs=100)

gives a different result on each run.

>>> run()
Epoch 1/100
100/100 [==============================] - 0s 4ms/step - loss: 10.1763 - acc: 0.1700
...
Epoch 100/100
100/100 [==============================] - 0s 2ms/step - loss: 4.5131 - acc: 0.4700

>>> run()
Epoch 1/100
100/100 [==============================] - 0s 5ms/step - loss: 7.2993 - acc: 0.2000
...
Epoch 1/100
100/100 [==============================] - 0s 2ms/step - loss: 0.8059 - acc: 0.7000

You can fix seed in keras random generator (which is numpy) for reproducibility.

>>> np.random.seed(1)
>>> run()
Epoch 1/100
100/100 [==============================] - 0s 5ms/step - loss: 7.6193 - acc: 0.1500
...
Epoch 100/100
100/100 [==============================] - 0s 2ms/step - loss: 0.3224 - acc: 0.6400

>>> np.random.seed(1)
>>> run()
Epoch 1/100
100/100 [==============================] - 0s 5ms/step - loss: 7.6193 - acc: 0.1500
...
Epoch 100/100
100/100 [==============================] - 0s 2ms/step - loss: 0.3224 - acc: 0.6400

https://github.com/keras-team/keras/issues/2743#issuecomment-219777627

P.S. Code may have very different results, if there are some problems with data/model (as in this mnist example with too small data and too easy model). 90% could be just overfitting. Check classifier on another independent test data.