How to Setup Adaptive Learning Rate in Keras

Question

Below is my code:

model = Sequential([
    Dense(32, input_shape=(32,), activation = 'relu'),
    Dense(100, activation='relu'),
    Dense(65, input_shape=(65,), activation='softmax')
])




model.summary()
model.compile(SGD(lr=.1), loss='binary_crossentropy', metrics=['accuracy'])
model.fit(train_samples, train_labels, batch_size=1000, epochs=1000,shuffle = True, verbose=2)

How will I set an adaptive learning rate of my model?

Answer 1

You need to replace SGD here

model.compile(SGD(lr=.1), loss='binary_crossentropy', metrics=['accuracy'])

with one of the provided optimizers, e.g. Adam:

model.compile(Adam(lr=.1), loss='binary_crossentropy', metrics=['accuracy'])

Read this https://keras.io/optimizers/

Answer 2

You may use a workaround.

For each_iteration in range(0, MaxEpoch):

1. Specify your own learning rate function that outputs a learning rate lr with respect to per epoch. The lr is then passed to your_optimiser

2. run model.compile(...optimizer=your_optimiser...)

3. run model.fit(...epochs = 1...)

4. After the ONE epoch, use model.save_weights(...)

5. Load weights by model.load_weights(...) for next iteration. See here for details https://keras.io/getting-started/faq/#how-can-i-save-a-keras-model

In fact, #4 and #5 enables you to do a transfer learning