Bug on Keras fit_generator, running few steps more

2019-07-08 11:57发布

问题:

I found fit_generator() would run few steps more than it should.
I set steps_per_epoch=100. i and k both start from 0. But at the end of training process, it will print out k = 109. This situation only occurs when validation data are added.

def data_generate(xfd, yfd, x_line_offset, y_line_offset):

    while True:
        k = 0

        x_line_offset, y_line_offset = shuffle_list(x_line_offset, y_line_offset)

        for i in range(100):
            print('i = {}'.format(i))
            print('k = {}'.format(k))
            k += 1

            x_train = get_line_by_offset(xfd, x_line_offset[i])
            x_train = rescaling(x_train, 0, 65535, 0, 1)
            y_train = get_line_by_offset(yfd, y_line_offset[i])

            yield x_train, y_train

train_generator = data_generate(xfd_train, yfd_train, x_train_line_offset, y_train_line_offset)
validation_generator = data_generate(xfd_valid, yfd_valid, x_valid_line_offset, y_valid_line_offset)

model.fit_generator(train_generator, steps_per_epoch=100,
                    validation_data=validation_generator,
                    validation_steps=len(fix_y_valid_line_offset), epochs=1)

Since it will print out k = 109, I assume it runs few more steps. I don't know if it's bug or not. But the keras log message doesn't show after k = 99.

回答1:

No bugs here, it's just some implementation details. The function fit_generator() has a default argument max_queue_size=10. The batches from train_generator and validation_generator will be inserted into queues before getting used to fit/evaluate the model.

When the first epoch ends, there are 100 batches generated (k = 99). However, the generator will continue to generate 10 batches to fill up the queue. That's why you're seeing k = 100 to k = 109. At the same time, the validation process begins, so you'll also see k = 0, ... coming from the validation_generator.