Keras: How to use fit_generator with multiple outp

问题:

In a Keras model with the Functional API I need to call fit_generator to train on augmented images data using an ImageDataGenerator. The problem is my model has two outputs: the mask I'm trying to predict and a binary value I obviously only want to augment the input and the mask output and not the binary value. How can I achieve this?

回答1:

The example below might be self-explanatory! The 'dummy' model takes 1 input (image) and it outputs 2 values. The model computes the MSE for each output.

x = Convolution2D(8, 5, 5, subsample=(1, 1))(image_input)
x = Activation('relu')(x)
x = Flatten()(x)
x = Dense(50, W_regularizer=l2(0.0001))(x)
x = Activation('relu')(x)

output1 = Dense(1, activation='linear', name='output1')(x)
output2 = Dense(1, activation='linear', name='output2')(x)

model = Model(input=image_input, output=[output1, output2])
model.compile(optimizer='adam', loss={'output1': 'mean_squared_error', 'output2': 'mean_squared_error'})

The function below generates batches to feed the model during training. It takes the training data x and the label y where y=[y1, y2]

batch_generator(x, y, batch_size, is_train):
    sample_idx = 0
    while True:
       X = np.zeros((batch_size, input_height, input_width, n_channels), dtype='float32')
       y1 = np.zeros((batch_size, mask_height, mask_width), dtype='float32')
       y2 = np.zeros((batch_size, 1), dtype='float32')

       # fill up the batch
       for row in range(batch_sz):
           image = x[sample_idx]
           mask = y[0][sample_idx]
           binary_value = y[1][sample_idx]
           # transform/preprocess image
           image = cv2.resize(image, (input_width, input_height))
           if is_train:
               image, mask = my_data_augmentation_function(image, mask)
           X_batch[row, ;, :, :] = image
           y1_batch[row, :, :] = mask
           y2_batch[row, 0] = binary_value
           sample_idx += 1

       # Normalize inputs
       X_batch = X_batch/255.
       yield(X_batch, {'output1': y1_batch, 'output2': y2_batch} ))

Finally, we call the fit_generator()

    model.fit_generator(batch_generator(X_train, y_train, batch_size, is_train=1))

回答2:

If you have separated both mask and binary value you can try something like this:

generator = ImageDataGenerator(rotation_range=5.,
                                width_shift_range=0.1, 
                                height_shift_range=0.1, 
                                horizontal_flip=True,  
                                vertical_flip=True)

def generate_data_generator(generator, X, Y1, Y2):
    genX = generator.flow(X, seed=7)
    genY1 = generator.flow(Y1, seed=7)
    while True:
            Xi = genX.next()
            Yi1 = genY1.next()
            Yi2 = function(Y2)
            yield Xi, [Yi1, Yi2]

So, you use the same generator for both input and mask with the same seed to define the same operation. You may change the binary value or not depending on your needs (Y2). Then, you call the fit_generator():

model.fit_generator(generate_data_generator(generator, X, Y1, Y2),
                epochs=epochs)

回答3:

The best way to achieve this seems to be to create a new generator class expanding the one provided by Keras that parses the data augmenting only the images and yielding all the outputs.