How to manually specify class labels in keras flow

Question

Problem: I am training a model for multilabel image recognition. My images are therefore associated with multiple y labels. This is conflicting with the convenient keras method "flow_from_directory" of the ImageDataGenerator, where each image is supposed to be in the folder of the corresponding label (https://keras.io/preprocessing/image/).

Workaround: Currently, I am reading all images into a numpy array and use the "flow" function from there. But this results in heavy memory loads and a slow read-in process.

Question: Is there a way to use the "flow_from_directory" method and to supply manually the (multiple) class labels?

---

Update: I ended up extending the DirectoryIterator class for the multilabel case. You can now set the attribute "class_mode" to the value "multilabel" and provide a dictionary "multlabel_classes" which maps filenames to their labels. Code: https://github.com/tholor/keras/commit/29ceafca3c4792cb480829c5768510e4bdb489c5

Answer 1

You could simply use the flow_from_directory and extend it to a multiclass in a following manner:

def multiclass_flow_from_directory(flow_from_directory_gen, multiclasses_getter):
    for x, y in flow_from_directory_gen:
        yield x, multiclasses_getter(x, y)

Where multiclasses_getter is assigning a multiclass vector / your multiclass representation to your images. Note that x and y are not a single examples but batches of examples, so this should be included in your multiclasses_getter design.

Answer 2

You could write a custom generator class that would read the files in from the directory and apply the labeling. That custom generator could also take in an ImageDataGenerator instance which would produce the batches using flow().

I am imagining something like this:

class Generator():

    def __init__(self, X, Y, img_data_gen, batch_size):
        self.X = X
        self.Y = Y  # Maybe a file that has the appropriate label mapping?
        self.img_data_gen = img_data_gen  # The ImageDataGenerator Instance
        self.batch_size = batch_size

    def apply_labels(self):
        # Code to apply labels to each sample based on self.X and self.Y

    def get_next_batch(self):
        """Get the next training batch"""
        self.img_data_gen.flow(self.X, self.Y, self.batch_size)

Then simply:

img_gen = ImageDataGenerator(...)
gen = Generator(X, Y, img_gen, 128)

model.fit_generator(gen.get_next_batch(), ...)

*Disclaimer: I haven't actually tested this, but it should work in theory.