LSTM with keras

Question

I have some training data x_train and some corresponding labels for this x_train called y_train. Here is how x_train and y_train are constructed:

train_x = np.array([np.random.rand(1, 1000)[0] for i in range(10000)])
train_y = (np.random.randint(1,150,10000))

train_x has 10000 rows and 1000 columns for each row. train_y has a label between 1 and 150 for each sample in train_x and represents a code for each train_x sample.

I also have a sample called sample, which is 1 row with 1000 columns, which I want to use for prediction on this LSTM model. This variable is defined as

sample = np.random.rand(1,1000)[0]

I am trying to train and predict an LSTM on this data using Keras. I want to take in this feature vector and use this LSTM to predict one of the codes in range 1 to 150. I know these are random arrays, but I cannot post the data I have. I have tried the following approach which I believe should work, but am facing some issues

    model = Sequential()
    model.add(LSTM(output_dim = 32, input_length = 10000, input_dim = 1000,return_sequences=True))
    model.add(Dense(150, activation='relu'))
    model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy']) 
    history = model.fit(train_x, train_y,
              batch_size=128, nb_epoch=1,
              verbose = 1)
    model.predict(sample)

Any help or adjustments to this pipeline would be great. I am not sure if the output_dim is correct. I want to pass train the LSTM on each sample of the 1000 dimension data and then reproduce a specific code that is in range 1 to 150. Thank you.

Answer 1

I see at least three things you need to change:

1. Change this line:

   model.add(Dense(150, activation='relu'))
   

   to:

   model.add(Dense(150, activation='softmax'))
   

   as leaving 'relu' as activation makes your output unbounded whereas it needs to have a probabilistic interpretation (as you use categorical_crossentropy).

2. Change loss or target:

   As you are using categorical_crossentropy you need to change your target to be a one-hot encoded vector of length 150. Another way is to leave your target but to change loss to sparse_categorical_crossentropy.

3. Change your target range:

   Keras has a 0-based array indexing (as in Python, C and C++ so your values should be in range [0, 150) instead [1, 150].