LSTM Autoencoder

Question

I'm trying to build a LSTM autoencoder with the goal of getting a fixed sized vector from a sequence, which represents the sequence as good as possible. This autoencoder consists of two parts:

• LSTM Encoder: Takes a sequence and returns an output vector (return_sequences = False)
• LSTM Decoder: Takes an output vector and returns a sequence (return_sequences = True)

So, in the end, the encoder is a many to one LSTM and the decoder is a one to many LSTM.

Image source: Andrej Karpathy

On a high level the coding looks like this (similar as described here):

encoder = Model(...)
decoder = Model(...)

autoencoder = Model(encoder.inputs, decoder(encoder(encoder.inputs)))

autoencoder.compile(loss='binary_crossentropy',
              optimizer='adam',
              metrics=['accuracy'])

autoencoder.fit(data, data,
          batch_size=100,
          epochs=1500)

The shape (number of training examples, sequence length, input dimension) of the data array is (1200, 10, 5) and looks like this:

array([[[1, 0, 0, 0, 0],
        [0, 1, 0, 0, 0],
        [0, 0, 1, 0, 0],
        ..., 
        [0, 0, 0, 0, 0],
        [0, 0, 0, 0, 0],
        [0, 0, 0, 0, 0]],
        ... ]

Problem: I am not sure how to proceed, especially how to integrate LSTM to Model and how to get the decoder to generate a sequence from a vector.

I am using keras with tensorflow backend.

EDIT: If someone wants to try out, here is my procedure to generate random sequences with moving ones (including padding):

import random
import math

def getNotSoRandomList(x):
    rlen = 8
    rlist = [0 for x in range(rlen)]
    if x <= 7:
        rlist[x] = 1
    return rlist


sequence = [[getNotSoRandomList(x) for x in range(round(random.uniform(0, 10)))] for y in range(5000)]

### Padding afterwards

from keras.preprocessing import sequence as seq

data = seq.pad_sequences(
    sequences = sequence,
    padding='post',
    maxlen=None,
    truncating='post',
    value=0.
)

Answer 1

Models can be any way you want. If I understood it right, you just want to know how to create models with LSTM?

Using LSTMs

Well, first, you have to define what your encoded vector looks like. Suppose you want it to be an array of 20 elements, a 1-dimension vector. So, shape (None,20). The size of it is up to you, and there is no clear rule to know the ideal one.

And your input must be three-dimensional, such as your (1200,10,5). In keras summaries and error messages, it will be shown as (None,10,5), as "None" represents the batch size, which can vary each time you train/predict.

There are many ways to do this, but, suppose you want only one LSTM layer:

from keras.layers import *
from keras.models import Model

inpE = Input((10,5)) #here, you don't define the batch size   
outE = LSTM(units = 20, return_sequences=False, ...optional parameters...)(inpE)

This is enough for a very very simple encoder resulting in an array with 20 elements (but you can stack more layers if you want). Let's create the model:

encoder = Model(inpE,outE)   

Now, for the decoder, it gets obscure. You don't have an actual sequence anymore, but a static meaningful vector. You may want to use LTSMs still, they will suppose the vector is a sequence.

But here, since the input has shape (None,20), you must first reshape it to some 3-dimensional array in order to attach an LSTM layer next.

The way you will reshape it is entirely up to you. 20 steps of 1 element? 1 step of 20 elements? 10 steps of 2 elements? Who knows?

inpD = Input((20,))   
outD = Reshape((10,2))(inpD) #supposing 10 steps of 2 elements    

It's important to notice that if you don't have 10 steps anymore, you won't be able to just enable "return_sequences" and have the output you want. You'll have to work a little. Acually, it's not necessary to use "return_sequences" or even to use LSTMs, but you may do that.

Since in my reshape I have 10 timesteps (intentionally), it will be ok to use "return_sequences", because the result will have 10 timesteps (as the initial input)

outD1 = LSTM(5,return_sequences=True,...optional parameters...)(outD)    
#5 cells because we want a (None,10,5) vector.   

You could work in many other ways, such as simply creating a 50 cell LSTM without returning sequences and then reshaping the result:

alternativeOut = LSTM(50,return_sequences=False,...)(outD)    
alternativeOut = Reshape((10,5))(alternativeOut)

And our model goes:

decoder = Model(inpD,outD1)  
alternativeDecoder = Model(inpD,alternativeOut)   

After that, you unite the models with your code and train the autoencoder. All three models will have the same weights, so you can make the encoder bring results just by using its predict method.

encoderPredictions = encoder.predict(data)

---

What I often see about LSTMs for generating sequences is something like predicting the next element.

You take just a few elements of the sequence and try to find the next element. And you take another segment one step forward and so on. This may be helpful in generating sequences.

Answer 2

You can find a simple of sequence to sequence autoencoder here: https://blog.keras.io/building-autoencoders-in-keras.html

Answer 3

Here is an example

Let's create a synthetic data consisting of a few sequence. The idea is looking into these sequences through the lens of an autoencoder. In other words, lowering the dimension or summarizing them into a fixed length.

# define input sequence
sequence = np.array([[0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9], 
                     [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8],
                     [0.2, 0.4, 0.6, 0.8],
                     [0.3, 0.6, 0.9, 1.2]])

# prepare to normalize
x = pd.DataFrame(sequence.tolist()).T.values
scaler = preprocessing.StandardScaler()
x_scaled = scaler.fit_transform(x)
sequence_normalized = [col[~np.isnan(col)] for col in  x_scaled.T]

# make sure to use dtype='float32' in padding otherwise with floating points
sequence = pad_sequences(sequence, padding='post', dtype='float32')

# reshape input into [samples, timesteps, features]
n_obs = len(sequence)
n_in = 9
sequence = sequence.reshape((n_obs, n_in, 1))

Let's device a simple LSTM

#define encoder
visible = Input(shape=(n_in, 1))
encoder = LSTM(2, activation='relu')(visible)

# define reconstruct decoder
decoder1 = RepeatVector(n_in)(encoder)
decoder1 = LSTM(100, activation='relu', return_sequences=True)(decoder1)
decoder1 = TimeDistributed(Dense(1))(decoder1)

# tie it together
myModel = Model(inputs=visible, outputs=decoder1)

# summarize layers
print(myModel.summary())


#sequence = tmp
myModel.compile(optimizer='adam', loss='mse')

history = myModel.fit(sequence, sequence, 
                      epochs=400, 
                      verbose=0, 
                      validation_split=0.1, 
                      shuffle=True)

plot_model(myModel, show_shapes=True, to_file='reconstruct_lstm_autoencoder.png')
# demonstrate recreation
yhat = myModel.predict(sequence, verbose=0)
# yhat

import matplotlib.pyplot as plt

#plot our loss 
plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('model train vs validation loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train', 'validation'], loc='upper right')
plt.show()

                                                

Lets build the autoencoder

# use our encoded layer to encode the training input
decoder_layer = myModel.layers[1]

encoded_input = Input(shape=(9, 1))
decoder = Model(encoded_input, decoder_layer(encoded_input))

# we are interested in seeing how the encoded sequences with lenght 2 (same as the dimension of the encoder looks like)
out = decoder.predict(sequence)

f = plt.figure()
myx = out[:,0]
myy = out[:,1]
s = plt.scatter(myx, myy)

for i, txt in enumerate(out[:,0]):
    plt.annotate(i+1, (myx[i], myy[i]))

And here is the representation of the sequences