Decoder's weights of Autoencoder with tied wei

问题:

I have implemented a tied weights Auto-encoder in Keras and have successfully trained it.

My goal is to use only the decoder part of the Auto-encoder as the last layer of another network, to fine tune both the network and the decoder.

Thing is, as you can see below from the summary, the decoder has no parameters with my tied weights implementation, so there is nothing to be fine tuned. (decoder.get_weights() returns [])

My question is: Should I change the implementation of the tied weights, so that the tied layer can still hold weights, that is the transposed weights of the encoder? If yes, how?

Or am I just way off?

Below is the summary of the autoencoder model as well as the class of the tied Dense layer (slightly modified from https://github.com/nanopony/keras-convautoencoder/blob/master/autoencoder_layers.py.)

---

Layer (type)                     Output Shape          Param #     Connected to                     
====================================================================================================
encoded (Dense)                  (None, Enc_dim)          33000       dense_input_1[0][0]              
____________________________________________________________________________________________________
tieddense_1 (TiedtDense)          (None, Out_Dim)            0           encoded[0][0]                    
====================================================================================================
Total params: 33,000
Trainable params: 33,000
Non-trainable params: 0
________________________________________________________________________


class TiedtDense(Dense):
def __init__(self, output_dim, master_layer, init='glorot_uniform', activation='linear', weights=None,
             W_regularizer=None, b_regularizer=None, activity_regularizer=None,
             W_constraint=None, b_constraint=None, input_dim=None, **kwargs):
    self.master_layer = master_layer
    super(TiedtDense, self).__init__(output_dim, **kwargs)

def build(self, input_shape):
    assert len(input_shape) >= 2
    input_dim = input_shape[-1]
    self.input_dim = input_dim


    self.W = tf.transpose(self.master_layer.W)
    self.b = K.zeros((self.output_dim,))
    self.params = [self.b]
    self.regularizers = []
    if self.W_regularizer:
        self.W_regularizer.set_param(self.W)
        self.regularizers.append(self.W_regularizer)

    if self.b_regularizer:
        self.b_regularizer.set_param(self.b)
        self.regularizers.append(self.b_regularizer)

    if self.activity_regularizer:
        self.activity_regularizer.set_layer(self)
        self.regularizers.append(self.activity_regularizer)

    if self.initial_weights is not None:
        self.set_weights(self.initial_weights)
        del self.initial_weights

回答1:

It's been more than 2 years since this question was asked, but this answer might still be relevant for some.

The function Layer.get_weights() retrieves from self.trainable_weights and self.non_trainable_weights (see keras.engine.base_layer.Layer.weights). In your custom layer, your weights self.W and self.b are not being added to any of these collections and that's why the layer has 0 parameters.

You could tweak your implementation as follows:

class TiedtDense(Dense):
    def __init__(self, output_dim, master_layer, **kwargs):
        self.master_layer = master_layer
        super(TiedtDense, self).__init__(output_dim, **kwargs)

    def build(self, input_shape):
        assert len(input_shape) >= 2
        input_dim = input_shape[-1]
        self.input_dim = input_dim

        self.kernel = tf.transpose(self.master_layer.kernel)
        self.bias = K.zeros((self.units,))
        self.trainable_weights.append(self.kernel)
        self.trainable_weights.append(self.bias)

NOTE: I am excluding the regularizers and constraints for simplicity. If you want those, please refer to keras.engine.base_layer.Layer.add_weight.