这是一个很难回答的问题,但我不妨试试。 我执行从本文的架构https://arxiv.org/pdf/1503.08895.pdf的语言模型。 对的图,第5页的关于位置或“暂时”编码部顶端参见第2页。 更多关于位置编码可以在这里找到, https://arxiv.org/pdf/1706.03762.pdf在第6页的第5页/顶部的底部(I是针对由所述第一的作者该第二纸张)
因此,这里是我的,简而言之keras实现:
word_seq = Input(shape = (SEQ_LEN,), dtype = "int32", name = "word_seq")
query = Input(shape = (EMBED_DIM, ), dtype = "float32", name = "q_input")
#the query for lang. modeling is a constant vector filled with 0.1, as described at the bottom of page 7 in the first linked paper
T_A = Added_Weights(input_dim = (SEQ_LEN, EMBED_DIM))
#Added_Weights is a custom layer I wrote, which I'll post below
#These are the "positional encoding" components
T_C = Added_Weights(input_dim = (SEQ_LEN, EMBED_DIM))
Emb_A = Embedding(output_dim = EMBED_DIM, input_dim = VOCAB_SIZE, input_length = SEQ_LEN, name = "Emb_A")
Emb_C = Embedding(output_dim = EMBED_DIM, input_dim = VOCAB_SIZE, input_length = SEQ_LEN, name = "Emb_C")
int_state_weights = Dense(units = EMBED_DIM, activation = 'linear',
kernel_initializer=RandomNormal(mean=0., stddev = 0.05, seed = None))
layer_output = query
#the loop uses the output from the previous layer as the query, but the first layer's query is just that constant vector
for i in range(0, NUM_LAYERS - 1):
memories = Emb_A(word_seq) #these all re-use the weights instantiated earlier.
memories = T_A(memories)
memories = Dropout(DROPOUT_R)(memories)
content = Emb_C(word_seq)
content = T_C(content)
mem_relevance = Dot(axes=[1, 2])([layer_output, memories])
weighted_internal_state = int_state_weights(mem_relevance)
mem_relevance = Softmax()(mem_relevance)
content_relevance = Dot(axes=1)([mem_relevance,
content]) # weight each piece of content by it's probability of being relevant
layer_output = Add()([content_relevance, weighted_internal_state])
layer_output = Dropout(DROPOUT_R)(layer_output)
final_output = Dense(units = VOCAB_SIZE, activation ='relu',
kernel_initializer=RandomNormal(mean=0., stddev = 0.05, seed = None))(layer_output)
model = Model(inputs = [word_seq, query], outputs = prediction)
model.compile(optimizer = SGD(lr = 0.01, clipnorm = 50.), loss = 'categorical_crossentropy', metrics = ['accuracy'])
model.fit(x = [td_seqs, td_query], y = [td_labels],
batch_size = BATCH_SIZE, callbacks = [lr_adjust, lr_termination, for_csv], epochs=200, verbose = 1)
BATCH_SIZE目前128之前我加入了T_A和T_C部分,在96%的准确率结束这很顺利上〜35000训练样本。 当我实现T_A和T_C(位置编码),在训练%的准确度约为10和5.2上下的训练损失结束。 我用的10倍提高训练数据,并没有看到任何真正的改善。 这是我的Added_Weights类:
class Added_Weights(Layer):
def __init__(self, input_dim, **kwargs):
super(Added_Weights, self).__init__(**kwargs)
self.input_dim = input_dim
def build(self, input_shape):
# Create a trainable weight variable for this layer.
self.kernel = self.add_weight(name='kernel',
shape=(self.input_dim[0], self.input_dim[1]),
initializer=RandomNormal(mean=0., stddev=0.05, seed=None),
trainable=True)
super(Added_Weights, self).build(input_shape)
def call(self, x, **kwargs):
return x + self.kernel
def compute_output_shape(self, input_shape):
return input_shape
我在为什么这是不行的,看完这两个真棒论文,明确指出它应该工作后痛苦。 如果任何人都可以设法解决这个问题,那将是惊人的。