replicate a row tensor using tf.tile?

Question

I have a tensor which is simply a vector, vector = [0.5 0.4] and tf.shape indicates that it has shape=(1,), I would like to replicate the vector m times and have the shape of [m, 2], so for m = 2, matrix = [[0.5 0.4], [0.5 0.4]]. How do I achieve that using tf.tile?

Answer 1

replicating / duplicating a tensor (be it a 1D vector, 2D matrix, or any dimension) can be done by creating a list of copies of this tensor (with pure python), and then using tf.stack - having both steps in one (short) line. Here is an example of duplicating a 2D Tensor:

import tensorflow as tf
tf.enable_eager_execution()

a = tf.constant([[1,2,3],[4,5,6]])  # shape=(2,3)
a_stack = tf.stack([a] * 4)  # shape=(4,2,3)

print(a)
print(a_stack)

"[a]*4" creates a list containing four copies of the same tensor (this is pure python). tf.stack then stack them one after the other, on the first axis (axis=0)

In graph mode:

import tensorflow as tf

a = tf.constant([[1,2,3],[4,5,6]])  # shape=(2,3)
a_stack = tf.stack([a] * 4)  # shape=(4,2,3)

sess = tf.Session()
print('original tensor:')
print(sess.run(a))
print('stacked tensor:')
print(sess.run(a_stack))

Answer 2

Tensorflow 2.0 solution: Refer to this link to read more about tf.tile

vector = [0.5, 0.4]
tf.reshape(tf.tile(vector, [2]), [2, tf.shape(vector)[0]])

# output 
<tf.Tensor: id=59, shape=(2, 2), dtype=float32, numpy=
array([[0.5, 0.4],
       [0.5, 0.4]], dtype=float32)>

Answer 3

Take the following, vec is a vector, multiply is your m, the number of times to repeat the vector. tf.tile is performed on the vector and then using tf.reshape it is reshaped into the desired structure.

import tensorflow as tf

vec = tf.constant([1, 2, 3, 4])
multiply = tf.constant([3])

matrix = tf.reshape(tf.tile(vec, multiply), [ multiply[0], tf.shape(vec)[0]])
with tf.Session() as sess:
    print(sess.run([matrix]))

This results in:

[array([[1, 2, 3, 4],
       [1, 2, 3, 4],
       [1, 2, 3, 4]], dtype=int32)]

Answer 4

The same can be achieved by multiplying a ones matrix with vec and let broadcasting do the trick:

tf.ones([m, 1]) * vec

vec = tf.constant([1., 2., 3., 4.])
m = 3
matrix = tf.ones([m, 1]) * vec

with tf.Session() as sess:
   print(sess.run([matrix]))

#Output: [[1., 2., 3., 4.],
#         [1., 2., 3., 4.],
#         [1., 2., 3., 4.]]

Answer 5

An answer without reshaping:

vec = tf.constant([[1, 2, 3, 4]])
multiply = tf.constant([3, 1])
tf.tile(vec, multiply)

Answer 6

I assume that the main use case of such replication is to match the dimensionality of two tensors (that you want to multiply?).

In that case, there is a much simpler solution. Let the tensorflow do the work of dimensionality matching for you:

import tensorflow as tf
tf.enable_eager_execution()

a = tf.constant([1, 2, 3])  # shape=(3)
b = tf.constant([[[1, 3], [1, 3], [1, 3]], [[2, 0], [2, 0], [2, 0]]])  # shape=(2, 3, 2)

print(tf.einsum('ijk,j->ijk', b, a))

# OUTPUT:
# tf.Tensor(
# [[[1 3]
#   [2 6]
#   [3 9]]
# 
#  [[2 0]
#   [4 0]
#   [6 0]]], shape=(2, 3, 2), dtype=int32)

As you can see it can work for much more complex situations: when you need to replicate on both first and last dimensions, when you are working with more complex shapes, etc. All you need to do is match the indices in the string description (above we match the dimension of a, labeled j with the second dimension of b (ijk).

Another example use case: I have a state per neuron, and since we simulate in batches, this state has dimensionality (n_batch, n_neuron). I need to use this state to modulate connections between neurons (weights of synapses), which in my case had additional dimension so they have the dimensionality (n_neuron, n_neuron, n_X).

Instead of making a mess with tiling, reshaping, etc. I can just write it in a single line like so:

W_modulated = tf.einsum('ijk,bi->bijk', self.W, ux)