tensorflow store training data on GPU memory

Question

I am pretty new to tensorflow. I used to use theano for deep learning development. I notice a difference between these two, that is where input data can be stored.

In Theano, it supports shared variable to store input data on GPU memory to reduce the data transfer between CPU and GPU.

In tensorflow, we need to feed data into placeholder, and the data can come from CPU memory or files.

My question is: is it possible to store input data on GPU memory for tensorflow? or does it already do it in some magic way?

Thanks.

Answer 1

If your data fits on the GPU, you can load it into a constant on GPU from e.g. a numpy array:

with tf.device('/gpu:0'):
  tensorflow_dataset = tf.constant(numpy_dataset)

One way to extract minibatches would be to slice that array at each step instead of feeding it using tf.slice:

  batch = tf.slice(tensorflow_dataset, [index, 0], [batch_size, -1])

There are many possible variations around that theme, including using queues to prefetch the data to GPU dynamically.

Answer 2

It is possible, as has been indicated, but make sure that it is actually useful before devoting too much effort to it. At least at present, not every operation has GPU support, and the list of operations without such support includes some common batching and shuffling operations. There may be no advantage to putting your data on GPU if the first stage of processing is to move it to CPU.

Before trying to refactor code to use on-GPU storage, try at least one of the following:

1) Start your session with device placement logging to log which ops are executed on which devices:

config = tf.ConfigProto(log_device_placement=True)
sess = tf.Session(config=config)

2) Try to manually place your graph on GPU by putting its definition in a with tf.device('/gpu:0'): block. This will throw exceptions if ops are not GPU-supported.