Can I measure the execution time of individual ope

Question

I know I can measure the execution time of a call to sess.run(), but is it possible to get a finer granularity and measure the execution time of individual operations?

Answer 1

There is not yet a way to do this in the public release. We are aware that it's an important feature and we are working on it.

Answer 2

Since this is high up when googling for "Tensorflow Profiling", note that the current (late 2017, TensorFlow 1.4) way of getting the Timeline is using a ProfilerHook. This works with the MonitoredSessions in tf.Estimator where tf.RunOptions are not available.

estimator = tf.estimator.Estimator(model_fn=...)
hook = tf.train.ProfilerHook(save_steps=10, output_dir='.')
estimator.train(input_fn=..., steps=..., hooks=[hook])

Answer 3

For the comments of fat-lobyte under Olivier Moindrot's answer, if you want to gather the timeline over all sessions, you can change "open('timeline.json', 'w')" to "open('timeline.json', 'a')".

Answer 4

To update this answer, we do have some functionality for CPU profiling, focused on inference. If you look at https://github.com/tensorflow/tensorflow/tree/master/tensorflow/tools/benchmark you'll see a program you can run on a model to get per-op timings.

Answer 5

To profile TensorFlow sessions automatically you can use the StackImpact profiler. No need to instrument sessions or add any options. You just need to initialize the profiler:

import stackimpact

agent = stackimpact.start(
    agent_key = 'agent key here',
    app_name = 'MyApp')

Both execution time and memory profiles will be available in the Dashboard.

Detailed info in this article: TensorFlow Profiling in Development and Production Environments.

Disclaimer: I work for StackImpact.

Answer 6

You can extract this information using runtime statistics. You will need to do something like this (check the full example in the above-mentioned link):

run_options = tf.RunOptions(trace_level=tf.RunOptions.FULL_TRACE)
run_metadata = tf.RunMetadata()
sess.run(<values_you_want_to_execute>, options=run_options, run_metadata=run_metadata)
your_writer.add_run_metadata(run_metadata, 'step%d' % i)

---

Better than just printing it you can see it in tensorboard:

Additionally, clicking on a node will display the exact total memory, compute time, and tensor output sizes.