Tensorboard Unble to get first event timestamp for

问题:

I am trying to visualize a training session I trained in a remote server. I used scp to copy the file in my local iMac. I tried to visualize the data by running tensorboard. It runs the tensorboard site but I can't get the visualization. Every chart has a single dot at zero. I get this warning on the terminal.

WARNING:tensorflow:Unable to get first event timestamp for run 
470_313_0.0001_2500_200/train
WARNING:tensorflow:Unable to get first event timestamp for run 
470_313_0.0001_2500_200/train
WARNING:tensorflow:Unable to get first event timestamp for run 
470_313_0.0001_2500_200/val
WARNING:tensorflow:Unable to get first event timestamp for run 
470_313_0.0001_2500_50/train
WARNING:tensorflow:Unable to get first event timestamp for run 
470_313_0.0001_2500_50/val

Any idea what is going on?

回答1:

I ran into a similar problem. There are 2 solutions to it -

• Delete all past tfevents files from the directory and keep the latest one (temporary solution).
• Create a new directory for building your logs (permanent solution).

In given below picture, first I build logs in directory 2 which resulted in same error/warnings. Later I changed the directory to 3 and build logs there and tensorboard ran successfully.

回答2:

I ran into the same problem (with a TensorFlow 1.4 trainer running in the cloud with Google Cloud ML Engine).

Explicitly closing tf.summary.FileWriters with close() solved it in my case.