The tensorflow documentation does not provide any example of how to perform a periodic evaluation of the model on an evaluation set.
Some people suggested the use of an Experiment, which sounds great but unfortunately does not work (depreciation and triggers an error).
Others suggested the use of SummarySaverHook, but I don't see how you can use that with an evaluation set (as opposed to the training set).
A solution would be to do the following
for i in range(number_of_epoch):
estimator.train(...) // on training set
estimator.evaluate(...) // on evaluation set
This architecture is explicitly discouraged in this paper (page 4 top right).
Any other idea/implementation?
EDIT:
The error message when running the experiment is the following:
File ".../anaconda2/lib/python2.7/site-packages/tensorflow/contrib/learn/python/learn/experiment.py", line 253, in train if (config.environment != run_config.Environment.LOCAL and
AttributeError: 'RunConfig' object has no attribute 'environment'
Tensorflow version 1.3