Finish Tensorflow training in progress

Question

Data loss: not an sstable (bad magic number): perhaps your file is in a different file format and you need to use a different restore operator?

I'm currently training Tensorflow-wavenet with a directory of .wav audio files. Now, I want to stop the training and generate output, however the commands are not working. Here is terminal training:

I have seen this stackoverflow post, but tf.train.export_meta_graph does nothing. Also, if I try to generate output in a new terminal tab, like so:

cd /Users/ruchirbaronia/Downloads/tensorflow-wavenet-master 
python generate.py --wav_out_path=generated.wav --samples 16000 logdir/train/2018-01-03T09-18-15/model.ckpt-0.index

I get this error which basically says I can't generate output since training data has not been saved. How can I save and finish training?

Answer 1

train.py has an option for how often to save a checkpoint. For example,

train.py --checkpoint_every=10 ...

to save the checkpoint every 10 steps. By default, it is saved every 50 steps. Once you have a checkpoint, you should be able to use generate.py.