TensorBoard - Plot training and validation losses

问题:

Is there a way to plot both the training losses and validation losses on the same graph?

It's easy to have two separate scalar summaries for each of them individually, but this puts them on separate graphs. If both are displayed in the same graph it's much easier to see the gap between them and whether or not they have begin to diverge due to overfitting.

Is there a built in way to do this? If not, a work around way? Thank you much!

回答1:

The work-around I have been doing is to use two SummaryWriter with different log dir for training set and cross-validation set respectively. And you will see something like this:

回答2:

Rather than displaying the two lines separately, you can instead plot the difference between validation and training losses as its own scalar summary to track the divergence.

This doesn't give as much information on a single plot (compared with adding two summaries), but it helps with being able to compare multiple runs (and not adding multiple summaries per run).

回答3:

Tensorboard is really nice tool but by its declarative nature can make it difficult to get it to do exactly what you want.

I recommend you checkout Losswise (https://losswise.com) for plotting and keeping track of loss functions as an alternative to Tensorboard. With Losswise you specify exactly what should be graphed together:

import losswise

losswise.set_api_key("project api key")
session = losswise.Session(tag='my_special_lstm', max_iter=10)
loss_graph = session.graph('loss', kind='min')

# train an iteration of your model...
loss_graph.append(x, {'train_loss': train_loss, 'validation_loss': validation_loss})
# keep training model...

session.done()

And then you get something that looks like:

Notice how the data is fed to a particular graph explicitly via the loss_graph.append call, the data for which then appears in your project's dashboard.

In addition, for the above example Losswise would automatically generate a table with columns for min(training_loss) and min(validation_loss) so you can easily compare summary statistics across your experiments. Very useful for comparing results across a large number of experiments.

回答4:

For completeness, since tensorboar 1.5.0 this is now possible.

You can use the custom scalars plugin. For this, you need to first make tensorboard layout configuration and write it to the event file. From the tensorboard example:

import tensorflow as tf
from tensorboard import summary
from tensorboard.plugins.custom_scalar import layout_pb2

# The layout has to be specified and written only once, not at every step

layout_summary = summary.custom_scalar_pb(layout_pb2.Layout(
  category=[
    layout_pb2.Category(
      title='losses',
      chart=[
          layout_pb2.Chart(
              title='losses',
              multiline=layout_pb2.MultilineChartContent(
                tag=[r'loss.*'],
              )),
          layout_pb2.Chart(
              title='baz',
              margin=layout_pb2.MarginChartContent(
                series=[
                  layout_pb2.MarginChartContent.Series(
                    value='loss/baz/scalar_summary',
                    lower='baz_lower/baz/scalar_summary',
                    upper='baz_upper/baz/scalar_summary'),
                ],
              )), 
      ]),
    layout_pb2.Category(
      title='trig functions',
      chart=[
          layout_pb2.Chart(
              title='wave trig functions',
              multiline=layout_pb2.MultilineChartContent(
                tag=[r'trigFunctions/cosine', r'trigFunctions/sine'],
              )),
          # The range of tangent is different. Let's give it its own chart.
          layout_pb2.Chart(
              title='tan',
              multiline=layout_pb2.MultilineChartContent(
                tag=[r'trigFunctions/tangent'],
              )),
      ],
      # This category we care less about. Let's make it initially closed.
      closed=True),
  ]))

writer = tf.summary.FileWriter(".")
writer.add_summary(layout_summary)
# ...
# Add any summary data you want to the file
# ...
writer.close()

A Category is group of Charts. Each Chart corresponds to a single plot which displays several scalars together. The Chart can plot simple scalars (MultilineChartContent) or filled areas (MarginChartContent, e.g. when you want to plot the deviation of some value). The tag member of MultilineChartContent must be a list of regex-es which match the tags of the scalars that you want to group in the Chart. For more details check the proto definitions of the objects in https://github.com/tensorflow/tensorboard/blob/master/tensorboard/plugins/custom_scalar/layout.proto. Note that if you have several FileWriters writing to the same directory, you need to write the layout in only one of the files. Writing it to a separate file also works.

To view the data in TensorBoard, you need to open the Custom Scalars tab. Here is an example image of what to expect https://user-images.githubusercontent.com/4221553/32865784-840edf52-ca19-11e7-88bc-1806b1243e0d.png