Hyperparameter Tuning of Tensorflow Model

Question

I've used Scikit-learn's GridSearchCV before to optimize the hyperparameters of my models, but just wondering if a similar tool exists to optimize hyperparameters for Tensorflow (for instance number of epochs, learning rate, sliding window size etc.)

And if not, how can I implement a snippet that effectively runs all different combinations?

Answer 1

Even though it does not seem to be explicitly documented (in version 1.2), the package tf.contrib.learn (included in TensorFlow) defines classifiers that are supposed to be compatible with scikit-learn... However, looking at the source, it seems you need to explicitly set the environment variable TENSORFLOW_SKLEARN (e.g. to "1") to actually get this compatibility. If this works, you can already use GridSearchCV (see this test case).

That said, there are a few alternatives. I don't know about any specific to TensorFlow, but hyperopt, Scikit-Optimize or SMAC3 should all be valid options. MOE and Spearmint look like used to be good choices but now don't seem too maintained.

Alternatively, you can look into a service like SigOpt (a company by the original author of MOE).

Edit

About running all possible combinations of parameters, the core logic, if you want to implement it yourself, is not really complicated. You can just define lists with the possible values for each parameter and then run through all the combinations with itertools.product. Something like:

from itertools import product

param1_values = [...]
param2_values = [...]
param3_values = [...]
for param1, param2, param3 in product(param1_values, param2_values param3_values):
    run_experiment(param1, param2, param3)

Note however that grid search can be prohibitively expensive to run in many cases, and even doing just a random search in the parameters space will probably be more efficient (more about that in this publication).

Answer 2

Another viable (and documented) option for grid search with Tensorflow is Ray Tune. It's a scalable framework for hyperparameter tuning, specifically for deep learning/reinforcement learning.

You can try out a fast tutorial here.

It also takes care of Tensorboard logging and efficient search algorithms (ie, HyperOpt integration and HyperBand) in about 10 lines of Python.

import ray
from ray import tune

def train_tf_model(config, tune_reporter):  # 1 new arg for reporting results
    # ... train here ....
    # ... train here ....
    # ... train here ....
    pass

ray.init()

tune.run_experiments({
    "my_experiment": {
        "run": train_tf_model,
        "stop": { "mean_accuracy": 100 },
        "config": {
            "alpha": tune.grid_search([0.2, 0.4, 0.6]),
            "beta": tune.grid_search([1, 2]),
        }}})

(Disclaimer: I contribute actively to this project!)