Example for Deploying a Tensorflow Model via a RES

Question

Is there any example code for deploying a Tensorflow Model via a RESTful API? I see examples for a command line program and for a mobile app. Is there a framework for this or people just load the model and expose the predict method via a web framework (like Flask)to take input (say via JSON) and return the response? By framework I mean scaling for large number of predict requests. Of course since the models are immutable we can launch multiple instances of our prediction server and put it behind a load balancer (like HAProxy). My question is, are people using some framework for this or doing this from scratch, or, maybe this is already available in Tensorflow and I have not noticed it.

Answer 1

https://github.com/sugyan/tensorflow-mnist shows a simple restAPI example by using Flask and loading pre-trained mode (restore).

@app.route('/api/mnist', methods=['POST'])
def mnist():
    input = ((255 - np.array(request.json, dtype=np.uint8)) / 255.0).reshape(1, 784)
    output1 = simple(input)
    output2 = convolutional(input)
    return jsonify(results=[output1, output2])

Also, see the online demo at https://tensorflow-mnist.herokuapp.com/. It seems the API is fast enough.

Answer 2

TensorFlow Serving is a high performance, open source serving system for machine learning models, designed for production environments and optimized for TensorFlow. The initial release contains examples built with gRPC, but you can easily replace the front-end (denoted as "client" in the diagram below) with a RESTful API to suit your needs.

To get started quickly, check out the tutorial.