Keras/Tensorflow: Get predictions or output of all

I am able to get the output/predictions of all layers as suggested in Keras Docs: how-can-i-obtain-the-output-of-an-intermediate-layer

def get_output_of_all_layers(model, test_input):
    output_of_all_layers = []

    for count, layer in enumerate(model.layers):

        # skip the input layer
        if count == 0:
            continue

        intermediate_layer_model = Model(inputs=model.input, outputs=model.get_layer(layer.name).output)
        intermediate_output = intermediate_layer_model.predict(test_input)[0]

        output_of_all_layers.append(intermediate_output)

    return np.array(output_of_all_layers)

But this is incredible slower and takes more than a minute (clocked at ~65s, in 6700HQ with GTX1070, that's ridiculously high, inference happens in less than a second...!) for models with about 50 layers. I guess this is because it is building a model every single time, load the model into the memory, pass inputs and get outputs. Clearly, you can't get the output of the last layer without getting results from the other layers, how do I save them all like above without having to create redundant models (or in a faster, more efficient way)?

Update: I also noticed that this does NOT utilize my GPU, this means all the conv layers are being executed by the CPU? Why wouldn't it use my GPU for this? I reckon it's gonna take way less if it used my GPU.

How do I do it more efficiently?

As suggested by Ben Usman, you can first wrap the model in a basic end-to-end Model, and provide its layers as outputs to a second Model:

import keras.backend as K
from keras.models import Model
from keras.layers import Input, Dense

input_layer = Input((10,))

layer_1 = Dense(10)(input_layer)
layer_2 = Dense(20)(layer_1)
layer_3 = Dense(5)(layer_2)

output_layer = Dense(1)(layer_3)

basic_model = Model(inputs=input_layer, outputs=output_layer)

# some random input
import numpy as np
features = np.random.rand(100,10)

# With a second Model
intermediate_model = Model(inputs=basic_model.layers[0].input, 
                              outputs=[l.output for l in basic_model.layers[1:]])
intermediate_model.predict(features) # outputs a list of 4 arrays

Or, you could use a Keras function in similar fashion:

# With a Keras function
get_all_layer_outputs = K.function([basic_model.layers[0].input],
                                  [l.output for l in basic_model.layers[1:]])

layer_output = get_all_layer_outputs([features]) # return the same thing