How to add new nodes / neurons dynamically in tens

Question

If I want to add new nodes to on of my tensorflow layers on the fly, how can I do that?

For example if I want to change the amount of hidden nodes from 10 to 11 after the model has been training for a while. Also, assume I know what value I want the weights coming in and out of this node/neuron to be.

I can create a whole new graph, but is there a different/better way?

Answer 1

Instead of creating a whole new graph you might be better off creating a graph which has initially more neurons than you need and mask it off by multiplying by a non-trainable variable which has ones and zeros. You can then change the value of this mask variable to allow effectively new neurons to act for the first time.

Answer 2

TensorFlow has many advantadges, but dynamic graph modification isn't one of them.

If you truly want to be able to change graphs on the fly, I recommend you PyTorch (http://pytorch.org/). Although it's much more recent than TensorFlow, so the documentation isn't as complete.

Answer 3

Let us assume that res is output taking into account there are 10 hidden layers and let us call 10th hidden layer layer10. So what you do is

res = f(layer10)

where f is some function which operates on layer10 and returns resulting tensor.

Pre-create layer11 as some operation on layer10 and res2 as output from layer11.

Now running you just evaluate res which will use on 10 layers. When you want to use 11 layers use res2. If you know the weights then you can assign it like tensor.assign(val).eval()