In OpenAI baselines code on DQN, tf.stop_gradient
is used on the q values of the target network during building the operation graph to prevent the contributions of the target q values to the minimization of the loss. (line 213)
However, when calling minimize
, the var_list
is specified as only the tf.Variable
with scope that falls under the q network being optimized, excluding the variables with scope under the target q network. (line 223)
I'm not sure why they do both. The two approaches seem to achieve the same result.
It's redundant. IMO code reads better - you know that gradient will not flow through that expression, and also you know exactly which variables will be affected.
One would indeed suffice to achieve equivalent effect.