What's the difference of name scope and a vari

Question

What's the differences between these functions?

tf.variable_op_scope(values, name, default_name, initializer=None)

Returns a context manager for defining an op that creates variables. This context manager validates that the given values are from the same graph, ensures that that graph is the default graph, and pushes a name scope and a variable scope.

---

tf.op_scope(values, name, default_name=None)

Returns a context manager for use when defining a Python op. This context manager validates that the given values are from the same graph, ensures that that graph is the default graph, and pushes a name scope.

---

tf.name_scope(name)

Wrapper for Graph.name_scope() using the default graph. See Graph.name_scope() for more details.

---

tf.variable_scope(name_or_scope, reuse=None, initializer=None)

Returns a context for variable scope. Variable scope allows to create new variables and to share already created ones while providing checks to not create or share by accident. For details, see the Variable Scope How To, here we present only a few basic examples.

Answer 1

You can think them as two groups: variable_op_scope and op_scope take a set of variables as input and are designed to create operations. The difference is in how they affect the creation of variables with tf.get_variable:

def mysum(a,b,name=None):
    with tf.op_scope([a,b],name,"mysum") as scope:
        v = tf.get_variable("v", 1)
        v2 = tf.Variable([0], name="v2")
        assert v.name == "v:0", v.name
        assert v2.name == "mysum/v2:0", v2.name
        return tf.add(a,b)

def mysum2(a,b,name=None):
    with tf.variable_op_scope([a,b],name,"mysum2") as scope:
        v = tf.get_variable("v", 1)
        v2 = tf.Variable([0], name="v2")
        assert v.name == "mysum2/v:0", v.name
        assert v2.name == "mysum2/v2:0", v2.name
        return tf.add(a,b)

with tf.Graph().as_default():
    op = mysum(tf.Variable(1), tf.Variable(2))
    op2 = mysum2(tf.Variable(1), tf.Variable(2))
    assert op.name == 'mysum/Add:0', op.name
    assert op2.name == 'mysum2/Add:0', op2.name

notice the name of the variable v in the two examples.

same for tf.name_scope and tf.variable_scope:

with tf.Graph().as_default():
    with tf.name_scope("name_scope") as scope:
        v = tf.get_variable("v", [1])
        op = tf.add(v, v)
        v2 = tf.Variable([0], name="v2")
        assert v.name == "v:0", v.name
        assert op.name == "name_scope/Add:0", op.name
        assert v2.name == "name_scope/v2:0", v2.name

with tf.Graph().as_default():
    with tf.variable_scope("name_scope") as scope:
        v = tf.get_variable("v", [1])
        op = tf.add(v, v)
        v2 = tf.Variable([0], name="v2")
        assert v.name == "name_scope/v:0", v.name
        assert op.name == "name_scope/Add:0", op.name
        assert v2.name == "name_scope/v2:0", v2.name

You can read more about variable scope in the tutorial. A similar question was asked before on Stack Overflow.

Answer 2

As for API r0.11, op_scope and variable_op_scope are both deprecated. name_scope and variable_scope can be nested:

with tf.name_scope('ns'):
    with tf.variable_scope('vs'): #scope creation
        v1 = tf.get_variable("v1",[1.0])   #v1.name = 'vs/v1:0'
        v2 = tf.Variable([2.0],name = 'v2')  #v2.name= 'ns/vs/v2:0'
        v3 = v1 + v2       #v3.name = 'ns/vs/add:0'

Answer 3

Namespaces is a way to organize names for variables and operators in hierarchical manner (e.g. "scopeA/scopeB/scopeC/op1")

• tf.name_scope creates namespace for operators in the default graph.

• tf.variable_scope creates namespace for both variables and operators in the default graph.

• tf.op_scope same as tf.name_scope, but for the graph in which specified variables were created.

• tf.variable_op_scope same as tf.variable_scope, but for the graph in which specified variables were created.

Links to the sources above help to disambiguate this documentation issue.

This example shows that all types of scopes define namespaces for both variables and operators with following differences:

1. scopes defined by tf.variable_op_scope or tf.variable_scope are compatible with tf.get_variable (it ignores two other scopes)
2. tf.op_scope and tf.variable_op_scope just select a graph from a list of specified variables to create a scope for. Other than than their behavior equal to tf.name_scope and tf.variable_scope accordingly
3. tf.variable_scope and variable_op_scope add specified or default initializer.

Answer 4

From the last section of this page of the tensorflow documentation: Names of ops in tf.variable_scope()

[...] when we do with tf.variable_scope("name"), this implicitly opens a tf.name_scope("name"). For example:

with tf.variable_scope("foo"):
  x = 1.0 + tf.get_variable("v", [1])
assert x.op.name == "foo/add"

Name scopes can be opened in addition to a variable scope, and then they will only affect the names of the ops, but not of variables.

with tf.variable_scope("foo"):
    with tf.name_scope("bar"):
        v = tf.get_variable("v", [1])
        x = 1.0 + v
assert v.name == "foo/v:0"
assert x.op.name == "foo/bar/add"

When opening a variable scope using a captured object instead of a string, we do not alter the current name scope for ops.

Answer 5

Let's make it simple: just use tf.variable_scope. Quoting a TF developer,:

Currently, we recommend everyone to use variable_scope and not use name_scope except for internal code and libraries.

Besides the fact that variable_scope's functionality basically extends those of name_scope, consider how they do not play so nice together:

with tf.name_scope('foo'):
  with tf.variable_scope('bar'):
    x = tf.get_variable('x', shape=())
    x2 = tf.square(x**2, name='x2')
print(x.name)
# bar/x:0
print(x2.name)
# foo/bar/x2:0

By sticking to variable_scope only you avoid some headaches due to this kind of incompatibility.

Answer 6

Both variable_op_scope and op_scope are now deprecated and should not be used at all.

Regarding the other two, I also had problems understanding the difference between variable_scope and name_scope (they looked almost the same) before I tried to visualize everything by creating a simple example:

import tensorflow as tf


def scoping(fn, scope1, scope2, vals):
    with fn(scope1):
        a = tf.Variable(vals[0], name='a')
        b = tf.get_variable('b', initializer=vals[1])
        c = tf.constant(vals[2], name='c')

        with fn(scope2):
            d = tf.add(a * b, c, name='res')

        print '\n  '.join([scope1, a.name, b.name, c.name, d.name]), '\n'
    return d

d1 = scoping(tf.variable_scope, 'scope_vars', 'res', [1, 2, 3])
d2 = scoping(tf.name_scope,     'scope_name', 'res', [1, 2, 3])

with tf.Session() as sess:
    writer = tf.summary.FileWriter('logs', sess.graph)
    sess.run(tf.global_variables_initializer())
    print sess.run([d1, d2])
    writer.close()

Here I create a function that creates some variables and constants and groups them in scopes (depending on the type I provided). In this function, I also print the names of all the variables. After that, I executes the graph to get values of the resulting values and save event-files to investigate them in TensorBoard. If you run this, you will get the following:

scope_vars
  scope_vars/a:0
  scope_vars/b:0
  scope_vars/c:0
  scope_vars/res/res:0 

scope_name
  scope_name/a:0
  b:0
  scope_name/c:0
  scope_name/res/res:0 

You see the similar pattern if you open TensorBoard (as you see b is outside of scope_name rectangular):

---

This gives you the answer:

Now you see that tf.variable_scope() adds a prefix to the names of all variables (no matter how you create them), ops, constants. On the other hand tf.name_scope() ignores variables created with tf.get_variable() because it assumes that you know which variable and in which scope you wanted to use.

A good documentation on Sharing variables tells you that

tf.variable_scope(): Manages namespaces for names passed to tf.get_variable().

The same documentation provides a more details how does Variable Scope work and when it is useful.