writing a custom cost function in tensorflow

Question

I'm trying to write my own cost function in tensor flow, however apparently I cannot 'slice' the tensor object?

import tensorflow as tf
import numpy as np

# Establish variables
x = tf.placeholder("float", [None, 3])
W = tf.Variable(tf.zeros([3,6]))
b = tf.Variable(tf.zeros([6]))

# Establish model
y = tf.nn.softmax(tf.matmul(x,W) + b)

# Truth
y_ = tf.placeholder("float", [None,6])

def angle(v1, v2):
  return np.arccos(np.sum(v1*v2,axis=1))

def normVec(y):
  return np.cross(y[:,[0,2,4]],y[:,[1,3,5]])

angle_distance = -tf.reduce_sum(angle(normVec(y_),normVec(y)))
# This is the example code they give for cross entropy
cross_entropy = -tf.reduce_sum(y_*tf.log(y))

I get the following error: TypeError: Bad slice index [0, 2, 4] of type <type 'list'>

Answer 1

I think you can use "Wraps Python function" method in tensorflow. Here's the link to the documentation.

And as for the people who answered "Why don't you just use tensorflow's built in function to construct it?" - sometimes the cost function people are looking for cannot be expressed in tf's functions or extremely difficult.

Answer 2

This is because you have not initialized your variable and because of this it does not have your Tensor there right now (can read more in my answer here)

Just do something like this:

def normVec(y):
    print y
    return np.cross(y[:,[0,2,4]],y[:,[1,3,5]])

t1 = normVec(y_)
# and comment everything after it.

To see that you do not have a Tensor now and only Tensor("Placeholder_1:0", shape=TensorShape([Dimension(None), Dimension(6)]), dtype=float32).

Try initializing your variables

init = tf.initialize_all_variables()
sess = tf.Session()
sess.run(init)

and evaluate your variable sess.run(y). P.S. you have not fed your placeholders up till now.

Answer 3

At present, tensorflow can't gather on axes other than the first - it's requested.

But for what you want to do in this specific situation, you can transpose, then gather 0,2,4, and then transpose back. It won't be crazy fast, but it works:

tf.transpose(tf.gather(tf.transpose(y), [0,2,4]))

This is a useful workaround for some of the limitations in the current implementation of gather.

(But it is also correct that you can't use a numpy slice on a tensorflow node - you can run it and slice the output, and also that you need to initialize those variables before you run. :). You're mixing tf and np in a way that doesn't work.

x = tf.Something(...)

is a tensorflow graph object. Numpy has no idea how to cope with such objects.

foo = tf.run(x)

is back to an object python can handle.

You typically want to keep your loss calculation in pure tensorflow, so do the cross and other functions in tf. You'll probably have to do the arccos the long way, as tf doesn't have a function for it.

Answer 4

just realized that the following failed:

cross_entropy = -tf.reduce_sum(y_*np.log(y))

you cant use numpy functions on tf objects, and the indexing my be different too.