I am trying to create a custom loss function for my deep learning model and I run into an error.
I am going to give here an example of a code that is not what I want to use but if I understand how to make this little loss function work, then I think I'll be able to make my long loss function work. So I am pretty much asking for help to make this next function work, here it is.
model.compile(optimizer='rmsprop',loss=try_loss(pic_try), metrics=
['accuracy'])
def try_loss(pic):
def try_2_loss(y_true,y_pred):
return tf.py_function(func=try_3_loss,inp=[y_pred,pic], Tout=tf.float32)
return try_2_loss
def try_3_loss(y_pred,pic):
return tf.reduce_mean(pic)
Now I want to know the following:
1. Does the pic that I am entering into my model.compile line need to be a tensor? Can it be a numpy array?
2. In my try_3_loss function, can I replace tf.reduce_mean to np.mean?
3. In my try_3_loss function, can I use normal numpy commands on y_pred, such as np.mean(y_pred)?
My main thing is that I want to use as many numpy commands as possible.
I tried to use all sorts of stuff, I tried to have my pic be a numpy array, I tried to use with that the np.mean (pic) in my try_3_loss function, I tried to make my pic be a tensor object and then use the tf.reduce_mean in my try_3_project and I tried to do sess.run(pic) before running the model.compile line and in all of the above situations I got the following error:
TypeError Traceback (most recent call
last)
<ipython-input-75-ff45de7120bc> in <module>()
----> 1 model.compile(optimizer='rmsprop',loss=try_loss(pic_try),
metrics=['accuracy'])
1 frames
/usr/local/lib/python3.6/dist-packages/keras/engine/training.py in
compile(self, optimizer, loss, metrics, loss_weights, sample_weight_mode,
weighted_metrics, target_tensors, **kwargs)
340 with K.name_scope(self.output_names[i] +
'_loss'):
341 output_loss = weighted_loss(y_true, y_pred,
--> 342 sample_weight,
mask)
343 if len(self.outputs) > 1:
344 self.metrics_tensors.append(output_loss)
/usr/local/lib/python3.6/dist-packages/keras/engine/training_utils.py in
weighted(y_true, y_pred, weights, mask)
418 weight_ndim = K.ndim(weights)
419 score_array = K.mean(score_array,
--> 420 axis=list(range(weight_ndim,
ndim)))
421 score_array *= weights
422 score_array /= K.mean(K.cast(K.not_equal(weights, 0),
K.floatx()))
TypeError: 'NoneType' object cannot be interpreted as an integer
Some test code:
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras.layers import *
from tensorflow.keras.models import Model
from tensorflow.keras import backend as K
@tf.custom_gradient
def py_loss_fn(y_true, y_pred):
""" This function takes eager tensors as inputs which can be explicitly
converted to np.arrays via EagerTensor.numpy() or implicitly converted
by applying numpy operations to them.
However, once tf operations are no longer used it means that the function has to
implement its own gradient function.
"""
def grad(dy):
""" Compute gradients for function inputs.
Ignore input[0] (y_true) since that is model.targets[0]
"""
g = np.mean(-dy * np.sign(y_true - y_pred), axis=1)[:, np.newaxis]
return None, g
return np.mean(np.abs(y_true - y_pred), axis=1), grad
def eager_loss_fn(y_true, y_pred):
""" If tf operations are used on eager tensors auto diff works without issues
"""
return tf.reduce_mean(tf.abs(y_true - y_pred))
def loss_fn(y_true, y_pred, **kw_args):
""" This function takes tensors as inputs. Numpy operations are not valid.
"""
# loss = tf.py_function(eager_loss_fn, inp=[y_true, y_pred], Tout=tf.float32)
loss = tf.py_function(py_loss_fn, inp=[y_true, y_pred], Tout=tf.float32)
return loss
def make_model():
""" Linear regression model with custom loss """
inp = Input(shape=(4,))
out = Dense(1, use_bias=False)(inp)
model = Model(inp, out)
model.compile('adam', loss_fn)
return model
model = make_model()
model.summary()
Test code to invoke the model:
import numpy as np
FACTORS = np.arange(4) + 1
def test_fn(x):
return np.dot(x, FACTORS.T)
X = np.random.rand(3, 4)
Y = np.apply_along_axis(test_fn, 1, X)
history = model.fit(X, Y, epochs=1000, verbose=False)
print(history.history['loss'][-1])
Thank you so much for your help! I actually decided to switch to tf 2.0 and writing functions there is MUCH easier, although it is a bit expensive in terms of efficiency, I can always very easily switch from np arrays to tensors and back so I just wrote it all in numpy array format and switched it back. So the inputs and outputs to all of my functions are tensors, but inside the functions I switch it to numpy arrays and before I return it back I switch it back to tensors, but I still have an error. The code goes like this:
model.compile(optimizer='rmsprop',loss=custom_loss(pic),
loss_weights=[None],metrics=['accuracy'])
def my_loss(y_true, y_pred):
return loss(y_pred,pic)
def custom_loss(pic):
return my_loss
And when I actually try to run the loss functions (not in the model.compile) as so:
my_loss(x0,x0)
I get the following:
orig shape x: (1, 2501)
shape x: (2501,)
shape pic: (100, 100)
shape a: ()
shape ms: (2500,)
r_size: 50
c_size: 50
<tf.Tensor: id=261, shape=(), dtype=float64, numpy=6.741635588952273>
So I do get an output of a tensor with the loss that I wanted. (printed are things that will help understand the error) HOWEVER when I try to run the compile command I get this:
orig shape x: ()
(...a bunch of unneccessary stuff...)
----> 4 x=np.reshape(x,(2501,1))
5 x=np.reshape(x,(2501,))
6 pic=np.array(pic)
/usr/local/lib/python3.6/dist-packages/numpy/core/fromnumeric.py in reshape(a,
newshape, order)
290 [5, 6]])
291 """
--> 292 return _wrapfunc(a, 'reshape', newshape, order=order)
293
294
/usr/local/lib/python3.6/dist-packages/numpy/core/fromnumeric.py in
_wrapfunc(obj, method, *args, **kwds)
54 def _wrapfunc(obj, method, *args, **kwds):
55 try:
---> 56 return getattr(obj, method)(*args, **kwds)
57
58 # An AttributeError occurs if the object does not have
ValueError: cannot reshape array of size 1 into shape (2501,1)
It's like the compiler doesn't understand that the y_pred will have the size of the output of my model.
My model:
model = tf.keras.Sequential()
#add model layers
model.add(layers.Conv2D(64, kernel_size=3,activation='linear',input_shape=
(inputs_shape_0,inputs_shape_1,1)))
#model.add(LeakyReLU(alpha=0.3))
model.add(layers.Conv2D(32, kernel_size=3,activation='linear'))
#model.add(LeakyReLU(alpha=0.3))
model.add(layers.Flatten())
model.add(layers.Dense(2501, activation='linear'))
Any ideas how to fix it? I will also look at the test code you sent me to get an idea.
Thank you!
