It seems like one (or more) of your label values are invalid, see this PR for information:
If you have an invalid ground truth label, "SoftmaxWithLoss" will silently access invalid memory [...] The old check only worked in DEBUG mode and also only worked for CPU.
Make sure your prediction vector length matches the number of labels you try to predict.
From your comments, it seems like you have labels in the range 0..10575, but on the other hand, your classification layer, "fc7" only predicts probabilities for 1000 classes. Thus, "SoftmaxWithLoss" layer tries to compute the loss for predicting label l>1000, and access memory outside the probability array, resulting with a segmentation fault.
It seems like one (or more) of your label values are invalid, see this PR for information:
Make sure your prediction vector length matches the number of labels you try to predict.
From your comments, it seems like you have labels in the range
0..10575
, but on the other hand, your classification layer,"fc7"
only predicts probabilities for 1000 classes. Thus,"SoftmaxWithLoss"
layer tries to compute the loss for predicting labell
>1000, and access memory outside the probability array, resulting with a segmentation fault.