caffe: Confused about regression

Question

I have a really weird problem I want to explain to you. I am not sure if this is a topic for SO but I hope it will be in the end.

My general problem task is depth estimation, i.e. I have an image as input and its corresponding ground_truth (depth image). Then I have my net (which should be considered as black box) and my last layers.

First of all depth estimation is rather a regression task than a classification task. Therefore I decided to use a EuclideanLoss layer where my num_output of my last Convolution layer is 1 --> output is 1xheightxwidth. The net converges and the loss value in the end is 3. When I try to create my estimated output image by using my deploy.prototxt where I remove the EuclideanLoss layer. My las layer then is a ReLU layer like this:

layer {
    name: "relu"
    type: "ReLU"
    bottom: "conv"
    top: "result"
    relu_param{
    negative_slope: 0.01
        engine: CUDNN
    }
}

My problem is: the estimated depth image does not even look like my ground_truth. Ok well first that sounds a bit like yeah you have a net and it does not work. But when I alter my task into a classification task which should be less suited for this approach I get really good results. By altering it to a classification task I mean exchanging the EculideanLoss layer with a SoftmaxLayer and rather than num_output = 1 in the last Convolution layer I have num_output = 256 For 256 depth values where 0 is black and 1 is white.

I am really confused since the regression task should work better in theory than the classification task. Can anyone help me with this problem? Is there any big mistake in my thoughts? Or did I forget something which is really important to do for regression?

Any answers are really appreciated!