Convolutional ImageNet network is invariant to fli

I am using Deep learning caffe framework for image classification.
I have coins with faces. Some of them are left directed some of them are right.

To classify them I am using common aproach - take weights and structure from pretrained ImageNet network that have already capture a lot of image patterns and train mostly the last layer to fit my training set.

But I have found that netowork does not works on this set: I have taken some coin for example leftdirected , generated horizontally flipped image for it and marked it as right sided.

For this set convolutional net gets ~50% accuracy, it is exactly random result.

I have also tried to train net on 2 images ( 2 flipped versions of "h" letter ). But with the same result - 50% . ( If I choose to diffetrent letters and train net on augemeneted dataset - i receive 100% accuracy very fast ) . But invariance to flipping brokes my classification.

My question is: is there exists some aproach that allowes me to use advantages of pretrained imagenet but broke somehow this invariance. And what layer on net make invariance possible.

I am using "caffe" for generating net based on this example approach:

https://github.com/BVLC/caffe/blob/master/examples/02-fine-tuning.ipynb

Caffe basic/baseline models trained on image net mostly use the very trivial image augmentation: flipping images horizontally. That is, imagenet classes are indeed the same when flipped horizontally. Thus, the weights you are trying to fine-tune were trained in a setting where horizontal flip should be ignored and I suppose what you see is a net that captured this quite well - it is no longer sensitive to this particular transformation.
It is not trivial to tell at what layer of the net this invariance is happening and therefore it is not easy to say what layers should be fine-tuned to overcome this behavior. I suppose this invariance is quite fundamental to the network and I will not be surprise if it required re-training of the entire net.