We need help to understand the parameters to use for smaller set of training (6000 jpgs) and val (170 jpgs) jpgs. Our execution was killed and exited after test score 0/1 in Iteration 0.
We are trying to run the imagenet sample on the caffe website tutorial at
http://caffe.berkeleyvision.org/gathered/examples/imagenet.html.
Instead of using the full set of ILSVRC2 images in the package, we use our own training set of 6000 jpegs and val set of 170 jpeg images. They are each 256 x 256 jpeg files in the train and val directories as instructed. We ran the script to get the auxillary data:
./data/ilsvrc12/get_ilsvrc_aux.sh
The train.txt and val.txt files are setup to describe one of two possible categories for each jpeg file. Then we ran the script to compute the mean image data which appeared to run correctly:
./examples/imagenet/make_imagenet_mean.sh
We used the model definitions supplied in the tutorial for imagenet_train.prototxt and imagenet_val.prototxt. Since we are training on much fewer images we modified the imagenet_solver.prototxt as follows:
train_net: "./imagenet_train.prototxt"
test_net: "./imagenet_val.prototxt"
test_iter: 3
test_interval: 10
base_lr: 0.01
lr_policy: "step"
gamma: 0.1
stepsize: 10
display: 20
max_iter: 45
momentum: 0.9
weight_decay: 0.0005
snapshot: 10
snapshot_prefix: "caffe_imagenet_train"
solver_mode: CPU
when we run it using:
./train_imagenet.sh
We get the following output where it hangs:
.......
.......
I0520 23:07:53.175761 4678 net.cpp:85] drop7 <- fc7
I0520 23:07:53.175791 4678 net.cpp:99] drop7 -> fc7 (in-place)
I0520 23:07:53.176246 4678 net.cpp:126] Top shape: 50 4096 1 1 (204800)
I0520 23:07:53.176275 4678 net.cpp:152] drop7 needs backward computation.
I0520 23:07:53.176296 4678 net.cpp:75] Creating Layer fc8
I0520 23:07:53.176306 4678 net.cpp:85] fc8 <- fc7
I0520 23:07:53.176314 4678 net.cpp:111] fc8 -> fc8
I0520 23:07:53.184213 4678 net.cpp:126] Top shape: 50 1000 1 1 (50000)
I0520 23:07:53.184908 4678 net.cpp:152] fc8 needs backward computation.
I0520 23:07:53.185607 4678 net.cpp:75] Creating Layer prob
I0520 23:07:53.186135 4678 net.cpp:85] prob <- fc8
I0520 23:07:53.186538 4678 net.cpp:111] prob -> prob
I0520 23:07:53.187166 4678 net.cpp:126] Top shape: 50 1000 1 1 (50000)
I0520 23:07:53.187696 4678 net.cpp:152] prob needs backward computation.
I0520 23:07:53.188244 4678 net.cpp:75] Creating Layer accuracy
I0520 23:07:53.188431 4678 net.cpp:85] accuracy <- prob
I0520 23:07:53.188540 4678 net.cpp:85] accuracy <- label
I0520 23:07:53.188870 4678 net.cpp:111] accuracy -> accuracy
I0520 23:07:53.188907 4678 net.cpp:126] Top shape: 1 2 1 1 (2)
I0520 23:07:53.188915 4678 net.cpp:152] accuracy needs backward computation.
I0520 23:07:53.188922 4678 net.cpp:163] This network produces output accuracy
I0520 23:07:53.188942 4678 net.cpp:181] Collecting Learning Rate and Weight Decay.
I0520 23:07:53.188954 4678 net.cpp:174] Network initialization done.
I0520 23:07:53.188961 4678 net.cpp:175] Memory required for Data 210114408
I0520 23:07:53.189008 4678 solver.cpp:49] Solver scaffolding done.
I0520 23:07:53.189018 4678 solver.cpp:61] Solving CaffeNet
I0520 23:07:53.189033 4678 solver.cpp:106] Iteration 0, Testing net
I0520 23:09:06.699695 4678 solver.cpp:142] Test score #0: 0
I0520 23:09:06.700203 4678 solver.cpp:142] Test score #1: 7.07406
Killed
Done.
