Example of 10-fold cross-validation with Neural ne

Question

I am looking for an example of applying 10-fold cross-validation in neural network.I need something link answer of this question: Example of 10-fold SVM classification in MATLAB

I would like to classify all 3 classes while in the example only two classes were considered.

Edit: here is the code I wrote for iris example

load fisheriris                              %# load iris dataset

k=10;
cvFolds = crossvalind('Kfold', species, k);   %# get indices of 10-fold CV
net = feedforwardnet(10);


for i = 1:k                                  %# for each fold
    testIdx = (cvFolds == i);                %# get indices of test instances
    trainIdx = ~testIdx;                     %# get indices training instances

    %# train 

    net = train(net,meas(trainIdx,:)',species(trainIdx)');
    %# test 
    outputs = net(meas(trainIdx,:)');
    errors = gsubtract(species(trainIdx)',outputs);
    performance = perform(net,species(trainIdx)',outputs)
    figure, plotconfusion(species(trainIdx)',outputs)
end

error given by matlab:

Error using nntraining.setup>setupPerWorker (line 62)
Targets T{1,1} is not numeric or logical.

Error in nntraining.setup (line 43)
    [net,data,tr,err] = setupPerWorker(net,trainFcn,X,Xi,Ai,T,EW,enableConfigure);

Error in network/train (line 335)
[net,data,tr,err] = nntraining.setup(net,net.trainFcn,X,Xi,Ai,T,EW,enableConfigure,isComposite);

Error in Untitled (line 17)
    net = train(net,meas(trainIdx,:)',species(trainIdx)');

Answer 1

It's a lot simpler to just use MATLAB's crossval function than to do it manually using crossvalind. Since you are just asking how to get the test "score" from cross-validation, as opposed to using it to choose an optimal parameter like for example the number of hidden nodes, your code will be as simple as this:

load fisheriris;

% // Split up species into 3 binary dummy variables
S = unique(species);
O = [];
for s = 1:numel(S)
    O(:,end+1) = strcmp(species, S{s});
end

% // Crossvalidation
vals = crossval(@(XTRAIN, YTRAIN, XTEST, YTEST)fun(XTRAIN, YTRAIN, XTEST, YTEST), meas, O);

All that remains is to write that function fun which takes in input and output training and test sets (all provided to it by the crossval function so you don't need to worry about splitting your data yourself), trains a neural net on the training set, tests it on the test set and then output a score using your preferred metric. So something like this:

function testval = fun(XTRAIN, YTRAIN, XTEST, YTEST)

    net = feedforwardnet(10);
    net = train(net, XTRAIN', YTRAIN');

    yNet = net(XTEST');
    %'// find which output (of the three dummy variables) has the highest probability
    [~,classNet] = max(yNet',[],2);

    %// convert YTEST into a format that can be compared with classNet
    [~,classTest] = find(YTEST);


    %'// Check the success of the classifier
    cp = classperf(classTest, classNet);
    testval = cp.CorrectRate; %// replace this with your preferred metric

end

I don't have the neural network toolbox so I am unable to test this I'm afraid. But it should demonstrate the principle.