How can I save individual iterations of a parfor

Question

I'm struggling to save output of each PARFOR iteration, but until now without success. I've searched a lot online, and according to some official newgroup replies, the correct way to deal with saving in a PARFOR (i.e., addressing the "transparency" issue) is to use an external function in the following way (elementary example):

parfor i=1:120
  display(i);
  [LL ll] = eig(rand(1000,1000));
  record(:,i) = diag(ll);
  samplesave('save.mat',record(:,i),i);
end

along with

function samplesave(fname, data,i)
   persistent st;
   store(:,i)=data;
   save(fname);
end

the problem is that declaring "st" as PERSISTENT make the variable "persistent" across calls of the same worker, but not across the PARFOR loop, so that at any given point you have only the whole history of the last worker that managed to save; the next PARFOR iteration (from a different worker) will overwrite this record with the whole history of (only) that worker, and so on.

How can I incrementally save history of all the workers?

Thank you,

Answer 1

I assume there is a reason why you can't wait until after the loop terminate to save the entire matrix.

One idea would be to save each iteration independent of the others using something like this:

baseFile = ('part_%03d.mat');
parfor i=1:120
  display(i);
  [LL ll] = eig(rand(1000,1000));
  singleRecord = diag(ll);
  record(:,i) = singleRecord;
  save(sprintf(baseFile, i), 'singleRecord');
end

Then load the files one by one and combine the values into a single matrix.

mat = [];
baseFile = ('part_%03d.mat');
for i = 1:120
  d = load(sprintf(baseFile, i));
  mat(:,i) = d.singleRecord;
end