Matlab Function Performance - Too many loops

Question

I have to write in a txt file several informations in a LOT of rows. The result is a file like:

result.txt:
RED;12;7;0;2;1;4;7;0.0140
RED;12;7;0;2;2;9;7;0.1484
RED;12;7;0;2;3;7;4;0.1787
RED;12;7;0;2;4;2;6;0.7891
RED;12;7;0;2;5;9;6;0.1160
RED;12;7;0;2;6;9;1;0.9893
...

Which is build by the code below (with some reduced dimensions):

/* the variables 'str1', 'num1', 'day', 'vect1', 'vect2' and 'MD' are inputs of this function
/* str1 is a string 1x1
/* num1 is a integer 1x1 
/* day is a vector 10x1
/* vect1 is a vector 7x1
/* vect2 is a vector 180x1
/* MD is a 4D matrix (7x180x10x15)*/

fid = fopen(path_result, 'Wt');    
for i1 = 1:15   
    for i2 = 1:10    
        for i3 = 1:7           
           for i4= 1:180
            /* print all the values */
                fprintf(fid,'%s%s%d%s%d%s%d%s%d%s%d%s%d%s%d%s%.4f \n',...
                str1,';',num1,';',i1,';',0,';',2,';',...
                day(i2,1),';',vect1(i3),';',...
                vect2(i4),';',MD(i3,i4,i2,i1));
            end
        end
    end
end

I saw some vectorization in here (stackoverflow post), but I don't think it's possible to apply in here. Any ideas?
Thanks in advance

Answer 1

One of the things you can do to optimize your code is to look for "repeated" bits of code. In your case, you format ALL of the result string in the innermost loop - although much of the string doesn't change. You also "format" the separator string ';' several times - you can have that directly in your formatting string (you can intersperse text and formatting commands in the format string). I combined these ideas in a few different ways, and timed them:

str1 = 'hello';
num1 = 123;
day = (1:10)';
vect1 = (1:7)';
vect2 = (1:180)';
MD = rand(7,180,10,15);
path_result = './mixedOutput1.txt';
fid = fopen(path_result, 'Wt');    
tic
for i1 = 1:15   
    for i2 = 1:10    
        for i3 = 1:7           
           for i4= 1:180
%             /* print all the values */
                fprintf(fid,'%s%s%d%s%d%s%d%s%d%s%d%s%d%s%d%s%.4f \n',...
                str1,';',num1,';',i1,';',0,';',2,';',...
                day(i2,1),';',vect1(i3),';',...
                vect2(i4),';',MD(i3,i4,i2,i1));
            end
        end
    end
end
fprintf(1, 'time for original loop: %.2f sec\n',toc)
fclose(fid);
%%
path_result = './mixedOutput2.txt';
fid = fopen(path_result, 'Wt');    
tic
for i1 = 1:15   
    for i2 = 1:10    
        for i3 = 1:7           
           for i4= 1:180
%             /* print all the values */
                fprintf(fid,'%s;%d;%d;0;2;%d;%d;%d;%.4f \n',...
                str1, num1, i1, day(i2,1), vect1(i3), vect2(i4),MD(i3,i4,i2,i1));
            end
        end
    end
end
fprintf(1, 'time for faster loop: %.2f sec\n',toc)
fclose(fid);
%%
path_result = './mixedOutput3.txt';
fid = fopen(path_result, 'Wt');    
tic
y = cell(1,15*10*7*180);
cc = 0;
for i1 = 1:15   
    for i2 = 1:10    
        for i3 = 1:7           
           for i4= 1:180
%             /* print all the values */
                cc = cc + 1;
                y{1,cc} = sprintf('%s;%d;%d;0;2;%d;%d;%d;%.4f \n',...
                str1, num1, i1, day(i2,1), vect1(i3), vect2(i4), MD(i3,i4,i2,i1));
            end
        end
    end
end
fprintf(1, 'time for loop with sprintf intermediate step: %.2f sec\n', toc)
fprintf(fid, '%s', y{:});
fprintf(1, 'time including file write: %.2f sec\n', toc);
fclose(fid);

%% optimize loop more:

path_result = './mixedOutput4.txt';
fid = fopen(path_result, 'Wt');    
tic
y = cell(1,15*10*7*180);
cc = 0;
for i1 = 1:15   
    for i2 = 1:10    
        for i3 = 1:7           
            x = sprintf('%s;%d;%d;0;2;%d;%d;', ...
                str1, num1, i1, day(i2,1), vect1(i3));
            for i4= 1:180
                fprintf(fid, '%s%d;%.4f \n', ...
                     x, vect2(i4), MD(i3,i4,i2,i1));
            end
        end
    end
end
fprintf(1, 'time for fastest loop: %.2f sec\n', toc);
fclose(fid);

On my machine, this resulted in the following benchmarks:

Original loop: 15.9 sec
Faster format:  9.2 sec
With sprintf:   8.2 sec
preformat:      6.2 sec

The "preformat" was not done as efficiently as possible - it was just there for illustration. The intermediate string x is computed much less frequently, then re-used.

Finally - I did create a "vectorized" version of the code - meaning that the entire sprintf happens in a single line. This requires creating a big cell array (Kahuna, below) with the right elements - it turns out that's actually marginally less efficient than the last code above (with the "preformatting"), but here it is just in case:

%% truly vectorized:
tic
Kahuna = cell(7, 15*10*7*180);
N = 15 * 10 * 7 * 180;
N1 = ones(1, N);
% final order needs to be [180 7 10 15] - inner loop first
Kahuna(1,:) = cellstr(repmat(str1, [N 1]))';
Kahuna(2,:) = mat2cell(repmat(num1, [N 1]), N1, 1);
Kahuna(3,:) = mat2cell(reshape(repmat(reshape(1:15,    1, 1, 1, 15), [180 7 10  1]), [], 1), N1, 1);
Kahuna(4,:) = mat2cell(reshape(repmat(reshape(day,     1, 1, 10, 1), [180 7  1 15]), [], 1), N1, 1);
Kahuna(5,:) = mat2cell(reshape(repmat(reshape(vect1,   1, 7, 1,  1), [180 1 10 15]), [], 1), N1, 1);
Kahuna(6,:) = mat2cell(reshape(repmat(reshape(vect2, 180, 1, 1,  1), [  1 7 10 15]), [], 1), N1, 1);
Kahuna(7,:) = mat2cell(reshape(permute(MD, [2 1 3 4]), [], 1), N1, 1);

x = sprintf('%s;%d;%d;0;2;%d;%d;%d;%.4f \n', Kahuna{:});
toc

Answer 2

Actually it seems that simply creating everything first and writing it later (the way I tried it) does not give a speed increase. My original thought was to save the data with save(file,data,'-ascii') but that gave unexpected results.

If you would only have numeric data you could perhaps use dlmwrite but I guess that is not an option now.

Here is the timing of my attempt compared to your original code, including some assumed inputs:

str1 = 'RED';
num1 = 4;
day = rand(10);
vect1 = 1:7;
vect2 = 1:180;
MD = rand(7,180,10,15);
y=[];

tic
for i1 = 1:15   
    for i2 = 1:10    
        for i3 = 1:7           
           for i4= 1:180

                x=sprintf('%s%s%d%s%d%s%d%s%d%s%d%s%d%s%d%s%.4f \n',...
                str1,';',num1,';',i1,';',0,';',2,';',...
                day(i2,1),';',vect1(i3),';',...
                vect2(i4),';',MD(i3,i4,i2,i1));
                y{end+1} = x;
            end
        end
    end
end

fid = fopen('test.txt','w');
for i=1:length(y)
         fprintf(fid,y{i});
end
fclose(fid)
t1=toc;

tic
fid = fopen('test.txt', 'Wt');    
for i1 = 1:15   
    for i2 = 1:10    
        for i3 = 1:7           
           for i4= 1:180
                fprintf(fid,'%s%s%d%s%d%s%d%s%d%s%d%s%d%s%d%s%.4f \n',...
                str1,';',num1,';',i1,';',0,';',2,';',...
                day(i2,1),';',vect1(i3),';',...
                vect2(i4),';',MD(i3,i4,i2,i1));
            end
        end
    end
end
t2=toc;

myTime = t1 % 56 secs
originalTime = t2 % 12 secs

Answer 3

A first optimization which can be applied to any situation is to generate a format string, let's call it fmt:

fmt       = sprintf('%s;%d;%%d;%d;%d;%%d;%%d;%%d;%%.4f \\n',str1,num1,0,2)
fmt =
RED;4;%d;0;2;%d;%d;%d;%.4f \n

After which the code inside the loops become:

x = sprintf(fmt, i1, day(i2,1), vect1(i3), vect2(i4), MD(i3,i4,i2,i1));

Now, a fully vectorized solution, which trades off on RAM but achieves an order of magnitude speedup, on my setup ~10.8x from 9.61 to 0.89 seconds.

tic
[a,b,c,d] = ndgrid(vect2,vect1,day,1:15);
out       = sprintf(fmt, [d(:), c(:), b(:), a(:), reshape(permute(MD,[2,1,3,4]),[],1)]'); 
toc