MATLAB write multipage tiff exponentially slow

Question

I am trying to write a single multipage tiff file which is 128 pixels x 128 pixels x 122000 frames of 16-bit unsigned integers. ImageJ or a short Python script can do this in less than one minute on a fast machine. On the same machine, MATLAB would likely take days to do so using any method I have tried. Here is what I have tried so far:

Use imwrite to write IMG to test.tif (recommended by MATLAB documentation)

    tic
    numframes=size(IMG,3);
    divider=10^(floor(log10(numframes))-1);
    imwrite(IMG(:,:,1),'test.tif');
    for i=2:numframes;
        imwrite(IMG(:,:,i),'test.tif','WriteMode','append');
        if (round(i/divider)==i/divider)
            fprintf('Frame %d written in %.0f seconds, %2d percent complete, time                 left=%.0f seconds \n', ...
                i, toc, i/numframes*100, (numframes - i)/(i/toc));
        end
    end

which results in the following output:

Frame 10000 written in 104 seconds, 8.196721e+00 percent complete, time left=1163 seconds Frame 20000 written in 296 seconds, 1.639344e+01 percent complete, time left=1509 seconds Frame 30000 written in 590 seconds, 2.459016e+01 percent complete, time left=1809 seconds Frame 40000 written in 1035 seconds, 3.278689e+01 percent complete, time left=2121 seconds Frame 50000 written in 1682 seconds, 4.098361e+01 percent complete, time left=2421 seconds

Notice the exponential increase in time as more frames are written.

Use the Tiff class directly

    if bigtiff
        t = Tiff(fname,'w8');
    else
        t = Tiff(fname,'w');
    end
    tagstruct.ImageLength = size(image,1);
    tagstruct.ImageWidth = size(image,2);
    tagstruct.Photometric = Tiff.Photometric.MinIsBlack;
    if bitspersamp==16
        tagstruct.BitsPerSample = 16;
    end
    if bitspersamp==32
        tagstruct.BitsPerSample = 32;
    end
    tagstruct.SamplesPerPixel = 1;
    tagstruct.RowsPerStrip = 256;
    tagstruct.PlanarConfiguration = Tiff.PlanarConfiguration.Chunky;
    tagstruct.Software = 'MATLAB';
    t.setTag(tagstruct);
    t.write(image(:,:,1));
    numframes = size(image,3);
    divider = 10^(floor(log10(numframes))-1);
    tic
    for i=2:numframes
        t.writeDirectory();
        t.setTag(tagstruct);
        t.write(image(:,:,i));
        if (round(i/divider)==i/divider)
            fprintf('Frame %d written in %.0f seconds, %2d percent complete, time left=%.0f seconds \n', ...
                i, toc, i/numframes*100, (numframes - i)/(i/toc));
        end
    end
    t.close();

which results in the following output:

Frame 10000 written in 66 seconds, 8.196721e+00 percent complete, time left=743 seconds 
Frame 20000 written in 225 seconds, 1.639344e+01 percent complete, time left=1145 seconds 
Frame 30000 written in 481 seconds, 2.459016e+01 percent complete, time left=1474 seconds 
Frame 40000 written in 915 seconds, 3.278689e+01 percent complete, time left=1877 seconds 
Frame 50000 written in 1512 seconds, 4.098361e+01 percent complete, time left=2177 seconds

Attempting to make use of the BigTIFF library does not work

Following the discussion here: http://blogs.mathworks.com/steve/2013/08/07/tiff-bigtiff-and-blockproc/

I attempted to convert the code to work with uint16 data by changing line 73 to:

    obj.TiffObject.setTag('BitsPerSample', 16);

but after writing with

outFileWriter = bigTiffWriter('test.tif', inFileInfo(1).Height, inFileInfo(1).Width, tileSize(1), tileSize(2));
for i=1:122000
    blockproc(IMG(:,:,i),tileSize,@(b) b.data,'Destination',outFileWriter);
    if rem(i,10000)==0
        fprintf('Frame %d done\n',i)
    end
end

I get the following error when trying to read back in:

Unexpected Standard exception from MEX file.
What() is:std::bad_alloc
..

Error in imtifinfo (line 27)
raw_tags = tifftagsread(filename,0,0,0);

Error in imfinfo (line 183)
info = feval(fmt_s.info, filename);

Error in TiffReader (line 11)
InfoImage=imfinfo(fname);

On a related note, preallocating the file with the correct size on the disk makes no difference

I thought there was a slim chance this was a file I/O problem, in which case preallocating the space on disk might be relevant, so I tried what was mentioned here: http://www.mathworks.co.uk/matlabcentral/newsreader/view_thread/241072, namely:

% Create the file
fh = javaObject('java.io.RandomAccessFile', 'test.dat', 'rw');
% Allocate the right amount of space
fh.setLength(1024);
% Close the file
fh.close();

but it made no difference.

Any help would be greatly appreciated.

Answer 1

I also once ran into this problem. It seems Matlab 2018b has two .mexw64 files that can handle tiff writing.

wtifc.mexw64 in \toolbox\matlab\imagesci\private called by writetif.mwhich is used by imwrite

and

tifflib.mexw64, same location, used by the Tiff object.

Both slow down with multi image tiff files. It is a lot faster to use fwrite to write all image data and end with the pointers to this data.

I tested three methods using this script and the difference is quite big. This was done using matlab 2018b.

clear all;close all; clc; fclose all;
%generate some data
N=1E3;
IM=imread('landOcean.jpg');
IM = uint16(sum(IM,3));
IM = IM(100:310,960:1170);
IM = IM-min(IM(:));
IM=IM*(2^15/max(IM(:)));
IM = repmat(IM,[1,1,N])+randi((2^15)-1,[size(IM,1),size(IM,2),N],'uint16');
S = (numel(IM)/N*2)/2^20;


%imread writespeed
methods = {'imwrite','tifflib','fTIF'};
for M = 1:length(methods)
    method = methods{M};
    %file
    filename = [method,'.tif'];
    if exist(filename,'file'), delete(filename);end
    switch method
        case 'imwrite'
            %timing vector
            t = zeros(1,100+1);
            tic;
            imwrite(IM(:,:,1),filename);
            t(2)=toc;
            for ct = 2:100
                imwrite(IM(:,:,ct),filename,'WriteMode','append');
                t(ct+1)=toc;
            end
        case 'tifflib'
            %timing vector
            t = zeros(1,200+1);
            tic;
            tf = Tiff(filename,'w');
            for ct = 1:200
                if ct>1,tf.writeDirectory;end
                tf.setTag('Photometric',Tiff.Photometric.MinIsBlack);
                tf.setTag('Compression',Tiff.Compression.None);
                tf.setTag('BitsPerSample',16);
                tf.setTag('SamplesPerPixel',1);
                tf.setTag('SampleFormat',Tiff.SampleFormat.UInt);
                tf.setTag('ExtraSamples',Tiff.ExtraSamples.Unspecified);
                tf.setTag('ImageLength',size(IM,1));
                tf.setTag('ImageWidth',size(IM,2));
                tf.setTag('PlanarConfiguration',Tiff.PlanarConfiguration.Chunky);
                tf.setTag('ImageDescription',sprintf('ImageJ=1.51j\nchannels=%.0f',size(IM,3)));
                tf.write(IM(:,:,ct));
                t(ct)=toc;
            end
            tf.close();
        case 'fTIF'
            %timing vector
            t = zeros(1,size(IM,3)+1);
            tic
            fTIF = Fast_Tiff(filename);
            for ct = 1:size(IM,3)
                fTIF = fTIF.WriteIMG(IM(:,:,ct)');
                t(ct)=toc;
            end
            tic
            fTIF.close;
            toc
        otherwise
            error('unknown method')
    end
    S = (size(IM,1)*size(IM,2)*2)/2^20; %MB/frame
    y = S./diff(t);
    subplot(1,length(methods),M)
    plot([1:length(y)],y);
    title(sprintf('Writing with %s; mean = %.2f MB/s',method,mean(y)))
    ylabel('Writing speed (MB/s)')
    xlabel('Frame');
    drawnow;
end

Code for the fTIF method:

https://github.com/rharkes/Fast_Tiff_Write

Answer 2

I ran into the same problem, but it's even worse: even if you write another tiff, the time to write a frame is increasing.

So a solution (which works better on my case with multiple files) is to call the restart matlab session per shell command.

So in your "writeTIFF.m", you should have a variable "startFrame" and put an "exit" at the end of your script. You can adress it with this kind of Batch (or equivalent under linux/unix) with this line :

@echo off
setlocal EnableDelayedExpansion 
:: count to 5 storing the results in a variable
set _tst=0
FOR /l %%G in (100,100,300) Do (
    echo matlab -r "startFrame=%%G;writeTIFF"  -nosplash -nodesktop -wait
    matlab -r "startFrame=%%G;writeTIFF"  -nosplash -nodesktop -wait
)
echo Done

I did not heavily tested, but it should do what's expected : restart matlab 3 times, and initialize the variable "startFrame" with the value 100, 200 and 300, successively.