Recording/playback Kinect with Matlab - upside dow

I am trying to write a program to record and playback both colour and depth streams from a xbox kinect to ease testing image processing programs. Currently I have the bulk done and the colour stream works fine. I am however having trouble with the depth stream.

Currently, the depth stream is playing back upside down and only in black and white. I have 3 thoughts as to why this may be the case: 1) The conversion to 8-bit from 11-bit 2) the Motion JPEG 2000 format (never used this before) 3) the colormap is wrong

Below is the code I am using. I feel I cannot be the only one trying to do this so any pointer and help would be much appreciated as I couldn't find anything this specific on the web.

    %------------------------------------------------
    %------------------------------------------------
    %Code to record kinect colour and sensor data
    %using code supplied on http://www.mathworks.co.uk/help/imaq/examples/using-the-                kinect-r-for-windows-r-from-image-acquisition-toolbox-tm.html
    %and http://www.mathworks.co.uk/help/imaq/examples/logging-data-to-disk.html
    %------------------------------------------------
    %------------------------------------------------

    dbstop if error

    imaqreset %deletes any image acquisition objects that exsist in memory and uploads         all adaptors loaded by the toolbox. As a result, image acquisition hardware is reset

    %------------------------------------------------
    %setting up video streams
    %------------------------------------------------
    disp('Setting up video streams');

    %Call up dicertory containing utility functions
    utilpath = fullfile(matlabroot, 'toolbox', 'imaq', 'imaqdemos', 'html', 'KinectForWindows');
    addpath(utilpath);

    %Create the videoinput objects for the colour and depth streams
    colourVid = videoinput('kinect', 1, 'RGB_640x480');
    %preview(colourVid);
    depthVid = videoinput('kinect', 2, 'Depth_640x480');

    %set backlight compensation with centre priority
    %set(colourVid, 'BacklightCompensation', 'CentrePriority');

    %Set camera angle to 0
    %set(colourVid, 'CameraElevationAngle', 0);

    disp('Video stream set-up complete');

    %------------------------------------------------
    %setting up record
    %------------------------------------------------

    % set the data streams to logging mode and to disk
    set(colourVid, 'LoggingMode', 'Disk&Memory');
    set(depthVid, 'LoggingMode', 'Disk&Memory');

    %Set a video timeout property limit to 50 seconds from
    %www.mathworks.com/matlabcentral/answers/103543-why-do-i-receive-the-error-getdata-timed-out-before-frames-were-available-when-using-getdata-in-im
    set(colourVid, 'Timeout',50);
    set(depthVid, 'Timeout',50);

    %Creat a VideoReader object
    colourLogfile = VideoWriter('colourTrial5.mj2', 'Motion JPEG 2000');
    depthLogfile = VideoWriter('depthTrial5.mj2', 'Motion JPEG 2000');

    %configure the video input object to use the VideoWriter object
    colourVid.DiskLogger = colourLogfile;
    depthVid.DiskLogger = depthLogfile;

    %set the triggering mode to 'manual'
    triggerconfig([colourVid depthVid], 'manual');

    %set the FramePerTrigger property of the VIDEOINPUT objects to 100 to
    %acquire 100 frames per trigger.
    set([colourVid depthVid], 'FramesPerTrigger', 200);

    disp('Video record set-up complete');

    %------------------------------------------------
    %Initiating the aquisition
    %------------------------------------------------
    disp('Starting Steam');

    %Start the colour and depth device. This begins acquisition, but does not
    %start logging of acquired data
    start([colourVid depthVid]);

    pause(20); %allow time for both streams to start

    %Trigger the devices to start logging of data.
    trigger([colourVid depthVid]);

    %Retrieve the acquired data
    [colourFrameData, colourTimeData, colourMetaData] = getdata(colourVid);
    [depthFrameData, depthTimeData, depthMetaData] = getdata(depthVid);

    stop([colourVid depthVid])

    disp('Recording Complete')

    %------------------------------------------------
    %Play back recordings
    %------------------------------------------------
    disp('Construct playback objects')

    colourPlayback = VideoReader('colourTrial5.mj2');
    depthPlayback = VideoReader('depthTrial5.mj2');

    %Set colour(c) playback parameters
    cFrames = colourPlayback.NumberOfFrames;
    cHeight = colourPlayback.Height;
    cWidth = colourPlayback.Width;

    %Preallocate movie structure
    colourMov(1:cFrames)=struct('cdata', zeros(cHeight,cWidth,3,'uint8'),'colormap',[]);

    disp('Reading colour frames one by one')

    %read one frame at a time
    for k = 1:cFrames
        colourMov(k).cdata=read(colourPlayback,k);
    end

    disp('Sizing figure for colour playback')

    %Size a figure based on the video's width and height
    hf1=figure;
    set(hf1,'position',[150 150 cWidth cHeight])

    disp('Playing Colour recording')

    %play back the movie once at the video's frame rate
    movie(hf1,colourMov,1,colourPlayback.FrameRate);

    %Set depth(d) playback parameters
    dFrames = depthPlayback.NumberOfFrames;
    dHeight = depthPlayback.Height;
    dWidth = depthPlayback.Width;

    %Preallocate movie structure
    depthMov(1:dFrames)=struct('cdata', zeros(dHeight,dWidth,3,'uint8'),'colormap',gray(256));

    disp('Reading depth frames one by one')

    %read one frame at a time
    for k = 1:dFrames
        depthMov(k).cdata=uint8(read(depthPlayback,k));
        %depthMov(k)=imrotate(depthMov(k),180); %tried this to no effect
    end

    disp('Sizing figure for depth playback')

    %Size a figure based on the video's width and height
    hf2=figure;
    set(hf2,'position',[150 150 dWidth dHeight])

    disp('Playing Depth recording')

    %play back the movie once at the video's frame rate
    movie(hf2,depthMov,1,depthPlayback.FrameRate);

    %clear videos from workspace
    delete([colourVid depthVid])
    clear [colourVid depthVid]

Your script is well-written and correct, except for how you display the depth data.

The Kinect records depth as 16-bit unsigned integers. The pixel value in the depth frame is the distance in millimeters of whatever object is in that pixel from the plane of the camera. So, if the depth value of a pixel of an object you care about reads out as, e.g., 1723, that means that part of the object is 1.723 meters away from the camera.

So, how does that relate to your broken display functionality? Here's your old display snippet:

%read one frame at a time
for k = 1:dFrames
    depthMov(k).cdata=uint8(read(depthPlayback,k));
end

The problem is right there at uint8. Many of your depth values I'm sure are beyond 255, meaning objects in your scene are farther than 0.2 meters from the camera. In fact, the Kinect cannot even sense data that close! The nearest you can detect depth, with DepthMode set to Near, is about 0.4 meters.

So, here's how to solve the display issue:

%read one frame at a time
maxDistFromCamera = 1600; % 1600 millimeters
for k = 1:dFrames
    % Depth frames are int16.
    depthFrame = read(depthPlayback,k); 
    % We'll rescale the image from [0,maxDistFromCamera] to [0,255]
    depthFrame = 255.0*single(depthFrame)/maxDistFromCamera;
    % And then recast it to uint8 for display.
    depthMov(k).cdata=uint8(depthFrame);
end

Below is reproduced the whole script, with my edits, for convenience.

%------------------------------------------------
%------------------------------------------------
%Code to record kinect colour and sensor data
%using code supplied on http://www.mathworks.co.uk/help/imaq/examples/using-the-                kinect-r-for-windows-r-from-image-acquisition-toolbox-tm.html
%and http://www.mathworks.co.uk/help/imaq/examples/logging-data-to-disk.html
%------------------------------------------------
%------------------------------------------------

imaqreset %deletes any image acquisition objects that exsist in memory and uploads         all adaptors loaded by the toolbox. As a result, image acquisition hardware is reset

%------------------------------------------------
%setting up video streams
%------------------------------------------------
disp('Setting up video streams');

%Call up dicertory containing utility functions
utilpath = fullfile(matlabroot, 'toolbox', 'imaq', 'imaqdemos', 'html', 'KinectForWindows');
addpath(utilpath);

%Create the videoinput objects for the colour and depth streams
colourVid = videoinput('kinect', 1, 'RGB_640x480');
%preview(colourVid);
depthVid = videoinput('kinect', 2, 'Depth_320x240');

% Set the depth mode to near.
srcDepth = getselectedsource(depthVid);
srcColor = getselectedsource(colourVid);
set(srcDepth, 'DepthMode' , 'Near');
set(srcDepth, 'CameraElevationAngle', 0);

%set backlight compensation with centre priority
set(srcColor, 'BacklightCompensation', 'CenterPriority');

disp('Video stream set-up complete');

%------------------------------------------------
%setting up record
%------------------------------------------------

% set the data streams to logging mode and to disk
set(colourVid, 'LoggingMode', 'Disk&Memory');
set(depthVid, 'LoggingMode', 'Disk&Memory');

%Set a video timeout property limit to 50 seconds from
%www.mathworks.com/matlabcentral/answers/103543-why-do-i-receive-the-error-getdata-timed-out-before-frames-were-available-when-using-getdata-in-im
set(colourVid, 'Timeout',50);
set(depthVid, 'Timeout',50);

%Creat a VideoReader object
colourLogfile = VideoWriter('colourTrial5.mj2', 'Motion JPEG 2000');
depthLogfile = VideoWriter('depthTrial5.mj2', 'Archival');

%configure the video input object to use the VideoWriter object
colourVid.DiskLogger = colourLogfile;
depthVid.DiskLogger = depthLogfile;

%set the triggering mode to 'manual'
triggerconfig([colourVid depthVid], 'manual');

%set the FramePerTrigger property of the VIDEOINPUT objects to 100 to
%acquire 100 frames per trigger.
set([colourVid depthVid], 'FramesPerTrigger', 30);

disp('Video record set-up complete');

%------------------------------------------------
%Initiating the aquisition
%------------------------------------------------
disp('Starting Steam');

%Start the colour and depth device. This begins acquisition, but does not
%start logging of acquired data
start([colourVid depthVid]);

pause(20); %allow time for both streams to start

disp('Starting Depth Stream');

%Trigger the devices to start logging of data.
trigger([colourVid depthVid]);

%Retrieve the acquired data
[colourFrameData, colourTimeData, colourMetaData] = getdata(colourVid);
[depthFrameData, depthTimeData, depthMetaData] = getdata(depthVid);

stop([colourVid depthVid])

disp('Recording Complete')

%------------------------------------------------
%Play back recordings
%------------------------------------------------
disp('Construct playback objects')

colourPlayback = VideoReader('colourTrial5.mj2');
depthPlayback = VideoReader('depthTrial5.mj2');

%Set colour(c) playback parameters
cFrames = colourPlayback.NumberOfFrames;
cHeight = colourPlayback.Height;
cWidth = colourPlayback.Width;

%Preallocate movie structure
colourMov(1:cFrames)=struct('cdata', zeros(cHeight,cWidth,3,'uint8'),'colormap',[]);

disp('Reading colour frames one by one')

%read one frame at a time
for k = 1:cFrames
    colourMov(k).cdata=read(colourPlayback,k);
end

disp('Sizing figure for colour playback')

%Size a figure based on the video's width and height
hf1=figure;
set(hf1,'position',[150 150 cWidth cHeight])

disp('Playing Colour recording')

%play back the movie once at the video's frame rate
movie(hf1,colourMov,1,colourPlayback.FrameRate);

%Set depth(d) playback parameters
dFrames = depthPlayback.NumberOfFrames;
dHeight = depthPlayback.Height;
dWidth = depthPlayback.Width;

%Preallocate movie structure
depthMov(1:dFrames)=struct('cdata', zeros(dHeight,dWidth,3,'uint8'),'colormap',gray(256));

disp('Reading depth frames one by one')

%read one frame at a time
maxDistFromCamera = 1600; % 1600 millimeters
for k = 1:dFrames
    % Depth frames are int16.
    depthFrame = read(depthPlayback,k); 
    % We'll rescale the image from [0,maxDistFromCamera] to [0,255]
    depthFrame = 255.0*single(depthFrame)/maxDistFromCamera;
    % And then recast it to uint8 for display.
    depthMov(k).cdata=uint8(depthFrame);
end

disp('Sizing figure for depth playback')

%Size a figure based on the video's width and height
hf2=figure;
set(hf2,'position',[150 150 dWidth dHeight])

disp('Playing Depth recording')

%play back the movie once at the video's frame rate
movie(hf2,depthMov,1,depthPlayback.FrameRate);

%clear videos from workspace
delete([colourVid depthVid])
clear [colourVid depthVid]

close all;

There are some minor textual differences, for instance, my Kinect SDK requires different spelling of some parameters, which I just fixed inline. I also disabled the aggressive debugging setting you had on at the top. Not important, just making a note of it. Lastly, I set the VideoWriter format to Archival for the depth stream, so that none of the depth values are corrupted on save (important for precise depth measurement applications, which is what I tend to be up to these days).