How to get the 'NextUSN' journal entry for

Question

after creating a VSS snapshot I'd like to be able to query the USN journal. Is this possible or is the USN journal not accessible from a VSS snapshot?

my goal is to be able to use the USN journal in an incremental backup between two VSS snapshots. The process for the backup would be to

1. take a VSS Snapshot and backup the volume, taking note of the USN entries for each file
2. ...use the filesystem, add/delete/modify files
3. take a second VSS snapshot, then use the USN journal to detect anything that changed during step #2

what I'm failing with right now is the part where I'm trying to get a hold of the highest USN entry on the VSS snapshot

1. create VSS Snapshot
2. open the snapshot with CreateFile(\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy25)
3. DeviceIoControl(FSCTL_QUERY_USN_JOURNAL) - this fails with GLE:1179 "the volume change journal is not active"

I can simulate this from the commandline as follows

C:\>vssadmin list shadows
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2005 Microsoft Corp.

Contents of shadow copy set ID: {54fc99fb-65f2-4558-8e12-9308979327f0}
   Contained 1 shadow copies at creation time: 5/10/2012 6:44:19 PM
      Shadow Copy ID: {a2d2c155-9916-47d3-96fd-94fae1c2f802}
         Original Volume: (T:)\\?\Volume{a420b1fa-9744-11e1-9082-889ffaf52b70}\
         Shadow Copy Volume: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy25
         Originating Machine: computer
         Service Machine: computer
         Provider: 'Microsoft Software Shadow Copy provider 1.0'
         Type: Backup
         Attributes: Differential


C:\>fsutil usn queryjournal \\?\Volume{a420b1fa-9744-11e1-9082-889ffaf52b70}
Usn Journal ID   : 0x01cd2ebe9c795b57
First Usn        : 0x0000000000000000
Next Usn         : 0x000000000001b5f8
Lowest Valid Usn : 0x0000000000000000
Max Usn          : 0x7fffffffffff0000
Maximum Size     : 0x0000000000100000
Allocation Delta : 0x0000000000040000

C:\>fsutil usn queryjournal \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy25
Error:  The volume change journal is not active.

Any ideas what I'm doing incorrectly, if this is possible?

Answer 1

You may want to give Ruben's answer a second thought:

The USN Journal in a snapped volume is definitely readable by reading a special file inside the snapped VSS Volume. If the Windows API won't allow you to read the USN journal of a snapped volume then this may be a viable option although I'm sure it feels like a hack.

The thing is although NTFS does not have an open specification its been figured out by more than one project amongst which is the Linux implementations of NTFS drivers. The document that Ruben posted for you was originally written to help development of this driver.

Like I mentioned, the USN Journal content sits in a special file on your NTFS volume (like many things in NTFS e.g. The NTFS master file table. Actually it is said that everything in NTFS is a file). Special files in NTFS starts with a dollar sign $ and the one jou are looking for is named $UsnJrnl which in turn resides in a special directory named $Extend. So on your C: volume that file is

C:\$Extend\$UsnJrnl 

or for you snapshot it would be

\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy25\$Extend\$UsnJrnl

The info you are looking for sits in an Alternate Data Stream named the $J stream and it has entries in this format (See Ruben's referred to doc):

Offset(in hex) Size Description
0x00 4 Size of entry
0x04 2 Major Version
0x06 2 Minor Version
0x08 8 MFT Reference
0x10 8 Parent MFT Reference
0x18 8 Offset of this entry in $J
0x20 8 Timestamp
0x28 4 Reason (see table below)
0x2B 4 SourceInfo (see table below)
0x30 4 SecurityID
0x34 4 FileAttributes
0x38 2 Size of filename (in bytes)
0x3A 2 Offset to filename
0x3C V Filename
V+0x3C P Padding (align to 8 bytes)

So you could be reading the $J stream of this special file to get the USN entry that you want. I'm wanting to tell you how to derive the USN number that you will need but I'm a bit rusty. If I figure it out again I will update this answer. But have a look at reading special files this way, its quite fun ;-). I've used this method to read the master file table (Special file $MFT) inside an unmounted VHD file in order to enumerate all the files on the volume inside the VHD.

Answer 2

Perhaps this may be of use: Journal-entries do not cross cluster-boundaries. Every cluster ( normally 8 sectors per cluster) starts with a new entry. If towards the end of this cluster the next entry does not fit into the remaining cluster-space, this space is filled with zero, and the next entry ist stored at the start of the next cluster ( It's a pity this is not expressed in the "size of entry"). So you need not parse this space - just jump to the next cluster ( ! do not forget to use the RUN's of the journal to obtain the next valid disk-cluster). Robert

Btw. You can use the USN (Offset of this entry in $J), the cluster-number and the position of this entry within the Cluster to check the validity of the entry.

Answer 3

I think that it is impossible to use the WinAPI interface to query the USN journal while the volume is not mounted.

You can try to open the file "$UsnJrnl" and manually parse the information you need.

See:

NTFS Documentation by Richard Russon and Yuval Fledel

Answer 4

This question was very important for the project I'm working on, so I finally got it (almost) 100% working.

Note : all of the below code snippets are in C#

Thanks to the previous answers from Hannes de Jager who pointed me to the right direction and documentations, I can now read an USN journal from a VSS snapshot or any other special device that the regular API cannot work with ; in my case, I mean VMware snapshots mounted using the VDDK (VMware SDK for VM disks).

I also reused or imported code coming from great projects :

• USN journal explorer in C#, from StCroixSkipper (http://www.dreamincode.net/forums/blog/1017-stcroixskippers-blog/). Only reads USN using the official API (so no VSS here) but provides useful pinvokes and Win32 API structures as well as general information about how USN works

• AlphaFS (https://github.com/alphaleonis/AlphaFS/) , which mimics large parts of the System.IO namespace, but allows to access special windows paths (VSS snapshots, raw devices) and provides useful extensions as well.

In case anyone else is interested, I share the code I use righ now, still in a rather crude state, but working.

How does it work?

First, you must acess the required Usn journal componenents. They are at the root of the device as ADS (alternate data streams) in an hidden entry. They cannot be accessed using the standard System.IO namespace, that's why I previously told I used the AlphaFS project, but pinvoking CreateFile() and ReadFile() should be enough.

1/2

The entry \$Extend\$UsnJrnl:$Max has global information about the current state of the journal. The most important parts are the usn journal ID (that you can use to check that the journal has not been reset if you want to compare multiple VSS snapshots) and the lowest valid USN journal sequence number.

USN journal structure:

  // can be directly extracted from $MAX entry using Bitconverter.ToUint64
 public struct USN_JOURNAL_DATA{
        public UInt64 MaximumSize; //offset 0
        public UInt64 AllocationDelta; // offset 8
        public UInt64 UsnJournalID; // offset 16
        public Int64 LowestValidUsn; // offset 24
    }

2/2

The entry \$Extend\$UsnJrnl:$J contains the journal records. It is a sparse file so its disk usage is much lower than it size.

To answer the initial question, how can one know the Max used USN sequence from a previous VSS snapshot and compare it with that of another snapshot? Well, the NextUsn value is simply equals to the size of the $Usnjrnl:$J entry.

On your "new" vss snapshot USN journal, you can seek to the "reference" VSS snapshot max USN before starting parsing records, if you want to parse the records changed between the two snapshots.

Generally speaking, each USN journal entry as a unique ID (USN number) which is the offset inside $J at whch the journal entry itself is located. Each entry has a variable size, so to sequentially read then we have to calculate:

next entry offset inside $J = 
    offset of current entry (or its USN sequennce number + length of current entry

Fortunately the record length is a also a field of the USN entry record. Enough said, here is the USN record class:

public class UsnEntry : IComparable<UsnEntry>{
        private const int FR_OFFSET = 8;
        private const int PFR_OFFSET = 16;
        private const int USN_OFFSET = 24;
        private const int REASON_OFFSET = 40;
        private const int FA_OFFSET = 52;
        private const int FNL_OFFSET = 56;
        private const int FN_OFFSET = 58;


        public UInt32 RecordLength {get; private set;}
        public Int64 USN {get; private set;}
        public UInt64 FileReferenceNumber {get;private set;}
        public UInt64 ParentFileReferenceNumber {get; private set;}
        public UInt32 Reason{get; set;}
        public string Name {get; private set;}
        public string OldName{get; private set;}

        private UInt32 _fileAttributes;
        public bool IsFolder{
            get{
                bool bRtn = false;
                if (0 != (_fileAttributes & Win32Api.FILE_ATTRIBUTE_DIRECTORY))
                    bRtn = true;
                return bRtn;
            }
        }

        public bool IsFile{
            get{
                bool bRtn = false;
                if (0 == (_fileAttributes & Win32Api.FILE_ATTRIBUTE_DIRECTORY))
                    bRtn = true;
                return bRtn;
            }
        }

         /// <summary>
        /// USN Record Constructor
        /// </summary>
        /// <param name="p">Buffer pointer to first byte of the USN Record</param>
        public UsnEntry(IntPtr ptrToUsnRecord){
            RecordLength = (UInt32)Marshal.ReadInt32(ptrToUsnRecord); //record size
            FileReferenceNumber = (UInt64)Marshal.ReadInt64(ptrToUsnRecord, FR_OFFSET);
            ParentFileReferenceNumber = (UInt64)Marshal.ReadInt64(ptrToUsnRecord, PFR_OFFSET);
            USN = (Int64)Marshal.ReadInt64(ptrToUsnRecord, USN_OFFSET);
            Reason = (UInt32)Marshal.ReadInt32(ptrToUsnRecord, REASON_OFFSET);
            _fileAttributes = (UInt32)Marshal.ReadInt32(ptrToUsnRecord, FA_OFFSET);
            short fileNameLength = Marshal.ReadInt16(ptrToUsnRecord, FNL_OFFSET);
            short fileNameOffset = Marshal.ReadInt16(ptrToUsnRecord, FN_OFFSET);
            Name = Marshal.PtrToStringUni(new IntPtr(ptrToUsnRecord.ToInt32() + fileNameOffset), fileNameLength / sizeof(char));
        }


        public int CompareTo(UsnEntry other){
            return string.Compare(this.Name, other.Name, true);
        }

        public override string ToString(){
            return string.Format ("[UsnEntry: RecordLength={0}, USN={1}, FileReferenceNumber={2}, ParentFileReferenceNumber={3}, Reason={4}, Name={5}, OldName={6}, IsFolder={7}, IsFile={8}", RecordLength, USN, (int)FileReferenceNumber, (int)ParentFileReferenceNumber, Reason, Name, OldName, IsFolder, IsFile);
        }
    }

I tried to isolate the smallest part of code that can parse an USN journal and extract its entries, starting from the lowest valid one. Remember, a record has a variable length; also note that some records point to a next record that is empty (the first 4 bytes, which are normally the record length, are zeroed). In this case I seek 4 bytes and retry parsing, until I get the next record. This behavior also have been reported by people having written similar parsing tools in Python so I guess I'n not too wrong here.

string vol = @"\\?\path_to_your_VSS_snapshot";
string maxHandle = vol + @"\$Extend\$UsnJrnl:$Max";
string rawJournal= vol + @"\$Extend\$UsnJrnl:$J";

// cannot use regular System.IO here, but pinvoking ReadFile() should be enough
FileStream maxStream = Alphaleonis.Win32.Filesystem.File.OpenRead(maxHandle);
byte[] maxData = new byte[32];
maxStream.Read(maxData, 0, 32);

//first valid entry
long lowestUsn = BitConverter.ToInt64(maxData, 24);

// max (last) entry, is the size of the $J ADS
IntPtr journalDataHandle = Win32Api.CreateFile(rawJournal, 
            0, 
            Win32Api.FILE_SHARE_READ| Win32Api.FILE_SHARE_WRITE,
            IntPtr.Zero, Win32Api.OPEN_EXISTING,  
            0, IntPtr.Zero);
Win32Api.BY_HANDLE_FILE_INFORMATION fileInfo = new Win32Api.BY_HANDLE_FILE_INFORMATION();
Win32Api.GetFileInformationByHandle(journalDataHandle, out fileInfo);
Win32Api.CloseHandle(journalDataHandle);
long lastUsn = fileInfo.FileSizeLow;


int read = 0;
byte[] usnrecord;
byte[] usnraw = new byte[4]; // first byte array is to store the record length

// same here : pinvoke ReadFile() to avoid AlphaFS dependancy
FileStream rawJStream = Alphaleonis.Win32.Filesystem.File.OpenRead(rawJournal);
int recordSize = 0;
long pos = lowestUsn;

while(pos < newUsnState.NextUsn){
seeked = rawJStream.Seek(pos, SeekOrigin.Begin);
read = rawJStream.Read(usnraw, 0, usnraw.Length);
recordSize = BitConverter.ToInt32(usnraw, 0);
    if(recordSize == 0){
    pos = pos+4;
    continue;
}
    usnrecord = new byte[recordSize];
rawJStream.Read(usnrecord, 4, recordSize-4);
Array.Copy(usnraw, 0, usnrecord, 0, 4);
fixed (byte* p = usnrecord){
    IntPtr ptr = (IntPtr)p;
        // here we use the previously defined UsnEntry class
    Win32Api.UsnEntry entry = new Win32Api.UsnEntry(ptr);
    Console.WriteLine ("entry: "+entry.ToString());
    ptr = IntPtr.Zero;

}
    pos += recordSize;
}

Here are the pinvokes I use:

public class Win32Api{

   [StructLayout(LayoutKind.Sequential, Pack = 1)]
    public struct BY_HANDLE_FILE_INFORMATION{
        public uint FileAttributes;
        public FILETIME CreationTime;
        public FILETIME LastAccessTime;
        public FILETIME LastWriteTime;
        public uint VolumeSerialNumber;
        public uint FileSizeHigh;
        public uint FileSizeLow;
        public uint NumberOfLinks;
        /*public uint FileIndexHigh;
        public uint FileIndexLow;*/
        public FileID FileIndex;
    }

    [DllImport("kernel32.dll", SetLastError = true)]
    [return: MarshalAs(UnmanagedType.Bool)]
    public static extern bool 
        GetFileInformationByHandle(
        IntPtr hFile,
        out BY_HANDLE_FILE_INFORMATION lpFileInformation);

   [DllImport("kernel32.dll", SetLastError = true)]
    public static extern IntPtr 
        CreateFile(string lpFileName, 
        uint dwDesiredAccess,
        uint dwShareMode, 
        IntPtr lpSecurityAttributes, 
        uint dwCreationDisposition,
        uint dwFlagsAndAttributes, 
        IntPtr hTemplateFile);


}

This is definitely not the best piece of code on earth, but I think it will provide a good starting point for anyone having to do the same thing.