How does CMake detect changed files

Question

I have a "C"/C++ CMake project which works fine. However, I'm sometimes (re)building on a remote cluster where the time is slightly different. This machine runs Linux and I'm building using make. I'm wondering if there is some make/CMake way to change how the changes to the files are detected, e.g. to MD5 or diff rather than using timestamps. Otherwise I guess I'd either have to endure the constant make clean / make -j cycle or have to change my local time every time I'm working with that particular server.

I was poking CMake documentation to see if there is a flag which would change these settings but found none. How would this work on platforms which have no RTC (e.g. Raspberry)?

Answer 1

Right, so knowing that CMake / make does not do what I want and I don't want the hassle of synchronizing the time of my machine to the target, I came up with the following:

#!/bin/bash

touch src_hash.md5

echo -n make "$@" > mymake.sh

find `pwd`/../src `pwd`/../include -print0 | 
    while IFS= read -r -d $'\0' f; do 
        if [[ ! -d "$f" ]]; then
            MD5=`md5sum "$f" | awk -v fn="$f" '{ print "\"" fn "\" " $1; }'`
            echo $MD5 >> src_hash.md5.new

            OLDMD5=`grep -e "^\"$f\"" src_hash.md5`
            if [[ "$OLDMD5" == "" ]]; then
                echo "$MD5 -- [a new file]"
                continue # a new file, make can handle that well on its own
            fi

            HASH=`echo $MD5 | awk '{ print $2; }'`
            OLDHASH=`echo $OLDMD5 | awk '{ print $2; }'`
            if [[ "$HASH" != "$OLDHASH" ]]; then
                echo "$MD5 -- changed from $OLDHASH"
                echo -n " \"--what-if=${f}\"" >> mymake.sh
                # this is running elsewhere, can't pass stuff via variables
            fi
        fi
    done

touch src_hash.md5.new
mv src_hash.md5.new src_hash.md5

echo using: `cat mymake.sh`
echo >> mymake.sh # add a newline
chmod +x mymake.sh
./mymake.sh
rm -f mymake.sh

This keeps a list of source file hashes in src_hash.md5 and at each time it runs it compares the current files to those hashes (and updates the list accordingly).

At the end, it calls make, passing any arguments you give to the script (such as -j). It makes use of the --what-if= switch which tells make to act like the given file changed - that way the dependences of build targets on sources / headers are handled elegantly.

You might want to also pass the path to source / include files as arguments so that those wouldn't be hardcoded inside.

Or one more iteration on the said script, using touch to change and restore the file timestamps for situations when make is extra stubborn about not rebuilding anything:

#!/bin/bash

if [[ ! -d ../src ]]; then
    >&2 echo "error: ../src is not a directory or does not exist"
    exit -1
fi
if [[ ! -d ../include ]]; then
    >&2 echo "error: ../include is not a directory or does not exist"
    exit -1
fi

echo "Scanning for changed files in ../src and ../include"

touch src_hash.md5 # in case this runs for the first time

rm -f mymaketouch.sh
rm -f mymakerestore.sh
touch mymaketouch.sh
touch mymakerestore.sh

echo -n make "$@" > mymake.sh

CWD="`pwd`"
find ../src ../include -print0 | 
    while IFS= read -r -d $'\0' f; do 
        if [[ ! -d "$f" ]]; then
            fl=`readlink -f "$CWD/$f"`

            MD5=`md5sum "$fl" | awk -v fn="$fl" '{ print "\"" fn "\" " $1; }'`
            HASH=`echo $MD5 | awk '{ print $2; }'`
            echo $MD5 >> src_hash.md5.new

            OLDMD5=`grep -e "^\"$fl\"" src_hash.md5`
            OLDHASH=`echo $OLDMD5 | awk '{ print $2; }'`
            if [[ "$OLDMD5" == "" ]]; then
                echo "$f $HASH -- [a new file]"
                continue # a new file, make can handle that well on its own
            fi

            if [[ "$HASH" != "$OLDHASH" ]]; then
                echo "$f $HASH -- changed from $OLDHASH"

                echo "touch -m \"$fl\"" >> mymaketouch.sh # will touch it and change modification time
                stat "$fl" -c "touch -m -d \"%y\" \"%n\"" >> mymakerestore.sh # will restore it later on so that we do not run into problems when copying newer from a different system

                echo -n " \"--what-if=$fl\"" >> mymake.sh
                # this is running elsewhere, can't pass stuff via variables
            fi
        fi
    done

echo using: `cat mymake.sh`
echo >> mymake.sh # add a newline
echo 'exit $?' >> mymake.sh

chmod +x mymaketouch.sh
chmod +x mymakerestore.sh
chmod +x mymake.sh

control_c() # run if user hits control-c
{
    echo -en "\nrestoring modification times\n"
    ./mymakerestore.sh
    rm -f mymaketouch.sh
    rm -f mymakerestore.sh
    rm -f mymake.sh
    rm -f src_hash.md5.new 
    exit -1
}

trap control_c SIGINT

./mymaketouch.sh
./mymake.sh
RETVAL=$?
./mymakerestore.sh
rm -f mymaketouch.sh
rm -f mymakerestore.sh
rm -f mymake.sh

touch src_hash.md5.new # in case there was nothing new
mv src_hash.md5.new src_hash.md5
# do it now in case someone hits ctrl+c mid-build and not all files are built

exit $RETVAL

Or even run hashing in parallel in case you are building a large project:

#!/bin/bash

if [[ ! -d ../src ]]; then
    >&2 echo "error: ../src is not a directory or does not exist"
    exit -1
fi
if [[ ! -d ../include ]]; then
    >&2 echo "error: ../include is not a directory or does not exist"
    exit -1
fi

echo "Scanning for changed files in ../src and ../include"

touch src_hash.md5 # in case this runs for the first time

rm -f mymaketouch.sh
rm -f mymakerestore.sh
touch mymaketouch.sh
touch mymakerestore.sh

echo -n make "$@" > mymake.sh

CWD="`pwd`"
rm -f src_hash.md5.new # will use ">>", make sure to remove the file
find ../src ../include -print0 |
    while IFS= read -r -d $'\0' f; do
        if [[ ! -d "$f" ]]; then
            fl="$CWD/$f"
            (echo `md5sum "$f" | awk -v fn="$fl" '{ print "\"" fn "\" " $1; }'` ) & # parallel, echo is atomic (http://stackoverflow.com/questions/9926616/is-echo-atomic-when-writing-single-lines)
            # run in parallel (remove the ampersand if you run into trouble)
        fi
    done >> src_hash.md5.new # >> is atomic but > wouldn't be
# this is fast

cat src_hash.md5 > src_hash.md5.diff
echo separator >> src_hash.md5.diff
cat src_hash.md5.new >> src_hash.md5.diff
# make a compound file for awk (could also read the other file in awk but this seems simpler right now)

cat src_hash.md5.diff | awk 'BEGIN { FS="\""; had_sep = 0; }
    {
        if(!had_sep && $1 == "separator")
            had_sep = 1;
        else {
            sub(/[[:space:]]/, "", $3);
            if(!had_sep)
                old_hashes[$2] = $3;
            else {
                f = $2;
                if((idx = index(f, "../")) != 0)
                    f = substr(f, idx, length(f) - idx + 1);
                if($2 in old_hashes) {
                    if(old_hashes[$2] != $3)
                        print "\"" f "\" " $3 " -- changed from " old_hashes[$2];
                } else
                    print "\"" f "\" -- a new file " $3;
            }
        }
    }'
# print verbose for the user only

cat src_hash.md5.diff | awk 'BEGIN { FS="\""; had_sep = 0; }
    {
        if(!had_sep && $1 == "separator")
            had_sep = 1;
        else {
            sub(/[[:space:]]/, "", $3);
            if(!had_sep)
                old_hashes[$2] = $3;
            else {
                if($2 in old_hashes) {
                    if(old_hashes[$2] != $3)
                        printf($2 "\0"); /* use \0 as a line separator for the below loop */
                }
            }
        }
    }' |
    while IFS= read -r -d $'\0' fl; do
        echo "touch -m \"$fl\"" >> mymaketouch.sh # will touch it and change modification time
        stat "$fl" -c "touch -m -d \"%y\" \"%n\"" >> mymakerestore.sh # will restore it later on so that we do not run into problems when copying newer from a different system

        echo -n " \"--what-if=$fl\"" >> mymake.sh
        # this is running elsewhere, can't pass stuff via variables
    done
# run again, handle files that require change

rm -f src_hash.md5.diff

echo using: `cat mymake.sh`
echo >> mymake.sh # add a newline
echo 'exit $?' >> mymake.sh

chmod +x mymaketouch.sh
chmod +x mymakerestore.sh
chmod +x mymake.sh

control_c() # run if user hits control-c
{
    echo -en "\nrestoring modification times\n"
    ./mymakerestore.sh
    rm -f mymaketouch.sh
    rm -f mymakerestore.sh
    rm -f mymake.sh
    rm -f src_hash.md5.new 
    exit -1
}

trap control_c SIGINT

./mymaketouch.sh
./mymake.sh
RETVAL=$?
./mymakerestore.sh
rm -f mymaketouch.sh
rm -f mymakerestore.sh
rm -f mymake.sh

touch src_hash.md5.new # in case there was nothing new
mv src_hash.md5.new src_hash.md5
# do it now in case someone hits ctrl+c mid-build and not all files are built

exit $RETVAL