Post-mortem unit testing

Question

I do version control with Git, and unit testing with QUnit. Sometimes I find a bug in my software that was not present in a past version. It's easy for me to write a unit test specifically for that bug.

Given that unit test, can I easily go trough all my past commits and test the build with that unit test, so that I can pinpoint which commit caused the breakage?

Answer 1

Git has a command to do exactly what you want, git bisect

Find by binary search the change that introduced a bug

In your case you want to use it as git bisect run /path/to/script, which will automatically test commits and performs a check with each commit to find the bad commit.

Note that the script (my_script in the above example) should exit with code 0 if the current source code is good, and exit with a code between 1 and 127 (inclusive), except 125, if the current source code is bad.

Any other exit code will abort the bisect process. It should be noted that a program that terminates via "exit(-1)" leaves $? = 255, (see the exit(3) manual page), as the value is chopped with "& 0377".

The special exit code 125 should be used when the current source code cannot be tested. If the script exits with this code, the current revision will be skipped (see git bisect skip above). 125 was chosen as the highest sensible value to use for this purpose, because 126 and 127 are used by POSIX shells to signal specific error status (127 is for command not found, 126 is for command found but not executable---these details do not matter, as they are normal errors in the script, as far as "bisect run" is concerned).

So, your script would compile your sources, run your unit test suite and then give the corresponding exit code. The example section from the bisect manpage covers this nicely (including broken builds, merging hotfix commits, etc.)

Answer 2

Use git bisect for this, please see this page.

Since you're testing JavaScript, you will probably have to run the tests by hand and run git bisect good and git bisect bad as appropriate. However, if you can run your unit test from the command line, then you can use git bisect run to have Git execute the test repeatedly and track down the faulty commit automatically:

$ git bisect run my-script.sh

That's pure magic the first time you see it! :-)

Answer 3

You described the job of git bisect. The Git Book has a good tutorial.

There is also some confusion in the terms of your question: When a test is used to guard against re-introduction of previously fixed errors or for bisection of buggy commits, it is called a regression test, not a unit test. The latter test is purely testing if a given small unit of code works, and is under heavy time constraints (TDD people run unit tests several dozen times in a day). For a large project, you usually have much longer regression tests than unit tests, so you might want to get the categories clean.

Answer 4

Yes, this is called git bisect, and was first introduced in git.

Principle:

• grab a commit in the past where you know it worked, let's call it c1;
• grab a commit after c1 you know fails, call it c2;
• git bisect start c2 c1.

You can even restrict the bisect to subpaths if you know where it fails, and if you have a script which can run the test non interactively, use git bisect run.