I'm working on a scientific computing code (written in C++), and in addition to performing unit tests for the smaller components, I'd like to do regression testing on some of the numerical output by comparing to a "known-good" answer from previous revisions. There are a few features I'd like:
- Allow comparing numbers to a specified tolerance (for both roundoff error and looser expectations)
- Ability to distinguish between ints, doubles, etc, and to ignore text if necessary
- Well-formatted output to tell what went wrong and where: in a multi-column table of data, only show the column entry that differs
- Return
EXIT_SUCCESS
orEXIT_FAILURE
depending on whether the files match
Are there any good scripts or applications out there that do this, or will I have to roll my own in Python to read and compare output files? Surely I'm not the first person with these kind of requirements.
[The following is not strictly relevant, but it may factor into the decision of what to do. I use CMake and its embedded CTest functionality to drive unit tests that use the Google Test framework. I imagine that it shouldn't be hard to add a few add_custom_command
statements in my CMakeLists.txt
to call whatever regression software I need.]
I ended up writing a Python script to do more or less what I wanted.
I know I'm quite late to the party, but a few months ago I wrote the nrtest utility in an attempt to make this workflow easier. It sounds like it might help you too.
Here's a quick overview. Each test is defined by its input files and its expected output files. Following execution, output files are stored in a portable benchmark directory. A second step then compares this benchmark to a reference benchmark. A recent update has enabled user extensions, so you can define comparison functions for your custom data.
I hope it helps.
You should go for PyUnit, which is now part of the standard lib under the name
unittest
. It supports everything you asked for. The tolerance check, e.g., is done withassertAlmostEqual()
.The ndiff utility may be close to what you're looking for: it's like diff, but it will compare text files of numbers to a desired tolerance.