What makes a good failure message for testunit or

Question

In Ruby's test/unit, and other such nunit style frameworks, what makes a good failure message?

Should the failure message merely describe how the expected value does not match the expected value?

assert_match("hey", "hey this is a test", "The word does not exist in the string")

Should it describe what you expected to happen?

assert_match("hey", "hey this is a test", "I expected hey to be in the string")

Should it describe why you wanted the behavior to happen?

assert_match("hey", "hey this is a test", "Program should provide a greeting")

Should it describe why you thought the test may fail?

assert_match("konnichiwa", "konnichiwa this is a test",
  "Program failed to use supplied i18n configuration")

Should information about tests also exist in the name of the test method, and in the name of the test case?

This is based on Ruby "test/unit" , how do I display the messages in asserts

Answer 1

the failure message is supposed to add context to the failure message. So anything that saves you having to drill into the test code to know what failed.

So if the [method name, expected, actual] set is adequate for the above purpose, you can skip the failure message. If you need more information, then you add the optional failure message.

e.g. Expected true but was false, doesn't tell me anything.

You can use a failure message so that Return value should contain only multiples of 10. Expected true but was false

You can first try to use more descriptive matchers. So that failures read Expected all items to be divisible by 10 but was [10,20,35,40] does.

Personally I prefer matchers... use failure messages as the last resort. (because like comments, it decays. You need discipline to ensure that the failure message is updated if you change the check.)