I have an app with some specs written into minitest. As usual, I start them using rake.
Because some times I got a random results, my specs can pass one time, and fail an other time.
In this case, I can keep the sequence number and replay it later, after fixing.
Because I have this kind of tests (with a random result), I generally run rake many time, just to be sure that the app is okay.
I would like to know if there is a nice way to perform multiple rake tests (100 times for example), and stop them if there any failure or any error?
I created a dummy test with random result to simulate your situation:
#store it as file 'testcase.rb'
gem 'test-unit'
require 'test/unit'
class X < Test::Unit::TestCase
def test_1
num = rand(10)
assert_true( num < 5, "Value is #{num}")
end
end
The following task calls the test 10 times and stops after the first failure:
TEST_REPETION = 10
task :test do
TEST_REPETION.times{
stdout = `ruby testcase.rb`
if stdout =~ /\d+\) Failure/
puts "Failure occured"
puts stdout
exit
else
puts 'Tests ok'
end
}
end
For real usage I would adapt some parts:
- Instead
puts 'Tests ok' define a counter to see how often the test was succussfull
- Instead
puts stdoutyou may store the result in a result file?
I think you should think again about your test, not about the test call. A test with a random result looks wrong for me.
What's the random factor in your test? Can you write a mock-element for the random factor and repeat the test with different values for the mock-element. So you get a "complete" test.
