Testing for Random Value - Thoughts on this Approa

Question

OK, I have been working on a random image selector and queue system (so you don't see the same images too often).

All was going swimmingly (as far as my crappy code does) until I got to the random bit. I wanted to test it, but how do you test for it? There is no Debug.Assert(i.IsRandom) (sadly) :D

So, I got my brain on it after watering it with some tea and came up with the following, I was just wondering if I could have your thoughts?

• Basically I knew the random bit was the problem, so I ripped that out to a delegate (which would then be passed to the objects constructor).
• I then created a class that pretty much performs the same logic as the live code, but remembers the value selected in a private variable.
• I then threw that delegate to the live class and tested against that:

i.e.

Debug.Assert(myObj.RndVal == RndIntTester.ValuePassed);

But I couldn't help but think, was I wasting my time? I ran that through lots of iterations to see if it fell over at any time etc.

Do you think I was wasting my time with this? Or could I have got away with:

GateKiller's answer reminded me of this:

Update to Clarify

• I should add that I basically never want to see the same result more than X number of times from a pool of Y size.
• The addition of the test container basically allowed me to see if any of the previously selected images were "randomly" selected.
• I guess technically the thing here being tested in not the RNG (since I never wrote that code) but the fact that am I expecting random results from a limited pool, and I want to track them.

Answer 1

Test from the requirement : "so you don't see the same images too often"

Ask for 100 images. Did you see an image too often?

Answer 2

The generation of random numbers is too important to be left to chance. -- Robert R. Coveyou

To solve the psychological problem:

A decent way to prevent apparent repetitions is to select a few items at random from the full set, discarding duplicates. Play those, then select another few. How many is "a few" depends on how fast you're playing them and how big the full set is, but for example avoiding a repeat inside the larger of "20", and "5 minutes" might be OK. Do user testing - as the programmer you'll be so sick of slideshows you're not a good test subject.

To test randomising code, I would say:

Step 1: specify how the code MUST map the raw random numbers to choices in your domain, and make sure that your code correctly uses the output of the random number generator. Test this by Mocking the generator (or seeding it with a known test value if it's a PRNG).

Step 2: make sure the generator is sufficiently random for your purposes. If you used a library function, you do this by reading the documentation. If you wrote your own, why?

Step 3 (advanced statisticians only): run some statistical tests for randomness on the output of the generator. Make sure you know what the probability is of a false failure on the test.

Answer 3

Random numbers are generated from a distribution. In this case, every value should have the same propability of appearing. If you calculate an infinite amount of randoms, you get the exact distribution.

In practice, call the function many times and check the results. If you expect to have N images, calculate 100*N randoms, then count how many of each expected number were found. Most should appear 70-130 times. Re-run the test with different random-seed to see if the results are different.

If you find the generator you use now is not good enough, you can easily find something. Google for "Mersenne Twister" - that is much more random than you ever need.

To avoid images re-appearing, you need something less random. A simple approach would be to check for the unallowed values, if its one of those, re-calculate.

Answer 4

Random is Random. Even if the same picture shows up 4 times in a row, it could still be considered random.

Answer 5

Although you cannot test for randomness, you can test that for correlation, or distribution, of a sequence of numbers.

Hard to test goal: Each time we need an image, select 1 of 4 images at random.

Easy to test goal: For every 100 images we select, each of the 4 images must appear at least 20 times.

Answer 6

As others have pointed out, it is impossible to really test for randomness. You can (and should) have the randomness contained to one particular method, and then write unit tests for every other method. That way, you can test all of the other functionality, assuming that you can get a random number out of that one last part.