Quickcheck for runtime errors

Question

I have interest in using the quick check library but it seems that it is designed to test properties. What I would like to do is generate random data for my defined data types and test functions I have written. I do not care about what the result is, just if the function produces a runtime error when fed random data. All of the quick check examples I have seen are for testing properties of functions like is the result greater than 5 when fed random data. Is there a way to use quick check in this manner? Something like

data Result = A | B

fun :: Result -> Int
fun A = 5

main = check fun

In the above code I have a custom data type, and a incomplete function. This function will fail if passed B. There are of course more types of runtime errors than just incomplete functions. I would like to have check generate data, and feed it to the function. Not caring what the result is. Is quick check able to do this?

Edit - I should note that I am not looking for flags that check for incomplete patterns, and what not. I am interested in general purpose runtime error checks.

Answer 1

Rather make sure that functions that don't handle IO don't throw exceptions if possible. Exceptions can only get caught in IO, and they kind of break the things you otherwise expect from a pure function.

It's possible with some helpers from Control.Exception and Test.QuickCheck.Monadic:

> import           Control.Exception       (Exception, PatternMatchFail,
>                                           SomeException, evaluate,
>                                           fromException, try)
> import           Data.Either             (either)
> import           Test.Hspec              (anyException, describe, hspec, it,
>                                           shouldThrow)
> import           Test.QuickCheck         hiding (Result)
> import           Test.QuickCheck.Monadic (assert, monadicIO, run)

For starters, let's write a function that enables us to check that a certain exception is thrown:

> throwsExceptionOr :: Exception e => (e -> Bool) -> (a -> Bool) -> a -> IO Bool
> throwsExceptionOr pe pa = fmap (either pe pa) . try . evaluate

This enables you to write tests like this:

> prop_fun_1 x = monadicIO . run $
>   throwsExceptionOr (const True :: SomeException -> Bool)
>                     (== 5)
>                     (foo x)

throwsExceptionOr is very general, so that you can define your own helpers:

> -- | Should always throw an exception.
> throwsException :: Exception e => (e -> Bool) -> a -> IO Bool
> throwsException p = fmap (either p (const False)) . try . evaluate

> -- | Should either pass the test or throw an exception.
> exceptionOr :: a -> (a -> Bool) -> IO Bool
> exceptionOr x p = fmap (either anyException p) . try . evaluate $ x
>   where
>     anyException :: SomeException -> Bool
>     anyException = const True

Now you can write your tests as usual:

> data Result = A | B deriving (Enum, Show, Eq, Ord, Bounded, Read)
> 
> instance Arbitrary Result where
>   arbitrary = elements [A, B]
> 
> foo :: Result -> Int
> foo A = 5
> 
> prop_foo x = monadicIO . run $ foo x `exceptionOr` (== 5)

You can go further and move the monadicIO . run into another helper, but that's left as an exercise. Furthermore, you can make the functions compatible with other testing frameworks, such as hspec or tasty:

> main :: IO ()
> main = hspec $ do
>   describe "foo" $ do
>     it "either returns five or throws a pattern match fail" $ propertyIO $ \x ->
>       throwsExceptionOr patternMatchFail (==5) (foo x)
> 
>     it "throws an exception on input B" $
>       evaluate (foo B) `shouldThrow` anyException
> 
>  where
>   patternMatchFail :: PatternMatchFail -> Bool
>   patternMatchFail _ = True
> 
>   -- I think there is a better combinator for this
>   propertyIO f = property $ \x -> monadicIO . run $ f x

That being said, regardless of the used languages you want to get rid of possible runtime errors at compile time if possible. This includes getting rid of partial functions or possible type nonsense. This depends on the actual use, of course. If you can verify that head always gets called on a non-empty list throughout your program, go ahead and use it. If you can't, use pattern matching (see this discussion on head's type).

Either way, given that older versions of GHC don't provide stack calls, you rather want to have errors at compile time than errors without a stack trace at runtime (recent versions of GHC have some nice features for this).