What is this about?

QuickCheck, and property testing in general, are a quagmire of bad documentation and folklore. A lot of what material exists on the topic of either, or both, is old, vague, overly-general or straight-up wrong, sometimes at the same time. This makes it difficult for newcomers to understand how to use the tools provided, or leads to common errors that lead to frustration or bad test quality. While plutarch-quickcheck aims to make the more tedious parts of this invisible, there are many such issues that we cannot do for our users.

To remedy this, we have created this list. It contains a mixture of how-tos and best practices, alongside warnings; our goal is to reduce friction and learning curves, while helping our users avoid common pitfalls. We also give a collection of time-saving tips and tricks to help get good tests faster.

The suggestions, tips and tricks

Be careful of suchThat and suchThatMap

The following functions are provided by QuickCheck for the purpose of producing values in the Gen monad subject to constraints:

suchThat :: forall (a :: Type) . Gen a -> (a -> Bool) -> Gen a

suchThatMap :: forall (a :: Type) (b :: Type) . Gen a -> (a -> Maybe b) -> Gen b

Both of these are implemented in a rather naive way by QuickCheck:

  1. Generate a random a using the provided generator.
  2. Attempt to apply the predicate (for suchThat) or Kleisli arrow (for suchThatMap).
  3. If successful, return the generated value (for suchThat) or the result (for suchThatMap); otherwise, go back to 1 and retry.

Notably, there is absolutely no retry limit or timeout. What this means is that if the predicate or Kleisli arrow fail often (or even always, such as due to an implementation mistake), it can make generators run for a long time or possibly forever, without even indicating what might be wrong or why. Furthermore, QuickCheck doesn't (and indeed, can't) be any smarter than that, as this would require being able to inspect the provenance of the predicate or Kleisli arrow and force the generator to act accordingly, neither of which QuickCheck presently even attempts.

Thus, you should avoid using either of these functions whenever possible. There are two cases where their use is justifiable:

  1. When you know that the predicate is very likely to succeed; or
  2. When you need to use a smart constructor, and can provide the variety of input required, but don't want to use something like fromJust.

Additionally, for certain common types, you can use a modifier instead of these functions; in those cases, such issues are handled for you.

Use modifiers

QuickCheck has an entire module of modifiers: newtypes whose only purpose is to provide the ability to generate (and shrink) restricted subsets of the values of the type being wrapped. Some notable examples (whose names largely speak for themselves) include: