Tests and results

Everything here is imported from proptest. Functions that run tests take an optional location keyword argument, which defaults to the caller’s source location. It names the test in reports and keys it in the example database; you only need it when wrapping these functions in your own helpers.

Running tests

given

given(s1, test, settings=Settings())
given(s1, s2, test, settings=Settings())
...                                        # up to six strategies

Runs test(a1, ..., aN) on values drawn from the strategies, max_examples times (see How many times will proptest run my test?). If the test raises, shrinks the failing input and raises an Error whose message is the failure report. The test’s parameter types must be the strategies’ value types. It formats the arguments after the test runs, so generated Functions can print the calls made.

check

check(prop, settings=Settings())

Like given, with each argument drawn by from_type from the property’s parameter types. prop takes one to six arguments and either raises on failure, returns Bool (False fails with check: the property returned False), or returns an Outcome.

given_data

given_data(test, settings=Settings())   # test: def(mut ConjectureData) raises

Runs a test that draws its own values with data.draw(strategy, label) (Hypothesis’s st.data()). Reports list each draw as Draw n (label): value.

check_examples

check_examples(examples: List[V], test)

Runs test on each explicit example (Hypothesis’s @example), raising Falsifying explicit example for the first that fails. Explicit examples are not shrunk.

run_test

run_test(test_fn, settings, name) -> RunReport   # test_fn: def(mut ConjectureData) raises

The engine entry point used by the functions above: generate, shrink, and replay, then return a RunReport without raising for failures. name identifies the test in reports, and its settings.database_key should be set if you use the database.

Inside a test

assume

assume(condition: Bool) raises

Discards the current test case if condition is false. Discarded test cases don’t count towards max_examples; too many fail the filter_too_much health check.

fail

fail(msg: String) raises

Fails the test with msg. Like the std.testing assertions, the error records where it was raised, which is how proptest tells distinct bugs apart.

ConjectureData methods for tests

In given_data tests, composite functions, state machine rules, and Arbitrary.arbitrary:

The low-level methods, for making primitive choices, are in Writing strategies: the low-level API.

Exploring strategies

example

example(strategy, seed: Optional[UInt64] = None) -> S.Value

One random value from strategy, for exploring interactively; not for use in tests.

find

find(strategy, condition, settings=Settings(max_examples=2000, database=None)) -> S.Value

The simplest value satisfying condition (def(S.Value) raises -> Bool), found by searching and then shrinking. Raises NoSuchExample if none is found.

Results

Outcome

A result type for properties passed to check (quickcheck’s TestResult):

is_failure() says whether it’s a failure. Outcome implements the Testable trait (check_result(self) raises), which check accepts as a property’s return type.

run_property

run_property(strategy, test, settings=Settings()) -> PropertyResult[S.Value]

Runs given(strategy, test) and returns what happened instead of raising (QuickCheck’s quickCheckResult). Use tuples(...) for several arguments.

PropertyResult[T]

RunReport

expect_failure

expect_failure(test: def() raises) raises -> String

Runs test, which should fail with a proptest failure (a falsifying example, several distinct failures, or insufficient coverage), and returns its report. Raises if test passes; re-raises any other error (QuickCheck’s expectFailure).

Debugging

before_minimal_test_case

before_minimal_test_case(name: String)

Called just before the final replay of each shrunk failure, and before a reproduce= replay. It does nothing; it’s a place for a debugger breakpoint. name is the test’s location.

Failure messages