Article 5 of the series “From Specification to Execution.” The previous article closed on a simple question: What makes a test suite strong? This article answers it with a tool I find particularly useful.

In 2025, a team of researchers hooked an AI agent equipped with Hypothesis to 100 popular Python packages. After triaging, they obtained 984 bug reports, of which 56% were true bugs. Several fixes have already been merged into NumPy, AWS Lambda Powertools, Hugging Face Tokenizers, CloudFormation CLI, and python-dateutil. The important point is simple. The bugs were in the edges. Detailed study: Agentic Property-Based Testing: Finding Bugs Across the Python Ecosystem.

Property-based testing, or PBT, is the tool that finds these kinds of bugs. I will show six canonical patterns, stateful PBT for sequence bugs, and the role an AI agent can play in this work.

Why example-based tests miss the edges

When a developer writes example-based tests, they usually think of a few cases. Three, five, sometimes ten. The happy path first, and a few failures if the discipline is good. But how many tests cover a 100,000-character string? A Unicode surrogate character? An empty list passed to a function that has never seen this case? A float right at the boundary of IEEE 754 precision?

The idea that an AI agent fixes this problem is no more solid. AI often inherits the same biases as the humans who produced its corpus. The examples it proposes are, statistically, those humans already think of. The Agentic PBT study shows this well. NumPy’s random.wald() silently returned negative values. AWS Lambda Powertools’ slice_dictionary returned the first chunk multiple times instead of advancing. No one, human or AI, had written the test that would have targeted these cases.

PBT flips the logic. I don’t generate examples by hand. I declare a property, an invariant, a relationship, an equation that must always hold. The framework then generates hundreds, sometimes thousands of examples.

The useful bonus is shrinking. When a failure occurs, the framework looks for the smallest counter-example that reproduces the bug. I don’t get an unreadable failure on a 50,000-character string. I often get a minimal case, sometimes a single character. Diagnosis becomes immediate.

The six canonical patterns

I keep six patterns in mind. They cover the bulk of useful cases.

1. Invariant, something that must always hold

from hypothesis import given, strategies as st

@given(st.lists(st.integers()))
def test_sort_preserves_length(xs):
    assert len(sorted(xs)) == len(xs)

Sorting preserves length. It’s trivial to formulate, but useful in practice. I use it for anything that shouldn’t change: length, sum, set of elements, structure.

2. Round-trip, encode/decode, serialize/parse

@given(st.dictionaries(st.text(), st.integers()))
def test_json_roundtrip(d):
    assert json.loads(json.dumps(d)) == d

This is the highest-ROI pattern for parsers and serializers. In a telecom context, it quickly reveals SIP, SDP, and RTP bugs. The examples/pbt-sip/ demo in the public repo illustrates this pattern on a minimalist SIP parser.

3. Idempotence, f(f(x)) == f(x)

@given(st.text())
def test_normalize_idempotent(s):
    assert normalize(normalize(s)) == normalize(s)

I use it for a function that must remain stable after a first application, like normalize, dedupe, sort, or canonicalize. This pattern quickly catches functions that add something on the first pass and then remove it on the second.

4. Metamorphic, relationship between two executions

@given(st.lists(st.integers()), st.integers())
def test_appending_increases_length_by_one(xs, x):
    assert len(xs + [x]) == len(xs) + 1

I use it when I can’t write a direct oracle. The pattern was formalized by T. Y. Chen in 1998, and it remains useful for complex algorithms, optimizers, and simulations. In a telecom dialplan, for example, I can verify that adding a more specific route doesn’t change the cost for already covered numbers.

5. Alternative oracle, comparing to a simple model known to be correct

@given(st.lists(st.integers()))
def test_my_fast_sort_matches_python(xs):
    assert my_fast_sort(xs) == sorted(xs)

I use it when I have a fast implementation to validate against a simple version assumed to be correct. It’s very useful for complex data structures like skiplists, B-trees, or tries.

6. Algebraic, commutativity, associativity, distributivity

@given(st.integers(), st.integers())
def test_add_commutative(a, b):
    assert add(a, b) == add(b, a)

I use it for aggregations, merges, and distributed reductions. If an operation is supposed to be commutative, PBT can verify it across thousands of pairs.

Stateful PBT, the weapon against sequence bugs

Classic PBT tests functions. Stateful PBT tests sequences of operations on a system that keeps state.

This is where many subtle bugs hide. They only appear after a series of actions in a particular order.

I’ll take a simple example on a SIP dialog. The full code is in examples/pbt-sip/ of the public repo.

from hypothesis.stateful import RuleBasedStateMachine, rule, invariant

class DialogMachine(RuleBasedStateMachine):
    def __init__(self):
        super().__init__()
        self.dialog = Dialog(call_id="test-call")
        self.model_state = State.EARLY

    @rule()
    def receive_2xx(self):
        self.dialog.receive_2xx()
        self.model_state = State.CONFIRMED

    @rule()
    def receive_cancel(self):
        self.dialog.receive_cancel()

    @invariant()
    def state_matches_model(self):
        assert self.dialog.state is self.model_state

Hypothesis generates arbitrary sequences of receive_2xx, receive_cancel, and other transitions, then checks the invariant after each step. On this code, the framework finds an RFC 3261 §9.2 bug in less than thirty milliseconds. A CANCEL goes through after the final response in just two transitions. This is exactly the kind of sequence an example-based test never writes.

I keep a few preferred domains for this testing mode:

  • Telecom FSMs, SIP dialogs, RTP states, dialplan routing, SBC sessions.
  • Reducers, Redux, Zustand, useReducer on the React side.
  • Stateful services, shopping carts, message queues, user sessions.
  • Any API where the call order matters.

Why PBT is well-suited for AI

I see three reasons.

1. LLMs are quite good at proposing properties

Models know how to start from a signature, a docstring, or function names to propose useful invariants. Anthropic Red states this clearly in Property-Based Testing with Claude (2026). They identify semantic guarantees well if the input text is clear.

2. The PBT oracle is a relationship

The oracle is not an isolated value. It’s a relationship between inputs and outputs, or between two executions. As I wrote in article 3, this reduces the room for tautologies. I don’t need to invent a reference value. I state a rule.

3. The reflection loop filters false positives

The reflection loop documented by Anthropic Red verifies that the agent hasn’t accidentally masked the error with a try/except, or that it hasn’t written a property that is too strong. According to the arxiv 2510.09907 study, this loop increases the validity of reports from 56% to 81% for the highest-scoring reports.

The usage pattern seems simple to me:

  1. The agent proposes three to five candidate properties, with the sentence or signature justifying them.
  2. I choose the property to implement.
  3. The agent writes the Hypothesis test.
  4. The reflection loop runs.
  5. The result goes into code review.

The Agentic PBT study also quantified the total cost: $9.93 USD per validated bug, in Claude/GPT-4 inference cost. It’s not a universal truth, but it’s a useful order of magnitude.

Three properties to add this week

I can start small. Three patterns already provide a lot of value.

On a parser, the round-trip

@given(any_input_strategy())
def test_parse_format_roundtrip(x):
    assert parse(format(x)) == x

If I have a parser or a serializer somewhere, I can write this test. It takes few lines and often reveals very costly bugs.

On a calculation, a numerical invariant

@given(positive_floats(), positive_floats())
def test_total_is_at_least_each_component(a, b):
    t = total(a, b)
    assert t >= a and t >= b

I use it for any function that aggregates or composes values. For a telecom call rating, I can verify that the total cost is greater than or equal to the cost of each sub-period.

On a pure function, idempotence

@given(st.lists(st.integers()))
def test_dedupe_idempotent(xs):
    assert dedupe(dedupe(xs)) == dedupe(xs)

I use it for a function that claims to clean, normalize, or canonicalize. The second call must be a no-op.

Tools by stack

LanguageTool
PythonHypothesis
TypeScript / JavaScriptfast-check
Gorapid
Rustproptest

Hypothesis also has a deterministic bonus. hypothesis write my_module.parse_uri generates a PBT test skeleton from the signature and the docstring, without AI. It’s a tool I find still underused.

Three points to keep

1. Example-based tests miss the edges. Humans and AI share this limit. Generating more cases does not fix the underlying problem.

2. Stateful PBT (RuleBasedStateMachine in Hypothesis) finds sequence bugs that example-based tests do not cover.

3. PBT is very well suited to AI-assisted use. The property becomes a relationship, not a value, and that limits tautologies.

With the spec, the role breakdown, the measurement axes, and PBT, the workflow is now set. Next week, I’ll move on to a practical implementation on a modern stack, React + Go, with a runnable boilerplate.

Further reading

This article is the fifth in a seven-part series. The previous article covered quality measurement axes. The next one will discuss a practical implementation on a React + Go stack (Gin / GORM / PostgreSQL).