Differential Testing: The Only Gate That Stops Model Hallucination in Reverse Engineering

The model finishes reading your leaf function and hands you a clean verdict: "This is ChaCha20, only the round count has been changed to a key-derived variable." It sounds credible, and you're half-tempted to just write the rewrite from it.

Stop. Until you turn that sentence into a running assert, it is only a sentence. However many public implementations the model has seen, it cannot check whether it got this one right — that's a question of fact, not of language, and the only way to answer it is to put both sides' outputs side by side and compare byte for byte.

This piece is about the one machine that can answer that question: the differential test. It's Stage 3 of the four-stage flow, and the destination of every hypothesis that comes out of deviation hunting. The model generates suspicion; the differential test delivers the verdict.

"Can you confirm this is right?" is a dead-end question

The question a beginner most loves to ask the model is: "Can you confirm this implementation is correct?"

It's a dead-end question, on three levels.

First, the model is tuned to agree with you. "Is this implementation correct" carries its own prior — you obviously want it to be right, and the model tends to catch that tone and hand you "yes."

Second, it has no basis to judge on. The only basis for "correct" is the black box's output over the same batch of inputs, and that isn't in its context. It has no ground truth to compare against, so it can only guess from whether the code "reads reasonably" — and reading reasonably is exactly what obfuscated code is best at.

Third, it's the wrong audience. Correctness isn't a negotiable opinion; it's the fact of whether an equation holds. Don't hand the question to the model, hand it to assert: model says right but the test fails, the test wins; model says wrong but every test passes, the test still wins. The moment you hand off the verdict, you're building on hallucination.

How to choose boundary inputs

The skeleton of a differential test is simple: the original as a black box, your rewrite compared against it over the same batch of inputs, case by case. What's valuable isn't the loop — it's the batch of inputs you feed in.

Throw ten thousand random normal strings at it and a green run proves nothing — normal inputs take the main path, and deviation points usually hide in the corners. What actually carries information is the boundaries:

# Boundary cases: derived from block size B
def boundary_cases(B: int) -> list[bytes]:
    return [
        b"",                 # empty: exposes initial state and padding logic
        b"\x00",             # single zero byte
        b"\xff",             # single high byte: checks sign-bit / unsigned handling
        b"A" * (B - 1),      # one below the block boundary
        b"A" * B,            # exactly one block
        b"A" * (B + 1),      # one above: checks carry and padding
        bytes(range(256)),   # full byte coverage: checks the alphabet map covers the whole domain
    ]

Each one interrogates a specific branch. The high byte \xff forces sign-bit handling out into the open — the difference between >>> and >> in JS, whether there's a & 0xff in Python, all show up on this one case. The B-1 / B / B+1 triple is the fiercest; block-padding logic only reveals itself here. Full byte coverage is for custom alphabets — one character too many or too few in the mapping table, and this case fails without exception.

The differential skeleton itself is under ten lines:

# Original as black box, rewrite compared case by case
def diff_test(blackbox, rewritten, B: int) -> None:
    for case in boundary_cases(B):
        got, want = rewritten(case), blackbox(case)
        assert got == want, f"len={len(case)} hex={case.hex()}"

Worth noting: when your deviation-point hypothesis involves a standard algorithm, you don't even always need the original black box — public standards often ship authoritative test vectors. For instance, RFC 8439 §2.1.1 gives a fixed input/output pair for the ChaCha20 quarter round (feed a=0x11111111, b=0x01020304, c=0x9b8d6f43, d=0x01234567 and you get a=0xea2a92f4, …). Have your rewrite pass the standard vector first, then compare against the target black box, and you cleanly separate two classes of problem: "I wrote the ChaCha implementation itself wrong" versus "the target changed ChaCha."

Notice that the assert message carries len(case). This is the single most useful line of diagnostics in the whole suite. If only the B+1 case fails out of the seven, the problem is almost certainly in padding or carry; if only full byte coverage fails, the problem is in the alphabet map. The length of the failing case points you straight at the layer that's wrong — no guessing.

Run the same input twice

The most common sticking point when rewriting signature logic isn't a misidentified algorithm — it's an entropy source that hasn't been factored out.

Catching that sticking point takes one line: the same input, run twice.

# Same input twice: differing results mean a hidden RNG / timestamp
def assert_deterministic(fn, case: bytes) -> None:
    assert fn(case) == fn(case), "unfixed entropy source or timestamp present"

If the two results differ, the implementation has mixed in a time.time(), a nonce, a self-incrementing counter, or something else that changes on every call. At that point you can't even do a differential comparison — the black box gives a different answer every time, so what would you compare against?

The fix isn't to delete the entropy source (delete it and the signature is wrong) — it's to lift it out of the code into an injectable parameter, pinned to a fixed value during differential testing:

# Lift the entropy source into an injectable parameter, pin it with a stub for diffing
class Rewritten:
    def __init__(self, clock=time.time, rng=os.urandom):
        self._clock = clock          # formerly an inline time.time(), now injected
        self._rng = rng

    def __call__(self, data: bytes) -> bytes:
        ts = int(self._clock())      # for diffing, clock=lambda: 0
        nonce = self._rng(16)        # for diffing, rng=lambda n: b"\x00" * n
        ...

Do the same to the target black box — find the point where it injects the timestamp/random and find a way to pin it (hook Date.now in the page, pass a fixed seed to Node). With both sides' entropy pinned, the output returns to being deterministic and the diff becomes meaningful. Once the rewrite passes end to end, swap clock and rng back to the real implementations.

This is also a step the model can't help with: where the entropy hides and how it's injected is runtime behavior — you force it out by running twice, not by reading the code.

Peel layer by layer; don't compare the whole thing

Say your rewrite's overall output doesn't match the black box. Don't fixate on that final string of bytes — it's the product of several nested layers, and you don't know which one is wrong.

A signer is usually layered: an innermost hash or block cipher, wrapped in an encoding layer (a Base64 family, hex, some custom table), wrapped in an outermost assembly layer (concatenate a prefix, insert fields, add a length header). Peeling layer by layer means comparing from the innermost out, moving one layer out only once the current one passes:

# Peel layer by layer: match the innermost first, then move outward
LAYERS = ["digest", "encode", "assemble"]   # inner → outer

def first_divergent_layer(bb_dump, rw_dump, case: bytes) -> str | None:
    for layer in LAYERS:
        if bb_dump(case)[layer] != rw_dump(case)[layer]:
            return layer                     # the first layer to diverge is the faulty one
    return None

This assumes your rewrite can dump the intermediate state of each layer, and that you can dig the corresponding intermediate values out of the black box too (usually by instrumenting at runtime). That extra effort is well worth it — the first layer to diverge is the faulty one, and your attention narrows in an instant.

This hooks straight into the four categories of deviation point: divergence at the digest layer usually means a constant was perturbed or the round count was changed; at the encode layer, usually the alphabet was reordered; at the assemble layer, usually some material was embedded into the output. The layer that diverges tells you which category in the fingerprinting piece to go looking in.

The long-term value of fixtures

The moment a differential test goes green feels great, but it's a one-shot — upstream ships a new version tomorrow and the implementation you validated today may be entirely wrong.

The artifact that actually appreciates over time is the fixture: a table of known input → known output, committed to the repo.

# Fixture: known input → known output, committed to the repo
# Re-run on every upstream change; green yesterday, red today = upstream changed, not you
VECTORS = load_json("vectors.json")   # [{"in": "<hex>", "out": "<hex>"}, ...]

for v in VECTORS:
    got = rewritten(bytes.fromhex(v["in"])).hex()
    assert got == v["out"], v["in"]

Its value pays off the moment upstream changes. One day CI goes red, and it's the implementation you didn't touch yesterday failing against the fixture — that piece of information is priceless: it rules out "I made a mistake" and points the problem uniquely at "upstream changed." Without a fixture, you'll waste half a day debugging perfectly correct code, because you can't tell whether it's your fault or someone else moved the goalposts.

The best inputs for the fixture are exactly the boundary cases from earlier — they're already the highest-coverage set you have.

The model does exactly two things at this step

Let me state the division of labor in a differential test flatly: the model does two things, and gets no say in the verdict.

First, generate cases. Laying out boundary inputs in bulk across many primitives, assembling a batch of one-bit-apart inputs to probe a suspect constant — this is enumeration, no reasoning required, and a cheap high-concurrency tier is the most economical.

Second, explain a diff. When a case fails, you put the two layer dumps in front of it and have it explain, against specific bytes, where the first divergence is and which of the four deviation categories it most likely belongs to. This step needs mid reasoning and the ability to "explain against bytes" — it's where the model in this piece actually does work.

The verdict goes to assert, and this doesn't budge. The model's explanation is a lead, not a conclusion; leads pointing the wrong way is normal, and assert catches them.

Those two things ask different capabilities of the model; running one tier for the whole thing either wastes money or wastes precision:

The workhorse is the second tier. Whether difference attribution is worth choosing a model over, you'll know from one round of testing yourself. The protocol is short:

1. Pick one genuinely failing case from your differential suite, with its two layer dumps (the black box's and your rewrite's).

2. Feed the same diff to gpt-5.6-sol and claude-opus-5 separately, asking only two things: which layer the first divergence is in, and which of the four deviation categories it most likely is.

3. Look only at whether the attribution lands on a specific byte and a specific layer, or hands you a vague "might be a padding issue."

4. Attribution precision is your selection criterion — it directly decides how many rounds of edits you make to get this one case green.

One round shows you the difference, more directly than any benchmark leaderboard.

The switching cost is the real obstacle

These four tiers come from three vendors: three SDKs, three auth schemes, three error formats. Wiring up your client three times just to switch tiers between sub-steps isn't worth it — which is exactly why most people end up running one tier for everything, use a model that only speaks in vagaries for difference attribution, and burn several rounds of edits without knowing why.

AIReiter flattens that layer: one key, one OpenAI-compatible interface, all four tiers behind it, and switching is just changing the model field in the request body.

# Difference attribution: the mid-reasoning tier
curl https://aireiter.com/api/v1/chat/completions \
  -H "Authorization: Bearer $AIREITER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-5.6-sol",
    "messages": [{"role": "user", "content": "<diff of the two layer dumps + locate the first divergent layer and deviation category>"}]
  }'

# Attribution won't converge, escalate to dig the root cause: change one field
#   "model": "claude-opus-5"
# Bulk-generate boundary cases:
#   "model": "claude-sonnet-5"

Already on the OpenAI SDK? Point base_url at https://aireiter.com/api/v1 and change nothing else. On the Anthropic SDK? Hit POST /api/v1/messages with the same key.

On price, Claude models run at 30% off list and GPT models at half price. For this workflow the discount lands right on the densest step: difference attribution is the most frequently called part of differential testing — one round per failing case, and every upstream change means a fixture rebuild and a whole fresh batch of failing cases to attribute. The workhorse gpt-5.6-sol is a GPT model at half price, so the dense part is cut straight in half; the occasional escalation to claude-opus-5 to dig a deep root cause is few calls, but a Claude model at 30% off applies there too.

• Get an API key

• Try it without signing up — feed one real diff to both tiers by hand first, compare the attribution precision with your own eyes, then decide which to wire up.

In closing

The skeleton of reverse-engineering rewrites really has only two beams: the model gives the hypothesis, assert gives the verdict.

The model is a suspicion generator that has seen every public implementation — it can tell you in seconds "this might have been changed here," but it never knows whether it's right this time. The differential test is the machine that turns "might" into "is/isn't": boundary inputs force out the branches, running the same input twice forces out the entropy source, layer-by-layer peeling pins the faulty layer, and the fixture distinguishes "I wrote it wrong" from "they changed it."

Every judgment the four-stage flow and the deviation-hunting piece hand you has to pass through this gate in the end. Nothing the model says counts; only what assert says counts.