The page that never changed

One of the six faults in my corpus was never caught. Not by the step-level falsifiers, not by the plan-level ones I added afterwards, not by the arm with a verifier that is right by construction. Four trajectories in which the agent clicks and the page stays exactly as it was, and in all four the detector sees a valid page that contradicts nothing and lets the agent carry on. I spent a while trying to write a falsifier for it before I accepted that there isn't one. That miss is, I think, the most useful thing in this article, and it took a working detector to find it.

The detector is my rebuild of the checking half of FCPAgent, from a July paper by Guangyi Liu, Huan Zhao and Quanming Yao. Their idea is simple to state. Before a plan step runs, the browser agent writes down two things: the evidence that would show the step is still on course, and the evidence that would show it has gone wrong. Then it tests both against the live page, before the action and again after. Most agent reliability work does its checking after the fact, by reflecting on the trajectory or scoring the final answer. This puts the check in front of the click.

I built that check with no language model in the loop and ran it over 40 hand-written browser trajectories, 24 of them deliberately broken. It caught 79.2% of the broken ones, raised zero false alarms on the healthy ones, cost 0.044 ms per check, and landed 57.9% of its catches before the browser had been touched. The paper's own "confirming evidence only" ablation caught none of them. I did not expect that. Not fewer. None.

What I have not done is reproduce FCPAgent's WebArena numbers, test its planning or repair halves, or learn anything about whether the agent finishes its task; the prototype measures detection and only detection, and the verdict is scoped to match.

Seventy-nine percent caught, one class missed

Attach to each plan step the evidence that would confirm it and the evidence that would kill it, and test both against the page before and after the action. A contradiction is cheap to check and expensive to fake, which is why the fast path can be model-free. On my 40 trajectories across four enterprise workflows, falsifying evidence detected 79.2% of off-track runs against 0.0% for the paper's confirming-only ablation, with zero false alarms, 7.4% of checks escalated to a verifier, and 0.044 ms per check. A silent no-op, where the action changes nothing, was invisible to every configuration. My decision is to ship the detection layer on one bounded internal workflow behind a policy gate, and never to treat a passed check as authorization to act.

Four steps, none hard, one wrong ticket

Here is the workflow the numbers come from. Find ticket T-390, confirm the customer reported a duplicate charge, assign it to Payments, save. A conventional browser agent runs a loop over it: observe, choose, execute, repeat. The check that matters arrives at the end, and it is almost always a question about the goal. Did we assign the ticket?

Now look at what that question accepts. Yes, says the trajectory that opened T-411 instead, read a plausible complaint, and assigned it to the right-sounding team. Yes, says the one that assigned it after the session had quietly expired. Yes, says the one that clicked a button whose label changed meaning in a redesign three weeks ago. Every step looks locally reasonable, nothing throws, the outcome check sees a saved ticket, and you find out six weeks later when someone asks why Payments owns an invoice request. A loud failure costs you a retry. This kind costs you a wrong record in a system you have to trust.

The paper's name for a plan step is a commitment, and the argument runs that a commitment nobody can contradict is not worth much. So each step carries five things: a subgoal, an optional reusable skill, confirming evidence, falsifying evidence, and a confidence score. Together they are a Falsifiable Commitment Unit.

The two kinds of evidence are organised differently, and the asymmetry is deliberate. Confirming evidence is staged: some of it should hold before the step runs, some during, some once it is finished. Falsifying evidence is scoped, and the scope names what broke. Execution if the action or state drifted. Skill if the reused procedure no longer fits the page. Planning if the subgoal was wrong from the start.

The scope labels are the part with production consequences. They tell the repair step how much work to throw away: retry the click, swap the procedure, or rewrite the plan. Two seconds of recovery, or twenty steps from the top.

Where the falsifiers come from, and how a step is routed

The planner pulls three candidate skills and two past failure-repair cases out of offline libraries, then emits the commitment sequence. Skills supply the procedure template. The failure memories supply the falsifiers, on the reasonable theory that the cheapest source of "what could go wrong here" is what went wrong here last time. Both libraries come from 191 training tasks and are frozen at serving time, so none of this costs anything per request.

Every step is then scored by a fast matcher before the expensive model is allowed to speak. The paper combines a natural language inference model with an image-text matcher over the screenshot:

There are two test points. The action-level test scores the action the agent proposes, before it touches the browser. The state-level test scores the page that action produced. Only the first can stop a write, and it is the reason I care about this paper more than about the other reflection-style methods.

The routing is the part I reimplemented exactly, and the part worth reading closely:

def _route(self, a_plus, a_minus, confidence):
    t = self.thresholds
    # Safety first: a contradiction outranks a completion signal.
    if a_minus >= t.falsify_threshold(confidence) and (a_minus - a_plus) >= t.delta_minus:
        return REPAIR
    if a_plus >= t.confirm_threshold(confidence) and (a_plus - a_minus) >= t.delta_plus:
        return ADVANCE
    if max(a_plus, a_minus) < t.pi_low:
        return CONTINUE
    return VERIFY

Two tests pin it down. test_conflicting_strong_signals_route_to_verify asserts that α+ = 0.9, α− = 0.9 returns VERIFY, and test_contradiction_outranks_confirmation asserts that α+ = 0.55, α− = 0.95 returns REPAIR and never ADVANCE.

Read the order of the branches. The falsifying branch is tested first, so a step can never be advanced on a page that also contradicts it. And no route fires on an absolute score alone; each needs a margin over the other side, which the paper sets at 0.18 for completion and 0.25 for drift. When both sides are strong, no branch wins, and the step escalates to the model instead of guessing.

The gain lives in the long tasks

A gain of that size against ColorBrowserAgent, and not against some weak baseline, is what made me keep reading. It is still a benchmark number. The structure underneath it is what persuaded me: on Shopping tasks of one to five steps the improvement is 11%, and on Shopping tasks of eleven steps or more it is 161%. If the mechanism does what it says, you would predict exactly that shape. A short task barely has time to go wrong. A long one drifts, and with no per-step contradiction check, nothing notices until the trajectory is over and the damage is written down.

Do the falsifiers earn the credit?

The ablation says yes, and it is the reason I chose this paper.

A controlled comparison that isolates the exact component the paper is named after is rarer than it should be. It also made a small prototype viable: the claim is narrow enough to test without rebuilding the system around it.

The efficiency table matters just as much if you are the one paying for the inference:

Three runs, no standard deviations

The paper reports averages over three runs and no spread, so a 2.0-point ablation delta has no stated uncertainty around it. I would say that makes the smallest of the three ablation numbers unusable and the largest one still convincing. Success is measured on WebArena and WebChoreArena, both simulated environments with stable page structure. The redesigns, interstitials and permission errors that break real automation are not in the sample. And the authors are direct about the mechanism's own weak point:

"A limitation of the current framework is that it depends on the quality of generated evidence: overly broad falsifiers may trigger unnecessary repairs, whereas overly narrow ones may miss off-track execution."

Four papers, and the one that says fix the site instead

I screened four candidates published within the last three months against five questions: claim support, baseline strength, causal evidence, production transfer, and testability. Four papers do not make a literature review, and one person chose both the four and the five questions, so read the table as my judgement and weigh it accordingly.

Agent-Ready Websites deserves more than a table row, because it is the strongest challenge to this entire line of work. If changing a site's markup lifts agent success from 49.3% to 89.3%, then for any site you control, fixing the site beats instrumenting the agent and it is not close. So what is commitment testing for? The sites you do not control, which in an enterprise is most of the vendor tools your staff spend their day in.

Three expectations, and the one that was wrong the good way

My impression going in was that the interesting half of FCPAgent is the cheap one, and building it confirmed that. Not reproducing the paper was a choice, not a limitation I ran into halfway through: running WebArena with a 397B backbone tests the whole system, and I wanted one premise tested cheaply enough that you can read the entire harness in an afternoon and tell me where I went wrong.

Going in, I expected three things. Falsifiers would help. Pre-action testing would be the interesting part. And the cheap matcher would be too blunt for the wrong-record case, since a near-miss identifier shares most of its tokens with the right one; I expected to spend the write-up apologising for it. The first two held. The third was wrong in the agent's favour, as you will see, and the result I had not predicted at all was the one that came back first.

Forty trajectories, no model, two bounds

The corpus is 40 hand-authored trajectories across four enterprise workflows: support triage, expense approval, catalogue price update, order refund lookup. 250 steps in total, every observation written by hand to imitate a flattened accessibility tree. Sixteen are healthy. The other twenty-four each carry one injected fault, drawn from six types: wrong record opened, session dropped, stale filter, control missing after a redesign, a success banner for a different operation, and an action that changes nothing. Each plan carries four commitments, and I wrote all of them before inspecting a single trajectory.

git clone https://github.com/shravan1996/ship-the-paper-falsifiable-commitments
cd ship-the-paper-falsifiable-commitments
python3 -m unittest discover -s tests   # 37 tests
python3 eval.py

Python 3.9.6 on an Apple M-series laptop, no dependencies, no network, no GPU. One substitution matters: the matcher is inverse-document-frequency weighted content recall in place of the paper's entailment model, so what I am testing is the control loop and not the paper's accuracy. With no model in the loop, each arm gets reported twice. Once where escalations resolve as "keep going", which is a lower bound from the cheap matcher alone. Once where they always resolve correctly, which is an upper bound no real verifier will beat. The truth sits between them.

Zero for the ablation, 79.2% with plan-level falsifiers

The confirming-only row is the one I did not expect. Less, yes. Nothing, no. It detected nothing, and once I understood why, the zero stopped being a result and became a property of the loop.

Sit with the outcome-check row too. It detects 100% of failures and saves zero steps, because it fires after the trajectory is over. I had expected detection rate to be the headline of this article. The corpus changed my mind: for a drift problem, when you detect is the whole point, and the outcome check is the proof.

The wrong-record case, the one I had planned to apologise for, was caught three times out of four. The fourth is its own section below.

Session loss, caught once, then four times

Session loss was caught in 1 of 4 workflows with step-level falsifiers, then 4 of 4 once I attached the same falsifier to the plan instead of a step. Obvious in hindsight: a session can drop on any step, so a falsifier written for step 2 never sees it happen on step 4. The paper scopes every piece of falsifying evidence to a single commitment, which for environment failures seems to me the wrong scope. The fix was three lines. A three-line change moving detection from 25% to 100% on a whole fault class is a surprisingly large return, and I suspect the paper left value on the table by never letting a falsifier cover the whole plan.

Three ten-thousandths from a repair

In the expense workflow the agent opened EXP-1207 instead of EXP-1180. The identity falsifier was fully satisfied at α− = 1.0, but "Expense report EXP-1207 detail" still matched most of the confirming evidence, leaving a margin of 0.2497 against a threshold of 0.25. Three ten-thousandths. The matcher declined to decide and escalated, which is the design working exactly as specified. That the margin rule fired was expected; that it fired this close to the line surprised me, and it is the single result in this prototype I would least trust to hold on a different corpus. It is also why I report the fast path as a lower bound and not as the number.

A page that does not change contradicts nothing

The silent no-op was caught 0 out of 4 times, in every arm, including the perfect-verifier ceiling. I stared at this one for a while looking for a falsifier that would work, and there isn't one. I had assumed every failure mode has some observable fingerprint if you write the evidence statement carefully enough. That assumption is wrong, and this is the counterexample. The page is valid. Nothing on it contradicts the commitment. The agent is simply not progressing, and no evidence statement written over a single observation can see that. You need state diffing across observations, and this mechanism does not give you any.

Where I expect it to go wrong on real pages

The planner grades its own homework

In the paper, the planner that commits to a step also writes the evidence that could kill it. In my prototype I wrote both halves myself. Neither arrangement is a check by an independent party, and I do not have a good answer for that.

The failure mode is slow and quiet. Falsifiers drift toward statements that never fire, each technically true, none load-bearing, and detection decays while the dashboard stays green. So track fires-per-thousand-steps for every falsifier, not just the aggregate. One that has not fired in a month is describing something that cannot happen, or it is written too narrowly to notice when it does, and you cannot tell which from the aggregate.

An unfalsified step is still not permission

An unfalsified commitment means one thing: no page content contradicts this step. It does not mean the agent is allowed to take it. Those are two different questions, and only one of them is about the page.

WebDecept evaluates multimodal web agents against seven classes of deceptive interface and finds them broadly susceptible, with prompt-based safeguards insufficient. Here is why that lands hard on this mechanism: a page written to mislead can satisfy every confirming check and trip no falsifier at all, because the falsifiers were written to catch the agent's mistakes and not the page's intent. Commitment testing raises the floor on accidental failure and does approximately nothing about a hostile page.

My read is that the control here is ordering, not evidence quality. The evidence check runs first, the policy gate second, and the gate asks questions the page cannot answer: is this a write, is the target in scope, has a named human approved this class of action?

Repair is where the money goes back out

My prototype stops at detection, which is the cheap half. FCPAgent carries on into scope-aware repair, and repair calls the model. Note what the 19.5% saving in Table 5 is measured against: a baseline that reflects after every single action, not an agent that does no checking at all. Against the second one, this mechanism costs money.

An agent that repairs constantly can cost more than one that never checks anything, and it will look better on task success the whole time it is doing so. Budget repairs per task. Alert on the rate, not the total spend, or you will find out at the end of the month.

A redesign kills the skill and its falsifier together

The missing-control fault was caught 4 out of 4 times, which reads like good news until you ask what happens next. It is good news exactly once. The falsifier that catches it is written against the old page: the assign control should be here, and it is not. After the redesign, that falsifier is permanently true and the workflow is dead until someone rewrites the skill and its evidence together. A sustained spike in skill-scope repairs on one site is a schema-change alert with an owner's name on it, not flakiness to retry away.

A 4.1 that describes a mechanism, not a deployment

The scores below cover the commitment-testing layer: falsifiers, the fast matcher, the routing, the two test points. I reimplemented that part from the stated constants and it is the part I would deploy. The planning and repair libraries score lower and are not what the verdict covers.

The average comes out at 4.1, and the binding constraint is nowhere in it. Every number in this article rests on falsifiers that a human wrote for a workflow they already understood. The paper's planner generates them and does not publish how. Until a team measures its own generated falsifiers against hand-written ones on its own traces, the 4.1 describes a mechanism. I am not sure how far apart the mechanism and a deployment will turn out to be, and closing that gap is the experiment I would run first.

Evidence check first, policy gate second, a human for anything irreversible

Follow a request through it. The planner and the libraries are offline and cached. Per step, the fast matcher runs synchronously and costs microseconds. Only the verify route calls a model, and only where the two evidence scores are too close to separate: 7.4% of checks in my prototype.

The ordering is the design decision, and it is the part I would defend hardest. The evidence check runs before the policy gate, so a falsified commitment never reaches the question of whether the write is permitted. Reads pass on an unfalsified commitment alone. Writes need that and a policy decision the page had no part in. Anything irreversible, meaning purchase, delete, send, approve, or an update to a system of record, needs a named human every time until shadow evidence says otherwise.

Alongside the request path, an evidence log records every score, route and repair keyed to the step, and a kill switch parks a run for review when repairs on one workflow cluster. Both fail closed. A workflow that keeps needing repair is telling you the site changed.

From one read-only workflow to a permitted write

1. 01

   Bound one workflow with no writes

   Owner

   Platform

   Artifact

   One read-only browser workflow against one internal tool, with its commitments and falsifiers written by hand

   Signal

   The workflow completes end to end on 20 consecutive live runs

2. 02

   Freeze an evaluation set from real traces

   Owner

   Applied science

   Artifact

   150 recorded trajectories from the last month, labelled on-track or off-track with the first off-track step marked

   Signal

   Two labellers agree on the fault step in at least 90% of cases

3. 03

   Measure detection against your outcome check

   Owner

   Applied science

   Artifact

   A detection report: rate, median steps saved, false alarms, escalation share

   Signal

   Detection beats the outcome check by 20 points on steps saved, at under 10% false alarms

4. 04

   Add the policy gate and the evidence log

   Owner

   Platform and risk

   Artifact

   A write-classifying gate plus an append-only evidence log with step-level retention

   Signal

   No write executes without a logged unfalsified commitment, verified by replaying a week of traces

5. 05

   Run beside the current system on real traffic

   Owner

   Platform

   Artifact

   A shadow worker that tests commitments and logs verdicts without acting

   Signal

   Escalation stays under 15% of checks and false alarms under 5% for 10 consecutive days

6. 06

   Grant writes on one action class, or stop

   Owner

   Risk and platform

   Artifact

   A decision record naming the permitted action class, the approver, and the rollback path

   Signal

   Zero unreviewed writes to a system of record in the first 30 days

Before writes are granted, three gates have to pass. Detection saves at least 20 percentage points more wasted steps than the outcome check on the frozen trace set. Escalation stays under 15% of checks and the fast path adds under 50 ms per step at the 95th percentile. And every write path fails closed on a falsified commitment, an escalation timeout, or a missing evidence log entry.

Any one of these sends it back. False alarms above 10% of healthy runs for three consecutive days. More than a quarter of falsifiers silent for 30 days, which means the evidence has become decorative. Any write reaching a system of record without a logged commitment and a named approver.

The human review policy is the same throughout: every irreversible action requires named human approval before execution. Reads and reversible writes need review only when the run escalated or repaired. Approval requirements drop only when 30 days of shadow evidence shows zero unreviewed writes and a false-alarm rate under 5%.

Ship the tests, wait on the repair

Ship the commitment-testing layer for bounded internal browser workflows. Wait on the full planning and repair system.

The testing layer is unusually well suited to production for a result this recent. It is specified precisely enough that I reimplemented the routing with no dependencies and no model, it emits exactly the signals an operations team needs, and it makes the systems it replaces cheaper. The evidence log it produces is the closest thing to an audit trail I have seen come out of an agent paper.

The paper and my prototype agree on direction and disagree on emphasis. Its headline is task success; my measurement says the more valuable property is when the failure is detected, and 57.9% of catches landing before the browser changed is the number I would take to a risk review. Where they conflict I trust the paper on whether the mechanism improves outcomes and the prototype on what it costs to run, because those are the questions each one actually measured.

The path I would fund is narrow: one workflow, one internal tool, reads before writes, a policy gate the page cannot influence, and every irreversible action behind a human. I would not extend it to pages I do not control, and I would not treat an unfalsified commitment as permission to act. Commitment testing tells you the agent has not drifted. It cannot tell you the page is honest.

I would change this verdict if:

1. Model-generated falsifiers, measured against hand-written ones on the same traces, lose more than 10 points of detection. That would make the whole approach a human-authoring exercise and not an agent capability.
2. Escalation on live traffic exceeds 25% of checks, which would erase the cost argument that makes the fast path worth having.
3. A published replication shows the WebArena gain does not survive a second backbone, since every number in the paper comes from one model.

Three things to write down before the next agent acts

One experiment would settle most of my remaining doubt, and I cannot run it: model-generated falsifiers measured against hand-written ones on the same traces. I expect the generated ones to lose ground, because a human writing a falsifier already knows which mistake the workflow tends to make. If you have that comparison, or a state-diffing approach that catches the page that never changed, tell me and I will update this piece.

Acknowledgements

This article builds on Falsifiable Commitment Planning for Self-Correcting Web Agents by Guangyi Liu, Huan Zhao, and Quanming Yao. The routing constants, the four routes, the two test points and the three repair scopes are theirs; the errors in my reading of them are mine.

The prototype uses no third-party code. It is Python standard library only, under MIT.

Three papers shaped the boundaries of the argument, and are credited in full in the References below: Benchmarking Web Agent Safety under E-commerce Deceptive Interfaces by Zijing Shi, Meng Fang, and Ling Chen; Designing Agent-Ready Websites for AI Web Agents by Said Elnaffar and Farzad Rashidi; and Engineering Robustness into Personal Agents with the AI Workflow Store by Roxana Geambasu, Mariana Raykova, Pierre Tholoniat, Trishita Tiwari, Lillian Tsai, and Wen Zhang.

References

Primary research

• Falsifiable Commitment Planning for Self-Correcting Web Agents. Guangyi Liu, Huan Zhao, Quanming Yao, 2026 (arXiv:2607.24167, submitted 27 July 2026)

Supporting and contrary evidence

• Benchmarking Web Agent Safety under E-commerce Deceptive Interfaces. Bounds the claim: commitment testing checks the agent's drift, not the page's intent
• Designing Agent-Ready Websites for AI Web Agents. The competing explanation, with a larger effect size on much thinner evidence
• BaRA: Budget-constrained and Reliable Web Data Collection Agent. Supports the direction: verification checks inside the loop, with public code
• Engineering Robustness into Personal Agents with the AI Workflow Store. A position paper arguing for hardened, reusable workflows over on-the-fly synthesis

Prototype

• Source, setup, and evaluation
• Exact tested commit

Sources last checked: 2026-08-08