Browser agents fail quietly. Falsifiable commitments make them fail loudly

A browser agent that fails loudly costs you a retry. A browser agent that fails quietly costs you a wrong record in a system you have to trust.

The quiet failure is the common one. The agent opens ticket T-411 instead of T-390, reads a plausible complaint, assigns it to the right-sounding team, and saves. Every step looks locally reasonable. Nothing throws. The outcome check at the end sees a saved ticket and reports success. You find out six weeks later when someone asks why Payments owns an invoice request.

Most agent reliability work attacks this after the fact: reflect on the trajectory, score the final answer, retry on failure. FCPAgent inverts the order. Before a plan step runs, the agent writes down the evidence that would prove the step is still working, and the evidence that would prove it is not. Being on track stops being an assumption and becomes a statement you can check.

I rebuilt the checking half of that loop. On 24 deliberately broken trajectories it caught 79.2% with no language model in the loop, at zero false alarms and 0.044 ms per check, and 57.9% of the catches landed before the browser was modified. It also missed one whole class of failure completely, and I think that miss is the more useful result of the two.

This article is about where that line falls: which browser failures a pre-written falsifier catches cheaply, which ones it structurally cannot, and what has to sit around it before an agent may write to anything that matters.

What it does not do: reproduce FCPAgent's WebArena results, test the planning or repair halves of the system, or tell you anything about task success. It measures detection, and only detection.

The short version

• The idea: attach to each plan step the evidence that would confirm it and the evidence that would kill it, then test both against the live page.
• Why it might work: a contradiction is cheap to check and expensive to fake; a confidence score is the opposite.
• What I tested: falsifying evidence against the paper's own "confirming evidence only" ablation, on 40 hand-authored browser trajectories across four enterprise workflows.
• What happened: 79.2% of off-track trajectories detected against 0.0% for the ablation, zero false alarms, 7.4% of checks escalated, 0.044 ms per check.
• Where it breaks: an action that changes nothing contradicts nothing, so a silent no-op was never caught, by any configuration.
• Decision: ship the detection layer on one bounded internal workflow behind a policy gate; do not treat it as authorization.

Start with the problem

Take the support workflow from a moment ago. Find ticket T-390, confirm the customer reported a duplicate charge, assign it to Payments, save. Four steps, none of them hard.

A conventional browser agent runs a loop: observe, choose, execute, repeat. The check that matters arrives at the end, and it is almost always a question about the goal. Did we assign the ticket?

Look at what that question will accept. Yes, says the trajectory that assigned the wrong ticket. Yes, says the one that assigned it after the session quietly expired. Yes, says the one that clicked a button whose label changed meaning in a redesign three weeks ago. The check is satisfiable by accident, which makes it a metric rather than a test.

The paper's framing is that a plan step is a commitment, and a commitment nobody can contradict is not worth much. Each step therefore carries five things: a subgoal, an optional reusable skill, confirming evidence, falsifying evidence, and a confidence score. They call it a Falsifiable Commitment Unit.

The two kinds of evidence are organised differently, and the asymmetry is deliberate. Confirming evidence is staged: some of it holds before the step runs, some during, some once it is finished. Falsifying evidence is scoped, and the scope names what broke. Execution if the action or state drifted. Skill if the reused procedure no longer fits the page. Planning if the subgoal was wrong from the start.

Those scope labels are the part with production consequences, and they are not decoration. They tell the repair step how much work to throw away: retry the click, swap the procedure, rewrite the plan. Two seconds of recovery, or twenty steps from the top.

How the method works

1. Plan with falsifiers attached

The planner pulls three candidate skills and two past failure-repair cases out of offline libraries, then emits the commitment sequence. Skills supply the procedure template. The failure memories supply the falsifiers, on the reasonable theory that the cheapest source of "what could go wrong here" is what went wrong here last time. Both libraries come from 191 training tasks and are frozen at serving time, so none of this costs anything at request time.

2. Test cheaply, escalate rarely

Every step is scored by a fast matcher before the expensive one is allowed to speak. The paper combines a natural language inference model with an image-text matcher over the screenshot:

Two test points, not one. The action-level test scores the action the agent proposes, before it touches the browser. The state-level test scores the page that action produced. The first is the one that can stop a write.

3. Route on the margin, not the score

This is the part worth reading closely, and it is the part I reimplemented exactly:

def _route(self, a_plus, a_minus, confidence):
    t = self.thresholds
    # Safety first: a contradiction outranks a completion signal.
    if a_minus >= t.falsify_threshold(confidence) and (a_minus - a_plus) >= t.delta_minus:
        return REPAIR
    if a_plus >= t.confirm_threshold(confidence) and (a_plus - a_minus) >= t.delta_plus:
        return ADVANCE
    if max(a_plus, a_minus) < t.pi_low:
        return CONTINUE
    return VERIFY

Correctness check: test_conflicting_strong_signals_route_to_verify asserts that α+ = 0.9, α− = 0.9 returns VERIFY, and test_contradiction_outranks_confirmation asserts that α+ = 0.55, α− = 0.95 returns REPAIR rather than ADVANCE.

Two properties matter here. The falsifying branch is tested first, so a step can never be advanced on a page that also contradicts it. And no route fires on an absolute score alone: each needs a margin over the other side. The paper sets that margin at 0.18 for completion and 0.25 for drift. When both sides are strong, no branch wins, and the step escalates to the model rather than guessing.

What the research actually shows

The strongest result

A gain of that size against ColorBrowserAgent, rather than against a weak baseline, is what made me keep reading. It is also a benchmark number, and the interesting structure sits underneath it: the gain concentrates in long tasks. On Shopping tasks of one to five steps the improvement is 11%. On Shopping tasks of eleven steps or more it is 161%.

That shape is exactly what you would predict if the mechanism does what it says. A short task barely has time to go wrong. A long one drifts, and with no per-step contradiction check, nothing notices until the trajectory is over and the damage is written down.

Does the mechanism earn the credit?

The ablation is the reason I selected this paper.

A controlled comparison that isolates the exact component the paper is named after is rarer than it should be, and it is why I picked this paper over three others. It also made a small prototype viable: the claim is narrow enough to test without rebuilding the system around it.

The efficiency table matters just as much if you are the one paying for the inference:

Where the evidence is thin

The paper reports averages over three runs and no standard deviations, so a 2.0-point ablation delta has no stated uncertainty around it. I would say that makes the smallest of the three ablation numbers unusable and the largest one still convincing. Success is measured on WebArena and WebChoreArena, both simulated environments with stable page structure. The redesigns, interstitials and permission errors that break real automation are not in the sample. And the authors are direct about the mechanism's own weak point:

"A limitation of the current framework is that it depends on the quality of generated evidence: overly broad falsifiers may trigger unnecessary repairs, whereas overly narrow ones may miss off-track execution."

Why this paper earned the test

I screened four candidates published within the last three months against five questions: claim support, baseline strength, causal evidence, production transfer, and testability. Four is not a literature review, and one person chose both the four and the five questions, so read the table as a judgement rather than a survey.

Agent-Ready Websites deserves more than a table row, because it is the strongest challenge to this entire line of work. If changing a site's markup lifts agent success from 49.3% to 89.3%, then for any site you control, fixing the site beats instrumenting the agent and it is not close. So what is commitment testing for? The sites you do not control, which in an enterprise is most of the vendor tools your staff spend their day in.

Rebuilding the claim

Experimental setup

My impression going in was that the interesting half of FCPAgent is the cheap one, and building it confirmed that. Not reproducing the paper was a choice, not a limitation I ran into halfway through: running WebArena with a 397B backbone tests the whole system, and I wanted one premise tested cheaply enough that you can read the entire harness in an afternoon and tell me where I went wrong.

The corpus is 40 hand-authored trajectories across four enterprise workflows: support triage, expense approval, catalogue price update, order refund lookup. 250 steps in total, every observation written by hand to imitate a flattened accessibility tree. Sixteen are healthy. The other twenty-four each carry one injected fault, drawn from six types: wrong record opened, session dropped, stale filter, control missing after a redesign, a success banner for a different operation, and an action that changes nothing. Each plan carries four commitments, and I wrote all of them before inspecting a single trajectory.

git clone https://github.com/shravan1996/ship-the-paper-falsifiable-commitments
cd ship-the-paper-falsifiable-commitments
python3 -m unittest discover -s tests   # 37 tests
python3 eval.py

Python 3.9.6 on an Apple M-series laptop, no dependencies, no network, no GPU. One substitution matters: the matcher is inverse-document-frequency weighted content recall rather than the paper's entailment model, so what I am testing is the control loop and not the paper's accuracy. With no model in the loop, each arm gets reported twice. Once where escalations resolve as "keep going", which is a lower bound from the cheap matcher alone. Once where they always resolve correctly, which is an upper bound no real verifier will beat. The truth sits between them.

What I expected

Falsifiers would help. Pre-action testing would be the interesting part. The cheap matcher would be too blunt for the wrong-record case, since a near-miss identifier shares most of its tokens with the right one, and I expected to spend the write-up apologising for it.

What I did not expect, at all, was for the confirming-only baseline to detect nothing. Not "less". Nothing.

What happened

The outcome-check row is the one to sit with. It detects 100% of failures and saves zero steps, because it fires after the trajectory is over. Detection rate on its own is a misleading metric for this problem; when you detect is the whole point.

Three results changed how I would build this.

Cross-cutting failures need cross-cutting falsifiers. Session loss was caught in 1 of 4 workflows with step-level falsifiers, then 4 of 4 once I attached the same falsifier to the plan instead of a step. A three-line change moving detection from 25% to 100% on a whole fault class is a surprisingly large return, and it makes me suspect the paper left value on the table by scoping every falsifier to a single commitment. Obvious in hindsight: a session can drop on any step, so a falsifier written for step 2 never sees it happen on step 4. The paper scopes all falsifying evidence to a single commitment, which for environment failures is the wrong scope. The fix was three lines.

A page that does not change contradicts nothing. The silent no-op was caught 0 out of 4 times, in every arm, including the perfect-verifier ceiling that no real verifier can beat. I stared at this one for a while looking for a falsifier that would work, and there isn't one. I had assumed every failure mode has some observable fingerprint if you write the evidence statement carefully enough. That assumption is wrong, and this is the counterexample. The page is valid. Nothing on it contradicts the commitment. The agent is simply not progressing, and no evidence statement written over a single observation can see that. You need state diffing, and this mechanism does not give you any.

One near miss escalated instead of repairing, and it was right to. In the expense workflow the agent opened EXP-1207 instead of EXP-1180. The identity falsifier was fully satisfied at α− = 1.0, but "Expense report EXP-1207 detail" still matched most of the confirming evidence, leaving a margin of 0.2497 against a threshold of 0.25. Three ten-thousandths. That the margin rule fired was expected; that it fired this close to the line surprised me, and it is the single result in this prototype I would least trust to hold on a different corpus. The matcher declined to decide and escalated, which is the design working exactly as specified, and it is also why I report the fast path as a lower bound rather than as the number.

What breaks outside the experiment

The falsifiers are written by the thing being checked

In the paper, the planner that commits to a step also writes the evidence that could kill it. In my prototype I wrote both halves myself. Neither arrangement is a check by an independent party, and I do not have a good answer for that.

The failure mode is slow and quiet. Falsifiers drift toward statements that never fire, each technically true, none load-bearing, and detection decays while the dashboard stays green. So track fires-per-thousand-steps for every falsifier, not just the aggregate. One that has not fired in a month is describing something that cannot happen, or it is written too narrowly to notice when it does, and you cannot tell which from the aggregate.

Evidence checking is not authorization

An unfalsified commitment means one thing: no page content contradicts this step. It does not mean the agent is allowed to take it. Two different questions, and only one of them is about the page.

WebDecept evaluates multimodal web agents against seven classes of deceptive interface and finds them broadly susceptible, with prompt-based safeguards insufficient. Here is why that lands hard on this mechanism: a page written to mislead can satisfy every confirming check and trip no falsifier at all, because the falsifiers were written to catch the agent's mistakes rather than the page's intent. Commitment testing raises the floor on accidental failure and does approximately nothing about a hostile page.

My read is that the control here is ordering, not evidence quality. The evidence check runs first, the policy gate second, and the gate asks questions the page cannot answer: is this a write, is the target in scope, has a named human approved this class of action?

Detection is cheap; repair is where the cost returns

My prototype stops at detection, which is the cheap half. FCPAgent carries on into scope-aware repair, and repair calls the model. Note what the 19.5% saving in Table 5 is measured against: a baseline that reflects after every single action, not an agent that does no checking at all. Against the second one, this mechanism costs money rather than saving it.

An agent that repairs constantly can cost more than one that never checks anything, and it will look better on task success the whole time it is doing so. Budget repairs per task. Alert on the rate, not the total spend, or you will find out at the end of the month.

Site redesigns invalidate skills and falsifiers together

The missing-control fault was caught 4 out of 4 times, which reads like good news until you ask what happens next. It is good news exactly once. The falsifier that catches it is written against the old page: the assign control should be here, and it is not. After the redesign, that falsifier is permanently true and the workflow is dead until someone rewrites the skill and its evidence together. A sustained spike in skill-scope repairs on one site is a schema-change alert with an owner's name on it, not flakiness to retry away.

The production scorecard

The scores below cover the commitment-testing layer: falsifiers, the fast matcher, the routing, the two test points. That is the part I reimplemented from the stated constants and the part I would deploy. The planning and repair libraries score lower and are not what the verdict covers.

The average comes out at 4.1, and the binding constraint is nowhere in it. Every number in this article rests on falsifiers that a human wrote for a workflow they already understood. The paper's planner generates them and does not publish how. Until a team measures its own generated falsifiers against hand-written ones on its own traces, treat the 4.1 as describing a mechanism rather than a deployment. I am not sure how far apart those two numbers will turn out to be, and that is the experiment I would run first.

A production shape I would test

Read it as a request path. The planner and the libraries are offline and cached. Per step the fast matcher runs synchronously and costs microseconds. Only the verify route calls a model, and only where the two evidence scores are too close to separate: 7.4% of checks in my prototype.

The ordering is the design decision. The evidence check runs before the policy gate, so a falsified commitment never reaches the question of whether the write is permitted. Reads pass on an unfalsified commitment alone. Writes need that and a policy decision the page had no part in. Anything irreversible, meaning purchase, delete, send, approve, or an update to a system of record, needs a named human every time until shadow evidence says otherwise.

Two things run continuously and fail closed: an evidence log recording every score, route and repair keyed to the step, and a kill switch that parks a run for review when repairs on one workflow cluster. A workflow that keeps needing repair is telling you the site changed.

From prototype to shadow traffic

1. 01

   Bound one workflow with no writes

   Owner

   Platform

   Artifact

   One read-only browser workflow against one internal tool, with its commitments and falsifiers written by hand

   Signal

   The workflow completes end to end on 20 consecutive live runs

2. 02

   Freeze an evaluation set from real traces

   Owner

   Applied science

   Artifact

   150 recorded trajectories from the last month, labelled on-track or off-track with the first off-track step marked

   Signal

   Two labellers agree on the fault step in at least 90% of cases

3. 03

   Measure detection against your outcome check

   Owner

   Applied science

   Artifact

   A detection report: rate, median steps saved, false alarms, escalation share

   Signal

   Detection beats the outcome check by 20 points on steps saved, at under 10% false alarms

4. 04

   Add the policy gate and the evidence log

   Owner

   Platform and risk

   Artifact

   A write-classifying gate plus an append-only evidence log with step-level retention

   Signal

   No write executes without a logged unfalsified commitment, verified by replaying a week of traces

5. 05

   Run beside the current system on real traffic

   Owner

   Platform

   Artifact

   A shadow worker that tests commitments and logs verdicts without acting

   Signal

   Escalation stays under 15% of checks and false alarms under 5% for 10 consecutive days

6. 06

   Grant writes on one action class, or stop

   Owner

   Risk and platform

   Artifact

   A decision record naming the permitted action class, the approver, and the rollback path

   Signal

   Zero unreviewed writes to a system of record in the first 30 days

Ship gates

• Detection saves at least 20 percentage points more wasted steps than the outcome check on the frozen trace set.
• Escalation stays under 15% of checks and the fast path adds under 50 ms per step at the 95th percentile.
• Every write path fails closed on a falsified commitment, an escalation timeout, or a missing evidence log entry.

Kill criteria

• False alarms exceed 10% of healthy runs for three consecutive days.
• More than a quarter of falsifiers have not fired in 30 days, indicating decorative evidence.
• Any write reaches a system of record without a logged commitment and a named approver.

Human review policy: every irreversible action requires named human approval before execution. Reads and reversible writes need review only when the run escalated or repaired. Approval requirements drop only when 30 days of shadow evidence shows zero unreviewed writes and a false-alarm rate under 5%.

My verdict

Ship the commitment-testing layer for bounded internal browser workflows. Wait on the full planning and repair system.

The testing layer is unusually well suited to production for a result this recent. It is specified precisely enough that I reimplemented the routing with no dependencies and no model, it emits exactly the signals an operations team needs, and it makes the systems it replaces cheaper rather than taxing them. The evidence log it produces is the closest thing to an audit trail I have seen come out of an agent paper.

The paper and my prototype agree on direction and disagree on emphasis. Its headline is task success; my measurement says the more valuable property is when the failure is detected, and 57.9% of catches landing before the browser changed is the number I would take to a risk review. Where they conflict I trust the paper on whether the mechanism improves outcomes and the prototype on what it costs to run, because those are the questions each one actually measured.

The path I would fund is narrow: one workflow, one internal tool, reads before writes, a policy gate the page cannot influence, and every irreversible action behind a human. I would not extend it to pages I do not control, and I would not treat an unfalsified commitment as permission to act. Commitment testing tells you the agent has not drifted. It does not tell you the page is honest.

I would change this verdict if:

1. Model-generated falsifiers, measured against hand-written ones on the same traces, lose more than 10 points of detection. That would make the whole approach a human-authoring exercise rather than an agent capability.
2. Escalation on live traffic exceeds 25% of checks, which would erase the cost argument that makes the fast path worth having.
3. A published replication shows the WebArena gain does not survive a second backbone, since every number in the paper comes from one model.

What to remember

One experiment would settle most of my remaining doubt, and I cannot run it: model-generated falsifiers measured against hand-written ones on the same traces. I expect the generated ones to lose ground, because a human writing a falsifier already knows which mistake the workflow tends to make. If you have that comparison, or a state-diffing approach that catches the silent no-op I could not, tell me and I will update this piece.

Acknowledgements

This article builds on Falsifiable Commitment Planning for Self-Correcting Web Agents by Guangyi Liu, Huan Zhao, and Quanming Yao. The routing constants, the four routes, the two test points and the three repair scopes are theirs; the errors in my reading of them are mine.

The prototype uses no third-party code. It is Python standard library only, under MIT.

Three papers shaped the boundaries of the argument, and are credited in full in the References below: Benchmarking Web Agent Safety under E-commerce Deceptive Interfaces by Zijing Shi, Meng Fang, and Ling Chen; Designing Agent-Ready Websites for AI Web Agents by Said Elnaffar and Farzad Rashidi; and Engineering Robustness into Personal Agents with the AI Workflow Store by Roxana Geambasu, Mariana Raykova, Pierre Tholoniat, Trishita Tiwari, Lillian Tsai, and Wen Zhang.

References

Primary research

• Falsifiable Commitment Planning for Self-Correcting Web Agents. Guangyi Liu, Huan Zhao, Quanming Yao, 2026 (arXiv:2607.24167, submitted 27 July 2026)

Supporting and contrary evidence

• Benchmarking Web Agent Safety under E-commerce Deceptive Interfaces. Bounds the claim: commitment testing checks the agent's drift, not the page's intent
• Designing Agent-Ready Websites for AI Web Agents. The competing explanation, with a larger effect size on much thinner evidence
• BaRA: Budget-constrained and Reliable Web Data Collection Agent. Supports the direction: verification checks inside the loop, with public code
• Engineering Robustness into Personal Agents with the AI Workflow Store. A position paper arguing for hardened, reusable workflows over on-the-fly synthesis

Prototype

• Source, setup, and evaluation
• Exact tested commit

Sources last checked: 2026-08-08

A method is ready for production traffic when it can tell you it is wrong before you find out from a customer.