Verification method
How a criterion becomes evidence.
A verdict is only as good as the check behind it, and a check is only as good as what it kept. This page follows a criterion from the issue to a stored result, then publishes the aggregate numbers and their limits.
Observer
A criterion becomes a set of checks
Acceptance criteria live on the issue. Observer reads them there and plans a check against each one: the step to perform, the outcome the criterion describes, and the observation that would show the criterion violated rather than merely unverified.
The application is exercised as a black box — a browser and an HTTP client against a running deployment. No source access is required and none is used, so a check is written against what a user can actually reach.
Every criterion also gets the ways the application is supposed to refuse. Those categories are enumerated against the criterion rather than applied as one fixed list, and a system that accepts what it should reject is a finding, not a pass:
- Invalid input — empty, malformed, or the wrong shape.
- Boundary values — one below the limit, one above it, and exactly at it.
- Security payloads — input a form is not supposed to store or execute.
- Error and session states — an expired token, an invalid session, a revoked credential.
- Repeated submission — the same action sent twice in quick succession.
- State violation — a step skipped, a direct URL, the back button.
- Authorization — no credential, the wrong role, an expired session.
Two limits are stated rather than hidden: a black-box run cannot inject a downstream fault such as a service going down or a timeout, and it cannot force a true race. A check that could not run — because it needs an answer a headless run will not give, or because the target could not be reached — is recorded as blocked, and a blocked check is never counted as a pass.
Observer
What a check keeps
A check is a record, not a status word. It keeps the step that was performed and the input it carried, the response that came back, the capture taken while it ran, the timestamp, and the run it belonged to — and a report reads from those records rather than from a retelling of them.
A finding is filed against the exact criterion it violates, with its evidence attached at the moment it is written. A reader can follow a claim back to the observation that produced it without asking the person who ran the check.
- The step, and the input it carried.
- The response — the status and body of an HTTP exchange, or the page’s accessibility tree after a browser step.
- The capture taken at the time: a screenshot, or a checkpoint recorded for later comparison.
- The timestamp, and the run the check belonged to.
A check that fails once and passes on a repeat does not become a pass. The attempts are kept on the check and the verdict is derived from all of them, so an unstable check reads as unstable.
The parts that are not a judgment
Four words carry severity — critical, error, warning and info — and they are the vocabulary a report and a tracker field share. A claim at the alarm level is checked against that check’s own recorded evidence before the finding is written, because the severity prints in a client-facing document as a priority word and is not a field that can be quietly corrected afterwards.
Saved scenarios double as snapshot gates. A replay captures a checkpoint per step — the accessibility tree after a navigation, the status and body of an exchange — and compares it against an approved baseline. A first run records a provisional baseline instead of inventing a pass, and only a difference from an approved baseline turns the gate red.
A visual difference is measured against a budget rather than treated as a regression on sight: a change under a tenth of a percent of the image is not one by default. A second statistic is read beside the changed-pixel count — the mean per-channel difference — because a change spread thinly across a page can move a large number of pixels by very little.
In the pipeline posture the snapshot gate is strict: a replay that captured checkpoints but could not reach the baseline registry fails the run, rather than reporting a green one with the gate effectively off.
The verdict is arithmetic over the checks
A criterion ends a run in one of four states: passed, failed, blocked or ambiguous. A blocked result means the check could not run, and it is never counted as a pass; an ambiguous one means the behaviour might be intended, and it is flagged for a person. The report states the counts it holds, and the verdict is read from those counts rather than formed separately from them.
A dismissal is a record too. A finding that does not hold up stays in the report, marked dismissed, and remains visible there instead of disappearing from the next render. The count of findings that did not survive review is published beside the count that did, which is the point of publishing either.
The numbers
Every figure on this page is an aggregate, and two things are deliberately absent from it.
There is no row per application. Seven applications are covered, one of them is a bank whose engagement was private, and a per-application row would identify that work by arithmetic alone. There is also no split by model provider: the provider split is not what a reader is being asked to trust. The checks, and the evidence each one kept, are what carry a verdict.
| Measure | Value |
|---|---|
| Runs executed | 218 (202 against real applications, 16 fixture or self-test) |
| Applications covered | 7 |
| Findings filed | 197 |
| Confirmed by human review | 118 |
| Explicitly marked false positives | 2 |
| Total model spend | 2.72 US dollars |
Measured over 2026-08-19 to 2026-09-02, 14 days, from the QA server’s own store — its findings, runs, cost ledger and verification checks — and pulled on 2026-09-02.
What the numbers do not prove
- One reviewer confirmed the findings, so there is no inter-rater check. A second person working the same queue could have confirmed a different number.
- Only one of the applications had a pre-existing defect list to measure detection against.
- The other applications are measured by review outcome — a person decided each finding stood up — rather than by recall against a known list of what was broken.
- Rejections include near-identical findings from a later run that were consolidated into the earlier one. That is a cost of review rather than a claim about the application that proved wrong, and those findings are counted as rejections rather than as false positives.