Automated testing

Automated scanners are fast, repeatable and cheap enough to run on every commit. They are also blind to most of WCAG. Both halves of that sentence matter: teams that skip automation ship defects they could have caught for free, and teams that stop at automation ship inaccessible sites with a green badge.

What this repository's own test run reports

Run it yourself:

npm install
npm run test:a11y

The run asserts three things for every one of the 86 criteria:

pass.html produces zero violations.
fail.html produces at least one violation, and it is a rule that genuinely maps to that criterion. A demo that quietly stops failing is reported as loudly as a broken fix.
Where the failure is invisible to the scanner, the run says so explicitly, and the criterion's page explains why.

The summary it prints is the honest coverage map:

Criteria in WCAG 2.2                         86
With at least one mapped axe rule            31  (36%)
No automated test exists at all              55  (64%)

The number behind the number

36% of criteria have some rule mapped to them. That is not the same as 36% of accessibility problems being detectable. Within those 31 criteria the coverage is partial: a scanner sees that alt is missing, never that the alt text is wrong; that a label is absent, never that it is meaningless; that contrast fails on a solid background, rarely over an image.

Independent studies of automated tooling consistently put real-world detection at roughly a quarter to a third of actual defects. Plan your testing budget on that basis.

The three outcomes axe reports, and why the third one matters

How to read an axe result
Outcome	What it means	What you do
violations	The rule failed, with confidence	Fix it
passes	The rule was checked and satisfied	Nothing — but this is not proof the criterion is met
incomplete	The rule fired but axe will not decide. Also called "needs review"	Check it by hand. This is not a pass

Most CI setups report only violations and silently discard incomplete. That is how a page with no skip link, no landmarks and no headings is recorded as clean — 2.4.1 Bypass Blocks returns incomplete, not a violation. This repository's runner reports it as a distinct outcome for exactly that reason.

Traps that make automated results misleading

Scanning one state of the page. A form with excellent error handling and a form with none look identical until you submit it. Drive the page into each state and scan again.
Scanning inside iframes by accident. axe descends into same-origin frames by default. This suite has to pass .exclude('iframe') or every deliberate failure would be reported against the page teaching about it.
Best-practice rules mixed with WCAG rules. Some genuinely useful rules — such as tabindex, which catches positive tabindex values — are tagged best-practice, so a WCAG-scoped scan never runs them.
Experimental rules disabled by default. label-content-name-mismatch is exactly the rule for 2.5.3, and it is off unless you opt in.
Testing only the components, not the pages. A component library can be flawless and the assembled page still fail on heading order, landmarks, focus order and duplicate ids.

The tools

Automated accessibility tools and what each is for
Tool	Engine	Best for
axe DevTools	axe-core	Browser extension. The fastest way to check a page while building it.
axe-core	axe-core	The library. Use it in unit and end-to-end tests, where you can drive the page into each state first.
pa11y-ci	HTML_CodeSniffer or axe	Crawling many URLs in CI. A second engine cross-checks the first.
Lighthouse	axe-core (subset)	A quick score. Its accessibility number is a subset of axe and should never be treated as a conformance measure.
WAVE	Own	Visual, in-page annotation. Excellent for teaching, because it shows you where on the page each issue is.
ANDI	Own	A bookmarklet from the US SSA. Good for inspecting the accessible name of a specific control.

Wiring it into CI properly

Fail the build on violations, but also surface incomplete as output a human reads.
Scan the states, not just the routes: logged out and in, empty and populated, valid and error, dialogs open.
Pin the axe-core version. New rules arriving on a Friday afternoon should be a decision, not a surprise.
Never let a green pipeline be described as "accessible". It means "no machine-detectable defects found", and the difference is most of WCAG.