Web Accessibility Evaluation
- Introduced by
- Ben Caldwell, Michael Cooper, Loretta Guarino Reid & Gregg Vanderheiden (W3C WCAG Working Group)
- Year
- 2008
- Type
- Conformance-assessment pipeline for digital accessibility
- Family
- Disability studies / digital accessibility
What does it do?
Web accessibility evaluation assesses how well digital content conforms to the Web Content Accessibility Guidelines so that it can be used by people with a wide range of disabilities.
WCAG 2.0, published as a W3C Recommendation in 2008 by Caldwell, Cooper, Reid, and Vanderheiden, organizes requirements under four principles—content must be Perceivable, Operable, Understandable, and Robust (POUR)—each broken into testable success criteria graded at conformance levels A, AA, and AAA. A rigorous evaluation combines three complementary methods: automated tools that scan for machine-detectable issues, manual expert inspection against the success criteria, and testing with assistive technologies such as screen readers and keyboard-only navigation. The evaluation determines whether the content satisfies all success criteria up to a target level and yields a conformance claim. Because no single method catches every barrier, the strength of the assessment lies in layering all three.
The intuition behind the method
A website or app is just code until a person tries to use it—and people use the web in radically different ways. Someone who is blind hears the page through a screen reader and never sees the layout; someone with a motor impairment may navigate entirely by keyboard or switch and never touch a mouse; someone with low vision magnifies everything or relies on high contrast; someone with a cognitive disability needs clear language and predictable behavior. WCAG turns 'is this usable by all of them?' into a checkable set of requirements grouped under four common-sense principles: can people perceive the content (is there alt text, captions, sufficient contrast?), can they operate it (does everything work by keyboard, is there enough time?), can they understand it (is it readable and predictable?), and is it robust (will it work with assistive technologies now and in future?). Evaluation is the disciplined process of checking these. Crucially, you cannot just run a scanner: automated tools catch only a fraction of issues, so an expert must inspect the rest, and you must actually try the page with a screen reader and a keyboard to know whether a real user could get through.
History
Web accessibility standardization began at the World Wide Web Consortium's Web Accessibility Initiative, which published WCAG 1.0 in 1999. WCAG 2.0, released as a W3C Recommendation in December 2008 and authored by Ben Caldwell, Michael Cooper, Loretta Guarino Reid, and Gregg Vanderheiden, marked a major shift: it reorganized requirements under the four POUR principles, expressed them as technology-agnostic, testable success criteria graded at levels A, AA, and AAA, and decoupled the requirements from specific techniques. This testability made WCAG 2.0 the reference point for accessibility laws and policies worldwide and the basis for conformance evaluation methodologies that combine automated, expert, and assistive-technology testing. Later updates (WCAG 2.1 and 2.2) extended the criteria, particularly for mobile, low vision, and cognitive needs, but they build directly on the 2.0 structure that established the modern evaluation framework.
When to use it
Use web accessibility evaluation whenever digital content—websites, web applications, documents, or software interfaces—must be usable by disabled people or must meet legal, procurement, or policy requirements that reference WCAG. It is the appropriate method when you need a defensible conformance claim at a stated level (most often Level AA), when remediating an existing product, or when verifying that new development meets accessibility acceptance criteria. The approach assumes you can sample representative pages and states and that you have access to automated tools and assistive technologies for testing. It is less suited to evaluating physical environments (use an accessibility audit), to appraising broad inclusive-design quality beyond conformance (universal design evaluation), or to arranging an individual adjustment for a specific person (reasonable accommodation assessment). Even a perfect conformance result should be complemented by usability testing with disabled users for the most important journeys.
Limitations and considerations
- Conformance is technology-conformance, not a guarantee of usability; content can pass every success criterion yet still be hard for disabled users, so user testing remains necessary.
- Automated tools cover only a minority of success criteria, so over-reliance on scanners produces a false sense of accessibility.
- Sampling means untested pages and states may harbor barriers the evaluation never sees.
- Manual judgments (for example, whether alternative text is adequate) vary between evaluators without shared rubrics and experienced reviewers.
Source conditions do not replace independent scientific review or validation by an analysis engine.
Strengths
- Tests against a stable, internationally adopted standard with testable success criteria, yielding precise, comparable, and legally referenced conformance claims.
- Layers automated, manual-expert, and assistive-technology methods so that machine-detectable and judgment-dependent barriers are both caught.
- Organizes requirements under the intuitive POUR principles, giving teams a shared structure for diagnosing and discussing accessibility.
- Produces both a conformance determination and a prioritized, evidence-backed remediation list that feeds directly into development.
Applications
- Establishing a defensible WCAG conformance claim for a website or web application, typically at Level AA.
- Auditing existing digital products to find and prioritize accessibility barriers for remediation.
- Verifying that new features meet accessibility acceptance criteria before release as part of a development workflow.
- Supporting procurement and compliance by evaluating whether software or content meets policy-mandated accessibility standards.
- Evaluating documents, design systems, and component libraries so accessibility is built in at the source rather than patched per page.
How it works
How it works
The evaluation starts by fixing what will be tested and to what standard. Because exhaustively testing every page of a large site is impractical, the evaluator selects a representative sample—key templates, high-traffic pages, and critical user journeys with their interactive states such as forms, errors, and dialogs—so that the sample exercises the range of content types and components. At the same time the target WCAG version and conformance level are chosen, most commonly Level AA, since that is the level most laws and policies reference. Defining scope and target level up front determines exactly which success criteria apply and makes the resulting conformance claim precise and verifiable.
Formula
\text{Scope} = (W,\,L),\quad W = \{\text{representative pages/states}\},\ L\in\{A, AA, AAA\}\ \text{target level}Automated tools are then run across the in-scope pages to catch machine-detectable problems quickly—missing alternative text, empty form labels, low color contrast, invalid markup, and similar issues. Automated scanning is fast and reproducible and is excellent for surfacing high-volume, mechanical failures, but its coverage is fundamentally partial: it can reliably test only a minority of WCAG success criteria, because many criteria require human judgment about meaning, context, or equivalence. Treating the automated pass as a first sweep rather than the whole evaluation is essential; its real value is clearing obvious defects so expert effort concentrates on what tools cannot decide.
Formula
A_{\text{auto}} = \bigcup_{w\in W} \text{tool}(w) \subseteq \text{SC},\quad \text{coverage} = \frac{|A_{\text{auto}}|}{|\text{SC}|} < 1An expert then inspects the content against the success criteria organized under the four POUR principles, exercising the judgment that automation lacks. This is where the bulk of WCAG is actually tested: whether alternative text genuinely conveys the meaning of an image, whether reading and navigation order are logical, whether form errors are clearly described, whether content is keyboard-operable without traps, and whether name, role, and value are correctly exposed for custom components. Working principle by principle and criterion by criterion ensures full coverage of what applies at the target level, and the expert records each success criterion as pass, fail, or not applicable with the evidence behind the verdict.
Formula
\text{POUR} = \{\text{Perceivable},\text{Operable},\text{Understandable},\text{Robust}\},\ \ \text{SC} = \bigcup_{p\in\text{POUR}} \text{SC}_pExpert inspection is corroborated by operating the content with the assistive technologies real users rely on—screen readers, keyboard-only navigation, screen magnification, and speech input among them. This step verifies not just that markup is technically correct but that the experience actually works: that a screen-reader user can perceive and act on every element, that all functionality is reachable and usable by keyboard without traps, and that interactive widgets announce and behave as expected. Assistive-technology testing frequently exposes barriers that pass on paper, which is why a success criterion is treated as met only when the content is genuinely usable across the relevant assistive technologies.
Formula
\text{pass}(w, sc) = \bigwedge_{at\in AT} \text{usable}(w, sc, at),\quad AT = \{\text{screen reader}, \text{keyboard}, \text{magnifier}, \dots\}Finally, the results are synthesized into a conformance determination. The content conforms at a given level only if every applicable success criterion at that level and below is satisfied across all the pages and processes in scope—conformance is all-or-nothing per level, so a single unmet Level A or AA criterion defeats a Level AA claim. The evaluator documents the level achieved, the criteria that failed, and the evidence, producing both a defensible conformance statement and a prioritized list of issues to remediate. Re-evaluating after fixes confirms that the target level is actually reached, making the process iterative rather than a one-off snapshot.
Formula
\text{Conform}(L) \iff \forall sc\in \text{SC}_{\le L},\ \forall w\in W:\ \text{pass}(w, sc)Common pitfalls
- Treating an automated scan as a complete evaluation, when tools can reliably test only a fraction of WCAG success criteria.
- Claiming conformance at a level while one or more applicable criteria fail—conformance is all-or-nothing per level, not a percentage.
- Skipping assistive-technology testing, so content that is technically correct but unusable with a screen reader or keyboard passes undetected.
- Sampling unrepresentatively—testing only static marketing pages and missing interactive forms, dialogs, and error states where most barriers live.
Frequently asked questions
Can't an automated accessibility checker do the whole evaluation?
No. Automated tools are valuable for quickly catching machine-detectable issues like missing alternative text, empty labels, low contrast, and invalid markup, and they scale across many pages. But they can reliably test only a minority of WCAG success criteria, because most criteria depend on human judgment—whether alt text actually conveys an image's meaning, whether reading order is logical, whether a custom widget is operable and correctly announced. Relying on a scanner alone gives a false sense of accessibility. A sound evaluation uses automated checks as a first sweep, then adds manual expert inspection and testing with assistive technologies.
What do the POUR principles and the A/AA/AAA levels mean?
POUR is the four principles WCAG organizes around: content must be Perceivable (users can sense it, e.g., text alternatives and contrast), Operable (users can interact with it, e.g., keyboard access and enough time), Understandable (readable and predictable), and Robust (works with current and future assistive technologies). Each principle contains success criteria assigned a conformance level—A (essential), AA (addresses major barriers and the common legal target), and AAA (the highest, not expected for whole sites). Conforming at a level requires meeting every applicable criterion at that level and below; AA conformance therefore includes all A criteria too.
Does passing WCAG mean the content is actually usable by disabled people?
Not necessarily. WCAG conformance is a strong, testable baseline and is what laws and policies reference, but it certifies technical conformance to success criteria rather than a good experience. Content can satisfy every criterion and still be confusing, inefficient, or frustrating for real users. That is why a thorough evaluation includes testing with the assistive technologies people actually use, and why conformance work is best complemented by usability testing with disabled participants on the most important journeys. Conformance establishes the floor; lived usability confirms whether the content truly works.
Related methods
Sources
Caldwell, B., Cooper, M., Reid, L. G., & Vanderheiden, G. (2008). Web Content Accessibility Guidelines (WCAG) 2.0. W3C Recommendation.
Open publication referenceCopied.