Skip to content
Back to search results

Acceptability Judgment Task

Introduced by
Experimental-syntax researchers (Jon Sprouse; Ellen Bard, Antonella Sorace)
Year
1996
Type
Quantified rating of sentence well-formedness
Family
Experimental syntax
Font size

What does it do?

The acceptability judgment task is the modern, quantified successor to informal grammaticality judgments: instead of a single linguist marking a sentence grammatical or not, many participants rate carefully controlled sentences on a graded scale, and the ratings are analyzed statistically.

Built on factorial designs with fillers and counterbalancing, and on response formats from Likert scales to magnitude estimation to forced choice, it turns intuition into replicable, gradient data. The approach anchors the experimental-syntax program associated with Jon Sprouse and colleagues, which tests grammatical hypotheses with the same methodological rigor as psycholinguistic experiments.

The intuition behind the method

Well-formedness is rarely all-or-nothing — some sentences are perfect, some are word salad, and a great many sit somewhere in between, more or less odd. A binary grammatical/ungrammatical verdict throws that gradient away, and a single linguist's verdict cannot tell a robust contrast from a fragile one. The acceptability judgment task keeps the gradient and adds numbers: ask dozens of naive speakers to rate matched sentences, design the materials so that a single factor varies, and the size and reliability of the rating difference become a measurable effect you can test, plot, and replicate — exactly the move that brought syntax into the experimental fold.

History

Discontent with single-linguist introspection and Carson Schütze's 1996 critique of the empirical base of grammaticality judgments set the stage for a quantified turn. Bard, Robertson, and Sorace's 1996 Language paper imported magnitude estimation from psychophysics, arguing that acceptability is continuous and should be measured on a ratio scale. Through the 2000s, Jon Sprouse and colleagues built the experimental-syntax program, and their 2013 Lingua study tested a random sample of contrasts from a decade of Linguistic Inquiry, finding that formal experiments reproduce the great majority of informal judgments while supplying effect sizes and flagging the unreliable cases. The rise of online platforms made large-sample collection routine, cementing the acceptability judgment task as the field standard.

When to use it

Use an acceptability judgment task when you want quantified, replicable evidence about gradient well-formedness and the size and reliability of a grammatical contrast — the standard for theoretical syntax claims, for adjudicating between competing analyses, and for cross-linguistic or cross-population comparisons. It is the right instrument when a contrast is subtle or contested, when you need effect sizes and statistics, or when reviewers expect formal confirmation of an intuition. It is less necessary for sharp, uncontroversial contrasts where informal judgments already converge, and it does not measure processing — for that, pair it with reading-time or eye-tracking methods.

Limitations and considerations

  • Like all judgments, it taps acceptability — grammar plus performance — so processing and plausibility still color the ratings.
  • Requires many participants, well-built materials, and statistical expertise, making it slower and costlier than informal judgments.
  • Scale choice (Likert, magnitude estimation, forced choice) affects sensitivity and comparability across studies.
  • Cannot localize the source of an effect in real time; it measures the endpoint, not the moment-by-moment processing difficulty.

Source conditions do not replace independent scientific review or validation by an analysis engine.

Strengths

  • Captures gradient acceptability and quantifies the size of a contrast, not just its direction.
  • Replicable and statistically testable, with effect sizes, confidence, and crossed random effects for participants and items.
  • Factorial designs with fillers and counterbalancing control confounds and mask the target, reducing strategic responding.
  • Largely validates the traditional syntax literature while objectively flagging the minority of unreliable contrasts.

Applications

  • Testing whether a proposed syntactic constraint produces a reliable, sized acceptability contrast.
  • Adjudicating between competing grammatical analyses that predict different acceptability patterns or interactions.
  • Quantifying island effects, superiority, agreement attraction, and other graded syntactic phenomena.
  • Comparing native speakers, second-language learners, or clinical groups on the same controlled materials.
  • Validating or re-examining contested judgments from the theoretical literature with formal experiments.

How it works

How it works

1

Materials are built as a factorial design: the structure of interest is decomposed into crossed factors (for example, presence of an island and presence of a long-distance dependency), and multiple lexically distinct items are written for each condition. Items are distributed across lists by a Latin square so that no participant sees two versions of the same item, and a large set of unrelated filler sentences spanning the full acceptability range is added to mask the design and anchor the scale.

2

A response scale is selected. Common choices are an n-point Likert scale, a binary or forced-choice comparison between conditions, and magnitude estimation, in which participants assign each sentence a number proportional to its acceptability relative to a reference modulus. Bard, Robertson, and Sorace introduced magnitude estimation precisely to capture the continuous, ratio-scaled nature of acceptability; later work weighs its costs against simpler Likert and forced-choice formats that prove highly sensitive in practice.

3

Participants — typically dozens per study, often recruited online — rate each sentence after practice and instructions that distinguish acceptability ('does this sound natural?') from prescriptive correctness or meaning. Randomized presentation, interleaved fillers, and attention checks keep the target contrast non-transparent. As Schütze emphasizes, the framing of the instructions and the composition of the fillers materially affect the ratings and must be reported.

4

Ratings are z-score transformed (to remove scale-use differences across participants) and analyzed with mixed-effects models that include participant and item as crossed random effects, testing whether the predicted condition differences and interactions are reliable. Sprouse and colleagues showed that formal experiments largely confirm the traditional informal judgments in the syntax literature while quantifying effect sizes and flagging the minority of contested cases, putting the inferential weight on a measured, replicable difference rather than an analyst's verdict.

Common pitfalls

  • Too few fillers, or fillers that do not span the full acceptability range, so participants cannot calibrate the scale and the target stands out.
  • Failing to z-score or otherwise normalize ratings, so individual differences in scale use masquerade as condition effects.
  • Analyzing with by-participant ANOVA only and ignoring item variability, inflating false positives that crossed random effects would catch.
  • Confusing the scale with prescriptive correctness in the instructions, so participants rate 'proper' usage rather than naturalness.

Frequently asked questions

How does the acceptability judgment task differ from the older grammaticality judgment task?

They share a goal but differ in rigor and ontology. The grammaticality judgment task asks for a binary grammatical/ungrammatical verdict and was historically given by one linguist about the abstract grammar (competence). The acceptability judgment task asks many naive participants for graded ratings of how acceptable a sentence sounds, over factorially designed materials with fillers, and analyzes the ratings statistically. Because what people actually report is acceptability — grammar plus performance — the modern task is named accordingly, and grammaticality is inferred from the measured pattern.

Is this the same as the NLP 'linguistic acceptability' task?

No. The ScholarGate method 'linguistic-acceptability' refers to the computational NLP task in which a machine-learning classifier (trained, for example, on the CoLA benchmark) predicts whether a sentence is acceptable. The acceptability judgment task here is a behavioral, experimental method in which human participants rate sentences and researchers test grammatical hypotheses statistically. They are related in spirit — both about sentence well-formedness — but one models human judgments with an algorithm while the other elicits and analyzes the human judgments themselves.

Which response scale should I use — Likert, magnitude estimation, or forced choice?

It depends on the contrast and the goal. Magnitude estimation, introduced by Bard, Robertson, and Sorace, treats acceptability as a continuous ratio scale and can capture fine gradience, but it is demanding for participants and its ratio assumptions are debated. Likert scales are simple and, in method-comparison studies, prove highly sensitive. Forced-choice (two-alternative) tasks maximize statistical power for a single targeted contrast. Many experimental syntacticians now default to Likert or forced choice, reserving magnitude estimation for questions specifically about fine-grained gradience.

Related methods

Sources

  1. Sprouse, J., Schütze, C. T., & Almeida, D. (2013). A comparison of informal and formal acceptability judgments using a random sample from Linguistic Inquiry 2001–2010. Lingua, 134, 219–248.

    Open DOI reference
    Copied.
  2. Bard, E. G., Robertson, D., & Sorace, A. (1996). Magnitude estimation of linguistic acceptability. Language, 72(1), 32–68.

    Open DOI reference
    Copied.
  3. Schütze, C. T. (2016). The Empirical Base of Linguistics: Grammaticality Judgments and Linguistic Methodology. Language Science Press.

    Open DOI reference
    Copied.