Grammaticality Judgment Task
- Introduced by
- Noam Chomsky (generative-linguistics tradition)
- Year
- 1965
- Type
- Introspective elicitation of linguistic competence
- Family
- Experimental syntax / second-language acquisition
What does it do?
The grammaticality judgment task asks speakers to decide whether a sentence is grammatical — well-formed according to the rules of their language — and treats that decision as evidence about the mental grammar that produces it.
Rooted in Noam Chomsky's generative program, where the native speaker's intuition is the primary data of linguistics, the task ranges from a single linguist consulting their own intuitions to large controlled experiments with binary, scaled, or forced-choice responses. It is a workhorse of syntactic theory and of second-language acquisition research, where it probes what learners know about a target language beyond what they can produce.
The intuition behind the method
Speakers cannot recite the rules of their grammar, yet they reliably feel that 'the cat sat on the mat' is fine while 'sat the cat mat on the' is not — and even that 'who do you think that left?' is somehow off in a way that 'who do you think left?' is not. That feeling of well-formedness is a window onto knowledge that is otherwise invisible: production data show only what speakers happen to say, but judgments reveal what the grammar permits and forbids, including sentences too rare or too complex ever to appear spontaneously. The grammaticality judgment task simply asks speakers to report that feeling so the analyst can reconstruct the rules behind it.
History
The grammaticality judgment task is inseparable from Chomsky's 1965 Aspects of the Theory of Syntax, which made the idealized native speaker's intuition the empirical foundation of generative grammar and drew the competence/performance distinction. For decades syntacticians relied on informal introspective judgments, often the analyst's own marked with an asterisk. From the 1980s, second-language acquisition researchers such as Susan Gass adopted and scrutinized the method, raising questions about its reliability with learners. Carson Schütze's 1996 monograph (reissued in 2016) systematically examined the empirical base of these judgments, and that critical work fed directly into the modern, quantified experimental-syntax program built around acceptability judgments.
When to use it
Use a grammaticality judgment task when you need evidence about what a grammar permits and forbids, especially for structures that are rare, complex, or never produced spontaneously, where corpus or production data are silent. It is central to theoretical syntax, to testing predictions of competing grammatical analyses, and to second-language research that probes learners' implicit and explicit knowledge of a target structure. It is less appropriate when the question concerns frequency or processing rather than well-formedness, when introspection is unreliable for the population (e.g., young children or low-proficiency learners), or when gradient acceptability is expected — in which case a quantified acceptability judgment task is the better instrument.
Limitations and considerations
- Conflates grammatical competence with many performance factors — memory load, plausibility, processing difficulty — that also color judgments.
- Single-linguist introspection is unreplicated and vulnerable to theoretical bias toward the analyst's own predictions.
- Binary verdicts force a sharp line on phenomena that are often gradient, discarding information about degrees of well-formedness.
- Learner and non-linguist judgments can be unreliable, sensitive to instructions, response mode, and metalinguistic awareness.
Source conditions do not replace independent scientific review or validation by an analysis engine.
Strengths
- Yields negative evidence — what is impossible — that production and corpus data cannot provide.
- Accesses structures too rare or complex to appear in naturalistic data, giving direct purchase on grammatical theory.
- Is fast, inexpensive, and adaptable, from a single linguist's intuition to large multi-participant experiments.
- In second-language research, exposes knowledge of the target grammar that learners possess but do not reliably produce.
Applications
- Testing predictions of syntactic theories about movement, binding, agreement, and island constraints.
- Diagnosing learners' second-language grammars and the development of specific structures over time.
- Comparing dialects or varieties on whether a given construction is part of their grammar.
- Probing knowledge of subtle contrasts (e.g., that-trace effects) that almost never surface in spontaneous speech.
- Establishing baseline native-speaker judgments against which experimental or clinical populations are compared.
How it works
How it works
The researcher constructs sentences that target a specific grammatical contrast — for instance a well-formed sentence and a minimally different ungrammatical counterpart that violates one rule (a missing agreement marker, an illicit movement, a wrong word order). Pairs are matched on length, vocabulary, and plausibility so that only the structure of interest differs, and the prediction is that the grammar will accept one and reject the other.
A response format is chosen. The classic linguistic practice uses a binary grammatical/ungrammatical verdict (often marked with an asterisk for the ungrammatical case), but experimental work also uses graded scales, yes/no with confidence, or timed judgments. Schütze documents how the wording of the instructions, the scale, and the presence of practice items all shape what informants report, so the format is fixed deliberately and described in full.
Informants read or hear each sentence and render their judgment. In single-linguist practice this is introspection by the analyst; in experimental and second-language work many participants judge a randomized list that mixes target items with fillers and distractors so that the contrast under study is not transparent. Gass's work shows that learner judgments must be elicited carefully, since reliability depends on task conditions, item presentation, and whether time pressure is imposed.
The pattern of judgments is interpreted as evidence about the grammar. A systematic contrast — speakers consistently accepting the grammatical member and rejecting the ungrammatical member of each pair — supports the proposed rule or constraint; inconsistent or gradient responses prompt revision of the analysis or a switch to a quantified acceptability design. In second-language research the judgments are read as evidence about the learner's interlanguage grammar and its distance from the target.
Common pitfalls
- Treating an ungrammatical-feeling sentence as a grammar violation when the difficulty is really semantic implausibility or parsing complexity.
- Relying on a single self-generated intuition without independent informants or fillers, so theory contaminates the data.
- Presenting target pairs adjacently or without distractors, letting informants guess the contrast and respond strategically.
- Forcing a binary judgment on gradient phenomena and then reporting it as if the grammar drew an equally sharp line.
Frequently asked questions
What is the difference between grammaticality and acceptability?
Grammaticality is a property of the abstract grammar — whether a sentence is generated by the speaker's internalized rule system (competence). Acceptability is what a person actually reports or feels about a sentence, which reflects the grammar plus performance factors such as memory load, plausibility, and parsing difficulty. A sentence can be grammatical yet hard to accept (a deeply center-embedded sentence) or ungrammatical yet easy to process. Strictly, informants give acceptability judgments, and the analyst infers grammaticality from them, which is why modern practice often reframes the task as an acceptability judgment task.
Can a single linguist's intuition count as data?
In the traditional generative practice, yes — the analyst's own clear intuitions about sharp contrasts (such as classic island violations) have proved robust and are often confirmed when later tested formally. But single-linguist judgments are unreplicated and open to theoretical bias, and for subtle or contested contrasts the field increasingly demands judgments from multiple naive informants under controlled conditions, as the experimental-syntax program advocates.
How should grammaticality judgments be elicited from second-language learners?
Carefully, because learner judgments are sensitive to task conditions. Gass and others show that reliability depends on whether items are presented in writing or aurally, whether time pressure is applied (timed tasks tap more implicit knowledge, untimed tasks allow explicit rule application), how the instructions frame 'grammatical', and whether learners can correct or only classify. Mixing grammatical and ungrammatical items with fillers, randomizing order, and reporting these conditions are essential for interpretable learner data.
Related methods
Sources
Chomsky, N. (1965). Aspects of the Theory of Syntax. MIT Press.
ISBN: 9780262530071Copied.Schütze, C. T. (2016). The Empirical Base of Linguistics: Grammaticality Judgments and Linguistic Methodology. Language Science Press.
Open DOI referenceCopied.Gass, S. (1994). The reliability of second-language grammaticality judgments. In E. Tarone, S. Gass, & A. Cohen (Eds.), Research Methodology in Second-Language Acquisition (pp. 303–322). Lawrence Erlbaum.
ISBN: 9780805814569Copied.