Berrypicking Evaluation
- Introduced by
- Marcia J. Bates
- Year
- 1989
- Type
- Model and evaluative lens for evolving, non-linear online search
- Family
- Information behavior / information retrieval interaction
What does it do?
Marcia Bates's berrypicking model, introduced in her 1989 Online Review article 'The design of browsing and berrypicking techniques for the online search interface,' rejects the classic picture of information retrieval as a single query matched against a database to return one optimal set.
Real searches, Bates argued, are evolving: the query shifts as the searcher learns, and useful information is gathered bit-at-a-time, like picking scattered berries, from many different sources using many different techniques. Used as an evaluative lens, the berrypicking model judges search systems and interfaces not by how well they answer one fixed query but by how well they support a continually changing need — letting searchers move fluidly among footnote chasing, citation searching, journal runs, area scans and subject searches as their understanding develops.
The intuition behind the method
Picture picking wild berries: they do not grow in one convenient bunch but are scattered across the bushes, so you wander, picking a few here and a few there, adjusting your path as you go. Bates noticed that real research searches work the same way. You rarely type one perfect query and get one perfect answer set. Instead you find a useful article, which changes what you are looking for, so you chase its footnotes, then notice an author worth following, then scan a promising journal — and at each step the question in your head has subtly shifted. The classic retrieval model, which assumes a stable query and a single best result set, completely misses this. The berrypicking lens takes the wandering seriously and then asks the design question: does this search system help you keep moving and gathering as your need evolves, or does it trap you in one fixed query?
History
Marcia Bates introduced the berrypicking model in her 1989 Online Review article, building on her earlier work on search tactics and the idmove and istrategy levels of searching. Writing as online search systems were maturing, she contrasted the prevailing 'classic' model of information retrieval — a fixed query returning a single optimal set — with the evolving, bit-at-a-time reality she observed in real searchers, and used the contrast to argue for interfaces supporting browsing and multiple search techniques. The berrypicking model became one of the most cited frameworks in information behaviour and interactive information retrieval, anticipating later work on exploratory search, dynamic and faceted interfaces, and the design of discovery systems, and it remains a touchstone for evaluating how well search environments support evolving, exploratory information needs.
When to use it
Use the berrypicking model as an evaluative and design lens when searches are exploratory, open-ended and likely to evolve — literature reviews, scholarly research, complex sense-making tasks — rather than known-item lookups, and when you want to judge how well a system or interface supports a changing need across many strategies. It is well suited to evaluating discovery systems, digital libraries and exploratory search interfaces, and to interpreting search-log or think-aloud data where the query visibly shifts. It is less appropriate for evaluating systems on fixed-query benchmarks, for transactional searches with a single correct answer, or where you need a single precision/recall number. Applying it requires evidence of how the search unfolds over time, so it pairs with session-level logs, think-aloud protocols and observation rather than one-query relevance judgments.
Limitations and considerations
- Its richness resists reduction to a single quantitative score, making standardized benchmarking difficult.
- Faithful application needs session-level traces of how a search evolves, which are costlier to collect than one-query judgments.
- It is a descriptive and evaluative model, not a retrieval algorithm, so it does not by itself rank documents.
- Less informative for genuinely simple known-item searches where the query really is fixed and the classic model suffices.
Source conditions do not replace independent scientific review or validation by an analysis engine.
Strengths
- Captures the realistic, evolving and non-linear character of exploratory search that classic query-match models ignore.
- Reframes evaluation around support for a changing need and browsing, rather than precision/recall on a single fixed query.
- Recognizes the full repertoire of real search moves — footnote chasing, citation searching, journal runs, area scanning.
- Provides clear, actionable design guidance for interfaces that keep searchers moving and gathering across sources.
Applications
- Evaluating discovery systems and digital libraries by how well they support evolving, exploratory searches.
- Designing search interfaces that carry context across moves and enable browsing, citation chasing and area scanning.
- Interpreting search-log and think-aloud data where the query and need visibly shift across a session.
- Framing literature-review and research support tools around bit-at-a-time gathering rather than single answer sets.
- Critiquing fixed-query precision/recall evaluation for exploratory and open-ended search tasks.
How it works
How it works
The model's starting move is to reject the classic information retrieval assumption. In the traditional view, the user has a single, well-formed information need expressed as a fixed query, and the system's job is to return the one best-matching set of documents, evaluated by precision and recall against that query. Bates argues this misrepresents real searching, where the query is not fixed but changes throughout. Discarding the single-query, single-result-set assumption is what opens the door to modeling search as a dynamic, evolving process.
Formula
\text{classic IR: } Q\ \text{fixed},\ \text{output}=\{d:\text{match}(d,Q)\}.\ \ \text{Berrypicking: } Q=Q(t),\ \frac{dQ}{dt}\neq 0In its place the model treats the query as evolving: each thing the searcher finds reshapes their understanding and therefore their need, so the next query is a function of the previous one and of what was just encountered. The information need itself is not satisfied by a final retrieved set but is continually shifting as the search proceeds. Capturing this dependence of each move on the last — the search as a trajectory rather than a single shot — is the analytic heart of berrypicking.
Formula
Q_{t+1}=\phi\bigl(Q_t,\,r_t\bigr),\quad r_t=\text{what was found at move }t,\ \text{need shifts with each } r_tBecause the need evolves, satisfaction is achieved not by one comprehensive answer set but by gathering useful fragments bit-at-a-time. At each move the searcher plucks the relevant bits from what they encounter, and the sum of these scattered pickings — across the whole evolving search — constitutes the result. This 'berrypicking' of partial, distributed information replaces the classic notion of a single relevant set and is the image that gives the model its name.
Formula
\text{Satisfaction}=\bigcup_{t} \{b_t\},\ b_t\subset r_t\ (\text{useful bits}),\ \text{not a single retrieved set}The model also recognizes that searchers use a wide variety of techniques and sources, not just subject queries against one database. Bates catalogues moves such as footnote chasing (following references), citation searching (finding citing works), journal runs (scanning a relevant journal), area scanning (browsing the shelf or classification neighborhood), author searching, and subject searching. Real berrypicking weaves among these strategies, and a faithful account of a search must track this diversity of moves rather than reducing it to query matching.
Formula
\text{moves}\in\{\text{footnote chasing},\text{citation search},\text{journal run},\text{area scan},\text{author search},\text{subject search}\}Used evaluatively, the model scores search systems and interfaces by how well they support this evolving, multi-strategy process. A good interface lets the searcher carry context from one move to the next, browse fluidly, and switch among footnote chasing, citation searching, area scanning and the rest without friction; a poor one forces a fixed query and a static result set. Evaluation thus shifts from precision/recall on a single query to facilitation of need evolution and browsing — the design payoff Bates explicitly intended.
Formula
\text{score}(\text{system})=\textstyle\sum_t \text{support}\bigl(Q_t\to Q_{t+1}\bigr),\ \text{reward evolution \& browsing}Common pitfalls
- Evaluating an exploratory-search system with fixed-query precision/recall benchmarks that the berrypicking model explicitly rejects.
- Treating each query in a session as an independent search and missing how each move reshapes the need.
- Designing interfaces that discard context between moves, forcing searchers to restart instead of evolving their query.
- Reducing the diversity of search moves to subject querying and ignoring footnote chasing, citation searching and browsing.
Frequently asked questions
What does 'berrypicking' mean as a model of search?
It is Bates's metaphor for how real information seeking works: just as wild berries are scattered and picked a few at a time, useful information is gathered bit-by-bit from many sources across an evolving search rather than delivered as one comprehensive result set. As each find reshapes the searcher's understanding, the query and the need shift, and the searcher moves among different techniques — footnote chasing, citation searching, journal runs, area scanning — picking relevant fragments along the way. The accumulated pickings, not a single retrieved set, satisfy the need.
How does the berrypicking model differ from the classic IR model?
The classic model assumes a fixed, well-formed query and judges a system by how well it returns the single best-matching set, scored with precision and recall. Berrypicking rejects both assumptions: the query evolves as the searcher learns, and satisfaction comes from gathering scattered fragments across many moves and sources rather than from one answer set. Consequently the berrypicking model also changes evaluation — instead of scoring a single query, it asks how well a system supports a continually changing need and the browsing and varied search strategies that real seekers use.
How is the model used to evaluate search systems?
It is used as a lens that scores systems and interfaces by how well they facilitate an evolving, multi-strategy search rather than by one-query relevance. A system rates well if it lets searchers carry context from one move to the next, browse fluidly, follow citations and references, scan related areas, and reformulate as their need shifts; it rates poorly if it forces a fixed query and a static result set. Evaluators draw on session-level evidence — search logs, think-aloud protocols, observation — to see whether the interface supports the bit-at-a-time, berrypicking trajectory the model describes.
Related methods
Sources
Bates, M. J. (1989). The design of browsing and berrypicking techniques for the online search interface. Online Review, 13(5), 407-424.
Open DOI referenceCopied.