When Do Psychological Findings Generalize?

From a particular finding to a defensible claim about people, materials and settings.

Level: Intermediate
Reading load: 8 core readings in 6 units
Prerequisites: Familiarity with experiments, sampling, effect sizes and uncertainty; How to Read Psychological Science provides an entry. Some papers use regression, weighting and multilevel models. The aim is to understand their inferential role; reproducing the calculations is not required.
Last reviewed: September 2026

Introduction

Every study takes place somewhere, with particular people, materials, and tasks. Its conclusion may seek a wider world. What permits that extension: a theory, a sample, a pattern reproduced under changed conditions? Generalization asks us to state where a claim is meant to hold and why the evidence bears upon that destination. A larger sample alone cannot supply the answer.

This path begins by defining a claim’s reach and distinguishing a test of a theory from an estimate about a population. It then examines the selection of study materials, such as faces, words or tasks, the relationship between sample composition and effects of an intervention, and an attempt to extend a school trial to a specified population. A large international replication project provides a demanding case for interpreting variation across settings; a final critique asks whether the analysis and the verbal claim concern the same thing.

By the end, you should be able to specify what a study sampled and held fixed, identify assumptions behind extending its conclusions, and propose a study that could reveal an important boundary. Generalization is a reasoned, revisable inference. It is neither guaranteed by a realistic setting nor ruled out simply because a sample is convenient.

Unit 1 of 6

What is the claim meant to cover?

A generalization needs an object: a population, a set of situations, a process or a theoretical prediction. Begin with a proposal for making scope explicit, then ask whether every useful experiment aims to describe everyday behavior.

Reading
1 of 8
Constraints on Generality (COG): A Proposed Addition to All Empirical Papers
Simons, D. J., Shoda, Y., & Lindsay, D. S. (2017). Constraints on generality (COG): A proposed addition to all empirical papers. Perspectives on Psychological Science, 12(6), 1123–1128. https://doi.org/10.1177/1745691617708630
Methodological contribution · Accessible · Full work: 6 pages
Follow how the proposed statement connects the sample described in the method to the population named in the conclusion.
Why you’re reading it. Turn an implicit claim of generality into something that can be evaluated.
A question to consider. For a student memory experiment, write one supported scope statement and one broader hypothesis that still needs testing.
Reading
2 of 8
In Defense of External Invalidity
Mook, D. G. (1983). In defense of external invalidity. American Psychologist, 38(4), 379–387. https://doi.org/10.1037/0003-066x.38.4.379
Foundational argument · Critical reappraisal · Moderate · Full work: 9 pages
Set Mook’s distinction beside the scope statement. A deliberately unusual condition may be useful if a theory makes a clear prediction about it.
Why you’re reading it. Distinguish testing a mechanism from estimating how often a pattern occurs in everyday life.
A question to consider. How would your evaluation of the memory experiment change if its purpose were to challenge a theory rather than recommend a classroom practice?

Unit 2 of 6

What did we sample besides people?

Materials are often treated as a neutral route to a psychological process. But particular words, pictures or scenarios may be part of what produced a result. Examine what new participants can establish when the materials never change.

Reading
3 of 8
Replicating Studies in Which Samples of Participants Respond to Samples of Stimuli
Westfall, J., Judd, C. M., & Kenny, D. A. (2015). Replicating studies in which samples of participants respond to samples of stimuli. Perspectives on Psychological Science, 10(3), 390–399. https://doi.org/10.1177/1745691614564879
Methodological contribution · Moderate · Full work: 10 pages
Connect the power illustration to the historical questionnaire example. The issue is both how precisely an effect is estimated and which materials the claim concerns.
Why you’re reading it. Recognize stimulus sampling as a design decision rather than a detail repaired by a larger participant sample.
A question to consider. Design a replication that includes original and new word lists. What comparison would reveal whether the finding depends on the original materials?

Unit 3 of 6

When do sample differences matter?

Who participates and who is named in a conclusion both deserve scrutiny. Yet demographic representativeness and similarity of an experimental contrast are different questions. Pair an audit of publication practices with an empirical comparison of effects across samples.

Reading
4 of 8
Making Psychological Science More Representative
Rad, M. S., Martingano, A. J., & Ginges, J. (2018). Toward a psychology of Homo sapiens: Making psychological science more representative of the human population. Proceedings of the National Academy of Sciences, 115(45), 11401–11405. https://doi.org/10.1073/pnas.1721165115
Critical reappraisal · Methodological contribution · Accessible · Full work: 5 pages
Examine how the audit distinguishes participant composition from how authors describe the reach of their findings. Keep its journal and time period in view.
Why you’re reading it. Identify whose experiences are absent and where a claim exceeds the population information reported.
A question to consider. Rewrite a claim about “people” for a study with a narrowly described sample. What missing information would matter most for judging its intended reach?
Reading
5 of 8
Generalizability of Heterogeneous Treatment Effect Estimates Across Samples
Coppock, A., Leeper, T. J., & Mullinix, K. J. (2018). Generalizability of heterogeneous treatment effect estimates across samples. Proceedings of the National Academy of Sciences, 115(49), 12441–12446. https://doi.org/10.1073/pnas.1808083115
Replication · Methodological contribution · Moderate · Full work: 6 pages
Compare the subgroup and across-study results without treating a nonsignificant difference as proof of equality. Treatments, outcomes and the online mode remain tightly constrained.
Why you’re reading it. See why population differences matter for an effect when they are connected to how that effect varies.
A question to consider. What evidence would you request before extending one of these survey effects to a different language, outcome or mode of administration?

Unit 4 of 6

How can a trial inform a specified population?

A trial can support a causal contrast within its sample while leaving a policy population poorly represented. Here the target is explicit. Examine how observed population data help, and what remains an assumption when those data are incomplete.

Reading
6 of 8
Assessing the Generalizability of Randomized Trial Results to Target Populations
Stuart, E. A., Bradshaw, C. P., & Leaf, P. J. (2015). Assessing the generalizability of randomized trial results to target populations. Prevention Science, 16(3), 475–485. https://doi.org/10.1007/s11121-014-0513-z
Methodological contribution · Challenging · Full work: 11 pages
Trace the school-level participation model and the comparison of weighted and unweighted results. Separate random assignment of the intervention from the process selecting schools into the trial.
Why you’re reading it. Assess a concrete attempt to extend an effect, including the information and assumptions it requires.
A question to consider. If the target changed from Maryland elementary schools to secondary schools elsewhere, which parts of the argument would need new evidence before you reused the estimate?

Unit 5 of 6

What does a large collection of settings establish?

Many Labs 2 broadens the observed samples while standardizing much of the procedure. Its length reflects detailed reports of the selected effects. Allow several sittings to connect those reports to the overall argument about heterogeneity and scope.

Reading
7 of 8
Many Labs 2: Investigating Variation in Replicability Across Samples and Settings
Klein, R. A., Vianello, M., Hasselman, F., Adams, B. G., Adams, R. B., Jr., Alper, S., Aveyard, M., Axt, J. R., Babalola, M. T., Bahník, Š., Batra, R., Berkics, M., Bernstein, M. J., Berry, D. R., Bialobrzeska, O., Binan, E. D., Bocian, K., Brandt, M. J., Busching, R., … Nosek, B. A. (2018). Many Labs 2: Investigating variation in replicability across samples and settings. Advances in Methods and Practices in Psychological Science, 1(4), 443–490. https://doi.org/10.1177/2515245918810225
Replication · Landmark empirical study · Challenging · Full work: 48 pages
Keep three levels separate: the selected finding, its results across samples, and the collection of findings. Follow the selection rules and uncertainty as carefully as the pooled results.
Why you’re reading it. Interpret multi-site evidence without turning a replication percentage into a verdict on a discipline.
A question to consider. Choose one reported effect and specify a boundary that the participating sites test, a boundary they do not test, and a new design that would address the latter.

Unit 6 of 6

Does the analysis support the claim we want to make?

Return to the original question of scope with all of the examples in view. Yarkoni’s critique connects unmodeled variation to overstated conclusions, while raising further questions about how far statistical models can take us.

Reading
8 of 8
The Generalizability Crisis
Yarkoni, T. (2022). The generalizability crisis. Behavioral and Brain Sciences, 45, Article e1. https://doi.org/10.1017/s0140525x20001685
Critical reappraisal · Methodological contribution · Challenging · Full work: 16 pages
Compare the verbal-overshadowing example with the stimulus and multi-site arguments. Distinguish the target article’s recommendations from a guarantee that a more complicated model will generalize.
Why you’re reading it. Bring the target claim, the sampled dimensions and the analytical assumptions into one defensible argument.
A question to consider. For a finding from Many Labs 2, write a narrower claim supported by its design and name one design change needed for a broader claim. Would changing the statistical model alone suffice?

Scope and omissions

This path covers the reasoning behind claims that extend across people, materials, tasks and settings. It distinguishes theoretical tests, descriptive representation, variation in treatment effects and generalization to a defined population. The quantitative papers make it an intermediate path, and Many Labs 2 is substantially longer than the other works.

It is not a survey-sampling course, an implementation manual for transport estimators, or an exhaustive account of measurement equivalence, historical change and qualitative transferability. The empirical examples remain selective: online US surveys, Maryland schools and a large but nonrepresentative collection of international samples. Country labels do not stand in for cultural mechanisms, and demographic similarity does not establish equivalent measures or experiences. Other Methods paths address individual change, qualitative interpretation and synthesis of a literature.

Synthesis

A finding has a reach only in relation to a claim and a target. Theory, explicit sampling, models of variation, comparisons across settings, and estimates for a defined population offer different kinds of support. Each carries assumptions. The central question remains whether the measured outcome and studied condition represent what the wider claim concerns.

Imagine a memory strategy tested with university students, English word lists and a computerized task. A school district wants to use it for classroom learning. This is an invented design exercise. Write an argument that distinguishes the original theoretical claim from the proposed practical claim. Name the target learners, materials, outcomes and settings; state which dimensions were sampled, varied or held fixed; and identify at least one plausible effect modifier.

Propose a feasible next study, explaining what it would add and what it would still leave unresolved. Decide what would justify narrowing, extending or abandoning the practical claim. Include the information you would need about the target population, and explain why more participants or a more complicated model would not automatically answer every question.

For the particular problems of translating measures and comparing cultural contexts, continue with How Can We Compare Psychological Processes Across Cultures?.

For inquiry into situated accounts, practices and interpretations, continue with How Can We Study Experience and Meaning?.