Reading focus
Syllabus: When Do Psychological Findings Generalize?
Compare the subgroup and across-study results without treating a nonsignificant difference as proof of equality. Treatments, outcomes and the online mode remain tightly constrained.
Why you’re reading it. See why population differences matter for an effect when they are connected to how that effect varies.
A question to consider. What evidence would you request before extending one of these survey effects to a different language, outcome or mode of administration?
Why this reading
Samples can differ demographically while yielding similar experimental contrasts. This analysis asks why, comparing average and subgroup treatment effects across 27 pairs of survey experiments.
Before you read
A treatment effect compares outcomes under alternative assignments. Effect heterogeneity means that this contrast varies. A conditional average treatment effect describes a defined subgroup; it is not an observed treatment effect for each individual.
Read for
- Compare correspondence across different experiments with variation between subgroups within an experiment.
- Track which features changed between the probability-based and convenience samples, which stayed fixed, and how demographic categories were combined.
Consider afterwards
- Two samples give similar average effects. What evidence could distinguish nearly constant effects from different subgroup effects that happen to average out?
Editorial context
The evidence concerns particular online US survey experiments, largely about attitudes and judgments. Coarse covariates and limited subgroup precision can hide variation. The authors distinguish similarity from proof of equality; their results do not establish that any convenience sample can answer any question.