Reading focus
Syllabus: When Do Psychological Findings Generalize?
Trace the school-level participation model and the comparison of weighted and unweighted results. Separate random assignment of the intervention from the process selecting schools into the trial.
Why you’re reading it. Assess a concrete attempt to extend an effect, including the information and assumptions it requires.
A question to consider. If the target changed from Maryland elementary schools to secondary schools elsewhere, which parts of the argument would need new evidence before you reused the estimate?
Why this reading
Random assignment within a trial does not make its participants representative of a target population. Stuart and colleagues connect that distinction to a school-based example of estimating effects beyond the participating schools.
Before you read
The sample average treatment effect concerns the trial; the population effect concerns a specified target. A participation weight gives more influence to trial units whose observed characteristics are less represented relative to that target. It addresses a different selection problem from treatment assignment.
Read for
- Follow the move from 37 trial schools to a target of 717 Maryland elementary schools.
- Examine the variables used for weighting, the balance comparisons, and the assumptions about unmeasured effect modifiers and the participation model.
Consider afterwards
- School leadership might affect both joining a trial and implementing its intervention, but is absent from the target-population data. What does that omission prevent the weighting exercise from establishing?
Editorial context
Similar weighted and unweighted estimates do not prove effects are identical everywhere. The example has a defined state, school population, intervention and outcome. Its limitations include unmeasured characteristics and uncertainty about effect modification. Weighting cannot supply information about conditions absent from the data.