Reading focus
Syllabus: What Can Statistical Evidence Tell Us?
Use plausible effects from outside the focal dataset when considering the design’s possible outcomes.
Why you’re reading it. Explain how a significant pilot estimate can still be a poor basis for planning.
A question to consider. If the school plans a larger trial using only its most successful pilot, what alternative effect assumptions should it examine?
Why this reading
A study can produce a significant estimate whose direction or magnitude is misleading. Gelman and Carlin expand design analysis to examine these risks under plausible effect sizes supplied by external evidence.
Before you read
Power is the chance of crossing a specified testing threshold under an assumed effect and design. Here, Type S error concerns the wrong sign among significant estimates; Type M concerns their average exaggeration in magnitude. These are conditional quantities.
Read for
- Trace how assumed effect size and standard error determine the design calculations.
- Distinguish external information about plausible effects from inserting the same study’s observed estimate as the truth.
Consider afterwards
- A noisy pilot produces a large effect. How would planning only around that estimate differ from examining several smaller, externally justified effects?
Editorial context
The worked example illustrates sensitivity to plausible effects; it does not endorse the original biological claim. Conclusions depend on the assumed sampling model and effect sizes. The method concerns expected behavior across hypothetical studies, not a known probability that one reported estimate is wrong. Lu, Qiu and Deng’s later analysis extends its mathematical treatment.