Reading focus
Syllabus: What Can Statistical Evidence Tell Us?
Connect the survey’s estimated power to its search, meta-analytic effect estimates and publication environment.
Why you’re reading it. Evaluate a claim about research practice without turning it into a rule that small studies are always wrong.
A question to consider. What features of the neuroscience survey would you need to reassess before applying its conclusions to the school’s intervention research?
Why this reading
Button and colleagues connect a methodological argument with an empirical survey of neuroscience meta-analyses. The paper makes the assumed effect sizes, selection of studies and incentives surrounding reported findings part of the evaluation.
Before you read
Power depends on a specified effect, variability and design, not sample size alone. Their calculations use meta-analytic estimates as proxies for unknown effects. Pre-study odds express the assumed ratio of non-null to null effects among those investigated.
Read for
- Examine how the search and eligibility rules shape the 49 meta-analyses and 730 studies in the main survey.
- Separate estimated power, selection of significant results and the probability model used for claimed discoveries.
Consider afterwards
- If the meta-analytic effects are inflated, how would that affect the estimated power of the contributing studies?
Editorial context
The erratum corrects the definition of pre-study odds. These historical samples do not establish current power across all neuroscience. Bacchetti’s comment and the authors’ reply debate how design properties bear on completed results. Neither small nor large sample size alone determines credibility.