How Do We Know What a Psychological Test Measures?

Ten readings on constructs, validity, reliability, and the claims we make from scores.

Level: Intermediate
Reading load: 10 core readings in 5 units
Prerequisites: Familiarity with correlations and the distinction between a psychological construct and an observed score. Start Here is useful preparation. Basic ideas of averages and variation. No experience fitting factor models is required; the guides introduce the necessary vocabulary.
Last reviewed: September 2026

Introduction

A test gives us a score; its name invites us to see anxiety, persistence, reasoning, or well-being within it. Between the number and the attribute lies an argument. What connects the responses to the quality being measured? What evidence would justify comparing people, interpreting change, or making a decision on that basis? This path examines the strength of those connections.

This syllabus follows that problem from definitions to research practice. The opening readings establish why measurement choices and theory matter. Competing accounts of validity then reveal that psychologists disagree about exactly what a measurement claim commits us to. The middle units examine scale construction, evidence across methods, and the difference between coherent items, reliable scores, and meaningful interpretations. The final unit asks when evidence travels across populations or occasions and how well published research makes its measurement case.

By the end, you should be able to distinguish consistent scores from a justified interpretation, explain why predicting an outcome does not establish what a test measures, recognize when the procedure shapes a result or leaves out part of the intended attribute, and state what evidence a comparison requires. The aim is informed evaluation rather than proficiency with a particular analysis package. Several papers are conceptually or mathematically demanding; work through their examples in words and return to the definitions when needed.

Carry one example through the sequence: a proposed questionnaire called the Persistence Scale. Imagine that its creators want to predict course completion, compare students in two countries, and evaluate an educational programme. Treat these as three claims requiring justification, not as consequences of the scale’s name.

Unit 1 of 5

What connects a construct to a score?

Begin with the decisions between naming an attribute and reporting a result. Flake and Fried make those decisions visible; Cronbach and Meehl explain why an interpretation needs a network of theoretical expectations and evidence.

Reading
1 of 10
Measurement Schmeasurement: Questionable Measurement Practices and How to Avoid Them
Flake, J. K., & Fried, E. I. (2020). Measurement schmeasurement: Questionable measurement practices and how to avoid them. Advances in Methods and Practices in Psychological Science, 3(4), 456–465. https://doi.org/10.1177/2515245920952393
Methodological contribution · Critical reappraisal · Accessible · Full work: 10 pages
Use the six measurement questions to trace how a construct becomes an operational procedure and a reported score.
Why you’re reading it. Establish the questions that the rest of the syllabus will deepen. Readers returning from Start Here can apply the same framework to a different measurement problem.
A question to consider. For the proposed Persistence Scale, which decisions must be justified before a correlation with course completion can be interpreted?
Reading
2 of 10
Construct Validity in Psychological Tests
Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52(4), 281–302. https://doi.org/10.1037/h0040957
Foundational argument · Methodological contribution · Challenging · Full work: 22 pages
Follow how the nomological network connects observations, theoretical expectations, and revisions to an interpretation.
Why you’re reading it. Move from transparent measurement choices to an argument about what a score means.
A question to consider. Which predicted relationships would help distinguish persistence from prior attainment, and what would a failed prediction require you to reconsider?

Unit 2 of 5

What are we claiming when we call a measure valid?

Messick places score interpretation and use within a broad evaluative account. Borsboom and colleagues argue for a narrower causal definition. Compare their premises and consequences; neither article is a simple empirical refutation of the other.

Reading
3 of 10
Validity of Psychological Assessment: Validation of Inferences From Persons’ Responses and Performances as Scientific Inquiry Into Score Meaning
Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50(9), 741–749. https://doi.org/10.1037/0003-066X.50.9.741
Foundational argument · Major synthesis · Challenging · Full work: 9 pages
Distinguish the meaning of the Persistence Scale’s scores from proposed actions based on them, and notice how Messick relates both to evidence.
Why you’re reading it. Make the scope of a validity argument explicit, including its implications for use.
A question to consider. What additional justification would be needed to use the scale to allocate tutoring rather than simply study an association with completion?
Reading
4 of 10
The Concept of Validity
Borsboom, D., Mellenbergh, G. J., & van Heerden, J. (2004). The concept of validity. Psychological Review, 111(4), 1061–1071. https://doi.org/10.1037/0033-295X.111.4.1061
Foundational argument · Critical reappraisal · Challenging · Full work: 11 pages
Compare the proposed causal requirement with Messick’s account of justified interpretations and uses.
Why you’re reading it. Understand a substantive disagreement about the target of validation, rather than treating validity as an agreed checklist.
A question to consider. If family resources explain both scale responses and course completion, what would the causal account require before calling this a measure of persistence?

Unit 3 of 5

How can a measure earn its interpretation?

A definition must shape the items and the evidence collected. Clark and Watson develop that connection through scale construction; Campbell and Fiske show why several methods and neighboring constructs are needed to investigate what the resulting scores represent.

Reading
5 of 10
Constructing Validity: New Developments in Creating Objective Measuring Instruments
Clark, L. A., & Watson, D. (2019). Constructing validity: New developments in creating objective measuring instruments. Psychological Assessment, 31(12), 1412–1427. https://doi.org/10.1037/pas0000626
Methodological contribution · Contemporary update · Moderate · Full work: 16 pages
Trace the route from a construct definition to an item pool, validation samples, and decisions about what to retain.
Why you’re reading it. See how design choices can strengthen or compromise a measurement claim before the main study begins.
A question to consider. What should the Persistence Scale include—and exclude—to distinguish sustained effort from having abundant time and resources?
Reading
6 of 10
Convergent and Discriminant Validation by the Multitrait-Multimethod Matrix
Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56(2), 81–105. https://doi.org/10.1037/h0046016
Methodological contribution · Historical turning point · Challenging · Full work: 25 pages
Translate the matrix’s trait–method combinations into concrete comparisons involving questionnaires, observers, and behavior.
Why you’re reading it. Test whether agreement reflects the intended attribute or the way it was measured.
A question to consider. Design a two-trait, two-method comparison for persistence and conscientiousness. Which pattern would make shared questionnaire method a plausible explanation?

Unit 4 of 5

Reliable enough for which inference?

Consistency is indispensable but easy to misinterpret. Sijtsma examines what alpha can and cannot establish. Hedge and colleagues show why a robust experimental contrast and a reliable measure of differences between people answer different questions.

Reading
7 of 10
On the Use, the Misuse, and the Very Limited Usefulness of Cronbach’s Alpha
Sijtsma, K. (2009). On the use, the misuse, and the very limited usefulness of Cronbach’s alpha. Psychometrika, 74(1), 107–120. https://doi.org/10.1007/s11336-008-9101-0
Methodological contribution · Critical reappraisal · Challenging · Full work: 14 pages
Keep reliability, item structure, and construct interpretation separate when working through the examples.
Why you’re reading it. Recognize why a single coefficient cannot carry the whole measurement argument.
A question to consider. The Persistence Scale reaches alpha = .90 after several similar items are added. What can you conclude, and what remains unknown?
Reading
8 of 10
The Reliability Paradox: Why Robust Cognitive Tasks Do Not Produce Reliable Individual Differences
Hedge, C., Powell, G., & Sumner, P. (2018). The reliability paradox: Why robust cognitive tasks do not produce reliable individual differences. Behavior Research Methods, 50(3), 1166–1186. https://doi.org/10.3758/s13428-017-0935-1
Methodological contribution · Critical reappraisal · Moderate · Full work: 21 pages
Examine the evidence needed to use task performance as an individual-differences measure, rather than assuming a repeatable condition effect is enough.
Why you’re reading it. Extend measurement questions beyond questionnaires and interpret convergence between tasks and self-reports carefully.
A question to consider. A persistence task shows an average effect of distraction but unstable participant rankings. How should that affect interpretation of its weak correlation with the questionnaire?

Unit 5 of 5

When can a measurement claim travel?

A measure supported in one setting still needs a defensible interpretation elsewhere. Putnick and Bornstein examine comparability across groups and occasions. Flake, Pek, and Hehman then turn the lens on what published reports actually document, providing a final opportunity to evaluate the evidence rather than the label.

Reading
9 of 10
Measurement Invariance Conventions and Reporting: The State of the Art and Future Directions for Psychological Research
Putnick, D. L., & Bornstein, M. H. (2016). Measurement invariance conventions and reporting: The state of the art and future directions for psychological research. Developmental Review, 41, 71–90. https://doi.org/10.1016/j.dr.2016.06.004
Major synthesis · Methodological contribution · Challenging · Full work: 20 pages
Connect each level of invariance with the comparison it is meant to support, keeping the model’s assumptions visible.
Why you’re reading it. Understand why identical questions or translations do not automatically produce comparable measurements.
A question to consider. One Persistence Scale item concerns studying alone. What evidence would you need before comparing students in settings where study is usually solitary versus collaborative?
Reading
10 of 10
Construct Validation in Social and Personality Research: Current Practice and Recommendations
Flake, J. K., Pek, J., & Hehman, E. (2017). Construct validation in social and personality research: Current practice and recommendations. Social Psychological and Personality Science, 8(4), 370–378. https://doi.org/10.1177/1948550617693063
Critical reappraisal · Methodological contribution · Moderate · Full work: 9 pages
Apply the distinctions developed throughout the syllabus to both the audited reports and the audit’s own design.
Why you’re reading it. Finish with a practical appraisal of measurement evidence and the limits of conclusions drawn from reporting.
A question to consider. A paper cites an established scale, deletes three items, and reports alpha. Write the three most consequential questions you would ask before accepting its interpretation.

Scope and omissions

This is a conceptual introduction to evaluating psychological measurement, with an intermediate reading load: ten core articles totaling 157 journal pages. Examples lean toward questionnaires and cognitive tasks. The path covers construct definition, competing accounts of validity, scale development, method effects, reliability, and comparability across groups or occasions.

It does not provide training in administering clinical tests, selecting an individual’s treatment, fitting item-response or factor models, or making personnel decisions. It does not attempt a full history of intelligence testing, quantitative measurement theory, qualitative research quality, or fairness in assessment. Those topics require more than a short section on validity or invariance. Statistical terminology is introduced in the guides, but some papers will take several sittings.

The optional readings supply the earlier scale-development account, the mathematical foundation of factorial invariance, and later qualifications to the reliability debate. They extend the core rather than completing an unstated prerequisite.

Synthesis

Measurement gives observations a proposed meaning. A defensible claim identifies an attribute, explains how the observations bear upon it, and supports the interpretation needed for a particular question. Clear scoring, strong correlations, and reliability contribute without completing the argument. The readings also preserve a real disagreement: whether validity concerns justified interpretation and use, or a causal relation between an attribute and the procedure used to measure it.

Return to the proposed Persistence Scale. Write a short evaluation separating its definition, item content, response process, structure, reliability, relationships with other measures, and comparability across populations or occasions. For each intended use—prediction, international comparison, and programme evaluation—identify the strongest evidence, a plausible rival explanation, and the next investigation you would prioritize. Explain which claims are currently defensible rather than issuing one verdict on the scale.

Continue with What Is a Mental Disorder? to examine how measurement questions interact with the definition and classification of psychological problems. Readers who have completed that syllabus can revisit its distinction between diagnostic agreement and validity using the questions developed here.