Knowledge Foundations
Construct-irrelevant variance — score variation driven by factors extraneous to the trait being measured — is one of the two classic threats to validity. Diagnosing and removing it draws on several interconnected bodies of knowledge. The strongest assessment work holds all of these in view at once.
The eight knowledge foundations
1. Educational and Psychological Measurement
The home discipline. Validity is treated as an ongoing, evidence-based judgment about a particular interpretation and use of scores — not a fixed property of a test. Within this frame, construct-irrelevant variance (CIV) is the central threat: when scores are shaped by factors outside the construct, the inference drawn from them is weakened.
2. Construct Theory and Test Design
CIV can only be diagnosed against a clearly specified construct. Defining the trait — reading comprehension, clinical reasoning, statistical literacy — before building items is what makes it possible to say which demands legitimately belong to the measurement and which are incidental. The working rule: define the construct first, operationalize second, validate the interpretation third.
3. Psychometrics and Differential Item Functioning
Reliability, item analysis, and Differential Item Functioning (DIF) supply the empirical machinery for detecting when an item behaves differently across groups for reasons unrelated to ability. Methods including the Mantel–Haenszel procedure, logistic regression, and item response theory flag items for review — a key source of evidence that an assessment is functioning fairly.
4. Universal Design for Learning
The parent framework. Multiple means of action and expression directly counter CIV by separating the skill being assessed from the medium used to demonstrate it — so that an assessment of reasoning does not inadvertently become an assessment of writing, typing, or timed recall. Barriers are anticipated in the design rather than addressed after the fact.
5. Disability Studies and Accessibility
Assistive technology, the distinction between reactive accommodation and proactive design, and standards such as WCAG. Many classic sources of CIV — timing pressure, rigid format, unnecessary sensory or motor demands — are accessibility barriers in another guise. Designing them out benefits far more learners than those who would have requested an accommodation.
6. Authentic and Performance Assessment
Real-world, application-based tasks reduce some of the artificial barriers that standardized formats introduce, while keeping the focus on what is actually meant to be measured. Their design requires equal rigour: authenticity does not relax the need for clear constructs, consistent scoring, and defensible inferences.
7. Cognitive Load Theory
A practical lens for separating load that is intrinsic to the construct from extraneous load imposed by confusing instructions, cluttered stimuli, or irrelevant complexity. Extraneous load is a frequent and largely invisible source of CIV: it lowers scores for reasons that have nothing to do with the trait under measurement.
8. Culturally Responsive and Equity-Centered Assessment
Language demands, cultural assumptions embedded in prompts, and structured bias review. This body of work asks whether a task measures the intended construct fairly across linguistic and cultural backgrounds — or whether it quietly rewards familiarity with the test-maker's world.
Two further foundations
Two bodies of knowledge run underneath the work rather than sitting beside it. Accommodation law and policy — the ADA and Section 508 in the United States, and Canadian equivalents such as the Accessible Canada Act and provincial statutes — set the legal floor and shape what a defensible assessment must look like. Backward design and constructive alignment ensure the assessment maps to the stated learning goal in the first place; misalignment is itself a structural form of construct-irrelevant variance.
Removing barriers is not lowering the bar
The most common misconception in this area is that removing construct-irrelevant barriers makes an assessment easier or less rigorous. The opposite is true. When a barrier unrelated to the construct is removed — a needless reading load on a mathematics item, a time limit that measures speed rather than competence, a culturally specific reference that advantages some test-takers — the score becomes a more accurate reflection of the trait being measured. Validity increases. What changes is precision, not standards.
The construct is the boundary that makes the whole judgment possible. Without a clear definition of what is and is not part of what is being measured, there is no principled way to distinguish a legitimate demand from an incidental one. This is why construct specification is treated here as the first and most consequential step, not a formality. Diagnosis of CIV is only as good as the construct definition it is measured against.
Why the stakes sharpen the work
The cost of construct-irrelevant variance scales with the consequence of the decision it informs. In a low-stakes formative check, an imprecise score is a minor inconvenience. In a high-stakes or credentialing context, construct-irrelevant variance becomes a threat to the validity of the credential itself — the very thing the assessment exists to protect. In that setting, psychometric analysis, DIF review, and accommodation-law fluency move from useful refinements to defensibility-critical practice. The same principles apply across contexts; the tolerance for error is what differs.
Working Process
Identifying and removing construct-irrelevant variance is a structured, evidence-building process rather than a single review step. The stages below represent a comprehensive approach; in practice, depth is calibrated to the stakes of the assessment. Understanding the full process clarifies what is gained or lost when a stage is abbreviated.
Specify the Construct
Naming exactly what is — and is not — being measured
Before any item is written, the construct is defined with enough precision that its boundary is visible: which knowledge, skills, or abilities belong to it, and which adjacent demands do not. A test of statistical reasoning, for example, must decide whether reading load, arithmetic fluency, or software proficiency are part of the target — or incidental to it.
This definition is the reference point against which every later judgment about construct-irrelevant variance is made. Where the construct is left implicit, CIV cannot be diagnosed reliably, because there is no agreed standard for what counts as extraneous.
- A written construct definition with explicit inclusions and exclusions
- A statement of the intended interpretation and use of scores
- Shared agreement among stakeholders on the boundary of the construct
Map Intended vs. Incidental Demands
Separating the construct from the way it is delivered
Every task imposes demands. This stage inventories them and sorts each one as either intrinsic to the construct or incidental to it. Incidental demands are the candidate sources of construct-irrelevant variance: reading level above the construct's requirement, time pressure unrelated to the skill, fine-motor or sensory demands, cultural or contextual references, and extraneous cognitive load created by item design.
Cognitive load theory is useful here for distinguishing the complexity that genuinely belongs to the construct from complexity introduced by presentation. The aim is not to eliminate difficulty, but to ensure that difficulty comes from the trait being measured.
- A demand map classifying each task requirement as intended or incidental
- A prioritized list of likely CIV sources to address in design
- Design constraints carried forward into item and task construction
Design for Multiple Means
Building tasks that isolate the construct
Items and tasks are constructed to measure the construct while minimizing the incidental demands identified in the previous stage. Universal Design for Learning's principle of multiple means of action and expression is applied directly: where the medium of response is not part of the construct, learners are offered more than one way to demonstrate the same competence, with all options mapped to a single construct-aligned rubric.
Plain-language construction, uncluttered stimuli, and flexible timing reduce extraneous load. Authentic and performance tasks are introduced where they measure the construct more directly than a standardized format would.
- Draft items and tasks with incidental demands designed out
- Where appropriate, multiple response options mapped to one rubric
- A scoring approach aligned to the construct rather than to format
Bias, Language, and Accessibility Review
Structured human judgment before piloting
Draft materials undergo structured review for cultural assumptions, unnecessary language complexity, and accessibility barriers. Sensitivity and fairness review examines whether prompts assume background knowledge unrelated to the construct. Accessibility review checks digital materials against established standards and confirms compatibility with assistive technology.
This review is qualitative and anticipatory. It catches barriers that statistical analysis alone will not surface — particularly those affecting groups too small to register reliably in later item statistics.
- A documented bias and sensitivity review with recommended revisions
- An accessibility audit covering format, language, and assistive-technology compatibility
- Revised materials ready for piloting
Pilot and Analyze (Including DIF)
Empirical evidence that items behave fairly
Items are piloted with a representative sample and examined through item analysis, reliability estimation, and Differential Item Functioning analysis. DIF compares performance across groups matched on overall ability, flagging items that function differently for reasons unrelated to the trait being measured — the empirical signature of construct-irrelevant variance at the item level.
A flagged item is not automatically discarded. Statistical flags are reviewed qualitatively to determine whether the difference reflects genuine bias or a defensible feature of the construct, and items are revised or removed on that combined evidence.
- Item statistics, reliability estimates, and DIF results
- A documented adjudication of each flagged item
- A revised item set with sources of CIV removed or justified
DIF detection is sensitive to sample size; small or unevenly distributed groups can obscure real bias. Statistical results are strongest when paired with the qualitative review from the previous stage.
Accommodation and Policy Alignment
Proactive design first, defensible accommodation second
Proactive design removes the need for many accommodations, but not all. Remaining needs are met through a documented accommodation process aligned with the applicable legal floor — the ADA and Section 508 in the United States, and the Accessible Canada Act and provincial statutes such as the Accessible British Columbia Act and Ontario's accessibility legislation in Canada.
The distinction matters: accommodations adjust the conditions of administration without altering the construct, whereas a modification that changes the construct changes the meaning of the score. Keeping that line clear is what makes both the design and the accommodation defensible.
- A documented accommodation process consistent with applicable law
- A clear record of which adjustments preserve the construct and which would alter it
- Policy alignment suitable for review by an accrediting or governing body
Build the Validity Argument
Assembling the evidence into a defensible case
The evidence gathered across the process is assembled into an argument for the specific interpretation and use of scores. Following the contemporary, argument-based view of validity, this draws on multiple sources: content relevance, response processes, internal structure, relationships to other variables, and the consequences of use. The treatment of construct-irrelevant variance is documented as part of that argument.
Validation is treated as ongoing rather than a one-time certification. As populations, contexts, and uses change, the argument is revisited — which is also what allows an accrediting body to use the same evidence for its own quality assurance.
- A validity argument linking evidence to the intended interpretation and use
- Documentation of how CIV was identified, addressed, and monitored
- A basis for periodic revalidation as conditions change
Key Decisions That Shape the Work
No two assessments call for identical treatment. Decisions about purpose, construct boundary, format, and evidence determine how much construct-irrelevant variance can be tolerated — and how much effort its removal warrants. These dimensions are best settled collaboratively, early.
Purpose and stakes: what decision will this score inform?
A formative check and a credentialing exam tolerate very different levels of imprecision. The stakes of the decision set the standard for how rigorously construct-irrelevant variance must be identified and removed — and how much validity evidence the use requires.
Construct boundary: what exactly is — and isn't — being measured?
This is the decision that makes every other one possible. Until the construct's boundary is explicit, there is no principled way to call any demand extraneous. Time invested here prevents the most expensive errors later.
Format and medium: does the format add demands unrelated to the construct?
Multiple-choice, essay, oral, portfolio, and performance formats each carry their own incidental demands. The right format is the one that measures the construct most directly for this population — not the one that is most convenient to score.
Single vs. multiple means: can learners demonstrate the construct in more than one way?
Where the medium of response is not part of the construct, offering multiple means of expression — mapped to one rubric — removes a major source of CIV. Where the medium is the construct, flexibility would change what is being measured.
Population and context: who takes this, and what variability do they bring?
Linguistic, cultural, and access variability in the test-taking population determines which barriers are most consequential. The wider and less knowable the population, the more proactive design must do the work that reactive accommodation cannot.
Evidence and defensibility: how much validity evidence does this use require?
Higher-stakes uses call for deeper evidence: DIF analysis, documented bias review, and a complete validity argument. Deciding this early shapes the pilot design and determines whether the assessment can withstand external scrutiny.
The effort spent removing construct-irrelevant variance should be proportional to the stakes of the decision and the breadth of the population. A modest formative assessment does not require a full psychometric programme. A credential that travels across languages, jurisdictions, and health systems does — because there, an invalid score is not a single bad decision but a threat to the credential's meaning.
Core Competencies
The competencies below form the professional foundation of this work — spanning definition, design, analysis, and defensibility across formative, high-stakes, and credentialing contexts.
Construct specification
Defining a construct with enough precision that its boundary is operationally clear — distinguishing the trait to be measured from adjacent demands. This is the competence on which the reliable diagnosis of construct-irrelevant variance depends.
Validity argumentation
Treating validity as an evidence-based judgment about a specific interpretation and use, and assembling content, response-process, internal-structure, external, and consequential evidence into a coherent, defensible argument.
Item and task design
Constructing items and performance tasks that elicit the intended construct while designing out incidental demands — applying plain-language principles, uncluttered presentation, and, where appropriate, multiple means of expression mapped to a single rubric.
Psychometric analysis
Conducting item analysis, estimating reliability, and applying Differential Item Functioning methods — Mantel–Haenszel, logistic regression, and item-response-theory approaches — together with the qualitative review needed to interpret flagged items responsibly.
Accessibility and UDL application
Anticipating learner variability in the design itself: applying Universal Design for Learning, meeting recognized accessibility standards, and ensuring compatibility with assistive technology so that barriers are removed before they require accommodation.
Bias and sensitivity review
Leading structured review of cultural assumptions, language demands, and contextual references to ensure tasks measure the construct fairly across linguistic and cultural backgrounds rather than rewarding familiarity with a particular world.
Accommodation law and policy fluency
Working knowledge of the applicable legal floor — the ADA and Section 508, and Canadian frameworks including the Accessible Canada Act and provincial statutes — and the judgment to distinguish accommodations that preserve the construct from modifications that alter it.
Curricular alignment
Using backward design and constructive alignment to ensure the assessment actually maps to the stated learning goal. Misalignment is itself a structural form of construct-irrelevant variance, and correcting it is often the highest-value intervention available.
Equity-centered practice and collaboration
Designing assessment to surface and remove barriers rather than to sort, and working alongside subject-matter experts, faculty, and the communities being assessed — including co-design and the disaggregation of results to detect disparities and guide revision.
Outcomes
Removing construct-irrelevant variance creates value at two horizons. The shorter-term outcomes are concrete and measurable. The longer-term shift affects how an institution's results — and its credentials — hold up under scrutiny.
Shorter-term outcomes: what becomes visible and actionable
| Outcome | What this looks like in practice |
|---|---|
| The construct is clearly boundedA written definition makes explicit what is and is not being measured. | "This feels unfair" becomes a specific, addressable claim about a named, incidental demand. |
| Barriers are named and removedIncidental demands — reading load, timing, format, cultural reference — are designed out. | Scores reflect the trait being measured rather than the way the task happened to be delivered. |
| Scores mean what they claimWith extraneous variance reduced, the inference drawn from a score is better supported. | Validity improves precisely because precision improves — without lowering the standard. |
| Flagged items are resolved on evidenceItems that function differently across groups are surfaced through DIF and reviewed. | Item-level bias is revised or justified deliberately, rather than left to chance or to complaint. |
Longer-term change: what becomes embedded in practice
| Outcome | What this looks like in practice |
|---|---|
| Decisions and credentials are defensibleA documented validity argument allows results to withstand scrutiny. | Learners, accreditors, and governing bodies can see that the assessment measures what it claims. |
| Equity is built in, not bolted onProactive design removes barriers for the whole population. | Fewer individuals are routed through a reactive accommodation process that cannot scale. |
| A standing capacity to review developsConstruct definitions, demand maps, and analysis routines become reusable. | Each new assessment starts further along than the last, rather than from scratch. |
| Results earn trust across stakeholdersScores are seen to measure the intended construct fairly. | The assessment's authority rests on evidence rather than assertion. |
Construct-irrelevant variance is not only a technical concern. Because scores inform decisions about people, the consequences of an invalid inference are borne by the test-taker. Treating the removal of extraneous barriers as a core design responsibility — rather than a remediation step — is what aligns measurement rigour with fairness.
Practice Reflection
The questions below are intended to help surface useful considerations about how your current assessments handle construct-irrelevant variance. They are not a formal assessment. Take your time with them — the most useful answers are honest ones, not aspirational ones. Working through these with colleagues who hold different roles — faculty, psychometricians, accessibility specialists — tends to be more productive than working through them alone.
On defining what you measure
- For your most consequential assessment, is the construct written down with explicit inclusions and exclusions — or does it live implicitly in the items?
- Where might an assessment be measuring something narrower, broader, or simply different from what it claims to measure?
- How well does each assessment actually map to the stated learning goal it is meant to evidence?
On incidental demands and access
- Which demands in your tasks — reading load, timing, format, cultural reference — are intrinsic to the construct, and which are incidental to it?
- At what point does accessibility currently get considered: in the design, during review, or only when an individual requests an accommodation?
- Where could a single proactive design change remove a barrier for many learners at once?
On evidence of fairness
- What evidence do you currently gather that items function fairly across groups — and is item-level analysis part of it?
- When an item is flagged as performing differently across groups, how is that adjudicated, and by whom?
- Are results ever disaggregated to detect disparities, and do those findings feed back into design?
On stakes and defensibility
- For each assessment, what decision does the score inform — and is the effort spent removing construct-irrelevant variance proportional to that stake?
- If an external body asked you to justify a score's meaning, what evidence could you currently assemble?
- What would a realistic next step look like — given current resources, constraints, and the people who have the authority to move things forward?
The teams that make meaningful progress on fairness in assessment tend to be those that create space for honest conversations about where scores might be measuring the wrong thing — not where they would like to believe they are.
Citations
The knowledge claims, frameworks, and evidence in this resource draw on established scholarship, professional standards, and law. Sources are grouped by the area of the resource they primarily support, and are listed so that each can be independently verified.
Messick, S. "Validity." In R. L. Linn (Ed.), Educational Measurement (3rd ed., pp. 13–103). American Council on Education / Macmillan, 1989. Origin of construct-irrelevant variance and construct underrepresentation as the two principal threats to construct validity.
Messick, S. "Validity of Psychological Assessment." American Psychologist, 50(9), 741–749, 1995.
doi.org/10.1037/0003-066X.50.9.741AERA, APA, & NCME. Standards for Educational and Psychological Testing. American Educational Research Association, 2014. Authoritative reference for validity, fairness, and the treatment of construct-irrelevant variance.
Kane, M. T. "Validating the Interpretations and Uses of Test Scores." Journal of Educational Measurement, 50(1), 1–73, 2013.
doi.org/10.1111/jedm.12000Cronbach, L. J., & Meehl, P. E. "Construct Validity in Psychological Tests." Psychological Bulletin, 52(4), 281–302, 1955.
doi.org/10.1037/h0040957Holland, P. W., & Thayer, D. T. "Differential Item Performance and the Mantel–Haenszel Procedure." In H. Wainer & H. I. Braun (Eds.), Test Validity (pp. 129–145). Lawrence Erlbaum, 1988.
Holland, P. W., & Wainer, H. (Eds.). Differential Item Functioning. Lawrence Erlbaum, 1993. Standard reference for DIF theory and methods.
CAST. Universal Design for Learning Guidelines, Version 2.2. CAST, 2018. Source for multiple means of action and expression as applied to assessment.
udlguidelines.cast.orgWorld Wide Web Consortium (W3C). Web Content Accessibility Guidelines (WCAG) 2.1. W3C Recommendation, June 2018.
w3.org/TR/WCAG21/Sweller, J. "Cognitive Load During Problem Solving: Effects on Learning." Cognitive Science, 12(2), 257–285, 1988.
doi.org/10.1207/s15516709cog1202_4Sweller, J., Ayres, P., & Kalyuga, S. Cognitive Load Theory. Springer, 2011. Source for the intrinsic / extraneous / germane load distinction applied to item design.
Wiggins, G. Educative Assessment: Designing Assessments to Inform and Improve Student Performance. Jossey-Bass, 1998.
Montenegro, E., & Jankowski, N. A. Equity and Assessment: Moving Towards Culturally Responsive Assessment. Occasional Paper No. 29. National Institute for Learning Outcomes Assessment (NILOA), 2017.
files.eric.ed.gov/fulltext/ED574461.pdfMontenegro, E., & Jankowski, N. A. A New Decade for Assessment: Embedding Equity into Assessment Praxis. Occasional Paper No. 42. National Institute for Learning Outcomes Assessment (NILOA), 2020.
files.eric.ed.gov/fulltext/ED608774.pdfRandall, J. "Color-Neutral Is Not a Thing: Redefining Construct Definition and Representation Through a Justice-Oriented Critical Antiracist Lens." Educational Measurement: Issues and Practice, 40(4), 82–90, 2021.
doi.org/10.1111/emip.12429Inoue, A. B. Antiracist Writing Assessment Ecologies: Teaching and Assessing Writing for a Socially Just Future. The WAC Clearinghouse / Parlor Press, 2015. Open access.
wac.colostate.edu/books/perspectives/inoue/Parliament of Canada. Accessible Canada Act, S.C. 2019, c. 10. Establishes the goal of a barrier-free Canada and the federal duty to identify and remove barriers.
laws-lois.justice.gc.ca/eng/acts/A-0.6/Legislative Assembly of British Columbia. Accessible British Columbia Act, S.B.C. 2021, c. 19.
bclaws.gov.bc.ca — SBC 2021 c. 19Americans with Disabilities Act of 1990, as amended (42 U.S.C. § 12101 et seq.), and Section 508 of the Rehabilitation Act (29 U.S.C. § 794d), governing accessibility of information and technology.
ada.govWiggins, G., & McTighe, J. Understanding by Design. 2nd ed. ASCD, 2005. Source for backward design and alignment of assessment to stated goals.
Biggs, J. "Enhancing Teaching Through Constructive Alignment." Higher Education, 32(3), 347–364, 1996.
doi.org/10.1007/BF00138871