Evaluation

Standing Up an Evaluation Function

The first cycle, before there is anything to compare against

Practised · resource not yet written

Establishing evaluation in an organization that has not carried it before: deciding what can be instrumented immediately, what is worth waiting a cycle to measure well, and what can be reconstructed from records already held. The sequence matters, because the first cycle sets the baseline every later cycle will read against, and because consent, retention and reporting arrangements are easier to establish at the outset than to retrofit.

Grounding. Evaluation design; programme theory; baseline and instrumentation planning

Sources

• Reference 1: Patton, M. Q. (2008). Utilization-Focused Evaluation (4th ed.). SAGE Publications.
• Reference 2: Weiss, C. H. (1998). Evaluation: Methods for Studying Programs and Policies (2nd ed.). Prentice Hall.
• Reference 3: Kirkpatrick, D. L., & Kirkpatrick, J. D. (2006). Evaluating Training Programs: The Four Levels (3rd ed.). Berrett-Koehler.

Most related, by shared theory and theme

• Designing Evaluation Backwards from the Decision — Evaluation, Process, Systems · 2 shared sources
• Evaluating a Portfolio Larger Than Its Measurement Capacity — Evaluation · 2 shared sources
• Cost as an Evaluation Question — Evaluation, Systems · 1 shared source
• Theory of Change and Logic Models as Design Artifacts — Evaluation, Systems · 1 shared source
• Reporting to Funders Alongside Learning from the Work — Evaluation · 1 shared source

Connected contributions

• Designing Evaluation Backwards from the Decision

Where this work leads

• Evaluation, on the Evaluation page
• Learning Evaluation Framework Design, on the Learning Systems Design page
• Organizational Learning Design, on the Learning Systems Design page
• Learner Data, Privacy and Analytics Ethics, on the Learning Design page

Evaluating at Small Numbers

Sound inference without a comparison group

Practised · resource not yet written

Most evaluation method assumes a scale that many learning programmes will never reach. Single-case and n-of-1 designs, contribution analysis, most-significant-change, and information power as an alternative to a target sample size each support a defensible claim from a single cohort. The corresponding discipline is knowing which claims the design can carry and stating the rest as description.

Grounding. Contribution analysis (Mayne); single-case design (Kazdin); information power (Malterud); most significant change (Davies & Dart)

Sources

• Reference 4: Mayne, J. (2012). “Contribution Analysis: Coming of Age?” Evaluation, 18(3), 270–280.
• Reference 5: Kazdin, A. E. (2011). Single-Case Research Designs: Methods for Clinical and Applied Settings (2nd ed.). Oxford University Press.
• Reference 6: Malterud, K., Siersma, V. D., & Guassora, A. D. (2016). “Sample Size in Qualitative Interview Studies: Guided by Information Power.” Qualitative Health Research, 26(13), 1753–1760.
• Reference 7: Davies, R., & Dart, J. (2005). The ‘Most Significant Change’ (MSC) Technique: A Guide to Its Use.

Most related, by shared theory and theme

• Choosing and Using Validated Outcome Measures — Evaluation, Evidence
• Evaluating an Anonymous Encounter — Evaluation, Learner research · 1 shared source
• Designing Evaluation Backwards from the Decision — Evaluation
• Reporting to Funders Alongside Learning from the Work — Evaluation, Evidence
• Cost as an Evaluation Question — Evaluation, Evidence

Connected contributions

• Designing Evaluation Backwards from the Decision

Where this work leads

• Evaluation, on the Evaluation page
• KMb Evaluation Design, on the Knowledge Mobilization page
• The Construct Inside the Measure, on the Epistemology page
• Equity Evaluation Framework Design, on the Learning Systems Design page

Evaluating a Portfolio Larger Than Its Measurement Capacity

A stated rule for where the evidence effort goes

Practised · resource not yet written

When an organization runs more programmes than it can evaluate, measurement is tiered rather than uniform: full evaluation for a named few, a single agreed signal for many, and periodic sampling across the rest. Setting the tiering rule in advance, and recording it, makes the coverage legible to everyone whose work sits inside it and gives the portfolio a defensible account of what it knows and where.

Grounding. Evaluation planning; portfolio sampling; exploration and exploitation (March)

Sources

• Reference 1: Patton, M. Q. (2008). Utilization-Focused Evaluation (4th ed.). SAGE Publications.
• Reference 8: March, J. G. (1991). “Exploration and Exploitation in Organizational Learning.” Organization Science, 2(1), 71–87.
• Reference 3: Kirkpatrick, D. L., & Kirkpatrick, J. D. (2006). Evaluating Training Programs: The Four Levels (3rd ed.). Berrett-Koehler.

Most related, by shared theory and theme

• Standing Up an Evaluation Function — Evaluation · 2 shared sources
• Designing Evaluation Backwards from the Decision — Evaluation · 1 shared source
• Reporting to Funders Alongside Learning from the Work — Evaluation · 1 shared source
• Cost as an Evaluation Question — Evaluation · 1 shared source
• Theory of Change and Logic Models as Design Artifacts — Evaluation · 1 shared source

Where this work leads

• Evaluation, on the Evaluation page
• Triage When Demand Exceeds Capacity, on the Learning Systems Design page
• Learning Evaluation Framework Design, on the Learning Systems Design page
• Quality Assurance as a Documented Practice, on the Learning Design page

Reporting to Funders Alongside Learning from the Work

Two purposes, each served properly

Practised · resource not yet written

Accountability reporting and improvement evaluation answer to different readers and reward different things, and a single instrument asked to do both tends to serve one well. Designing them as a related pair — shared data collection, distinct analyses and outputs — lets a grant report stand up to scrutiny while the programme retains evidence it can act on. It also keeps what is promised in an application within what the design will be able to evidence later.

Grounding. Accountability and organizational learning (Ebrahim); measurement effects on programmes (Campbell)

Sources

• Reference 9: Ebrahim, A. (2005). “Accountability Myopia: Losing Sight of Organizational Learning.” Nonprofit and Voluntary Sector Quarterly, 34(1), 56–87.
• Reference 10: Campbell, D. T. (1979). “Assessing the Impact of Planned Social Change.” Evaluation and Program Planning, 2(1), 67–90.
• Reference 1: Patton, M. Q. (2008). Utilization-Focused Evaluation (4th ed.). SAGE Publications.

Most related, by shared theory and theme

• Cost as an Evaluation Question — Evaluation, Evidence · 1 shared source
• Designing Evaluation Backwards from the Decision — Evaluation · 1 shared source
• Evaluating at Small Numbers — Evaluation, Evidence
• Choosing and Using Validated Outcome Measures — Evaluation, Evidence
• Standing Up an Evaluation Function — Evaluation · 1 shared source

Connected contributions

• Designing Evaluation Backwards from the Decision

Where this work leads

• Evaluation, on the Evaluation page
• KMb Evaluation Design, on the Knowledge Mobilization page
• Arguing for the Work Before the Constraint Is Set, on the Learning Design page
• Making the Case for Public Learning Outcomes, on the Public Learning page

Designing Evaluation Backwards from the Decision

Naming the decision each measure informs

Practised · resource not yet written

Evaluation earns its cost when each measure is attached to a decision someone is already committed to making, with the threshold agreed in advance and the route to acting on the result identified before data collection begins. Utilization-focused design starts from intended use by intended users, which places the question of who will act, and on what, at the beginning rather than at the report stage.

Grounding. Utilization-focused evaluation (Patton); evaluation use (Weiss)

Sources

• Reference 1: Patton, M. Q. (2008). Utilization-Focused Evaluation (4th ed.). SAGE Publications.
• Reference 2: Weiss, C. H. (1998). Evaluation: Methods for Studying Programs and Policies (2nd ed.). Prentice Hall.

Most related, by shared theory and theme

• Standing Up an Evaluation Function — Evaluation, Systems, Process · 2 shared sources
• Theory of Change and Logic Models as Design Artifacts — Evaluation, Systems · 1 shared source
• Reporting to Funders Alongside Learning from the Work — Evaluation · 1 shared source
• Evaluating at Small Numbers — Evaluation
• Cost as an Evaluation Question — Evaluation, Systems · 1 shared source

Connected contributions

• Standing Up an Evaluation Function
• Evaluating at Small Numbers

Where this work leads

• Evaluation, on the Evaluation page
• Learning Evaluation Framework Design, on the Learning Systems Design page
• Arguing for the Work Before the Constraint Is Set, on the Learning Design page
• Reading Course Signals Without Overreading Them, on the Learning Design page

Theory of Change and Logic Models as Design Artifacts

The causal chain, written down and open to test

Practised · resource not yet written

A logic model sets out inputs, activities, outputs and outcomes; a theory of change states the reasoning connecting them and the conditions each link depends on. Written at the design stage rather than for a report, it names what would have to be true for the programme to work, which lets the measurement plan follow from the reasoning instead of the reverse, and gives a later reader a specific claim to examine.

Grounding. Programme theory (Funnell & Rogers); logic model development (W. K. Kellogg Foundation); utilization-focused evaluation (Patton); contribution analysis (Mayne)

Sources

• Reference 11: Funnell, S. C., & Rogers, P. J. (2011). Purposeful Program Theory: Effective Use of Theories of Change and Logic Models. Jossey-Bass.
• Reference 12: W. K. Kellogg Foundation (2004). Logic Model Development Guide. Battle Creek, MI: W. K. Kellogg Foundation.
• Reference 1: Patton, M. Q. (2008). Utilization-Focused Evaluation (4th ed.). SAGE Publications.
• Reference 4: Mayne, J. (2012). “Contribution Analysis: Coming of Age?” Evaluation, 18(3), 270–280.

Most related, by shared theory and theme

• Designing Evaluation Backwards from the Decision — Evaluation, Systems · 1 shared source
• Standing Up an Evaluation Function — Evaluation, Systems · 1 shared source
• Cost as an Evaluation Question — Evaluation, Systems · 1 shared source
• Evaluating a Portfolio Larger Than Its Measurement Capacity — Evaluation · 1 shared source
• Reporting to Funders Alongside Learning from the Work — Evaluation · 1 shared source

Connected contributions

• Designing Evaluation Backwards from the Decision

Where this work leads

• Evaluation, on the Evaluation page
• Learning Evaluation Framework Design, on the Learning Systems Design page
• Arguing for the Work Before the Constraint Is Set, on the Learning Design page

Cost as an Evaluation Question

What a programme takes, alongside what it returns

Practised · resource not yet written

Cost per participant, cost per completion, and the resources a programme needs in order to continue are answerable from records the work already produces. Economic evaluation methods in health and in education set out how these are assembled — the ingredients counted, the perspective declared, the time horizon stated — so that a comparison between options rests on one basis and the reader can see what was included and what was left out.

Grounding. Economic evaluation of health care programmes (Drummond et al.); economic evaluation in education (Levin et al.); utilization-focused evaluation (Patton)

Sources

• Reference 13: Drummond, M. F., Sculpher, M. J., Claxton, K., Stoddart, G. L., & Torrance, G. W. (2015). Methods for the Economic Evaluation of Health Care Programmes (4th ed.). Oxford University Press.
• Reference 14: Levin, H. M., McEwan, P. J., Belfield, C., Bowden, A. B., & Shand, R. (2018). Economic Evaluation in Education: Cost-Effectiveness and Benefit-Cost Analysis (3rd ed.). SAGE Publications.
• Reference 1: Patton, M. Q. (2008). Utilization-Focused Evaluation (4th ed.). SAGE Publications.

Most related, by shared theory and theme

• Reporting to Funders Alongside Learning from the Work — Evaluation, Evidence · 1 shared source
• Designing Evaluation Backwards from the Decision — Evaluation, Systems · 1 shared source
• Standing Up an Evaluation Function — Evaluation, Systems · 1 shared source
• Theory of Change and Logic Models as Design Artifacts — Evaluation, Systems · 1 shared source
• Evaluating at Small Numbers — Evaluation, Evidence

Connected contributions

• Reporting to Funders Alongside Learning from the Work

Where this work leads

• Evaluation, on the Evaluation page
• Designing Under Constraint, on the Learning Design page
• Triage When Demand Exceeds Capacity, on the Learning Systems Design page

Choosing and Using Validated Outcome Measures

What an established instrument offers, and what it asks in return

Practised · resource not yet written

An established measure brings comparability, published psychometric properties, and a body of prior results to read against; it also brings its own construct, the population it was developed with, and its administration and licensing requirements. Measurement standards set out the properties to look for and the evidence supporting each, which turns the choice between adopting an instrument and developing one into a documented decision rather than a default.

Grounding. COSMIN taxonomy (Mokkink et al.); health measurement scales (Streiner, Norman & Cairney); construct validity (Messick; Borsboom)

Sources

• Reference 15: Mokkink, L. B., Terwee, C. B., Patrick, D. L., et al. (2010). “The COSMIN Study Reached International Consensus on Taxonomy, Terminology, and Definitions of Measurement Properties for Health-Related Patient-Reported Outcomes.” Journal of Clinical Epidemiology, 63(7), 737–745.
• Reference 16: Streiner, D. L., Norman, G. R., & Cairney, J. (2015). Health Measurement Scales: A Practical Guide to Their Development and Use (5th ed.). Oxford University Press.
• Reference 17: Messick, S. (1995). “Validity of Psychological Assessment: Validation of Inferences from Persons’ Responses and Performances as Scientific Inquiry into Score Meaning.” American Psychologist, 50(9), 741–749.
• Reference 18: Borsboom, D., Mellenbergh, G. J., & van Heerden, J. (2004). “The Concept of Validity.” Psychological Review, 111(4), 1061–1071.

Most related, by shared theory and theme

• Evaluating at Small Numbers — Evaluation, Evidence
• Reporting to Funders Alongside Learning from the Work — Evaluation, Evidence
• Cost as an Evaluation Question — Evaluation, Evidence
• Standing Up an Evaluation Function — Evaluation
• Evaluating a Portfolio Larger Than Its Measurement Capacity — Evaluation

Connected contributions

• Evaluating at Small Numbers

Where this work leads

• Evaluation, on the Evaluation page
• The Construct Inside the Measure, on the Epistemology page
• Eliminating Construct-Irrelevant Barriers in Assessment, on the Universal Design for Learning page

Evaluating for Harm

Unintended effects, anticipated in advance and looked for

Practised · resource not yet written

Programmes produce effects beyond the ones they intend, and those effects are found reliably only where they have been anticipated and measured. Dark logic models set out plausible harmful pathways at the design stage in the same form as the intended ones; conceptual frameworks for adverse effects supply categories to look across, including opportunity cost and effects on people the programme did not reach. Recording what was examined, including where nothing was found, is part of the result.

Grounding. Dark logic models (Bonell et al.); adverse effects framework (Lorenc & Oliver); programme theory (Funnell & Rogers); research ethics (TCPS 2)

Sources

• Reference 19: Bonell, C., Jamal, F., Melendez-Torres, G. J., & Cummins, S. (2015). “‘Dark Logic’: Theorising the Harmful Consequences of Public Health Interventions.” Journal of Epidemiology and Community Health, 69(1), 95–98.
• Reference 20: Lorenc, T., & Oliver, K. (2014). “Adverse Effects of Public Health Interventions: A Conceptual Framework.” Journal of Epidemiology and Community Health, 68(3), 288–290.
• Reference 11: Funnell, S. C., & Rogers, P. J. (2011). Purposeful Program Theory: Effective Use of Theories of Change and Logic Models. Jossey-Bass.
• Reference 21: Canadian Institutes of Health Research, Natural Sciences and Engineering Research Council, & Social Sciences and Humanities Research Council (2022). Tri-Council Policy Statement: Ethical Conduct for Research Involving Humans — TCPS 2 (2022).

Most related, by shared theory and theme

• Theory of Change and Logic Models as Design Artifacts — Evaluation · 1 shared source
• Standing Up an Evaluation Function — Evaluation
• Evaluating at Small Numbers — Evaluation
• Evaluating a Portfolio Larger Than Its Measurement Capacity — Evaluation
• Reporting to Funders Alongside Learning from the Work — Evaluation

Where this work leads

• Evaluation, on the Evaluation page
• Teaching Mental Health Without Causing Harm, on the Health Workforce Education page
• Ethics and Consent in Learner Research, on the Learning Design page
• Incidents and Near-Misses as Design Inputs, on the Learning Systems Design page
• The Designer’s Own Position, on the Epistemology page

Evaluating an Anonymous Encounter

No roster, no follow-up, and evidence all the same

Practised · resource not yet written

Public settings rarely permit named participants or later contact, so evaluation is built from what can be observed and asked in the moment: tracking and timing, structured observation, brief exit conversations, and repeat cross-sectional sampling across a season. Each method answers a narrow question well, and stating which question it answers keeps the resulting claims proportionate.

A resource in two areas. Its home is Public Learning, and it is also part of the Evaluation set.

Grounding. Tracking and timing (Serrell); visitor studies methods; informal learning evaluation

Sources

• Reference 22: Serrell, B. (1997). “Paying Attention: The Duration and Allocation of Visitors’ Time in Museum Exhibitions.” Curator: The Museum Journal, 40(2), 108–125.
• Reference 23: National Research Council (2009). Learning Science in Informal Environments: People, Places, and Pursuits. National Academies Press.
• Reference 4: Mayne, J. (2012). “Contribution Analysis: Coming of Age?” Evaluation, 18(3), 270–280.

Most related, by shared theory and theme

• Evaluating at Small Numbers — Evaluation, Learner research · 1 shared source
• Theory of Change and Logic Models as Design Artifacts — Evaluation · 1 shared source

Connected contributions

• Evaluating at Small Numbers

Where this work leads

• Public Learning, on the Public Learning page
• Learning Nobody Is Required to Do, on the Public Learning page
• Making the Case for Public Learning Outcomes, on the Public Learning page
• Reading Course Signals Without Overreading Them, on the Learning Design page