Will your retrospective impact evaluation draw reputation-damaging criticism, yawns, or (worse) both?
Build your real-world evaluation know-how with this engaging case-based online learning experience from Drs. Jane Davidson and Thomaz Chianca.
Find out how we delighted our clients with a sound, credible, and actionable retrospective impact evaluation – all within budget – despite dead ends on ALL of our searches for secondary outcome data!
Now you too can stand out from the crowd as one of the few evaluation teams that can actually pull this off!
Scoping and Framing the Evaluation
- Stakeholder engagement
- Identifying who and what the evaluation is for
- Determining the appropriate evaluation approach, guiding principles, and values
- Crafting the Key Evaluation Questions (KEQs) to guide the work
- Connecting the OECD/DAC criteria
Evaluating Design, Implementation, and Durability
- Challenges with obtaining historical data
- Using GIS maps to show clearly what was implemented and where
- Assessing durability: A mixed method approach
- Designing and using rubrics for evidence synthesis in the field
Evaluating Outcomes, Impacts, and Sustainment
- Sustainment vs. sustainability
- Identifying the relevant outcomes and impacts
- What to do when secondary evidence is unavailable
- Triangulation with different sources of evidence
- Mixed method causal inference
- Using rubrics to assess baseline, outcomes, impacts, and sustainment
Value for Investment Analysis
- Options for answering the VfI KEQ (and why we opted for a cost-benefit analysis here)
- Identifying the big-ticket items to include in a cost-benefit analysis
- Figuring out the cost component
- Monetizing benefits
- Estimating a cost-benefit ratio
- VfI lessons learned
Sensemaking, Synthesis, and Reporting
- Sensemaking and synthesis - bringing it back to the big picture
- Extracting the lessons learned for lasting change
- What kinds of things help or limits durability and sustainment?
- What makes for a good transition and exit?
- Dataviz and reporting tips and tricks
- 10 tips for doing great retrospective impact evaluation
e-Course Topics
Introduction and Overview
Available in
days
days
after you enroll
Section 1: Scoping and Framing the Evaluation
Available in
days
days
after you enroll
- 1a. What were we evaluating and who were the primary intended users? (3:01)
- 1b. Key RIE challenges; determining the evaluation approach & guiding principles. (8:01)
- 1c. Where, how, why, which, and whose values we use in evaluation (4:44)
- 1d. How to write good Key Evaluation Questions (KEQs) (11:18)
- 1e. The Retrospective Impact Evaluation KEQ Cheat Sheet (3:20)
- 1f. Adapting the generic KEQs for a real evaluation (5:30)
- 1g. Using the KEQs to address the OECD-DAC evaluation criteria (4:40)
Section 2: Evaluating Design, Implementation, and Durability
Available in
days
days
after you enroll
- 2a. Answering the design, implementation, & durability KEQ – an education example (4:07)
- 2b. Data challenges: Finding historical records, mapping out what was done (5:45)
- 2c. Evaluating design, adaptation, and implementation – what to consider (4:57)
- 2d. What does equitable design and implementation look like? (5:40)
- 2e. Assessing the OECD/DAC criteria of Alignment and Coherence (2:30)
- 2f. Assessing durability and sustainability of component design and implementation (6:02)
- 2g. Rubric-guided evaluation of durability - a health and nutrition example (4:52)
- 2h. Where did the definitions in the rubrics come from? (1:57)
- 2i. How mixed methods and triangulation help support robust conclusions (2:06)
Section 3: Evaluating Outcomes, Impacts, and Sustainment
Available in
days
days
after you enroll
- 3a. The difference between sustainment and sustainability of outcomes and impacts (1:52)
- 3b. Answering the outcomes, impacts and sustainment KEQ (1:34)
- 3c. Figuring out which outcomes and impacts to track on education – Plan A (3:06)
- 3d. What do you do when Plan A doesn’t work out and you need a Plan B (2:05)
- 3e. Feasible student outcomes and impacts to explore – the key questions to ask (1:37)
- 3f. The importance of asking the “why” question for causal inference (2:01)
- 3g. What do equitable outcomes and impacts look like (1:11)
- 3h. Representing Save the Children’s impact on access to education (3:09)
- 3i. Assessing outcomes for young people trained as teachers by Save the Children (3:55)
- 3j. Strategies for collecting and analyzing outcome data on teachers’ income (2:35)
- 3k. Searching for outcomes and impacts related to health and nutrition (4:35)
- 3l. Figuring out which outcomes and impacts to track on health and nutrition – Plans A, B & C (8:16)
- 3m. Dataviz showing outcomes and impacts on health and nutrition (2:03)
- 3n. How to infer causation - tips and tricks (6:09)
- 3o. Tracking down ripple effects and unanticipated impacts (9:37)
Section 4: Assessing Value for Investment
Available in
days
days
after you enroll
- 4a. Answering the Value for Investment KEQ (4:45)
- 4b. Top tips and steps for doing Value for Investment analyses (2:56)
- 4c. Finding the big-ticket items for Value for Investment studies (5:25)
- 4d. Estimating the total investment made – the cost component of VfI studies (7:23)
- 4e. Monetizing benefits and estimating a cost-benefit ratio (2:00)
- 4f. Estimating a cost-benefit ratio for increased years of schooling (12:26)
- 4g. Estimating a cost-benefit ratio for potential lifetime earnings for trained teachers (11:00)
- 4h. Estimating a cost-benefit ratio for health savings due to access to clean water and sanitation (4:42)
- 4i. Six top tips for doing sound Value for Investment studies (2:51)
Section 5: Sensemaking, Synthesis, and Reporting
Available in
days
days
after you enroll
- 5a. Answering the KEQ about lessons learned for lasting change (2:56)
- 5b. The durability and sustainment evaluation cheat sheet (2:23)
- 5c. What helped or limited durability and sustainment in Save the Children’s project (5:28)
- 5d. The 3 Cs influencing how well transition and exit went (5:06)
- 5e. Cool tricks to make beautiful and effective dataviz (10:06)
- 5f. How to make clear, compelling, and insightful reports (7:11)
- 5g. Reporting pro tips (5:08)
- 5h. Ten tips for doing great Retrospective Impact Evaluations (24:15)
- 5i. Useful additional resources (2:40)
Check your inbox to confirm your subscription