Learning measurement & evaluation
Building a learning measurement function.
I set the direction, built a team of four colleagues, and rebuilt learning measurement and reporting across five Claims lines of business. A 90-day workplace study helped us decide where reinforcement and support were needed.
Read the executive briefRead this case
Start with the brief or explore the method and artifacts.
Executive brief
What would it take for leaders to trust and use learning measurement?
- My contribution
- I set the measurement direction across five Claims lines of business and grew the team from one colleague to four. I created the Levels 1–3 strategy and conducted a mixed-method study after 90 days, using surveys, learner and leader interviews, and stratified sampling.
- What we put into practice
- We established monthly and quarterly reporting and data-quality controls. The study led to a buddy system, improved monitoring, and job aids and refresher learning designed for rollout. The team revised and adopted the evaluation approach across Levels 1–4.
- Outcome and evidence
- Leaders had a more dependable reporting practice and a wider evaluation approach. By my estimate, final reporting accuracy reached roughly 90–95%, and senior leaders gave positive feedback on the reports.
Follow the evidence
How the evidence fits.
Select a step to inspect its source and what it supports.
The function
I built the measurement function across five Claims business lines and grew the team from one colleague to four.
- Source basis
- My account of leading the team and establishing the measurement practice.
- What to inspect
- Read the starting conditions, team decisions, and reporting work.
01 / The question
What would it take for leaders to trust and use learning measurement?
Reporting did not consistently match what leaders saw in the field. Learner surveys and assessments varied in quality, and the group had no systematic approach to measuring behavior transfer or business results. I began by assessing the measurement landscape, the source-data problems, and the skills the team needed. I met with colleagues who supplied LMS data, investigated the bottlenecks, and built the case for a more useful measurement function.
Leadership & practice
I set the direction and built the team to deliver it.
I led learning measurement and reporting across five Claims lines of business. I used job task analysis to clarify responsibilities, rewrote role descriptions, and brought in capabilities the group lacked. The team grew from one colleague to four, covering learning management, knowledge management, data analytics, and instructional design. I presented the strategy to senior executives and introduced the changes in manageable stages.
Build the measurement practice
I introduced the Kirkpatrick New World Model, set a timeline and project charter, and created the Levels 1–3 strategy. The team revised and adopted the wider approach across Levels 1–4.
Make reporting dependable
My analytics consultant designed a process for tracking incoming data and identifying cleaning needs. I secured senior leadership support for cleaning, restructuring, and quality controls from Excel inputs through dashboard outputs.
Develop the team and its partners
We established monthly and quarterly reporting, moved from Tableau toward Power BI, and built the team's skills. I also led lunch-and-learn sessions on measurable objectives for designers and business partners.
The wider build
A measurement practice that reached into the work.
Make the objectives measurable.
Weak objectives and assessment design were part of the reporting problem. I worked with designers and business partners on measurable objectives and connected them to the evaluation questions leaders needed to answer.
Plan the business follow-up.
The Level 4 approach centered on each Claims business line’s top two business metrics, with thresholds and follow-up at six, nine, and twelve months. That gave the team a plan for examining business signals alongside the learning measures.
Protect the history during migration.
I led the Claims L&D LMS team’s part of the Cornerstone-to-Workday transition. Before the old system went dark, I had the team export and track the data and created a SharePoint backup for continued collection. That backup preserved our historical data when the migration lost records. I also shortened the Level 1 survey from twelve items to five while the analysis system was unavailable.
One part of the build / Workplace research
What got in the way of application.
I conducted a mixed-method Level 3 study after 90 days, using stratified sampling, surveys, and interviews with learners and leaders. The study examined what people applied and what helped or got in the way.
Leader support
Learners described gaps in the guidance and support they needed from managers after training.
Peer capacity
Experienced colleagues had limited capacity for the buddy system, and other peers were helping newer colleagues.
Fit with field work
Managers and learners questioned how closely parts of the curriculum matched what people faced in the field.
Knowledge access
The knowledge base was described as outdated and disjointed, making useful information difficult to find.
From findings to action
What I recommended, and what followed.
I recommended support that continued after the course: accessible job aids, focused refreshers, leader reinforcement, peer support, and recognition. The purpose was to help people keep using the skills as time passed.
Make support available in the work.
For gaps in curriculum fit and knowledge access, I recommended job aids and short refresher learning, with particular attention to support around 90–120 days. Instructional designers developed job aids and training for rollout.
Give reinforcement an owner.
To address gaps in leader and peer support, I recommended manager one-on-ones, mentoring, and recognition for applying skills. The buddy system was implemented, and leaders had opportunities to recognize employees through organizational rewards and public acknowledgment.
Keep the follow-up visible.
I recommended stronger monitoring so the team could track the support and application measures. We implemented dashboard and monitoring changes immediately.
What changed
A function leaders could use.
Strategy adoption
A wider evaluation practice
The team revised and adopted the approach across Levels 1 to 4. We connected reaction and learning measures with workplace follow-up and a plan for examining business signals.
Reporting and trust
Cleaner data and more dependable reporting
By my estimate, reporting accuracy reached roughly 90–95% after the cleaning and quality-control work. Senior leaders' feedback indicated greater confidence in the reports.
The approach
The research and evaluation method.
- 01
Frame the evaluation
I created the strategy for reaction, learning, and behavior transfer, with attention to what people applied at work.
- 02
Plan the follow-up
The Level 3 study looked at behavior and understanding after 90 days, using stratified sampling.
- 03
Hear both perspectives
Surveys and interviews with learners and leaders gave the study quantitative and qualitative evidence.
- 04
Review the findings
I brought together the findings on leader support, peer capacity, curriculum fit, and access to current information.
Watch the process
How a finding could guide the next step.
An illustrative handoff example connects an observable behavior, the evidence to collect, and a decision.
A closer look · 32 seconds
How I would measure learning transfer
A handoff example that connects a behavior to evidence and a leadership decision.
Loading the motion demo…
Make the handoff clear. Use the job aid to name an owner, the next step, and a due date.
I created this example to explain my measurement approach. The handoff and findings are synthetic, and the decision is conditional.
Read the complete story
- 01 / Define the behavior
Make the handoff clear.
Use the job aid to name an owner, the next step, and a due date.
- 02 / Plan the evidence
Look at the work over time.
Compare work samples before and after learning. Observe the work and ask follow-up questions.
- 03 / Understand the gap
Find out why the gap remains.
In this synthetic example, the owner is missing. Does the workflow make responsibility clear?
- 04 / Choose the next step
Use the findings to choose a response.
If responsibility is unclear, work with the process owner to clarify it. Review another handoff to see what happens.
This is a practice example. Measured improvement, adoption, and return on investment have not been established.
Selected work
Look at the work more closely.
Select a visual to enlarge it and read its caption.
Measurement architecture
What each measure can tell us
Each level answers a different question. To understand transfer, I would look for people using the learning in their work. A positive course rating gives us information about the learning experience.
An illustration of the questions I use to connect learning and workplace application.Worked example
A plan for checking learning transfer
The behavior in this example is using a job aid during a handoff. The plan connects that behavior to the evidence we would collect and the decision it could support.
A handoff scenario that shows how I would plan the evidence.Interface concept
A dashboard for the next decision
The concept puts the behavior and available evidence first. Questions about barriers help the leader work out what needs attention.
A dashboard concept for exploring a finding and the next decision.Have a question about this work?
Email Dennis