Beyond completion rates: how to tell whether training actually worked
A 96% completion rate tells you that people opened the module and clicked through to the end. That's attendance. It says nothing about whether anyone can now do the thing the training was built to fix — and when an executive asks "did the training work?", attendance is not the answer they're asking for.
The good news is that useful measurement doesn't require a research department. It requires a baseline, an assessment that tests performance rather than memory, one behaviour check a month or two later, and a business metric you agreed on before launch. Here's the working version.
Kirkpatrick, without the ceremony
The Kirkpatrick model — reaction, learning, behaviour, results — is sixty years old and still the most useful map, provided you treat it as a map and not a compliance exercise. The pragmatic reading:
Level 1: reaction (cheap, and weaker than it looks)
Satisfaction scores correlate weakly with actual learning — learners reliably rate entertaining sessions above effortful ones, and effortful ones are where learning happens. Keep Level 1, but use it as a smoke detector, not a success metric: a sudden drop flags a problem worth investigating. And swap "did you enjoy the session?" for the one reaction question with predictive value: "what will you do differently on Monday?" A blank answer to that is genuine signal.
Level 2: learning (only as good as the assessment)
Most Level 2 measurement is a recall quiz, and a recall quiz measures short-term memory of the module's wording. It's better than nothing, but only just. Level 2 becomes meaningful when the assessment is performance-based — more on that below, because it's the part most worth fixing.
Level 3: behaviour (the expensive one, and the one that matters)
Did anything change on the job? This is where measurement gets costly, because someone has to look: a manager check-in at 30 and 60 days against a short list of observable behaviours, a sample of real work products reviewed before and after, or QA/error data re-cut post-training. It doesn't need to be exhaustive — sampling 10 pieces of work beats surveying 200 people about their confidence. If you only have budget for one level beyond completion, spend it here.
Level 4: results (be honest about attribution)
Business results have many parents — market shifts, staffing, tooling, and, occasionally, your training. Claiming sole credit for a revenue movement destroys credibility with the finance-literate people you most need to convince. The honest posture: present directional evidence ("errors fell 30% in the trained group and 5% elsewhere over the same period") and let the triangulation speak.
Assessment that proves competency, not recall
The fastest upgrade to your measurement is upgrading what your assessments ask people to do:
- Recall question:"Which of the following is a stage of the escalation process?" Measures memory of a list.
- Scenario question:"A customer emails this (real, messy) complaint. What do you do first, and why?" Measures judgement — and wrong-answer options built from real past mistakes tell you exactly which misconception survives the training.
- Performance task: for tool and process skills, have them do the actual thing. For spreadsheet skills, that means building in a real workbook and grading the artefact — which is exactly what Mltitude's auto-graded Excel assessments exist for, because hand-marking workbooks is the reason most teams retreat to multiple-choice.
A useful discipline: for every learning objective from your needs analysis, ask "could someone pass this assessment and still be unable to do the job?" If yes, the assessment is testing the wrong thing.
Control-ish comparisons
You will almost never get a randomised controlled trial inside a company, and you don't need one. You need some comparison, because a before/after number on its own is nearly uninterpretable. Three designs that work in practice:
- Phased rollout: train region A this quarter and region B next quarter — which you were probably going to do anyway for capacity reasons. Region B is your comparison group for a quarter, free.
- Trend, not snapshot:plot the metric for six months before training, not one. A single before/after pair can't distinguish training impact from a trend already underway.
- Trained vs. not-yet-trained cohorts: where rollout is by team or intake, compare error rates across cohorts at the same tenure. Imperfect — cohorts differ — but stated honestly, far better than nothing.
Metrics leaders actually respect
Executives discount learning-native metrics (completions, satisfaction, even test scores) because they can't connect them to anything they manage. The metrics that land are the ones already on someone's dashboard:
- Time-to-competency: how long until a new starter passes the competency gate or handles the standard workload unaided. Onboarding programmes live or die by this number — see the 30-60-90 checklist for where the gates go.
- Error and rework rates:QA failures, corrections, credit notes, re-opened tickets — whatever your operation already counts as "done wrong."
- Escalation volume: how often frontline work gets bumped to expensive senior people. A drop here converts directly into senior hours, which converts directly into money.
- First-time resolution / first-pass yield: the fraction of work done right without a second touch.
The pattern: report training impact in the operational units the business already uses, and the conversation changes from "justify your budget" to "what else can you fix?"
The minimal measurement plan
For any programme worth building, four commitments — all made before launch:
- Baseline: capture the chosen business metric and, if possible, a pre-assessment before anyone is trained. You cannot reconstruct this later.
- A performance-based assessment gate: scenario questions or a real task, not recall. Record item-level results — they tell you which part of the training is weak.
- One behaviour check at 30–60 days: a structured manager observation or a work-product sample against three to five observable behaviours.
- One business metric with a comparison: phased rollout or trend line, reviewed a quarter after launch.
That's four artefacts, none exotic, and together they answer the executive's question with a straight face: here's what people could do before, what they could do after, what changed on the job, and what it moved in the business. Completion rate doesn't appear anywhere on the list — which is rather the point.