When we talk about evaluation in training, we tend to think of tests administered in the class to check our understanding of the content. However, let’s look at assessments from a broader perspective.
Let’s look at a system of evaluation that has become a training industry standard: the Kirkpatrick model of evaluation. This model acknowledges that we evaluate for a number of different — and unique — reasons and in a number of different — and unique — ways.
Level 1: Reaction
Evaluation reaction to training usually takes the form of end-of-course evaluations. While they can be used to improve the training, they should NOT be interpreted to show how effective the training was. End-of-course evaluations (or “smile sheets”) identify whether someone liked the training or whether they thought the training was effective.
Here are some tips for making the most out of this type of evaluation:
- Determine the purpose.
- Create specifically for each training: Avoid using “one-size fits all”. If you have a standard form, customize it by adding questions specific to your course.
- Write majority of questions close-ended: Getting anecdotal quotes from students is interesting and makes for good marketing. However, few of us have time to conduct a content analysis on the comments. Use more quantitative measures.
- Balance positive and negative questions: Asking participants to list the most important things about the training without asking about the least important things leads to an imbalanced, biased feedback.
- Ask questions about transfer and impact: Go beyond the traditional smile sheet. Ask “What, if anything, will make it difficult for you to use your new skills on the job?” “Will your manager be able to help you with your new skills?” “How confident are you that you will be able to use your new skills on the job?” “How do you expect your job to change as a result of using these new skills?”
- Provide time: Give time and expect participants to use the allotted time to complete the feedback thoroughly and thoughtfully.
- Consider handing the evaluation out at the beginning of the training and encourage participants to complete it throughout the training.
Level 2: Learning
Just because we know that participants like our training, doesn’t mean that they’ve learned anything. We need to determine whether they learned something.
Level 2 evaluations can be used to determine if learning occurred. This information is good for three primary reasons:
Does the participant have the required skill and knowledge? These level 2 evaluations are also known as summative tests or certification exams.
Where does the participant require remediation or additional development? These level 2 evaluations are known as formative tests or developmental evaluations.
How can the training be improved? Sometimes that absence of learning indicated in level 2 evaluations is because of the teaching style or the materials. Level 2 assessments can tell you WHAT needs attention; and with time and analysis, can tell you WHY some participants are picking up the content and others aren’t.
Tips to using level 2 evaluations effectively:
- Determine the purpose: Why are you evaluating the learning? Who needs to know (students, managers, the instructor)? Why (improve the training, hire/fire/promote, give participants a gauge)?
- Link tests to performance: If participants need to know something, design the test to measure knowledge (multiple choice, true-false, essay, matching, short-answer). If participants need to do something, design behavioral evaluations (i.e., role plays or demonstrations) with a measurement instrument (Behavior Checklist, Behaviorally Anchored Rating Scale, Behavior Frequency Checklist, Best-Solution Approach). See the examples on the right for simple performance-based checklists.
- We’re making a clear distinction between knowledge and performance for a very good reason. Just because you know something doesn’t mean that you can do it. Having knowledge of brain surgery is not the same as having experience performing brain surgery.
- There are two primary types of Performance-Based Evaluations: labs for procedural performance and role plays/demonstrations for non-procedural (principles).
- Pilot the test and validate: Strong performers should do better than weaker performers; the test should give similar results over time.
Level 3: Transfer
Just because we know participants liked our training (level 1) and they learned something (level 2), doesn’t mean that they can apply it back on the job (level 3)
As performance consultants, we need to measure if participants are using their newly acquired skills and knowledge on the job. Level 3 evaluation can be used to improve the training and to identify work environment barriers that prevent participants from using their skills and knowledge.
Level 3 evaluation can be as simple as observing performance; gathering information from participants, managers, and others; or accessing records/artifacts such as performance data, error data, customer satisfaction information, etc.
While evaluating learning transfer may be an expensive and thankless task involving control groups and quantitative statistics, the goal is to ensure that learning is transferring to the job. Here are some suggestions for making sure that training transfers:
If you are management (any level above supervisor):
- Use an advisory committee to ensure training addresses business needs.
- Communicate the importance of training in general and this training specifically.
- Coach supervisors appropriately; model coaching behavior.
- Establish standards for training follow-up and coaching by supervisors.
- Provide time for training and follow-up.
If you are a supervisor (the trainees’ immediate manager):
- Develop performance measurements to measure gains from training.
- Attend a training orientation for supervisors.
- Select trainees based on specific criteria.
- Conduct a pre-training meeting with trainees.
- Use graduates of previous training to prepare trainees.
- Allow time to prepare for the training.
- Plan for after training follow-up.
- Self-assess his/her skills to identify and focus needs.
- Complete precourse work.
- Identify performance needs; conducts task analyses to determine content and flow.
- Identify other elements impacting performance (job design, information, resources, feedback, etc.)
- Provide an advance letter to the supervisor.
- Coach the supervisor on ways to increase training transfer and effectiveness.
- Provide an advance letter to the trainees.
- Provide precourse work.
- Plan for after training follow-up.
Level 4: Business Impact
Just because we know that 1) participants liked our training, 2) learned something in our training, and 3) applied what they learned on the job, we still don’t know if the business is better off.
Training impact refers to the measurable change in profits, revenue, errors, or other operational factors linked to your training. This is also called Return on Training Investment. (See Entelechy’s Return On Training Investment Calculator.)
How do we determine the benefits of our training?
- Work with the client to identify — before the training — what performance measures would be important and credible.
- Determine what data would point to success in your training. Determine how best to collect the data (without excessive administrative work).
- Determine what other factors are important to the desired performance.
Knowing WHAT you’re evaluating and WHY you’re evaluating will help you determine HOW best to evaluate. Using Kirkpatrick’s four levels of evaluation will help you use training evaluations effectively.