Determining Levels for Training Evaluations

There are generally considered five different levels at which a training or learning intervention can be evaluated, ranging from “Level 1” (which simply captures learners’ reactions to the intervention) up to “Level 5” (which calculates business-level return on investment). [This widely-used labeling system is based on an evaluation model first developed by Donald Kirkpatrick.] Simpler evaluations are less expensive than more advanced levels of evaluation, which can require significant investments of time and effort.

Therefore, careful consideration should be given to determining the appropriate level of evaluation for any given specific curriculum, course, or learning initiative. Different courses or curricula need different evaluation strategies, and you should always keep in mind that it may not be necessary to do all levels of evaluation. For example, a Level 2 evaluation alone may be most appropriate for a particular course, while Levels 1 and 3 are appropriate for another course.

You should also be aware that some levels of evaluation require prerequisites. For example, a prerequisite for Level 3 evaluation is that a Level 1 evaluation has been conducted. And it will almost always be the case that a Level 3 evaluation is a prerequisite for Level 4 and Level 5 evaluation.

So how should you determine what type(s) of evaluations should be used for a given course or training or learning intervention? While there is no simple algorithmic formula for making this decision, your decision will be much easier if you take into account a small number of specific factors that should help determine your decision in each case.

The checklists that follow summarize the major requirements for each level of evaluation. They are intended to help you in planning your evaluation strategy.

Directions for Using the Checklists

First, identify the learning intervention that is under consideration. For each specific one, you should work sequentially through each checklist level below. (As noted above, recall that some levels are prerequisites for others.)

Next, read each element in the checklist. If you believe the specific course or intervention you are considering meets that statement’s requirements, place a check in the box beside the statement.

Finally, review the number of statements you have checked. While there is no pre-set number of statements that should be checked in order for you to proceed with a given level of evaluation, you probably should check at least 75 to 80 percent of the items in order to ensure wise expenditures of the resources that will be required for evaluation.

Level 1 Factors Checklist

Level 1 evaluations measure learners’ reactions to the learning intervention.

• There is a need to systematically monitor the results of a learning initiative/event over time
• Data from the evaluation will be used to improve the learning initiative or event
• Data from the evaluation will be used to decide whether to continue, refine, or discontinue the learning initiative or event

Level 2 Factors Checklist

Level 2 evaluations measure the extent to which a learning intervention has improved the learners’ knowledge of the specific content covered in the curriculum.

• The learning intervention meets all of the considerations for a Level 1 evaluation (Note: this item is required)
• A reliable measure of the learner’s competency (skills, knowledge) relative to specific job position or role is required
• There is a need to certify that learners have obtained knowledge that increases the likelihood that they can safely and effectively perform key tasks in the workplace
• A pre-test or assessment can be conducted before the learning initiative or event occurs to establish a baseline
• Management is likely to request and review data reporting on the specific results of the learning initiative or event
• Assessment expertise or tools exists to ensure construction of a reliable instrument
• Adequate time is available to construct, test, and analyze evaluation items to ensure that they are valid and discriminate between learners of differing abilities and skill levels

Level 3 Factors Checklist

Level 3 evaluations measure the extent to which specific learning initiatives bring about changes in learners’ behaviors that directly support business goals, performance needs and/or productivity on the job.

• The learning intervention meets all of the considerations for a Level 1 evaluation (Note: this item is required)
• The learning initiative/event is costly to conduct
• The learning initiative/event is resource-intensive
• The learning initiative/event is in support of key business goals or strategic initiatives
• A high level of participation by the work force in the learning initiative/event is expected
• The learning initiative/event is expected to have longevity (i.e., will be in place for multiple years)
• Career development and/or promotions are linked to the initiative/event
• Learning initiative/event receives, or is likely to receive, management scrutiny
• It is realistic to expect that the learning initiative will result in changes in behavior that can be observed within 30 days after the event (to isolate the impact of the learning intervention)
• It is reasonable to assume that managers will be able to observe changes in behaviors as a result of the learning initiative
• The evaluation/observation is not likely to impact performance
• The learning initiative/event specifies clear, measurable behavioral objectives or goals

Level 4 Factors Checklist

Level 4 evaluations analyze the extent to which the learning produced the intended results/impact at the level of the business as a whole.

• The course or intervention meets all of the requirements for Level 1 evaluation and most of the considerations for Level 3 evaluations (Note: this item is required.)
• It is possible to estimate the business outcomes with sufficient precision to provide a credible, rigorous foundation for conducting a Level 4 evaluation
• There is reason to believe that the aggregate business-level impact of a learning intervention might be different than (either greater or less than) the sum of the impacts on individuals
• The ultimate outcome of the learning intervention can only be observed at the level of the business unit, rather than at the individual level

Level 5 Factors Checklist

Level 5 evaluations calculate the return on investment (ROI) for learning interventions by monetizing both the business impacts that have been identified in a Level 4 evaluation and the costs of the learning intervention

• The course or intervention meets all of the requirements for Level 1 and most of the considerations for Level 4 evaluations (Note: this item is required.)
• The course or intervention meets a majority of the specific considerations for Level 4 evaluations (Note: this item is required.)
• It is imperative to know whether the benefits of a learning intervention justify its costs.
• The learning intervention of sufficient criticality—in terms of the resources that are devoted to it or the strategic objectives that it is designed to achieve—to merit the trouble and expense of conducting a ROI analysis.
• It is possible to identify all aspects of costs (direct, indirect and opportunity) and estimate the monetary value of the benefits with sufficient precision to provide a credible, rigorous foundation for conducting a Level 5 evaluation.

This evaluation checklist was provided by McBassi & Company with permission from Laurie Bassi, CEO of McBassi & Company. She is former vice president of American Society for Training and Development (ASTD) where she created internationally recognized standards for measuring and valuing firms' investments in education and training, and a core set of quantitative indicators for measuring the effectiveness of knowledge management initiatives. She holds a Ph.D. in economics from Princeton University. McBassi & Company helps organizations discover how their Human Capital and Human Resources programs are influencing business.

This article was re-uploaded from our internal archive.