4 Indexes for Measuring Large-Scale Technical Training Effectiveness
4 Indexes for Measuring Large-Scale Technical Training Effectiveness
Dr Raman K Attri 
This article describes guidelines for to use of an intuitive and practical 4-tier model for the measurement of technical training effectiveness through 4 simple indices: training reaction index, improvement index, effectiveness index, and impact trends.
I wrote some papers and articles on measuring the effectiveness of training (including impact and ROI) back in 2005. I conceptualized a simple, intuitive, and practical 4-tier “Return on Effectiveness (ROE)” model for measuring training effectiveness and ROI as an engineer would do. Since then I have received quite many requests, particularly from technical training managers to elaborate on the approach a bit more. Somehow, the approach I originally published showed quite an instant connection with a technical audience who had not a whole lot of business responsibilities directly. Sometimes Kirkpatrick’s Training Evaluation model and Phillips’ ROI models, though widely used, either turn too abstract or turn too infeasible for engineering managers to even apply. This post is not an attempt to discredit existing models of training effectiveness or ROI measurement. The model may initially look like generic models like Kirkpatrick’s model. However, the attempt here is not to over-confuse new technical training managers with training lingo.
All in all, I believe professionals and training managers needed a ‘common-sense’ measurement approach – a simple feasible way to measure the effectiveness of the training. The model which I proposed was geared toward the language of ratio and indices, and a practical way to implement this approach that makes an instant connection with technical training managers. In this post, I will share a 4-tier workable model of measuring training effectiveness of complex training operations with a focus on briefly how to implement or use it.
The whole picture
While the model below is described from a training standpoint but the moment we talk about on-the-job measurement, it is a collective measurement of learning and on-the-job performance which could be a result of several other support factors. Complex training dynamics and factors impacting the outcome may be hard to isolate as the sole effect of training per se. The basic premise of the approach described in this post is that training does not work in isolation. It has a close connection with other support functions like direct management of the employee, performance support systems, and continuous learning that happens post-training. The business performance of any company comes as a result of the learning and performance of employees, not just because of training. Therefore it should actually be a measurement of learning impact in the business rather than training alone.
The general outline of the model is described in the figure. It has 4-tiers, in line with the most popular training effectiveness models. The model contains 4 tiers.
- Tier-1: Reaction index
- Tier 2: Improvement index
- Tier 3: Effectiveness index
- Tier 4: Impact index
4-Tier Training Effectiveness Measurement Modle (Image copyrights: Raman K. Attri)
Tier-1: REACTION INDEX: How closely training program meets the expectations of trainees?
- Objective: Measure the training reaction of employees in terms of perceived skills/knowledge
- Who: Training Participants
- When: At the end of the training
- Outcome: Reaction Index (RI)
- Source: End-of-the-training self-assessment and pre-training baseline
Measurement
This tier measures the training reaction – a feedback score collected from each trainee at end of the class using a well-drafted survey form. You might say –what’s new. You have been doing it all your life. No. what you might have been doing is measuring the reaction of trainees to how they ‘feel’ about the training.
In complex business settings, your goal is to get how soon you can get your trainee ready for the job. You are interested in the results which give that impact the business. You would not want to go smiley sheet ‘how well the trainer taught the class and whether or not objectives were clearly stated or whether or not the material was comprehensive’.
Data Collection
The thing is that employees come to a training event to learn on-the-job skills that can make them successful in a short time. That’s what the ‘expectations’ are. And what you want to do is to measure how well you meet the expectations. With changing business needs, what makes sense is to measure the instant confidence gained by trainees from the training even on the on-the-job skills required by them to do their job. At this stage, you can use participants’ post-training self-assessment of skills (knowledge and attitudes as well) they just acquired in the training in terms of their “perceived” confidence and competence level. These scores can be normalized or calibrated by instructors’ measurement of their skills as indicated by training assessments. This will provide a quantitative measure of their readiness for on-job assignments for which they came to training.
Intent
Put it simply, the higher the scores are, the closer you are to meeting the business needs. Remember that these employees are the ones who help you meet the business needs in the field and your business needs cannot be separate from their on-the-job skill needs or their on-the-job skills needs cannot be different from your real business needs.
Applying the model
In order to use this information to improve your training alignment with business, you need to run the exact same set of surveys before trainees come to the training on the exact same skills/measures. That data will give you the baseline scores of their pre-training skills and other performance measures. With the perceived confidence and competence scores and baseline pre-training scores available to you, you can compute an index called Reaction Index.
REACTION INDEX = ratio of post-training perceived skills score vs. pre-training baseline score
Making Training Decisions
- Ideally, Reaction Index should be > 1.
- A large Training Reaction Index will mean that the training program has high relevance to the participants and they see the training aligned well with the way they see their on-the-job challenges. Remember that RI is based on the perception of the participants and it may not hold reality. However, it is an important measure.
- What if Reaction Index < 1? It means that you need to re-assess the scope of the training and critically review your measurement instrument to see if it reflects on-the-job requirements correctly.
Tier-2: IMPROVEMENT INDEX: How well has training improved the on-the-job skills of trainees?
- Objective: Measure observed improvement in on-the-job skills as a result of training
- Who: Employee Supervisor
- When: At the end of 3 months post-training
- Outcome: Improvement Index (II)
- Source: Business unit employee performance data sources
Measurement
Now since the employee has started doing his job and applied skills he learned during training, a supervisor is bound to see visible indicators of changes in job behavior, skills, and attitudes. Though there are several factors, including the influence of the direct manager, responsible for this, the changes in skills and attitude in the first 3 months can be attributed to training. The 3 months are just good enough period to assert the improvement due to the training program but short enough to establish the long-term effectiveness of the course or its impact on the business. Therefore you need an intermediate measure called ‘improvement’ with a measurement instrument (survey or through job-performance data) to be put in place 3-months post-training.
Data Collection
The complex organizational settings make use of a single instrument infeasible. Therefore you may have to use an integrated system that combines data inputs (qualitative and quantitative) from various sources like company CRM, escalation matrix, task trackers, customer satisfaction scores, stakeholder 360 degrees, assignment data, and so forth. The data from these sources can be used to get the supervisor to rate the skills (knowledge and attitude included if applicable) learned in training. If only skills are involved then the data can be consolidated to reflect the supervisor’s rating of the confidence and competence of the employee on all the skills received during training. The instrument can be subjective or objective or can be one in which subjective measures are objective measures.
Intent
The instrument is primarily for the supervisors and should be aligned very well with the baseline instrument and the instrument used during the training program to measure the expectations/reactions of trainees. As you see, we have isolated it from any “business” measurement at this point. Let’s first establish training indeed resulted in the improvement of skills and impacted the trained employee before we start measuring the impact on the business.
What you get out of this is the measure of observed performance of the employee in terms of skills (not in terms of business indicators) in the field. This data would show you improvement after training and how it is improving over time. This exercise will give a “relative” improvement before and after training in terms of meaningful “acquired & applied” skills, improvement in the attitudes, enhancement of knowledge, level of handling of assignments, and level of fulfillment of supervisor’s expectations.
Applying the model
You can use this information to compute the Training Improvement Index for the training program.
IMPROVEMENT INDEX = Ratio of post-training observed skills score vs. pre-training skills baseline score.
Making Training Decisions
- An ideal index would be > 1. The larger the ratio better the improvement training has created.
- What does an Improvement Index of 1 mean? It means training has not shown any improvement in the skills yet. This could be alarming – you need to watch it.
- You can compare this value to the Reaction Index which was computed using the same set of skills. By rule, due to field exposure and an increase in confidence, the training improvement index should be more than Training Reaction Index.
- What if the Training Reaction Index < Improvement Index? You have organizational trouble for which training is no longer a solution. Possibly the post-training performance support system and other supporting mechanisms do not reinforce the learning in the training. This also tells another story that training may not have been correctly designed in line with field requirements or field requirements may have changed since the training.
- A combination of the Reaction Index and Improvement Index will give you an assessment of your strategy and if you need to revisit your training program in light of new information.
Tier-3: EFFECTIVENESS INDEX: How effective is the training in changing employee performance parameters?
- Objective: Measure the impact of training on employee performance indicators
- Who: Employee Supervisor
- When: Quarterly
- Outcome: Effectiveness Index (EI)
- Source: Business Unit employee performance data sources
Measurement
By now you already have a good idea of improvement in skills training kicked in the employee group who received the training. Assuming that the ratio is more than 1, you need to move forward to see if employees have produced meaningful and measurable target results using their skills learned during the training program. This is the supervisor and BU owner’s assessment of the quantifiable impact on employee performance indicators due to training.
Data Collection
At this stage, instead of numeric scale numbers based on subjective judgment, a data-driven approach is taken. Data can be collected from an organization or BU’s CRM/ SAP or other integrated database repositories which track performance data for each business transaction and which can be broken down to an employee. Some of the examples of parameters indicating an employee’s performance are Service time, time to repair, time to close escalations, Productivity increase, Number of escalations handled, Case closure time, the success rate in closing support calls, etc. This data should be gathered as a baseline as a pre-training exercise and later should be gathered on a quarterly basis.
The employee performance parameters used here should be the one that was used to analyze and draft the learning outcomes of the training program which generated the skill inventory used for pre-training baseline measurements. The tough part is mapping the skills to on-the-job performance indicators. However, if a good job has been done to ensure training objectives reflect the real-world performance of the employee group, then the mapping may be easy.
Intent
The goal of this tier is to gather nearly the “quantified” impact of the training on key performance factors like time to service, man-hours, service cost involved, etc. Note that at this point we are still not getting into the business unit’s business performance collectively. If you want to measure the effectiveness of the training you need to first see the impact of training on the employee performance indicators. In short, it is called the effectiveness of the training in terms of making employees more productive for business needs.
Applying the model
The above information can be used to compute the Effectiveness Index if the pre-training employee performance baseline data is available. This index measures how much training contributed to or impacted meeting the individual employee’s performance goals as a business contributor
EFFECTIVENESS INDEX = Ratio of post-training numbers vs. pre-training average employee performance baseline data
What if an employee is new and baseline data does not exist? Well, in that case, the organization’s average baseline data can be used. In some instances, it is advisable to use organizational baseline data on average to take an integrated business approach.
Making Training Decisions
- Ideally, you want an Effectiveness Index > 1
- A large value of this index indicates a good translation of training into measurable employee job performance indicators. It also indicates that the training program is designed correctly and is effective.
- What if Improvement Index is high but Effectiveness Index is low? In that case, you need to analyze the mapping between training objectives and on-the-job performance indicators.
- What if Improvement Index is low but Effectiveness Index is high? This is likely not possible. You would have gone back to my recommendation of the Improvement Index being low and fixed the training before any other action. If the situation indeed happens, then training is not the contributory factor in improving the employee performance parameters.
Tier-4: IMPACT TRENDS: To what extent has training impacted the business metrics specific to an employee group, BU, or organization as a whole?
- Objective: Measure the impact of training on employee groups or BU or organization business KPI
- Who: BU owner
- When: 6-monthly or quarterly (whoever lines up with business measurements)
- Outcome: Impact Trends
- Source: Business unit and organizational performance data sources
If the above 3 tiers are in place, then BU will have enough data which can be averaged and normalized quarterly over the entire organization for various training programs and for various business units. Now the three indexes computed (averaged or normalized) above can be plotted quarterly in relationship to the key performance measurements or metrics of the given business.
Every organization has some sort of metrics in place for tracking its performance. Some examples of business KPIs are: Start to close Service time per call, Cost per call, Man-hours per case, Number of cases handled per engineer, Success rates of cases, Cost of labor per call, etc. During the training analysis exercise, the training objectives are mapped to the business indicators. Not all business indicators are used to develop the training program and hence not all parameters are required to compute the impact of the training program. For example, for a repair-related business, the measurements should be related to the number of cases handled, revenue from service, average hours spent in service, contract margins, and so forth as opposed to the organizational P&L. If the training program was aligned with BU’s objectives then it makes sense to use BU business performance data. Depending on the business unit’s business model, it is possible to compute Impact Index in relationship to some critical measurements. For example, if a business service call duration (SCD) is a critical business-driving factor, then the ratio of this SCD can be plotted with respect to two parameters: BU’s baseline data on SCD vs. quarterly average on SCD and Quarterly MTTR data vs. Effectiveness Index. In a nutshell, the possibilities are endless depending on the nature of the business. The measurement cycle should line up with the measurement cycle for the business. If a business is monitored quarterly then plotting this relationship makes sense quarterly.
Summary of Steps in Data Collection
The above approach involves a good amount of systematic data collection. This requires the implementation of a framework for such data collection to as much objectivity as possible.
Step 1: Pre-Training baseline data
– Employee performance indicator data
– Employee self-assessment on skills targeted in the training program
– BU or organizational baseline data on key business performance indicators (if available/ applicable)
Step 2: End-of-the-training program self-assessment data of each participant
Step 3: 3 months post-training supervisor’s rating on participant’s on-the-job skills
Step 4: Quarterly employee on-the-job performance indicator data
Step 5: Quarterly or six-monthly BU or Organizational key business performance indicators/matrix
For a detailed procedure for data collection, refer to the Return on Effectiveness (ROE): A Working Model for Measuring Effectiveness of Service Training.
Application
As a note, this model was developed back in 2005 and has not been updated with the more contemporary challenges of today’s business. Readers are advised to modify or adapt the model to their specific requirements. In general, I have applied the model in the context of complex technical skills and it works without much overhead other than instituting systematic data collection. In fact, the model expands thinking to stay aware of data that may be needed to implement such a model. Further, since the model presents the results in form of indexes, which may be easily trackable and easy to explain to upper management, rather than chasing the ROI in terms of investment and outcome with actual dollars.
SUGGESTED CITATION
Attri, RK (2014), ‘Training Effectiveness Measurement Demystified with A 4-Tier Model for New Technical Training Managers’, [Blog post], Speed To Proficiency Research: S2PRo©, Available online at <https://get-there-faster.com/blog/technical-training-effectiveness-measurement/>.
REFERENCES
- Attri, Raman K. 2005, “Return on Effectiveness (ROE): A Working Model for Measuring Effectiveness of Service Training”. https://dx.doi.org/10.13140/RG.2.1.1085.5765 and https://www.researchgate.net/publication/285587862.
- Kirkpatrick, DL & Kirkpatrick, JD 2009, Transferring Learning to Behavior: Using the Four Levels to Improve Performance, Berrett-Koehler.
- Kirkpatrick Partners, (no date). The Kirkpatrick Model. https://www.kirkpatrickpartners.com/Our-Philosophy/The-Kirkpatrick-Model
- Vance, A. 2017. The Bottomline on ROI: How to Measure the Results of Your Training, ROI Institute. https://roiinstitute.net/bottomline-roi-measure-results-training/
Image credits: Pixabay CC0 attribution
Image of “ROE” model- copyright Raman K Attri
Article originally published: Sept 11, 2014
