The Official Blog of Benifys HR Solutions

Delivering a training program is not enough, the cycle does not end there. Measuring the effectiveness of training programs gives a realistic idea of how effective the training program really has been, it’s technical preciseness nowithstanding. Today, we look at one of the most widely used tool for this purpose, which is Donald Kirspatrick’s four level model, for evaluating the impact of training programs on short term or long term goals of the organization.

The need to measure training effectiveness

• To gauge the effectiveness of the current ongoing training programs in an organization
• To identify gaps in training output and use the information to improve the training initiatives
• To evaluate whether the budget allocated for the training activity is justified, and whether the program has a good ROI.

Donald Kirkpatrick and his Four Level evaluation model

Donald L Kirkpatrick, Professor Emeritus, University Of Wisconsin (where he achieved his BBA, MBA and PhD), first published his ideas in 1959, in a series of articles in the US Training and Development Journal. Kirkpatrick then went on to write a book (first published in 1975) called “Evaluating Training Programs” in which he devised a new model that he called the four levels of evaluating training program.

Now widely recognized as the authority on measuring training effectiveness, his four level theory has since then gained worldwide popularity and is arguably the most widely used model for the evaluation of training and learning programs. Kirkpatrick’s four-level model is now considered an industry standard across the HR and training communities.

According to Kirkpatrick, each one of these four levels is equally important and every level has an impact on the next level. Also, as you move along one level to the next one, the process progressively becomes more difficult and time consuming, but it also provides more valuable information.

The Four Levels – A Brief Overview

Level 1 : Reaction

Kirkpatrick likes to call this step a measure of customer satisfaction, which is because this step involves gathering feedback from the trainees about they the felt about the training program, not much unlike how customers are asked for feedback about services. This step is important because it evaluates the engagement level and hence the receptivity of your audience. It is an understatement that without receptivity, the entire training exercise would be meaningless.

In his own words – “It is important not only to get a reaction but to get a positive reaction. The future of the program depends on positive reaction. In addition, if participants do not react favorably, they probably will not be motivated to learn. Positive reaction may not ensure learning, but negative reaction almost certainly reduces the possibility of its occurrence.”

Level 2 : Learning

By learning, Kirkpatrick means measuring the extent of change in the competency of the participant in terms of positive attitude change, improved knowledge and/or increase in skills as a result of attending the program. A lot depends on the use of an appropriate learning channel and the consideration paid while designing the training program to the needs of different learning styles of trainees (i.e., visual learners, kinesthetic learners, etc).

As Kirkpatrick puts it himself – “Some trainers say that no learning has taken place unless change in behavior occurs. Learning has taken place when one or more of the following occurs: Attitudes are changed. Knowledge is increased. Skill is improved. One or more of these changes must take place if a change in behavior has to occur”

Level 3 : Behavior

It is one thing to learn and absorb information and quite another to apply it to actual situations, this step is intended to measure the degree to which the trainee has internalized the learning. Some trainers tend to bypass the first two levels, i.e., reaction and learning, and get straight to measuring changes in behavior. This is a serious mistake. The problem is that the obvious conclusion that will be drawn will be that the training was ineffective and that it should be discontinued. This solution may not be very accurate, because even though the training might have been technically flawless and does well on the first two levels, it may not guarantee a change in behavior.

According to Kirkpatrick, four conditions are necessary to bring about behavioral changes:

1. The person must have the desire to change
2. The person must know what to do and how to do it
3. The person must work in the right climate
4. The person must be rewarded for changing

If no change in behavior is observed after training, the reaction and learning should be re-evaluated to check whether the training delivery or content was effective, if it was, then maybe the climate is not conducive to change. Trainers should actively find out what kind of climate the trainees will face after the completion of the training; a failure to provide the right climate is a major reason why most training programs don’t achieve their set objectives.

This is what Kirkpatrick says about providing the right climate – “One way to create a positive job climate is to involve bosses in the development of the program. It is suggested that that bosses should be asked to help determine the needs of sub-ordinates. Such involvement helps to ensure that a program teaches practical concepts, principles, and techniques. Another approach is to present the training program, or at least a condensed version of it, to the bosses before supervisors are trained”

Level 4 : Result

The final evaluation of the training program is measuring the final results that occurred because the participants attended the program. The various metrics it includes could be increase in production, decline in costs, reduction in employee turnover, higher profits. It is on these results that the scope and future of the training department depends on. It is important to mention here how pertinent it is to set objectives before the commencement of the training program, if this is not done right, it becomes very difficult to determine just how effective the training was.

In the words of the Master himself – “It is difficult if not impossible to measure final results for programs on such topics as leadership, communication, motivation, time management, empowerment, decision making, or managing change. We can state and evaluate desired behaviors, but the final results have to be measured in terms of improved morale or other nonfinancial terms. It is hoped that such things as higher morale or improved quality of work life will automatically result in tangible results”

A word of caution – implementing the Kirkpatrick model requires engagement and sponsorship of the executive management. Very often, the feedback derived from an assessment of the first two levels is positive, but the third and fourth levels leave a lot to be desired. Two primary reasons, as elaborated above, come in the way of training activities achieving their potential – lack of a conducive environment, and lack of a direct correlation between business results and the training activity. It is important for organisations to a) set clear objectives, b) obtain executive level approval of the content, c) provide concrete and visible support to the training activity and a playground to enable the participants apply the newly learnt skills at the workplace, and d) establish clear monitoring and reporting mechanisms so that the influence of non-training variables is minimized in the fourth level assessment.

360° – An Overview:

360 degree feedback is arguably the most comprehensive feedback process involving an all-round assessment of performance of an individual by people at all levels. It is not just an appraisal method, but also a very powerful development tool. It can be instrumental in not just measuring the current performance, but also identifying the gaps in the knowledge/skills/abilities/attitudes/behaviours required for the job and those held by the incumbent. It not only reflects the judgment/opinion of those up the line, but also the opinions/perspective of those down the line. Professionally managed, 360 degree feedback increases individual self-awareness, and as part of a strategic organisational process can promote:

• Increased understanding of the behaviours required to improve both individual and organisational effectiveness.
• More focused development activities, built around the skills and competencies required for successful organisational performance.
• Increased involvement of people at all levels of the organization.
• Increased individual ownership for self-development and learning.
• Increased familiarity with the implications of cultural or strategic change.

360° as an Appraisal Tool:

360 degree feedback is a process whereby an individual (the recipient) is rated on their performance by people who know something about their work (the raters). This can include direct reports, peers and managers and in some cases customers or clients, in fact anybody who is credible to the individual and is familiar with their work can be included in the feedback process. This is usually in addition to completing a self-assessment on performance. The resulting information is presented to the individual with the aim of helping them to gain a better understanding of their skills and development areas. Each source can provide a different perspective on the individual’s skills, attributes and other job relevant characteristics and thus help to build up a richer, more complete and accurate picture than could be obtained from any one source. 360 degree appraisals involve the appraisee receiving feedback from people (named or anonymous) whose views are considered helpful and relevant. The feedback is typically provided on a form showing job skills/abilities/attitudinal/behavioural criteria and some sort of scoring or value judgment system. The appraisee should also assess themselves using the same feedback instrument or form.

As an appraisal tool, 360 degree feedback ensures more participation and involvement on the part of employees at all levels. It also gives a more balanced view of the individual’s performance as it eliminates some defects like halo effect, fear of persecution, political manipulation etc. inherent in traditional appraisal methods.

360° as a Development Tool:

360 degree feedback is also a powerful tool for training and development. Evaluation on the 360 degree model helps identify the gaps in not just the qualifications and technical aspects of the job requirement, but also in the behavioural and attitudinal aspects. This provides a basis for planning training and orientation activity. 360 degree is also useful in succession planning, career planning and leadership development. Giving an all-round assessment of the skills, abilities and attitudes of an individual, it helps in identifying his/her strong and weak traits. This, in turn, helps chalk out a person’s career path and also find out the likeliest successors for all positions. Also, taking the same test at different stages in one’s career helps to evaluate the training and coaching initiatives.

The benefit of 360 degree employee feedback is the identification of strengths, weaknesses, mentoring insights, and areas needing professional development

360° – Process:

1. Planning

Planning involves establishing the purpose and the process for the 360 degree feedback to be put into place. This means finding out whether the organization is culturally ready for 360 degree feedback to be implemented or not, whether the objectives are clear, the management is committed to the process and those who are going to take part are involved at the planning stage. Also, there has to be a clear process to identify the raters, the questionnaire has to be reviewed to ensure it is relevant, the structure of the feedback has to be decided and, where computers are being used, system security has to be ensured.

2. Piloting

Running a pilot is of vital importance to the process as it will help iron out any difficulties related to the relevance of the questionnaire, the effectiveness of the feedback survey, the efficiency and security of the systems as well as the responsiveness of the employees to the survey.

3. Implementation

Once a pilot run is done, the implementation becomes much smoother as the snags are ironed out in the pilot stage itself. For the process to be a success, it has to be ensured that clear and positive communication is open and everyone is made aware of the role that he/she is to play. It is beneficial to have a team constituted to administer the process. Also, a contact point, or ‘helpline’, should be established to whom queries can be addressed. Deadlines should be clearly and firmly communicated and the confidentiality of the process should be stressed.

4. Feedback

Feedback can be communicated to the individual in two ways i.e. in a face-to-face interview with a facilitator or as a report without any person-to-person interaction. The best practice, however, would be to take the help of a facilitator so that the feedback can be interpreted in the correct light and in a positive manner. The facilitator in question should be someone who is conversant with the process, the criteria and the policies. The facilitator can help focus the discussion on the future development planning.

5. Review

Reviewing the success of the 360 degree feedback is of vital importance as it will help to further smoothen and refine the process.

360° – Advantages and Disadvantages:

Advantages

As a tool for performance management and training and development, 360 degree feed back has the following advantages:

• It enhances performance quality
• Provides specific, itemized performance feedback.
• Identifies the target development areas.
• It can be a strong motivational tool.
• Facilitates performance improvement be pinpointing the areas that need improvement.
• Is useful in measuring the effectiveness of training programs.
• It enhances self-knowledge.
• It is supportive to continuous learning.
• It improves the reliability and validity of performance information.

Disadvantages

360 degree feedback is not, however, without drawbacks; some of which may be:

• It might result in the undermining of authority of managers due to the pressure of upward appraisal.
• Low evaluation might result in loss of credibility among subordinates.
• Managers may focus on trying to please subordinates in order to get higher appraisals.
• It cannot be used as an absolute measure of job-specific skills.
• Managers may be at a loss on how to interpret ratings from subordinates relative to those from superiors and peers.

Things to Be Aware Of

Given the nature of the 360 degree feedback and the level of participation required, certain things have to be borne in mind before the system is put into implementation:

• It is imperative to gain organization-wide acceptance of the system to ensure the fair and smooth functioning of the process.
• Prior information be given to all those involved and communication channels kept open so that all queries can be cleared.
• Ensure that the instrument in use is valid, clear, relevant and reliable.
• Selection of raters should be made carefully keeping in mind that they have to be familiar with the appraisee’s work and give them the opportunity of opting out if they are not.
• Proper training and guidance be provided to the raters to help them understand their role.
• Promote participation at all levels.
• Administer the process consistently and regularly.
• Be sensitive and respectful towards the employees.
• Ensure that offensive feedback by coworkers is not communicated to the employee.