Practitioners tell us that creating the Outcome Statement is one of their biggest challenges in implementing TDRP because it is often hard to get sponsors to agree on the percentage contribution of the HR initiative to their goal. We understand the challenge this presents and offer the following guidance for using qualitative assessments and proxies when it is not possible to make a quantitative assessment of impact.
Let us know what you think and where else you need help.
Dave Vance
Executive Director
Introduction
Frequently, the use of quantitative impact assessments (like training will lead to a 2% increase in sales) in the Outcome Statement and Summary Report is not possible. This may be because the sponsor is uncomfortable with the process of assigning a percentage contribution from the HR initiative (like a 20% contribution toward the goal of increasing sales by 10% = 2% increase in sales due to training) or because the sponsor is simply uncomfortable with the concept of an isolated, quantitative impact. This is often the case when organizations first adopt TDRp and it may take a year or two to get the sponsors comfortable with this new approach. Quantitative impact assessments may also not always be appropriate for “softer” goals like increasing employee engagement or improving leadership. While the goal should always be specific (like a 3 point increase in the employee engagement score), there may be a feeling that it is not appropriate or possible to assign a percentage contribution to each of the different HR initiatives supporting the goal.
Qualitative Impact Assessment
So, when there is resistance to quantitative assessments, consider the use of qualitative assessments. A qualitative assessment still focuses on the expected, isolated impact of the HR initiative on the business goal, but characterizes the impact as High, Medium, Low, or Essential. For example, the impact of a new performance management process might be expected to have a medium impact on employee engagement, especially if last year’s survey had indicated significant dissatisfaction with the old process. The sponsor impact assessment methodology (see note at end) recommended for quantitative assessment may still be used, but the process stops after the sponsor has identified and rank ordered the factors expected to contribute to achieving the goal. Simply look at the list and see where the HR initiative appears in the ranked order. If it is at the top, then consider assigning it a High impact. If it is at or near the bottom, then consider assigning it a Low impact. If it is in or near the middle, then consider assigning it a Medium impact. Before making the final assessment, however, you will need to consider the relative importance of the factors.
A High impact should mean that the factor makes the biggest contribution to achieving the goal and that it will, by itself, be responsible for most of the change in the goal (equivalent to a 50% or more percentage contribution in a quantitative assessment). At most there will be one High impact factor. A Medium impact should have a considerable but not dominant impact (equivalent to a 25%-50% percentage contribution). Typically, there would be just one or two Medium impact factors. The rest will be Low impact factors (equivalent to a 5%-20% percentage contribution). There may be cases where there are no factors which qualify as High. This could occur when there are just two factors of equal impact (both would be Medium) or when there is one Medium and several Low, or when there are five or more factors of equal impact (all would be Low).
For example, suppose the sponsor (SVP of HR in this case) provided the following rank ordered list of factors expected to contribute to increasing the employee engagement survey results by 3 points:
Planned salary increases due to an improving economy
Leadership development initiative to improve frontline supervisors
Increased reach of L&D and increased spending on L&D
New performance management process
Expansion of career planning discussions to all employees
New CEO
Assume the first is the predominant factor but that the second will also play a major role while the last four will have a limited impact. In this case the first would be a High factor, the second a Medium factor, and the last four would be Low.
There will also be times where the sponsor and you agree that the initiative is Essential to accomplishing the business goal. “Essential” should only be used if the business goal cannot be accomplished without it. Many goals can be accomplished without the help of HR but it will take longer and incur greater cost. So, there is good business reason to invest in the initiative, but this doesn’t make the initiative Essential. Examples of Essential initiatives would be training for combat troops (they cannot be sent into combat without training), compliance with regulations that require training (the regulation cannot be satisfied without the training), and call center employees (they cannot be allowed to interact with customers until they are trained).
So, there is still some professional judgment required in a qualitative assessment and it is less precise than a quantitative assessment, but it still accomplishes the key goal of conducting a thoughtful discussion with the sponsor where all the key factors are identified and rank ordered resulting in agreement on the expected, isolated impact of an initiative.
Proxies
In contrast to both quantitative and qualitative assessments, the use of a proxy does not attempt to assess the direct impact of the initiative on the goal, but instead provides a measure which is believed to be either a leading indicator or at least highly correlated to the (unobservable) impact. So, if you run into resistance with the qualitative assessment, or if it does not seem appropriate to attempt it, then you can always fall back on a proxy. The object here is to reach agreement on a proxy which is acceptable to you and the sponsor, and which each of you believes to be highly correlated with the impact. In TDRp terminology, proxies for impact will be effectiveness or efficiency measures. So, if you need a proxy, review the effectiveness and efficiency measures in the library for candidates. Typically, the chosen proxy will be easily measured and in many cases is already being collected. When multiple proxies are available, choose the one (or perhaps two) that are expected to be most highly correlated with impact on the goal. In other words, choose the one which appears to be strongest “stand in” for the isolated impact of the initiative on the goal. In L&D, the proxy of choice is usually the application rate (level 3) which is just one step from impact (level 4). The application rate may be observable or may be self-reported. (In fact, the self-reported application rate is itself a proxy for the actual application rate!)
When you use a proxy, consider supplementing it with an additional efficiency or effectiveness measure. For example, if you choose the application rate as a proxy for an L&D initiative, also show the number of participants which will allow you to tell a more complete and convincing story. For example, “The sponsor and I agree that if we deliver this particular training program to 250 specially targeted sales reps by March 31 and 80% apply the three key lessons from the training within 60 days and continue to apply it throughout the year, this training should have a significant impact on the goal to increase sales by 10%”
Now, it would be nice to reach agreement on the percentage contribution or at least a High/Medium/Low contribution, but the next best thing in this case is to get agreement on a proxy like the application rate and combine it with at least one other efficiency or effectiveness measure.
Conclusion
When it is challenging, difficult, or impossible to obtain sponsor agreement on the quantitative isolated expected impact of an initiative, try to reach agreement on a qualitative assessment of impact which still accomplishes the key TDRp goal of conducting a thoughtful discussion with the sponsor where all the key factors are identified and rank ordered resulting in agreement on the expected, isolated impact of an initiative. If that is not possible, then agree on a proxy to represent the expected, isolated impact of the initiative and consider supplementing it with an additional effectiveness or efficiency measure to tell a more complete story. The use of a proxy will still accomplish a key goal of TDRp to have a thoughtful, proactive discussion with the sponsor resulting in agreement on measures of success.
In other words, don’t allow your efforts to implement TDRp to be sidetracked over difficulty in get sponsor agreement on expected, isolated impact. If you encounter resistance go to Plan B (qualitative assessment), and if you still encounter resistance go to Plan C (proxies). Keep moving forward and remember Progress over Perfection!
Note on Sponsor Impact Assessment Methodology
This is the process in which the sponsor identifies all the key drivers (contributors) expected to lead to achieving the business goal, prioritizes the factors, and then assigns a percentage contribution to each one. The percentage contribution assigned to the initiative in question is then used to calculate the expected isolated impact of the initiative on the business goal. For example, suppose that the SVP of sales identified nine factors which should lead to the planned 10% increase in sales for next year. She rank ordered them and the sales training initiative (consultative selling skills combined product knowledge) was ranked second behind economic growth (an expanding market). She assigned a percentage to each factor so that the percentages for all nine factors totaled 100%. Now, whatever percentage is next to the sales training initiative is her best guess of the expected isolated impact of training on achieving the 10% increase in sales. If the percentage is 20%, then training is expected to deliver one fifth of the 10% sales increase, or more precisely, training is expected to lead to a 2% increase in sales (20% contribution x 10% goal to increase sales = 2% higher sales due just to training).