Six levels of evaluation
The ROI Methodology sets objectives and measures results at six levels.
Objectives are first set for Level 5, the desired ROI or profit from the event, its contribution to shareholder value in the business world, or its contribution to the mission of a non-profit organization. It ends with defining the target audience. However during all planning phases objectives are clearly assigend to the targeted category of potential participants. Only the detailed definition of „Who should we invite to achieve a desired impact (level 4)“ happens at the end.
Objectives on each of the 6 levels become measurable by setting clear success criteria or KPIs (key performance indicators) at the initial stages of planning. This provides the data to measure just how successful it was. The evaluation sequence works in the reverse order, starting at Level 0 and working step by step towards the top. Results are measured by using different data collection methods. There is a close interconnection between all levels with a knock-on effect if one is not carefully designed to serve the next level objective.
5 Return on investment
ROI is another way of expressing the contribution to profit made by an event in the corporate world. The profit is the net value created by the event minus the event costs. For association and government events, profit is usually not an objective, the ultimate value is the mission or political purpose for which the organisation exists, this is the mission that replaces ROI.
4 Impact
The Impact, or Business Impact, is the ultimate value contribution of the event to its stakeholders. The impact is used in the profit and ROI calculations. For a customer event, the Impact is usually sales. For an internal event it is typically organisational effectiveness. The business impact data may be obtained from company records, such as sales or cost accounting. More ‘soft’ data may be obtained by asking participants to estimate the business impact of certain changes in behaviour.
3 Behaviour
Behaviour is the application of learning. Learning without application is in most circumstances without value. You may give the participant a certain product experience, but if he or she has no opportunity to apply the newly acquired knowledge or skill, the learning was wasted. “What exactly do you know that participants should do differently after the event?” Self-reporting is a quickest and cheapest method, if maybe not as accurate.
2 Learning
What cognitive change (i.e. learning) is required for the participants to change their behaviour? At an exhibition for example you want to increase the requests for appointments with sales post-event to drive revenue for a specific product. So what should participants learn to buy more of this product? Where is the gap? Just about all meeting or event activities fall under the definition of learning, which includes the learning of information, skills, attitudes and relationship learning.
The method of self-reporting makes it quick and easy to measure all forms of learning. It may not be as accurate as tests or other methods, but the greater degree of accuracy rarely justifies the extra time and complication of measurement.
1 Learning Environment & Satisfaction
How can we design a learning environment which will make cognitive change most effective? Learning is influenced by the state of mind of the learner as well as ambient factors. We measure delegate satisfaction (i.e. through online surveys) with facilities, logistics, form and content. But participant satisfaction is essentially a proxy variable for the quality of the learning environment, which is what really matters if learning, in one form or other, is what must happen in order to change delegate behaviour and provide value to stakeholders.
0 Target Audience
Finally, how can we ensure that the right people are attending. The target delegates are those with the greatest learning (level 2) and behavioral gap (level 3) in the population of potential participants. There is no need to ‘speak to the converted’, similarly, there is no need for participants to learn something for which they have no application.
Isolating the effects of the meeting
Whenever we measure business impact, we must always use at least one method to isolate the effect of the meeting. If sales went up after the customer event, how do you know that it was because of the event and not the advertising campaign which was launched at the same time?
There are several methods for isolating the effect of the meeting and in general it can always be done. The best method is to do a control group experiment but the method can be difficult and costly, in particular the careful selection of the control group. An alternative method is to ask participants to estimate the effect of the event compared to other factors.
[hr height=”30″ style=”default” line=”zigzag” themecolor=”1″]