Innovation Metrics is about determining, measuring and interpreting the indicators of development and flow of an innovation through the innovation process. Ultimately, innovation accounting enables transparent and fast decision-making by all involved in the process of Experimentation, Scaling Up and Embedding of an innovation.
Details
Measuring the success of an innovation is more difficult than it may seem. Especially in the early stages of the innovation funnel, the business value that an innovation generates is either still an assumption or it is growing at an incomparable speed towards an unknown potential. Traditional methods of measurement of such value, such as Return on Investment, Net Present Value or Discounted Cash Flow do not offer any insights, since both the investment required for the development of the innovation and the earning potential are unknown. In much the same way, existing KPI’s for product sales, market growth, profitability etc. will not apply to the innovation in early stages.
Since 2010 a number of lean/agile methods of measurement have gained traction that has proven to be particularly effective in early-stage development environments, such as start-ups and innovation. Exactly these methods are useful during the development and Scaling Up Phases of innovations, when monetary indicators are insufficiently representative. Examples of such methods are The Lean Startup, David McClure’s Pirate Metrics or Sean Elli’s Growth Pyramid. Each of these methods serves the same purposes, which are almost identical to the purposes of Innovation Accounting.
Innovation Accounting is about managing:
- Investment decisions at different points in the innovation funnel
- The progression of success of innovation projects
- The impact that innovation is having on the business as a whole
In their book Lean Analytics, Alistair Croll and Benjamin Yoskovitz propose a metrics framework called ‘Lean Analytics Stages’. Although the authors specifically refer to their stage as ‘start-up-stages’, the ‘stages and gates’ that they propose apply equally to the development and growth of innovations. The table below lists Croll and Yoskovits’ stages and gates and applies them to the innovation funnel phases:
| Innovation phase | Lean Analytics Stage | Gate |
| Experiment | Empathy | Proof that the innovation solves a problem and that a market exists |
| Experiment/Scaling Up | Stickiness | The innovation works and people are willing to pay for it |
| Scaling Up | Virality | The functionality keeps users engaged and on board |
| Scaling Up | Revenue | Users and features fuel organic and artificial growth |
| Scaling Up/Embedding | Scale | The innovation provides a sustainable, profitable business |
The Importance of Growth Rate
In addition to the stages and gates above, Croll and Yoskovitz measure the ‘growth rate’: the speed at which the innovation grows throughout the Scaling Phase. Where in start-ups this growth rate is important to investors and incubators, in incumbent organizations the growth rate is especially important for the Continuous Innovation Board (CIB) to determine the rational investment in money and time. The exact measurement of this growth rate differs per stage and per situation. However, the growth rate primarily measures the speed at which the innovation moves from one stage to the next rather than the adoption rate within a stage.
Four Steps to Effectively Use Metrics in Every Stage
Although the use of metrics is different in each phase of the innovation, the same steps are required to measure the effect of an innovation, regardless of the phase it is in.
- Identify drivers for success and the goals that define success
- Define metrics that measure these drivers and goals
- Collect data related to the defined metrics
- Interpret the data and take action in short intervals
One Metric That Matters
Innovation analytics are particularly difficult to use and interpret because they often lack a benchmark against which to measure success. Or, in case of innovation in larger corporate environments, the organization may be tempted to use the existing business as a benchmark, even though this may be a poor indicator with regards to the innovation. Therefore, defining the metrics that measure success (step 2 in the list above) is far more difficult than it may seem at first. Our natural response is often to add more data to create a ‘bigger picture’. In practice, however, because of our lack of understanding of the market mechanics of the innovation, adding more data usually leads to less understanding.
Successful start-ups have proven that of their primary keys to growth is in their focus on a single performance indicator. Even though they track multiple indicators, they stick to the ‘One Metric That Matters’ for the stage of development that they are in. In the Lean Analytics Stages these metrics provide a single view of the empathy, stickiness, virality, revenue and scale. In a similar way, a single metric should be used to measure the Scaling Up and Embedding of the innovations in your organization.
Which exact metric is favorable as ‘the One Metric That Matters’ differs per innovation. Two things are certain though: the one metric should be discussed, decided and adhered to by the entire innovation team and Business Owner during the Lean Analytics Stage that you are in. Your One Metric That Matters also defines when you believe you are successful in each stage. Until you have reached this point, keep making changes for the better before you switch focus or strategy. If you cannot seem to achieve your target, rethink your innovation!
Metrics during the Experimentation Phase
As a starting point of experimentation, all validated ideas in the innovation funnel have a feasibility hypothesis and a rough business case (the value hypothesis). During the Experimentation Phase, the innovation is tested against these hypothesis.
The Experimentation Phase goal is twofold:
- Prove that the innovation can work
- Get early customer feedback on usability and perceived value
Both goals have different measurements, each clearly defined in the hypothesis statement “We believe that…. To prove this we will…. Our hypothesis is validated when….”. The metrics for measurement of success (ie. validation of the hypothesis) are therefore defined in the hypothesis statement. Each Experimentation Phase can have multiple hypotheses validated and therefore track multiple metrics in a single experiment. It is pivotal though that each hypothesis statement formulates a SMART-defined measurement.
Metrics during the Scaling Up Phase
Moving from proof-of-concept to scaled-up rollout of the innovation in the business requires different metrics than the Experimentation Phase. Success is no longer defined as ‘getting a working version’ but as ‘getting impact in the market’. Measuring such impact requires actionable metrics and governance focused on transforming from achieving growth in development to achieving growth in impact. Focus shifts from ‘proving our earlier assumptions’ to ‘enabling decision making towards the future’. In his book ‘How to measure anything’ Douglas Hubbard states that ‘management needs a method to analyze options for reducing uncertainty about decisions.’ Even though in the Scaling Up Phase the existence of the market is proven, it’s size is probably still unknown. Revenue potential is based on assumptions, and the key competency to be developed is the ability to capture the market.
By now, management should be convinced that the product will work and has a fair chance of creating and growing revenue. Now it needs proof that this potential is real and can be realized with sufficient profit margin and a sense of direction when choices need to be made. The purpose of metrics in this stage is exactly that: find the best way to create stickiness and virality to generate sustainable turnover from the innovation and find the optimal means to grow revenue from existing customers and through market expansion. This is the time and place where the Innovation Team and Business Owner decide on the innovation’s Key Performance Indicators and define the ‘One Metric That Matters’.
The key metric is a ratio
The one metric that matters should be a ratio which compares a given result to an earlier measurement. A metric could be ‘the total number of users’. An example of a ratio is ‘the number of new users per week’. Better still, the ratio can reveal insights into multiple metrics combined, such as the number of new users and the frequency of use per user. The ratio for these two metrics might be ‘average number of logins per user per week’. Ratios are a good way to describe intended behavior by clients without the limitation of a single metric. This is why ratios are excellent drivers for creative and diverse actions. If your aim is to attract more and more avid users for your innovation, the metric ‘new signups this week’ can only tell you whether you should spend more attention to acquisition. If your ratio combined the new signups with the ‘average number of logins per user per week’ might point you to acquire a specific group of users that turn out to be the one most likely to login often.
Set up testing
The score in a metric represents the status of your innovation at a given point in time. The point of measuring this status is to create a benchmark for change. By taking action you intend to change the current status of the innovation for the better. Obviously, there are multiple ways to achieve this improvement and no guarantees that either way will achieve the desired result. Therefore, testing changes is a structured process in which multiple changes are executed in parallel and measured to find out which of these change achieved the best results.
In innovation, both the product usage and the way its user base responds to it are new to your organization. This is why in the analysis of each change you make you need to take into account not only the change to the product but also the response of specific users or groups to this change. Hence, apart from defining the specific product change(s) by which you expect to achieve the improvement, you should also consider the target group to which you present these changes. This is done by selecting the right kind of tests to run the comparisons for product changes and their user group(s). Tests can be run across different types of comparison:
- Segments: the response of different groups of users with a similar characteristic (for instance age, region, product usage etc.) to the same product is measured.
- Cohorts: the response of different groups of users based on the time they started using the product (early adopters, long-term users, novices) to the same product is measured.
- A/B or split-run tests: Two similar groups of the total userbase get presented different versions of the product.
- Multivariate Analysis: multiple A/B tests run are simultaneous, with statistical analysis used to determine which change is most beneficial.
Qualitative analysis
Although automated and statistical tests provide many insights in the development and scaling up phase of an innovation, qualitative tests provide insights that no data analysis will easily reveal: the why behind people’s liking of an innovation. This key metric is also referred to as the ‘product-market fit’. There is no better feedback from customers than the direct contact between the innovation team and the innovation users. For this reason, surveys and interviews remain key to achieving success in the scale-up of innovations. An interesting example of such a survey is the one that is offered at http://survey.io/survey/demo. This particular survey was used by the creators of survey.io to measure their own start-up product, which is now a successful online surveying tool.
Measure stage gates
The measurement of each of the Lean Analytics stages has 2 goals:
- measure the effect of changes that you make to the product (or the way you present or promote it) in order to achieve your scale-up target
- measure whether you have achieved your target
Even though the latter seems obvious, it is important to realize clearly when you have achieved your goal within the stage, because this is the point where you should change your growth strategy. Moving from stickiness to virality to revenue means changing goals and with it the measure for success. Although the exact metrics to choose at each stage differ highly per innovation, the list below does provide you with insights into what you could measure per stage:
Stickiness
Are people using the innovation as expected? If not, is there something blocking them from using it the way you intended? Or are they using it to solve a problem you were not aware of? In either case, you can decide to make changes to the product or pivot your plan to support the success that your innovation is displaying unintended.
Are your users experiencing enough value from your innovation? Are they coming back for more after the first trials? Are they willing to pay (enough)? Do they refer the innovation to other users? Is it being discussed in the company or in the press?
Virality
Is the adoption rate of your innovation increasing fast enough? Are users inviting new users? If not, what is keeping them? Do you actively provide them with options for a providing a referral?
What is the cycle time of referrals? Are users promoting your innovation straight away or do they take some time to before they start promoting your innovation? What can you do to increase this cycle time?
Revenue
Is your revenue growing (metric) or better: is your revenue per user growing (ratio)? If not, what causes this? Are your acquisition costs per user growing? Are you losing users faster than you are acquiring them? Is your pricing wrong?
Metrics for Non-Commercial Innovations
Not all innovations are actually commercial products or services. Some are process innovations or internal services. Even though the overall indicator for success for such innovations is not the total revenue they generate, much of the same metrics that are used for commercial products do apply to non-commercial innovations. Ratios for stickiness and virality equally apply to the adoption of non-commercial innovations. Think of the adoption of a new HR-app, a new security measure, colleague location tracker in your new flex-office etc.
Do keep in mind that when you work on a non-commercial innovation, there is always a cost/benefit ratio involved. No organization will use an innovation unless it has a problem that needs solving. The cost of the innovation needs to be lower than the cost of the problem, the cost of the solution that is being replaced by the innovation or the cost of not having the additional benefit that the innovation delivers. This perspective of cost/benefit accounting should have been formulated in the value hypothesis of the innovation early on in the validation phase. During the Experimentation Phase it should have become clear that it is at least reasonable to assume that such a cost/benefit analysis would result in a positive outcome during Scaling. Therefore, metrics similar to commercial innovations can and should be applied to non-commercial initiatives.
Metrics during the Embedding Phase
In the Embedding Phase, the innovation becomes the full responsibility of Business Owners in the line management of the organization. For commercial products, each innovation should get a revenue-accountability, for non-commercial innovations a cost-accountability should be brought into place. In the Embedding Phase, the key metrics for success are defined by the existing organizational process for target-setting and innovations seize to be exceptions to the rule.
This does not mean that during the Embedding Phase, the innovation needs to be ‘squeezed into the existing mold’. Rather, the organization should decide which targets are viable (and challenging) for this innovation to achieve and which requirements are set on the existing organization to be able to meet these targets. If not already done so, can the organization incur costs for changing the existing architecture? How do we train existing staff to sell and maintain the innovation? Do we change reporting lines and target metrics?
It should be clear to everyone involved in the innovation process when the innovation is considered ‘embedded’ into the organization. It is therefore advisable to determine a specific moment in time when the innovation metrics are considered to have become integrated with the regular reporting processes of the organization. From the Scaling Up Phase, the revenue metrics should be tuned in such a way that during the Embedding Phase the revenue metrics are gradually (but as quickly) converted to business-as-usual metrics, or vice-versa when the business-as-usual metrics have been adapted to the scale-up metrics.
References
- Hubbard, D. (2007). How to Measure Anything. Newark: Wiley.
- Croll, A. and Yoskovitz, B. (2016). Lean analytics. Beijing: O’Reilly®.