The scientific method for venture creation and growth
Founders must apply a scientific approach to venture building for a higher chance of successRead more
Without the element of uncertainty, the bringing off of even the greatest business triumph would be dull, routine, and eminently unsatisfying. – J. Paul Getty
The task of developing good innovation metrics is complicated. Research reveals hundreds of suggestions about what to measure about the effectiveness of your innovation system and the value of the products, services and businesses that result.
An analysis was undertaken of hundreds of actual and proposed innovation metrics documented in the literature. This analysis was used to create an innovation system framework and model to help understand and categorize the various metrics. What emerged were the following five categories of innovation measurement. These can be used as an organizing framework to create a measurement capability for any organization’s innovation system.
When these are effectively measured (with specific KPIs), tracked over time and normalized by the costs incurred, a picture of both activity and impact emerges. With this picture, we can now begin to accurately assess the effectiveness of an innovation system.
How do you measure something that could have an (uncertain) impact 3, 5, 10 or even more years from now? What measures are causal and what are merely correlative? Measuring innovation is difficult for many reasons but the ultimate cause of the difficulty is that current actions and decisions are often widely separated in time from ultimate outcomes. Unlike other business processes, innovation can take years and substantial amounts of capital and expense before its outcomes can be effectively measured. During this time, many things can intervene that are not at all related to the quality of the underlying opportunity. The intended outcomes are often far into the future and, thus, they are inherently uncertain and difficult to predict.
To create a credible method of measuring an innovation system, it is useful to create a framework that describes the various dimensions of a measurement system. A framework that has proven to be useful in creating an effective measurement system consists of the following components:
Activity and impact – Companies can use (1) metrics of activity, which are usually objective, precise and relatively easy to gather, and (2) metrics of predicted outcomes which are most always subjective, imprecise and also relatively easy to gather. Knowing which is which and using them in the proper context is important.
A-priori and post-hoc – Predict what future outcomes or wait to get actual data. This is a constant tension in any innovation management system and an area in which new, AI-based tools will have significant impact in this area.
Individual opportunity and opportunity pipeline – Individual opportunities need to be measured along a variety of dimensions (not covered here) that are collectively used to predict the opportunity’s potential. However, individual opportunities should never be assessed in isolation but always in relation to other opportunity options. The innovation system needs to be measured on its ability to produce a steady stream of high potential opportunities.
Real-time and long-term – Immediate course corrections and long-term improvements are both desirable and an innovation measurement system should provide information for both.
Measurement Categories, KPI’s, Instrumentation, Data and Dashboards – In many cases, defining what and how to measure gets confused. Clear distinctions between Key Performance Indicators (KPIs), the instrumentation needed to track the KPIs, the data produced by the instrument and how the data is used are confused. An explicit measurement taxonomy is useful in clearing up much of the confusion.
None of these framework dimensions is an either-or proposition and they all interact with each other. For example, activity and outcome are intimately entwined and both need to be measured both predictively and post-hoc. The complex interactions between these various measurement dimensions can best be understood and developed using an explicit model of the innovation system.
We know that certain activities and decisions lead to better innovation outcomes but often don’t explicitly connect the two in our measurement systems. We know that it is awfully hard to predict outcomes (especially for very early disruptive opportunities) but we want to try to do it anyway. We want to make decisions about individual opportunities, about collections of opportunities at similar stages of development and about the entire opportunity pipeline. We want to use our measurements in real-time, but also take a long-term view so we can improve the system over time.
To do all of this and to connect the various parts of an innovation measurement system and design the proper KPI’s, measurement instruments and information representations, it is extremely useful to have a model of the system you want to measure. A model can indicate to us precisely where we need to instrument the innovation system, what we need to measure and how to connect activity (process) with outcomes at each stage.
The following diagram depicts a (simplified) model that can inform the measurements we need to take to see how well the innovation system is operating and how it could be improved.
Figure 2 – A System Model for Innovation Measurement
This model has four (4) phases labeled A, B, C and D (phase X is important but outside the scope of this article), five (5) operational outcomes labeled 1-5, and five (5) categories of measurements to be implemented. The four phases are as follows:
X. Strategy – This phase establishes the environment within which the innovation system operates. It is critical to the operation of the innovation system, but the activities involved in developing strategy usually take place outside of the innovation system itself (although they shouldn’t). Measurements of the Strategy process are not addressed here except in the context of the overall environment within which the innovation system operates and the ‘jobs’ that the innovation system is doing for the company.
A. Domain – Domains and the activity of ‘Domaining’ translate strategic intent into innovative action by defining where the organization should look for new opportunities. The key thing to measure about your Domain phase is whether it is stretching the boundaries of the organization enough and in the right directions.
B. Discover – The Discover phase is where new opportunities are identified. Employing a number of different channels and means of discovery is useful here and each will have its particular characteristics. Volume and quality are important but the primary measurement goal for discovery is assessing the breadth, depth and pace of learning (and as a result the creative concept formation) from the various networks employed for that purpose.
C. Incubate – This phase is where specific opportunities get tested through real-world experimentation and iteration. The flow of opportunities through this phase needs to be measured to ensure that the right opportunities enter and ‘graduate” when they are ready. It is also necessary to ensure that the company has the absorptive capacity to accelerate the opportunities that graduate.
D. Accelerate – It is only at this phase that actual economic and adoption outcomes start to unfold, and can therefore be directly measured.
With a model such as this, one can identify specific operational outcomes that can and should be measured and start to answer ‘what-if’ questions.
A recent article in HBR describes a study of 154 companies that use the Spigit idea management system to analyze data on new concepts as they move from initial idea to reality. The key conclusion of the study is:
“The key variable that we identified across all the companies in our analysis is the ideation rate, which we define as the number of ideas approved by management divided by the total number of active users in the system.”
Four rate factors were identified
No one will argue that volume is not significant (and needs to be measured) but the definition of ‘success’ used by the study investigators is problematic:
“… the number of ideas approved by management”
This definition is part of the problem! Despite our sincere belief in our ability to identify which ideas are the best, experience has shown that human judgement is not up to the task.
There’s a long way to go from defining success as ‘approved by management’ to defining success as ‘adopted by the market’. The metrics we use should strive for the latter.
In the above model, four phases of an innovation system are identified and connected to create a continuous flow of opportunities from concept to reality. In this system, five high-level operational outcomes have been identified.
Note that one thing that many people focus on, how much money (revenue, profit, ROI etc.) will an opportunity create, is not in this list. That is not because it is not important, it is because it is a dependent variable that can be (and should be) derived from these outcomes. Other ‘size of potential market’ factors are taken into account when estimating potential adoption size and rates as well as overall opportunity value (see below).
Given these 5 operational outcomes, we can ‘run’ the model to see what the results of improvement in each of these areas would be. The following is a chart showing what a 10% improvement in all 5 of these outcomes would result in. 
Figure 2 – Effect of Improving the Innovation System by 10%
This chart shows the expected difference in revenue between baseline (existing innovation system performance) and an innovation system that operates 10% better in all five of the listed performance areas. As you can see, there is a potential to more than double the revenue created by an innovation system over 5 years.
The model above identifies key operational outcomes of an innovation system and lists five (5) measurement categories needed for monitoring these operational components. This model can be used to identify specifically what to measure, how to measure it and how to use what is measured
Each measurement system setup will be different. There are different processes, methods and tools used, different governance rules and responsibilities, different organizational structures and different people involved. Each measurement system needs to be customized to the specific situation by choosing the specific KPI’s to use, how to instrument the system, how the data is collected and processed, how the data is visualized and what is acted on. Despite this customization, the five general categories of measurements identified above are the ones that are important to keep in mind when creating a specific set of KPIs, instruments, data analytics and visualization.
Value – What is the uncertainty-adjusted value of specific opportunities as well as the total innovation opportunity pipeline?
You need different measurement tools and methods for opportunities at different phases.
Flow – What is the rate of opportunity generation, throughput and yield of the pipeline? What is the flow per $ spend and how is this changing over time?
This is a set of values and ratios based on when opportunities achieve certain levels of readiness (i.e. specific uncertainties have been eliminated). From this data, the following things can be calculated.
Network – How well are the internal and external knowledge creating networks working to facilitate learning and creativity? Are the internal and external networks robust and effective for knowledge and relationship creation and assimilation?
Measures of how connected the organization is, internally and externally and how effective the network is at creating and assimilating new knowledge.
Stretch – How is the company’s strategic intent being pursued and how are the company’s boundaries being extended in appropriate ways?
Measures of how the organization is expanding into adjacent and new areas that are strategically aligned and important.
Jobs – Are we doing the right innovation jobs and doing these jobs right? How are jobs changing? Note that these are not the jobs of individual people, they are the jobs that the innovation system is doing for the company.
Perform a periodic assessment of the jobs the innovation system is doing and should be doing.
With a model and its corresponding measurement strategy, we can focus on developing and using better instruments for measuring activities and predicting outcomes.
The fundamental condition at the heart of any innovation endeavor will always remain – the future is uncertain. Uncertainty about the future is, after all, what an innovation system is set up to exploit.. Paradoxically, a well-functioning innovation system should both create uncertainty (for yourself and for others) and resolve it. It creates uncertainty through the constant imagining of new things the world has never seen. It should manage and reduce uncertainty by changing how the world behaves in ways that allow the innovations it produces to be adopted.
Uncertainty plays a central role in innovation and, therefore, it makes sense for an innovation measurement system to directly measure and track this. As far as we know, this has not been tried. Consider what follows as a modest proposal to experiment with a different way to measure innovation.
The principle behind a new measurement approach is that what really matters for an innovation system is how well you can create claims about the future in the face of inherent uncertainty about those claims. The approach goes something like the following.
There are numerous details to be worked out and experiments to be done before a measurement approach such as this can be validated, but it is intriguing nonetheless since it points to some very real areas of research and development using new AI and machine learning based tools and methods.
Predicting the impact of opportunities, especially transformational opportunities that are at a very early stage of the innovation journey, is very difficult for people to do. The track record shows an estimated 30% or less success rate with these types of opportunities, which are arguably the most important ones.
New tools are emerging, however, that use AI, predictive analytics and big data sets to help with the problem. One such tool is the Predictive Testing of Opportunities (PTO) service that is based on technology developed by Growth Science.
You don’t have to wait 10 – 20 years for your own innovation pipeline to produce results. With PTO, you can take advantage of the success or failure of thousands of new products and services that have been introduced over the years. By comparing the opportunity with this vast data set, the AI system can accurately predict whether it will succeed or fail and what the approximate growth rate will be.
As a company’s innovation system becomes more critical to the long-term success of the company, the desire to measure and improve it increases. Innovation, however, unlike other business processes, is an area that defies the development of precise and unambiguous metrics.
The temporal decoupling of cause and effect, the ambiguous and complex nature of information and knowledge and the inherent irrationality of human behavior create tremendous uncertainties that are not seen elsewhere in business and which complicate the creation of meaningful metrics and the means to measure what makes a difference.
One of the things that can help in the effort to create a meaningful innovation measurement system is to have a clear framework and model of how the innovation system is constructed and how it works. With such a framework and model, it is possible to identify exactly what should be measured to increase or decrease, compress or expand the desired outcomes. It is then possible to design the most useful KPI’s, instrumentation, data gathering and analysis and decision support tools possible.
In the future, as our innovation tools get better, as they become more automated with AI and machine learning, the measurements required will simultaneously become simpler and more complex. Simpler for the humans who are running the system but more complex in their internal data gathering, analysis and predictive capability.
The effectiveness of our future innovation systems depends on our ability to measure how effectively they are operating and our ability to use these measurements to keep evolving our innovation systems. If we do not do this well, we will ultimately be the victim of those who do.
 One of the issues of innovation metrics is that the word ‘Innovation’ can be used as a verb, as in ‘we are innovating’, and as a noun, as in ‘our innovations drive growth’. Innovation, as used here, refers to the activity (verb) of innovating and the term ‘opportunity’ is used to refer to the output of the activity of innovating. Both need to be measured.
 For just one example of how many and varied innovation metrics can be see The Conference Board Working Paper ‘Signposts of Innovation: A Review of Innovation Metrics’
 Recent years have seen significant efforts to reduce the time between action and outcome (and thereby make measurement much easier). The whole lean-startup, test and learn movement is directed at doing precisely this.
 The results shown here are based on model parameters such as rate of opportunity concept generation, % of opportunities that make it to next phase, % of commercial offerings that fall short, meet or exceed ROI targets, average time per phase, time to midpoint on adoption curve (months), and many others. For more details on the model, please contact the author.
Founders must apply a scientific approach to venture building for a higher chance of successRead more
Discover how to build a successful strategic innovation teamRead more
Discover why Innovation is an indispensable catalyst for business success.Read more