Five Things You Should Measure about Your Innovation System

Larry Schmitt

Innovation Partner

Highlight

• Innovation measurement is difficult because the future is inherently uncertain. Current actions and decisions are necessary for, but not determinative of, outcomes that may be realized many years in the future.
• To address this inherent uncertainty, it is useful to have a framework and model of the innovation system which defines the structural and operational components of the system as well as operational outcomes.
• Using such a model, five key categories of innovation measurement can be identified that apply to individual opportunities, to the opportunity pipeline or portfolio, and to the operation of the innovation system itself.

Without the element of uncertainty, the bringing off of even the greatest business triumph would be dull, routine, and eminently unsatisfying. – J. Paul Getty

The task of developing good innovation metrics[1]  is complicated. Research[2] reveals hundreds of suggestions about what to measure about the effectiveness of your innovation system and the value of the products, services and businesses that result.

An analysis was undertaken of hundreds of actual and proposed innovation metrics documented in the literature. This analysis was used to create an innovation system framework and model to help understand and categorize the various metrics. What emerged were the following five categories of innovation measurement. These can be used as an organizing framework to create a measurement capability for any organization’s innovation system.

1. Value – What is the uncertainty-adjusted value of the innovation pipeline?
2. Flow – What is the rate of opportunity generation, throughput and yield of the pipeline?
3. Network – How well are the internal and external knowledge-creating networks working?
4. Stretch – Are the company’s extended boundaries compatible with strategic intent?
5. Jobs – Is our innovation system doing the right jobs and is it doing these jobs the right way?

When these are effectively measured (with specific KPIs), tracked over time and normalized by the costs incurred, a picture of both activity and impact emerges. With this picture, we can now begin to accurately assess the effectiveness of an innovation system.

The Problem with Measuring Innovation

How do you measure something that could have an (uncertain) impact 3, 5, 10 or even more years from now? What measures are causal and what are merely correlative? Measuring innovation is difficult for many reasons but the ultimate cause of the difficulty is that current actions and decisions are often widely separated in time from ultimate outcomes[3].  Unlike other business processes, innovation can take years and substantial amounts of capital and expense before its outcomes can be effectively measured.  During this time, many things can intervene that are not at all related to the quality of the underlying opportunity. The intended outcomes are often far into the future and, thus, they are inherently uncertain and difficult to predict.

To create a credible method of measuring an innovation system, it is useful to create a framework that describes the various dimensions of a measurement system. A framework that has proven to be useful in creating an effective measurement system consists of the following components:

Activity and impact – Companies can use (1) metrics of activity, which are usually objective, precise and relatively easy to gather, and (2) metrics of predicted outcomes which are most always subjective, imprecise and also relatively easy to gather. Knowing which is which and using them in the proper context is important.

A-priori and post-hoc – Predict what future outcomes or wait to get actual data. This is a constant tension in any innovation management system and an area in which new, AI-based tools will have significant impact in this area.

Individual opportunity and opportunity pipeline – Individual opportunities need to be measured along a variety of dimensions (not covered here) that are collectively used to predict the opportunity’s potential. However, individual opportunities should never be assessed in isolation but always in relation to other opportunity options. The innovation system needs to be measured on its ability to produce a steady stream of high potential opportunities.

Real-time and long-term – Immediate course corrections and long-term improvements are both desirable and an innovation measurement system should provide information for both.

Measurement Categories, KPI’s, Instrumentation, Data and Dashboards – In many cases, defining what and how to measure gets confused. Clear distinctions between Key Performance Indicators (KPIs), the instrumentation needed to track the KPIs, the data produced by the instrument and how the data is used are confused. An explicit measurement taxonomy is useful in clearing up much of the confusion.

None of these framework dimensions is an either-or proposition and they all interact with each other. For example, activity and outcome are intimately entwined and both need to be measured both predictively and post-hoc. The complex interactions between these various measurement dimensions can best be understood and developed using an explicit model of the innovation system.

A Model Indicates What and How to Measure

We know that certain activities and decisions lead to better innovation outcomes but often don’t explicitly connect the two in our measurement systems. We know that it is awfully hard to predict outcomes (especially for very early disruptive opportunities) but we want to try to do it anyway. We want to make decisions about individual opportunities, about collections of opportunities at similar stages of development and about the entire opportunity pipeline. We want to use our measurements in real-time, but also take a long-term view so we can improve the system over time.

To do all of this and to connect the various parts of an innovation measurement system and design the proper KPI’s, measurement instruments and information representations, it is extremely useful to have a model of the system you want to measure. A model can indicate to us precisely where we need to instrument the innovation system, what we need to measure and how to connect activity (process) with outcomes at each stage.

The following diagram depicts a (simplified) model that can inform the measurements we need to take to see how well the innovation system is operating and how it could be improved.

Figure 2 – A System Model for Innovation Measurement

This model has four (4) phases labeled A, B, C and D (phase X is important but outside the scope of this article), five (5) operational outcomes labeled 1-5, and five (5) categories of measurements to be implemented. The four phases are as follows:

X. Strategy – This phase establishes the environment within which the innovation system operates. It is critical to the operation of the innovation system, but the activities involved in developing strategy usually take place outside of the innovation system itself (although they shouldn’t). Measurements of the Strategy process are not addressed here except in the context of the overall environment within which the innovation system operates and the ‘jobs’ that the innovation system is doing for the company.

A. Domain – Domains and the activity of ‘Domaining’ translate strategic intent into innovative action by defining where the organization should look for new opportunities. The key thing to measure about your Domain phase is whether it is stretching the boundaries of the organization enough and in the right directions.

B. Discover – The Discover phase is where new opportunities are identified. Employing a number of different channels and means of discovery is useful here and each will have its particular characteristics. Volume and quality are important but the primary measurement goal for discovery is assessing the breadth, depth and pace of learning (and as a result the creative concept formation) from the various networks employed for that purpose.

C. Incubate – This phase is where specific opportunities get tested through real-world experimentation and iteration. The flow of opportunities through this phase needs to be measured to ensure that the right opportunities enter and ‘graduate” when they are ready. It is also necessary to ensure that the company has the absorptive capacity to accelerate the opportunities that graduate.

D. Accelerate – It is only at this phase that actual economic and adoption outcomes start to unfold, and can therefore be directly measured.

With a model such as this, one can identify specific operational outcomes that can and should be measured and start to answer ‘what-if’ questions.

A recent article in HBR describes a study of 154 companies that use the Spigit idea management system to analyze data on new concepts as they move from initial idea to reality. The key conclusion of the study is:

“The key variable that we identified across all the companies in our analysis is the ideation rate, which we define as the number of ideas approved by management divided by the total number of active users in the system.”

Four rate factors were identified

• Scale – more participants.
• Frequency – more ideas
• Engagement –more people evaluating ideas
• Diversity –more kinds of people contributing

No one will argue that volume is not significant (and needs to be measured) but the definition of ‘success’ used by the study investigators is problematic:

“… the number of ideas approved by management”

This definition is part of the problem! Despite our sincere belief in our ability to identify which ideas are the best, experience has shown that human judgement is not up to the task.

There’s a long way to go from defining success as ‘approved by management’ to defining success as ‘adopted by the market’. The metrics we use should strive for the latter.

With a Model,  You Now Can Model

In the above model, four phases of an innovation system are identified and connected to create a continuous flow of opportunities from concept to reality. In this system, five high-level operational outcomes have been identified.

1. Compress Time – The faster you can take a concept from inception to reality (or kill it along the way) the better. The velocity of opportunity creation and development influences the mechanisms for identifying new concepts (see below) and for learning, evaluating and selecting (decision making) among opportunities to invest in.
2. Increase Volume – Volume makes a difference because it lets you select among opportunities instead of evaluating each one in isolation (see sidebar). In this way, opportunity volume is like deal flow to a venture capitalist, the higher the deal flow, the more selective (and presumably better) you can be.
3. Increase Yield – Yield typically refers to the error-rate of a process. How many defects per X number of opportunities. In this case, a defect is an opportunity that won’t be successful. Letting these opportunities through the system lowers its overall yield.
4. Increase Adoption – To the extent that you can get a new product, service or business model to be more widely adopted, its value will increase. Increasing adoption among the targeted customers, or, in some cases, unexpected customers, is a desired outcome.
5. Speed Adoption – The faster the adoption of a product, service or business model, the more value it creates. Speed of adoption depends on many factors but measuring the speed in real time is pretty straight forward.

Note that one thing that many people focus on, how much money (revenue, profit, ROI etc.) will an opportunity create, is not in this list. That is not because it is not important, it is because it is a dependent variable that can be (and should be) derived from these outcomes. Other ‘size of potential market’ factors are taken into account when estimating potential adoption size and rates as well as overall opportunity value (see below).

Given these 5 operational outcomes, we can ‘run’ the model to see what the results of improvement in each of these areas would be. The following is a chart showing what a 10% improvement in all 5 of these outcomes would result in. [4]

Figure 2 – Effect of Improving the Innovation System by 10%

This chart shows the expected difference in revenue between baseline (existing innovation system performance) and an innovation system that operates 10% better in all five of the listed performance areas. As you can see, there is a potential to more than double the revenue created by an innovation system over 5 years.

Categories of Measurement and Some Potential KPIs

The model above identifies key operational outcomes of an innovation system and lists five (5) measurement categories needed for monitoring these operational components. This model can be used to identify specifically what to measure, how to measure it and how to use what is measured

Each measurement system setup will be different. There are different processes, methods and tools used, different governance rules and responsibilities, different organizational structures and different people involved. Each measurement system needs to be customized to the specific situation by choosing the specific KPI’s to use, how to instrument the system, how the data is collected and processed, how the data is visualized and what is acted on. Despite this customization, the five general categories of measurements identified above are the ones that are important to keep in mind when creating a specific set of KPIs, instruments, data analytics and visualization.

Value – What is the uncertainty-adjusted value of specific opportunities as well as the total innovation opportunity pipeline?

You need different measurement tools and methods for opportunities at different phases.

• Discovery – 1^st order approximation of yearly revenue. Estimates based on educated guesses derived from triangulation of supply – demand – system and existing analogs in the market
• Incubation – 2^nd order approximation of yearly revenue. Prediction of survival, growth and size based on experimental data and advanced analytics.
• Acceleration – 3^rd order approximation of yearly revenue and costs. Company standard methodology (e.g. BASES)

Flow – What is the rate of opportunity generation, throughput and yield of the pipeline? What is the flow per $ spend and how is this changing over time?

This is a set of values and ratios based on when opportunities achieve certain levels of readiness (i.e. specific uncertainties have been eliminated). From this data, the following things can be calculated.

• Number at level – Count of the # of opportunities at each readiness level
• Time in level – Average time an opportunity spends at each level
• Level attrition – % of opportunities that do not move on to next level
• Resources at level – total $ and FTEs spent on each level
• Throughput – Phase transitions (Discovery to Incubation to Acceleration per unit time
• Yield – Throughput/ Value, how it is trending

Network – How well are the internal and external knowledge creating networks working to facilitate learning and creativity? Are the internal and external networks robust and effective for knowledge and relationship creation and assimilation?

Measures of how connected the organization is, internally and externally and how effective the network is at creating and assimilating new knowledge.

• Automated analysis – Natural language analysis of e-mail, phone calls, calendars, channel conversations etc. for network structure, contact frequency and content value.
• Number and depth of network events and activities – meetings, workshops, fairs etc.
• Self-assessment – Surveys of network participants

Stretch – How is the company’s strategic intent being pursued and how are the company’s boundaries being extended in appropriate ways?

Measures of how the organization is expanding into adjacent and new areas that are strategically aligned and important.

• Expanded Ansoff Score – Technology, Market
• Strategic Innovation Score – New to World, New to Company
• Advanced portfolio measures using core/non-core, sustaining/transformational dimensions

Jobs – Are we doing the right innovation jobs and doing these jobs right? How are jobs changing? Note that these are not the jobs of individual people, they are the jobs that the innovation system is doing for the company.

Perform a periodic assessment of the jobs the innovation system is doing and should be doing.

• Innovation Strategy Canvas and Roadmap – Benchmark and plan for the evolution of the innovation system
• Job analysis and assessment – Update list of innovation system jobs and rating of each job on importance and performance.
• Gap assessment – analysis of gaps, roles, responsibilities, proficiencies and team that are revealed by the analysis of jobs to do.

With a model and its corresponding measurement strategy, we can focus on developing and using better instruments for measuring activities and predicting outcomes.

Future Uncertainty – The Heart of Innovation

The fundamental condition at the heart of any innovation endeavor will always remain – the future is uncertain. Uncertainty about the future is, after all, what an innovation system is set up to exploit.. Paradoxically, a well-functioning innovation system should both create uncertainty (for yourself and for others) and resolve it. It creates uncertainty through the constant imagining of new things the world has never seen. It should manage and reduce uncertainty by changing how the world behaves in ways that allow the innovations it produces to be adopted.

• Every innovation we create is a statement about what we believe the future will be. How well we anticipate the future is a critical aspect of how well we innovate. Our ability to formulate potential futures in a structured, precise, and unbiased way can be measured.
• Creating new-to-the-world artifacts is an uncertain (not to be confused with risky) endeavor. How well we create, and then resolve, uncertainty, is a critical aspect of how we innovate. Uncertainty can, and should, be measured.

Uncertainty plays a central role in innovation and, therefore, it makes sense for an innovation measurement system to directly measure and track this. As far as we know, this has not been tried. Consider what follows as a modest proposal to experiment with a different way to measure innovation.

Innovation Measurement using Future Claims and Uncertainty

The principle behind a new measurement approach is that what really matters for an innovation system is how well you can create claims about the future in the face of inherent uncertainty about those claims. The approach goes something like the following.

1. Any opportunity concept is, in large part, represented as a collection of claims about the future. One need look no further than the ‘pitch decks’ used by entrepreneurs and intrapreneurs to sell their ideas to investors).
2. Each claim about the future has derivable levels of uncertainty assigned to it. This uncertainty can have multiple dimensions and, hopefully, declines over time.
3. Each claim about the future has a derivable measure of impact (along multiple dimensions) and likelihood of happening. The impact and likelihood of the future claim can change over time. Low levels of change (i.e. low variability) is a property of an effective innovation system that can create accurate future claims.

4. The effectiveness of the innovation system is measured through a combination of how accurate the future claims are, how impactful and likely those claims are and how quickly uncertainty is reduced across all claims of all opportunities.

There are numerous details to be worked out and experiments to be done before a measurement approach such as this can be validated, but it is intriguing nonetheless since it points to some very real areas of research and development using new AI and machine learning based tools and methods.

New Tools for Innovation Measurement

Predicting the impact of opportunities, especially transformational opportunities that are at a very early stage of the innovation journey, is very difficult for people to do. The track record shows an estimated 30% or less success rate with these types of opportunities, which are arguably the most important ones.

New tools are emerging, however, that use AI, predictive analytics and big data sets to help with the problem. One such tool is the Predictive Testing of Opportunities (PTO) service that is based on technology developed by Growth Science.

You don’t have to wait 10 – 20 years for your own innovation pipeline to produce results. With PTO, you can take advantage of the success or failure of thousands of new products and services that have been introduced over the years. By comparing the opportunity with this vast data set, the AI system can accurately predict whether it will succeed or fail and what the approximate growth rate will be.

Conclusion

As a company’s innovation system becomes more critical to the long-term success of the company, the desire to measure and improve it increases. Innovation, however, unlike other business processes, is an area that defies the development of precise and unambiguous metrics.

The temporal decoupling of cause and effect, the ambiguous and complex nature of information and knowledge and the inherent irrationality of human behavior create tremendous uncertainties that are not seen elsewhere in business and which complicate the creation of meaningful metrics and the means to measure what makes a difference.

One of the things that can help in the effort to create a meaningful innovation measurement system is to have a clear framework and model of how the innovation system is constructed and how it works. With such a framework and model, it is possible to identify exactly what should be measured to increase or decrease, compress or expand the desired outcomes. It is then possible to design the most useful KPI’s, instrumentation, data gathering and analysis and decision support tools possible.

In the future, as our innovation tools get better, as they become more automated with AI and machine learning, the measurements required will simultaneously become simpler and more complex. Simpler for the humans who are running the system but more complex in their internal data gathering, analysis and predictive capability.

The effectiveness of our future innovation systems depends on our ability to measure how effectively they are operating and our ability to use these measurements to keep evolving our innovation systems. If we do not do this well, we will ultimately be the victim of those who do.

[1] One of the issues of innovation metrics is that the word ‘Innovation’ can be used as a verb, as in ‘we are innovating’, and as a noun, as in ‘our innovations drive growth’. Innovation, as used here, refers to the activity (verb) of innovating and the term ‘opportunity’ is used to refer to the output of the activity of innovating. Both need to be measured.

[2] For just one example of how many and varied innovation metrics can be see The Conference Board Working Paper ‘Signposts of Innovation: A Review of Innovation Metrics’

[3] Recent years have seen significant efforts to reduce the time between action and outcome (and thereby make measurement much easier). The whole lean-startup, test and learn movement is directed at doing precisely this.

[4] The results shown here are based on model parameters such as rate of opportunity concept generation, % of opportunities that make it to next phase, % of commercial offerings that fall short, meet or exceed ROI targets, average time per phase, time to midpoint on adoption curve (months), and many others. For more details on the model, please contact the author.