“Measure Better,” an ImpactAlpha series in partnership with Acumen, features investment practitioners and thought leaders who are putting customers at the center of their impact strategies.
Acumen’s impact team reached out to me three years ago about a fundamental problem facing impact investors: existing methods for assessing impact were too slow and too expensive. More damning, they were irrelevant as a management tool.
Early-stage entrepreneurs needed data so they could improve their business models and theories of change as quickly as possible. This gap in useful methods presented an opportunity. Acumen had just piloted a novel approach to measurement in a small number of its portfolio companies. The method bridged customer feedback with social impact data.
Measuring the impact of impact investments – by talking to customers themselves
Our task was to clarify the emerging fundamentals of this new approach to impact measurement so that it could reliably be used not only in enterprises, but across an investor’s entire portfolio. That, we believed, had potential to influence the larger industry.
We articulated Acumen’s approach in an article, “The Power of Lean Data,” published in the Stanford Social Innovation Review in 2016. Lean Data shifted impact measurement “toward the use of simple, inexpensive tools,” we wrote. “With high-quality data in hand, impact-driven companies can iterate faster and achieve their missions with greater efficiency.”
Fast forward to today. Acumen has surveyed more than 65,000 customers for more than 100 enterprises across more than 200 Lean Data projects. Acumen’s own Energy Impact Report showed how Lean Data could be used at the portfolio level as well. It is an opportune moment to take stock of the advances and gaps in impact measurement, including a critical look at Lean Data itself.
Below, I pose five questions and offer my initial perspectives on Lean Data and the state of impact measurement. Thought leaders in the field will weigh in in coming days, and we look forward to your thoughts as well.
No. 1. Don’t most businesses already use feedback from customers to improve their performance?
Among my initial observations: The single most important feature of Lean Data is its focus on listening to customers in order to serve them better. Lean Data opens up a channel for listening to customers or beneficiaries, and it has the potential to be deployed across portfolios. For example, the data collected so far show that some companies are reaching the poor better than others. That provides investors with critical information on how well their portfolio is serving its target markets.
It is unrealistic to expect early-stage social enterprises to develop useful performance measurement practices on their own. As Acumen has demonstrated, the responsibility falls on investors to support the development of usable and cost-effective tools of measurement.
The idea of gathering information from customers for improving products or services is, of course, not new. But the challenge for early-stage enterprises lies in collecting such data quickly, reliably, and inexpensively from customers who are typically hard to reach and about whom there is little reliable demographic data. Add to that the need for information on social impact (and not just whether customers like or would buy a product or service), and you have a data-collection problem that has not been adequately addressed at scale.
This is a two-pronged challenge: measuring and improving what the enterprise can control (its product or service), and assessing long-term results over which it has only limited control or influence (its social impact).
Lean Data leverages the increasing ubiquity of mobile technology—text messaging, interactive voice response, call centers, and sensors— with robust survey instruments in order to gather quality data from far flung customers in a way that simply wasn’t possible a decade ago.
Over the past few years, Acumen has demonstrated that it is possible to deploy this method quickly, cost-effectively, and across an investor’s entire portfolio. Acumen claims that the cost of such data collection efforts are typically one-twentieth that of standard evaluations, and require a matter of six to eight weeks to collect, analyze, and feed into managerial decision making, depending on the degree of complexity and customization.
Lean Data is not the only way to gather reliable customer feedback on social impact. It builds on a suite of approaches to improving “downward accountability” to customers or beneficiaries – such as methods of Constituent Voice developed over the past decade.
No. 2. What kinds of changes in customers lives can Lean Data illuminate?
Acumen’s Energy Impact Report marks a first attempt to apply Lean Data across a portfolio—with data from 5,500 customers of 21 energy sector companies in 11 countries. It demonstrates both the value of the method as well as its limitations.
The deep insight for investors is that good measurement can help to clarify the theory of change for a portfolio. For example, Acumen’s theory of change for its cookstove investments might be summed up as follows: if it invests in cookstoves that are so appealing to customers that they eliminate other cooking methods (i.e. no product stacking), then there will be sustained improvements in customer quality of life.
Customer insight and industry research. The report focuses on 18 energy sector indicators. But rather than identifying these impact metrics a priori in a top-down fashion, Acumen used a hybrid approach: it asked customers what they found most valuable about the products or services they had purchased. It also drew on existing metrics used by industry associations. such as the Global Off-Grid Lighting Association (GOGLA) and the Global Alliance for Clean Cookstoves. This approach to developing portfolio metrics—by listening to customers and drawing on established industry research—offers a pragmatic way of developing standardized indicators that can be replicated across various industries.
Room for improvement. Acumen claims that its indicators are “standardized, outcome-based customer impact metrics across an entire portfolio” (p. 18). This is an overstatement. First, some of the indicators are not standardized in a way that yields managerially useful information.
For instance, a key indicator in the report is “improvement in quality of life” as measured by the percentage of customers “saying their quality of life has ‘very much improved’ due to product/service”. This indicator tells us little about what key aspects of quality of life have improved. What are the key components of quality of life that matter to customers, and which the enterprise can actually influence?
The indicator can be standardized either by unpacking what customers mean by it, or by drawing on existing research to define and operationalize the term so that it can be consistently assessed. Arguably, some of the other indicators in the report—such as increased hours of daily light, or change in hours of daily study—might constitute components of an aggregate and standardized quality of life indicator.
Outputs and outcomes. It would be useful to prioritize among indicators and to clarify whether they are outputs or outcomes. Contrary to conventional wisdom, a well-chosen output measure can be at least as valuable as an outcome metric.
In Acumen’s metrics for cookstove companies, for example, one of the most important indicators turns out to be an output— “no product stacking”—which measures whether a household stops using other more polluting and less efficient types of cooking methods such as kerosene stoves or fuelwood burning.
A key insight here is that a reduction in stacking is a “prerequisite for stove sales to translate to reduced household air pollution, savings, and less pressures to cut down trees for firewood” (p. 36). In other words, the output metric of stacking serves as a proxy for outcomes in quality of life (such as improved air quality, household savings, and natural resources).
Acumen is right to prioritize this metric for all of its cookstove investments. But let’s be clear that it is not an outcome but a valuable output metric. For early-stage entrepreneurs, it may be sufficient to measure outputs where they serve as a reasonable proxy for outcomes.
No. 3. What kinds of investment decisions does Lean Data inform?
There are two instances in the Energy Impact Report that yield important insights for comparing investments, and thus potentially for informing decisions about exit or reinvestment.
The first is the collection of reliable data on the poverty levels of the customers served by Acumen’s portfolio companies. The data show what proportion of customers of each firm are living in poverty (earning below $3.10 per day), or are low income (below $6 per day), as compared to those that are better off.
Because the data show that some companies are reaching the poor better than others, they provide the investor with critical information on how well it is serving its target markets.
Based on such data, some investors may choose to double down on supporting companies that reach the poorest, while others may decide to spread their portfolio across income levels based on a cross-subsidy model. This is useful information in designing a portfolio.
In addition, the report makes creative use of spider charts to enable a comparison of performance based on a subset of five key indicators. They help visualize which enterprises substantially outperform Acumen’s benchmark on key indicators and which ones underperform on all key indicators. Ultimately, in order for metrics to provide a consistent basis for investment decision making, it will be necessary to converge on a core set of metrics at a portfolio or sub-portfolio level.
Does this mean Acumen should exit from the low performers? Not necessarily. The data may provide insights on how to better support those enterprises in order to improve their performance. It also suggests that high performance on one or two key indicators may be more feasible than high performance across all. Over time, however, consistent underperformance across a majority of indicators could provide a reasonable basis for exit.
No. 4. Does using Lean Data across a portfolio help investors think about system-level impact?
Building a portfolio for social impact requires identifying and addressing systemic gaps, not simply investing in good companies. Acumen’s customer feedback on its enterprises points to a consistent set of such gaps: one-off energy products (such as lighting) are simply not enough to meet the diverse energy needs of households, but more complex products and services (such as off-grid solar systems) are unaffordable to many low income families .
In other words, investing in companies that provide good products and services is not enough. Acumen has learned that it needs to ramp up its investments across the energy ecosystem. That means better upstream financing for households, deeper integration of various products and services, and collaborative funding with other investors, philanthropists and governments in later-stage energy companies order to scale impact.
There are two challenges for impact investors here. The first is about the purpose of a portfolio: How can investments be combined to produce synergies in addressing a social problem, rather than simply being a collection of companies acting in isolation?
The second is about measurement: How can Lean Data and other measurement approaches capture and drive system-level impacts?
We do not yet have good answers to these questions. Acumen’s energy report suggests that part of the way forward lies in listening more deeply to customers.
“All of this work must be grounded in ongoing conversation with customers to understand impact, so we can identify companies that best meet their short- and long-term energy needs,” the report concludes. “It is only by building the full ecosystem that we will achieve our shared goal having efficient, affordable, clean power met the full energy needs of one billion customers still living off the grid.”
No. 5. How might Lean Data influence the impact measurement landscape more broadly?
Investors require different types of performance-measurement tools at various points in their investment decision making. For most investors, this process can be broken down into two pre-investment and two post-investment stages:
- Search for investment opportunities;
- Diligence to assess the potential for success;
- Improvement to identify mid-course changes;
- Evaluation to assess performance.
The Lean Data approach seems to be especially useful during the third stage of mid-course improvement, or what in international development circles is called “monitoring” rather than “evaluation.”
The data collected through the process can, of course, provide invaluable information on impacts important to customers and thus for evaluating those impacts in the long run. But Lean Data is better seen as a complement to evaluation approaches such as randomized control trials rather than a substitute. It is especially suited to early-stage enterprises which need reliable data at high speed and low cost.
Alnoor Ebrahim is a professor of management at the The Fletcher School at Tufts University. He’s also an advisor to Acumen.
