Features | October 15, 2018

Measuring the impact of impact investments – by talking to customers themselves

Sasha Dichter and Tom Adams
Guest Author

Sasha Dichter

Guest Author

Tom Adams

Measure Better,” an ImpactAlpha series in partnership with Acumen, features investment practitioners and thought leaders who are putting customers at the center of their impact strategies.

We all know that understanding customers is at the core of running a successful business. This is especially true for businesses that aim to improve customers’ lives.

Yet too often, especially for those aiming to create social change, we end up relying on well-developed strategies and desk research to understand what should happen for customers, and we don’t regularly gather customer data to understand whether these strategies are playing out in their lives.

How can this be right? Shouldn’t regularly and systematically talking to customers help us with our two most important, intertwined goals: creating successful businesses that create social change?

The question of how best to measure social impact has been at the core for Acumen since our founding in 2001. We’ve re-engaged with this question over the last five years with a new, bottoms’-up, customer-centric perspective that we call Lean Data.

Our thinking was: since our portfolio companies were in daily contact with their customers, could we leverage these touch-points in some useful way to understand social impact? With this premise, we tried a number of things that did not work, like having water delivery workers in Karachi ask questions of their customers, and attempting to incorporate customer surveys into in-person sales calls.

Then, we partnered with one of our investees’ call centers and had them ask questions as part of customer service calls. Success! We found that using mobile phones for data collection was quick and unobtrusive and, if executed well, it produced great data.

Suddenly, our conversation with our investees flipped: we had spent years making a data “ask” of these companies, and now we could show up with a data “offer” that gave great results and was hassle-free. A typical Lean Data project takes three to eight weeks from start to finish, it demands as little as two to four hours of our investees’ staff time, and it gives them high-quality, actionable data about how better to serve their customers and create social impact.

While we’ve seen great results with Lean Data, we are still at the very beginnings of shifting perceptions of impact measurement in our sector.

For example, Sasha recently ran an impact measurement workshop for a group of the Schwab social entrepreneurs at the Schwab Foundation Social Entrepreneurs Summit. He led off the session asking the entrepreneurs to describe their on-the-ground reality of social impact measurement.

They responded with phrases like “out of our hands,” “RCTs (randomized control trials) are still king,” “we must focus on what donors want, not what matters to our customers” and, worst of all, “colonial.”

This resignation, bordering on anger, from a group of entrepreneurs taking on some of the world’s toughest, most intractable problems was sobering to say the least.

The Metrics Myth

Ironically, while entrepreneurs are feeling the burden of impact measurement, the impact investing sector as a whole has struggled to fully integrate social impact metrics into the day-to-day operations of impact investing funds.

Few stories illustrate this challenge as clearly as one shared by Jed Emerson in his provocative article, The Metrics Myth. In it, Jed recounts a post-presentation conversation with the head of impact performance at a well-respected fund. When Jed asked him how they used the impact data they collected, the manager chuckled, “Oh—we don’t actually use these metrics in our work. We just need them to give our funders!” He was articulating the worst-kept secret in our field.

Jed goes on to suggest that social impact metrics were a “dance of deceit, set to the music of politics, perception and persuasion.” High-quality social impact metrics, Jed muses, are supposed to be the “Holy Grail of our professional quest.” In fact, they are more akin to Big Foot: “‘widely known yet seldom seen.” And the metrics discussion, is, according to Jed, a myth, “a force living in some deep wood, visions of which are obscured by branches, brush and bullshit.”

We took Jed’s article as a challenge: surely there was a way to bust this metrics myth, to find an ever-elusive, right-sized approach to social impact measurement.

A Middle Way

Traditional, academically-grounded approaches, which are the norm for the social and public sectors, have been regarded by impact investors as too expensive and slow to make part of ongoing data collection from companies.

The other approach, favored by most impact investors and their investees, is to report operational “outputs,” rather than actual outcomes. This means measuring, for example,  number of units sold and multiplying by people reached (i.e. five people in a household) and, for the most progressive impact investors, multiplying again by the potential social impact per product.  

We found that neither approach – heavy academic, or light-touch analysis – was telling Acumen enough about what we or our companies wanted to know: who, exactly, these companies were reaching, and how much their products and services were (or were not) improving customers’ lives?

There had to be a better way. How could we build a sector that delivers large-scale social impact if we didn’t invest in gathering data directly from customers so they could tell us if their lives are improving?

We developed Lean Data to address these core questions. Our goal has been to find a way to hear directly from customers, even those in the most remote locations, in ways that creates value for them, for the companies that serve them, and for investors.

Since early 2014, Lean Data has completed more than 120 projects with 55 Acumen investees in 14 countries. The data we gather are helping our entrepreneurs build more impactful businesses to better serve these customers, and they are enabling us to learn where our investment capital is having the greatest impact on customers’ lives.

For example, our report on the impact of Acumen’s off-grid energy investments found that $20 million in Acumen investments in 20 energy companies across South Asia and Africa had helped these companies reach 81 million people. Digging a level deeper, our Lean Data surveys of more 5,500 customers revealed that 58 million of these people had access to energy for the first time, and that they have stopped using 18 million kerosene lamps, the equivalent of taking 1 million cars off our roads.

We’ve also completed another 120 Lean Data projects for clients ranging from the Omidyar Network to the Gates Foundation to CDC group, in sectors as far-ranging as energy, agriculture, education, financial inclusion, and workforce development. We have found that Lean Data’s approach is most powerful when complemented by a global evidence base on what works – in the form of more traditional impact studies that uncover causality.

The results have been encouraging. Our Lean Data clients tell us that we are gathering data on the social impact that they value. Most importantly, they say the data that is being used to drive day-to-day decisions about how to better serve these customers and beneficiaries.

Between our work with these clients and the Acumen portfolio, we have heard from more than 65,000 customers in 32 languages across 27 countries, and gotten detailed data on customers’ poverty levels, customer experience, and self-reported social impact. In addition, over the last 18 months, we’ve been working closely with the Impact Management Project, developing and testing questions across the five dimensions of impact so that anyone who wants to use the Impact Management Project approach has ready-to-go questions that they can apply to their fund or business.

Over time, as we gather more and better data using standard questions, we aim to build the kind of benchmarks of impact performance that our sector has been talking about for decades, but that, until recently, have been hard to spot.

We hope that we’ve started to chip away at the Metrics Myth. Bigfoot really does exist (in the form of useful, available, actionable impact metrics).

Measure Better

Like the entrepreneurs who use Lean Data, our aim is to be relentlessly practical and focused on what will help companies and not-for-profits better serve their customers.

So, we’ve taken this moment to pause and get feedback, to hear how much progress we’ve made on cracking the Metrics Myth and to gain perspective on the broader landscape of impact measurement in impact investing. We wanted to learn more about what’s holding enterprises and investors back from truly putting customers at the center? And we wanted to hear about other innovations in social impact measurement, especially those that make sense for investors and create excitement and value for social businesses. What is already working, and where do we, as a field, need to aim higher?

To explore these questions, we’ve worked with ImpactAlpha to ask Jed and a number of other practitioners and thought leaders for their perspective on impact measurement.

  • Alnoor Ebrahim, a professor at the Fletcher School at Tufts University, calls out the need for data that can identify mid-course corrections that can help a social enterprise get back on track.
  • Jed Emerson emphasizes the need for high-quality data that puts the customer’s voice at the center.
  • Jessica Kiessel of Omidyar Network is keen on measurement tools as a “portfolio service” the investment office can provide to its investees.
  • Chiara Kovarik and Laura Birx of the Gates Foundation describe how the foundation’s agricultural team was able to dig into social and economic outcomes beyond simple productivity.
  • Martina Castro and Sara Taylor at CDC Group are collecting data to test the proxy metrics that the development finance institution has used at the beginning of its investments.
  • Clara Barby, who leads the Impact Management Project, shows how customer data can help create commercial value for businesses and investors, as well as mitigate risks.

Over the next two weeks, ImpactAlpha will present their perspectives in a series we’re calling Measure Better. We look forward to hearing from all of you as well. In a sense, we’re collecting data on Lean Data itself, to inform the role it plays today and how it might evolve in the future to be even more useful to everyone working to create social impact.

Sasha Dichter is Acumen’s chief innovation officer. Tom Adams is Acumen’s chief impact officer. They are co-founders of Lean Data.