“Measure Better,” an ImpactAlpha series in partnership with Acumen, features investment practitioners and thought leaders who are putting customers at the center of their impact strategies.
Jessica Kiessel believes that social policy should be subjected to the rigors of randomized control trials.
But such rigorous trials, which compare groups that receive an intervention with a control group are often not feasible or appropriate.
“We have limited impact data that rolls up to the portfolio level,” says Kiessel, the head of learning and impact at Omidyar Network, one of the world’s most active family investment offices and impact investors, backed by eBay co-founder Pierre Omidyar and his wife Pam.
Measuring the impact of impact investments – by talking to customers themselves
So what’s she to do? ImpactAlpha’s Measure Better series, produced in partnership with Acumen, has tried to assess Acumen’s Lean Data approach to impact measurement, in which trained call-center operators reach out to real customers.
“Just talk to the people,” Kiessel says. “It’s not a perfect solution. It doesn’t fit everything, but its simplicity is reassuring.”
Kiessel cut her impact chops conducting randomized control trials on global development programs for Innovations for Poverty Action, the research organization known for its rigorous approach to evaluations. Now she’s tasked with evaluating the impact of products and services of early-stage firms and other companies in the Omidyar Network portfolio that are tackling challenges in education, financial services and property rights. ImpactAlpha caught up with her to take the measure of impact measurement.
ImpactAlpha: What’s wrong with impact measurement as it’s currently practiced?
Jessica Kiessel: Organizations, especially nonprofits, often look to their donors as their client more than they look to their beneficiaries. It’s on us (as a funder) to make sure that we’re listening (to beneficiaries). That’s a very important founding premise behind Omidyar Network’s work with the Lean Data team – that we need to be more connected.
ImpactAlpha: How’s the level of rigor around impact measurement in impact investing?
Kiessel: We take a systems-based approach to investing. It would be foolish for us to try to make our portfolio have one single measure of impact. That would shortchange good measurements that would actually give us meaningful feedback on our investments, which are all quite different.
Within the impact investing, we often fix this by creating definitional categories and making assumptions to roll things up to the portfolio level.
Impact investors can gain real feedback from their portfolio companies’ customers. Do they want it?
If we really care about metrics for decision-making, not just for showing off with how great we’re doing, we need to evolve as an industry.
We are not selling a one-size-fits-all solution. It might not be pretty. We need to aim to have numbers and metrics that will lead to better conversations about how we can improve.
You can infuse rigor into that process. Aiming for a simple number to match both bottom lines might actually short-change leadership. That’s where I started.
ImpactAlpha: Can take me through an example of how you think about an investment’s potential and actual impact?
Our investment process starts with initiative strategy because we’re taking a systems lens to our approach. We’re not exclusively interested in investment-level impact. We set big goals. We have theories of change behind our investments to understand how we’re going to get to that impact, and then link our investment strategy to our theory of change. When we’re sitting down at the table to talk about an investment, it’s backed by a very strong hypothesis within the strategy that we’re trying to test.
Because we’re investing in things that are very early stage, we’re not going to be able to have rigorous evidence on all of these things. We talk about, ‘What’s the alternative to this? What are the expected impacts of this?’ We still have a ways to go to link our indicators with our hypotheses so that we can actually say, ‘Are we on track or not?’
We have limited impact data that rolls up to the portfolio level. We do have qualitative discussions around impact. From an impact perspective, we’re working to be clearer about how our portfolio is performing. In doing so, we often revisit our original assumptions.
ImpactAlpha: Omidyar Network is a supporter of the Impact Management Project. How does IMP connect to Lean Data?
Kiessel: The Impact Management Project can help you define what impact you’re seeking. Lean Data can help you understand progress towards that impact. It’s one thing to understand it conceptually. It’s another thing to actually have data.
We feel confident that the Lean Data tools are providing insight. But we have some real thinking to do. What can you see from a one-off survey and what can’t you? Is it that you need to measure more frequently or are there just some things that you can’t capture very easily using surveys?
ImpactAlpha: What are some of the other ways you use Lean Data?
Kiessel: I’m currently working the Lean Data team to conduct surveys with 24 of our education investees so that we can produce a portfolio education report that is somewhat similar to (Acumen’s) energy report. Our education portfolio is quite diverse, geographically and topically. It can be hard to roll things up and have a good portfolio-lens on what’s happening. There is a lot of excitement about being able to look at 24 companies at one time and be able to compare across meaningful indicators to understand whether we’re making progress.
Lean Data is also valuable for our portfolio; the company provides a service to our investees that can help them improve.
ImpactAlpha: What are some limitations of the real-time customer feedback?
Kiessel: We need to be really cautious about what we can know and say with these kinds of measures. It’s one tool in a toolkit. A team really needs to be able to deploy different measurement tools. The risk is that we feel complacent and stop pushing ourselves to better understand efficacy or build up the evidence base.
We have to meaningfully walk the walk with our investees. The results aren’t always positive. That means, as investors, we have to sit down and talk about things with our investees. There should not be a penalty that comes with having taken the time and devoted resources to learning; otherwise organizations aren’t going to be open to doing this exercise. As an investor, we need to be able to open ourselves up to feedback and show our investees that we’re doing this as well.
ImpactAlpha: Do you think measuring the impact of investments is so complex that it’s holding back certain investors from making investments?
Kiessel: I don’t know that we can place that on any one player for not having solved the challenge of measuring impact. There’s now greater understanding that having sustainable business practices isn’t just good for the world or for people. It’s the best business decision to make. It’s what’s going to keep you in business.
There is greater awareness. I’m just hoping that it brings more energy into the space that allows us to get collectively crisper. That’s one reason why I’m excited about Lean Data. Just talk to the people. It’s not a perfect solution. It doesn’t fit everything, but its simplicity is reassuring.
ImpactAlpha: Don’t forget the customer.
Kiessel: I keep trying to tell people: impact analytics are business analytics. Talking to your customer is just part of good business analytics. You can build off of your business analytics to incorporate impact analytics. This can open up a lens into things that you couldn’t see before.
We can get stuck using the language of international development, focusing on accountability. But if we talk about insights and analytics, good businesses are already doing that. Impact cannot just be something you do on the side. It should be part of how we work.
