“Measure Better,” an ImpactAlpha series in partnership with Acumen, features investment practitioners and thought leaders who are putting customers at the center of their impact strategies.
Global agricultural development doesn’t have the long trail of evidence-based practice that, let’s say, health policy does. The evidence that does exist focuses on agricultural productivity.
The world’s largest foundation wanted to know more. What impact was its agricultural funding having on smallholder incomes, women’s empowerment and diversity of diets?
The Bill & Melinda Gates Foundation knew it had to go beyond traditional measurement tools to collect data that might shed light on the broader social and economic implications of its agriculture portfolio. It turned to Acumen’s Lean Data approach and other creative tools.
Measuring the impact of impact investments – by talking to customers themselves
“One of the opportunities with Lean Data, and these other lighter evaluation tools, is the opportunity to get innovative with the metrics and the measurement,” said Chiara Kovarik, a program officer on the foundation’s agricultural development team. “These are quick surveys, both to administer and analyze.”
ImpactAlpha spoke with members of the Gates Foundation’s Ag team for our Measure Better series with Acumen. Kovarik, along with Laura Birx, deputy director for strategy on the agricultural development team, are using Lean Data to evaluate global development grants. The Gates Foundation gives away (and invests) about $4.7 billion a year.
Lean Data is helping the foundation gather “clustered insights” across its portfolio. The data Acumen is collecting across organizations lets Gates “benchmark our investments and the organizations that we work with against those in different sectors or even within the same sector but in different geographies,” Kovarik says.
ImpactAlpha: What’s the state of impact measurement and evaluation in global development?
Laura Birx: In agriculture there’s historically been a dearth of the gold standard impact evaluations – randomized control trials – that have traditionally been used in the health sector. We have done more of these in the last decade than were previously done. They give you more concrete evidence about causality.
What we’re also finding is that they are long-term and expensive. For something like agriculture, they may not necessarily be the most conducive to what you’re trying to do and the outcomes you’re trying to get, in part because of the complexity and multi-sectoral nature of agriculture and poverty outcomes.
Impact investors can gain real feedback from their portfolio companies’ customers. Do they want it?
One of the reasons that we have appreciated the partnership with Acumen’s Lean Data team and other monitoring and evaluation providers is that we’re trying to utilize creative evaluation tools that make evaluations either higher quality, or more cost effective, or faster. The efficiency aspect is really important.
The vast majority of agriculture-related impact evaluations have been focused on agricultural productivity. We know that three other things in particular for us are really important, which are related to smallholder income, women’s empowerment, and diversity of diets. We’re using some of these more creative tools, whether it’s Lean Data or others, to actually look at other social or economic-related outcomes as a result of our intervention.
Chiara Kovarik: One of the opportunities with Lean Data, and these other lighter evaluation tools, is the opportunity to get innovative with the metrics and the measurement. These are quick surveys, both to administer and analyze. You are not spending a lot of time with an individual. You’re able to iterate and try different things out and get innovative with measurement of these topics that traditionally we haven’t spent as much time thinking about or measuring.
ImpactAlpha: Do you see a push in global development toward standardization of the impact measurement process?
Laura Birx: There is. Less so in the process itself. More so an attempt to have some standardization around the indicators that we measure as a result of the evaluation. The Sustainable Development Goals at a high-level give the development community some standard against which to measure. Those are obviously high-level outcomes and long-term outcomes, but they’ve done a better job than the previous Millennium Development Goals about getting a little bit more granular on the indicators.
We are really focused on 1) making sure that we’re actually measuring impact, not just counting outputs, but actually measuring what really matters. And, 2) trying to have a conversation with other partners about how we evaluate the causal pathways from the outputs to those higher-level impacts.
ImpactAlpha: How have you used Lean Data with an actual company or an organization?
Chiara Kovarik: The Gates Foundation had given a grant to a poultry company in Ethiopia to expand its scope and reach to smallholder farmers, to reach rural and marginalized farmers. Acumen had also made an investment in the company and that’s how we got to know the Lean Data work.
The poultry company did not have an established monitoring and evaluation infrastructure. They didn’t have a good sense of who they were reaching or any kind of feedback from the customers. Did they like the product? Were they happy with the chickens? Would they have liked something to be done differently?
The Gates Foundation’s Ag team gave an additional grant to Acumen’s Lean Data team and asked, “Can you help this poultry company set up a light-touch monitoring and evaluation infrastructure to help them get feedback from their customers as they expand into these different regions of Ethiopia?” We did this not only because we’re interested in the social and business impact of the company as a donor, but because we thought it would be useful for them as a company to understand whether or not their product and service is satisfying to the customers, and whether there is anything they can do differently.
Customer survey data that makes it simpler to embrace the complexity of impact investing
Over the course of the past two and a half years, Acumen’s Lean Data team has worked with this poultry company to design multiple Lean Data surveys, and collect them from farmers across multiple regions fairly frequently. Every couple of months they’ll do data collection surveys within one of the different regions. The insights have been really useful to us internally to see, yes, this company is having an impact on the social outcomes that we’re interested in, but also for the company because it’s provided them with useful insights into what they might be able to do better with regards to their marketing strategy or the way in which they engage with their sales agents.
ImpactAlpha: Has this changed your relationship with your grantee?
Chiara Kovarik: Definitely. It’s less of a ‘compliance-driven’ monitoring and evaluation system where a grantee is collecting indicators because it’s what the donor requested. Instead grantees are collecting these indicators because doing so will actually help them to become a better business and it’s the sort of thing that they’re interested in as a socially-minded company. And it’s the kind of thing that the donor is interested in to understand whether or not our investments are having an impact.
ImpactAlpha: Where has Lean Data fit best within your monitoring and evaluation work?
Chiara Kovarik: We see Lean Data as one tool within a toolbox of different monitoring and evaluation options that we would consider for any given grant. In some instances, including in the case of the poultry company we spoke about, we have supported both the collection of Lean Data as well as a more standard impact evaluation with a quasi-experimental design. We’re able to get the statistically-significant impacts that might end up in peer-reviewed literature, but we also have the quicker and more customer-centric feedback that Lean Data delivers.
ImpactAlpha: Tell me about your project with Acumen to look at Lean Data across a portfolio.
Chiara Kovarik: Within the agriculture team, we have a series of different topical portfolios. Livestock is one of them. We also have food systems, and seed systems and varietal improvement, etc. We’ve asked the Lean Data team to work with some of those portfolios to collect standardized Lean Data metrics so that we can start to get some consistent social impact indicators across our different investments. Acumen is partnering with 20 of our grantees to do this. We were inspired by the Acumen Energy Report. We really like how they can start to get some clustered insights.
One thing that we really liked with Acumen’s Energy Report is this ability to look across different companies and see how they perform on a variety of different indicators that are standardized. This goes to Laura’s earlier point about the push and the desire to have more standardized indicators. Initially we saw this as a huge value add for our investments; if we’re looking at six different livestock grants, for example, we can start to see how they’re doing with regards to one another on a specific aspect and dig in to understand why one may be having more or less success in that area.
As Acumen continues to collect Lean Data across a variety of organizations globally, there’s this ability to benchmark our investments and the organizations that we work with against those in different sectors or even within the same sector but in different geographies.
ImpactAlpha: On cross-cutting indicators like income?
Chiara Kovarik: You might be able to. I was thinking more of things like the Net Promoter Score, which is a score that Acumen collects for all of its Lean Data engagements. With it we can understand whether or not customers or clients are satisfied with that product or service. And once we have that we can dig into why they may be satisfied or unsatisfied, which could be for a variety of reasons. Perhaps from other some sectors, there are things that we can learn that might be applicable to our work in agriculture.
ImpactAlpha: What are some of the risks or challenges in using Lean Data as opposed to other tools?
Laura Birx: One thing that we don’t have from Lean Data is a counterfactual. Lean Data is more of a monitoring mechanism. Especially when we get these results so quickly, there is a temptation to interpret them as being maybe more of a study design than what it actually is. That’s one thing that it doesn’t get. That’s what makes it so fast and efficient so you wouldn’t want to give that up. Perhaps it’s more on the interpretation side that needs work.
Chiara Kovarik: One of the reasons it’s so valuable and so quick is that the interviews are conducted via phone. This can also be a downside. We see that the response rates may or may not be as high as they would be with in-person surveys, where you’re going household-to-household. From the gender lens perspective, lots of times women don’t own a mobile phone. There is the chance that we might reach more men via phone calls; and therefore, get a slightly skewed response with regards to gender dynamics, which is something to be aware of. I don’t think it’s necessarily a con, but something to keep in mind when interpreting the results.
