When people find out that I reported from Africa for many years and am now producing a series called Tracking Charity, they frequently ask me this: "Which charities do you think are doing really good work on the ground overseas?"
Honestly, I have trouble answering.
Certainly, many charities are doing good work, but even after all my years covering conflict, food crises, HIV/AIDS, and refugees, I still find it difficult to define effective aid. How should one measure success? Should all charities keep overhead low, or can high expenses be justified if they allow a charity to hire the best people? Even if an aid program improves lives in the short term, might it create a culture of dependency in the long run?
Thursday, August 8, we're giving you the chance to discuss these and related questions with people who have devoted their careers to answering them. I'll be moderating the conversation and will be joined by:
Iqbal Dhaliwal, an economist who grew up in Delhi, is director of policy at MIT's Abdul Latif Jameel Poverty Action Lab. When people ask him where to donate money, he advises, "Don't just think process, but think of the final impact that you are interested in."
Dayna Brown is director of the Listening Program at CDA Collaborative Learning Projects, a nonprofit in Cambridge, Massachusetts. She is co-author of Time To Listen: Hearing People on the Receiving End of International Aid.
Holden Karnofsky is co-founder of GiveWell, a nonprofit that conducts cost-benefit analyses of charities "to help donors decide where to give." A graduate of Harvard University, he previously worked in the hedge fund industry.
Our discussion will take place in the section below where you can leave your questions and comments. You can follow the discussion as it evolves by subscribing to the comment thread by RSS, or by clicking "Subscribe via email" at the bottom of the discussion box.
Editor's Note: Here is an archived version of this discussion, which took place on an older version of our website. Feel free to continue the conversation in the live comment section below.
Good morning to our panelists, Iqbal, Dayna and Holden! Thank you for joining today's discussion. Panelists, please introduce yourselves and answer this question by hitting "Reply": What's the biggest misunderstanding out there about what makes aid effective?
- Iqbal Dhaliwal amy_costello
Thanks Amy and PRI for hosting this discussion on a very important topic. The Jameel Poverty Action Lab (J-PAL) started as a center at MIT’s economics department with a mission to promote evidence informed development policy. Our network of 87 professors from 42 universities have ongoing or completed evaluations of over 400 programs in about 40 countries. I lead the policy group at J-PAL that tries to make the research results more accessible to implementers, policymakers, donors, and the civil-society, and works with them to scale-up programs and policies that are found to be effective.
These underlying programs are designed and implemented by NGOs, foundations, governments and the private sector with varying degrees of inputs from policymakers, researchers, local communities, donors, and other stakeholders. Using rigorous impact evaluations, our researchers working in the field, try to understand what policies and programs work or not, and why, thus generating original evidence that can be used by implementing organizations and donors to make decisions based on hard evidence and not just rely on instincts or ideology, or selective anecdotes from the community.
To answer your question about common “misunderstanding about
what makes aid effective”, many organizations trying to create social change or improve people's lives believe that if their program(s) are based on strong logic or a sound theory of change, then combined with a motived organization, they will be effective, when in reality we need to be careful and really test our assumptions about what works and why in different contexts. There is often an under-appreciation for the complexity of constraints that affect an individual, household, community, or economy and the administrative capacity of the implementing organizations (i.e. their ability to effectively delivery and monitor the program).
For instance, a charity, its donors, and even many of the parents in the local community may believe that the reason the children are not learning in schools is because of unaffordable textbooks and hence they will try and provide free textbooks directly to the children and this will lead to an increase in learning. The charity then measures its success in terms of number of textbooks delivered, the donors see pictures of smiling children with books in their hands and a few parents provide nice anecdotal tales of how their child who never had a book before now comes home and reads a book. Very often, no attempt is made to rigorously measure the one outcome that really matters and that motivated this entire program – the learning levels of the children.
In a few cases, reading tests are administered to the students receiving the books at the beginning and end of the school year and the increase in these test scores are shown as proof of how the program improved learning outcomes. In other cases, comparisons are made between schools that received the program and neighboring district schools that did not. These commonly used methods ignore the learning that anyway happens during the year or the impact of any other factors or programs that may have run concurrently (e.g. a new motivated teacher may be driving the before-after improvements, or greater scrutiny by administrators in schools that got the free textbooks may lead to decreased absenteeism among teachers and be driving the inter-district improvements).
This is why it is so important to not just rely on an untested theory of change, however sound it may seem (far less just the instincts, ideology or inertia of either the implementer, the donor, the researcher or even the beneficiary), but also rigorously measure the aggregate impact of a program on all the beneficiaries, and on the outcome that we are really
interested in. Thus adding rigorous evidence to the mix of factors used to make funding and program decisions can increase the impact of development spending.
- Holden Karnofsky amy_costello
Thanks Amy! I am the co-founder of GiveWell, which aims to find outstanding giving opportunities and publish the full details of our analysis to help donors decide where to give. We've spent years looking for charities that donors can give to and be confident of their impact, and we've found it to be a huge struggle. I think the biggest misunderstanding out there is simply overestimating how much is known about effective aid. Many people imagine that as long as the charity they're supporting is well-intentioned, honest, isn't spending too much on overhead & fundraising, etc., that their money is well spent and accomplishing good. In reality, we know very little about how most aid programs affect the people they're trying to help. It's hard to collect good data, it's hard to generalize from data, and it's hard even for someone in the field to really connect with and understand people with radically different cultures and living conditions. We've tried to find the "easiest" bets for donors - cases in which we can answer nearly every question might ask - and even for these (our top charities), there are huge numbers of unanswered questions.
That isn't to say the challenge isn't worthwhile. I believe that aid does a great deal of good in aggregate, and the track record of health interventions in particular is strong. But assessing the isolated impact of a particular organization is very difficult, and there aren't established reliable methods for doing so.
- amy_costello Holden Karnofsky
So fascinating, Holden. I, too, have struggled to find reliable data on many well-intentioned initiatives. And it's especially hard to evaluate the impact of international charities when you can't actually visit the field to see how their initiatives are working for recipients. Instead, we're too often left to rely on what an org tells us. What else would you suggest our listeners do to gauge the effectiveness of a charity, assuming we all can't devote our careers to it the way you have!
- Holden Karnofsky amy_costello
Amy, it definitely takes more work to gain confidence in even a single charity than most people have time to do. For all of our top charities, we've reviewed the academic and other evidence behind what they're doing, reviewed large amounts of internal documents on their activities, budgets, processes, monitoring data, and funding needs, spent substantial time in person with the people running the organization, and performed extended visits to see their work in the field - and we still have large numbers of unanswered questions. For donors who don't have time for that sort of investigation, I'd recommend our 6 tips for giving like a pro (or, of course, simply donating to our top charities).
- Dayna Brown amy_costello
Hi, I have worked in international development for over 20 years and my thoughts on what is good development have always been shaped by my experience in Peace Corps, where I lived in local communities and tried to understand the local resources, challenges, opportunites, politics, power dynamics, etc. Much of our understanding and discussion about what is "effective" however, has been focused on what we, as outsiders, think is effective. What we tried to do in the Listening Project was to hear what local people--aid recipients, government officials, civil society activists, teachers, students, village residents, observers of aid efforts, etc--thought about what made different aid efforts effective. Their voices and ideas are captured in our recently published book, Time to Listen: Hearing People on the Receiving End of International Aid. One of the biggest misunderstandings is focusing too much on the "what" and not enough on the "how" we provide assistance to ensure it is effective in the long-term in strengthening local capacities and supporting local priorities.
- amy_costello Dayna Brown
You say that what's considered "effective" is what we, the outsiders, think is effective. So what very concrete things would you tell listeners to look for when deciding which international charities to support?
- Dayna Brown amy_costello
How do the charities engage with those they are meaning to assist or support? Do they meaningfully involve local people in assessing the resources and needs in their communities? How do they know what local priorities are and how well are their programs addressing them (versus what the donors' priorities may be)? Do charities involved local people in coming up with options and ideas for addressing the challenges and priorities they have identified? Do local people have a voice in determining what programs or projects are funded by the charity? Are those who are meant to benefit involved in the monitoring and evaluation of programs/projects or even of the charity itself? Are there people from the communities charities aim to assist on their governance structures or are they part of the decision-making processes?
These are hard questions to answer from afar and good donors have to work hard sometimes to get beyond what charities put in fundraising materials and reports. Try to find program reports, evaluations, audits, and external reviews of their work if you cannot visit and talk to local people yourself. Ask the charities what kinds of feedback they gather from those they aim to serve and to share it. The more educated a donor is the more effective their donations will be as they ask better questions!
I had a question regarding impact evaluation, which I know JPAL has championed. What are the challenges to implementing a widescale standardized system of evaluations of charity projects?
- Iqbal Dhaliwal notjanetjackson
The main challenge to a standardized system of evaluating charities (or any development programs) is (a) the tendency to focus on outputs not outcomes, and (b) not being able to distinguish good evidence used to evaluate one program from the not so good evidence being touted by another organization.
(a) Output vs. Outcome: While it is true as Joel said that "charities are engaged in very different activities and employing very different theories of change", this variation is often much lesser in terms of outcomes that they are trying to affect. For instance, charities could be trying to supply different types of school inputs (black boards, chalks, textbooks, new buildings), or instead be focusing on improving teaching quality (training courses, pedagogy) or quantity (to reduce student-teacher ratio), or trying to increase demand for schooling among parents and kids (scholarships, free uniforms, free school based lunch/meals) or teach to the right level (remedial education for the weaker students instead of teaching just the syllabus) - yet despite the large number of theories of change that these interventions are based on, most of them will have the common goal of either increasing student attendance or improving learning outcomes (ability to read or do math). So if we can shift the way we evaluate charities from looking at outputs (number of books delivered, schools built, teachers trained, textbooks supplied etc.) to outcomes (attendance or learning), then we have a far better way of comparing organizations.
(b) All evidence is not equally rigorous: Even when organizations try to measure outcomes, they often do it in a way that makes it wrong to attribute causation (see my earlier discussion in response to Amy's opening question on the problems with widely used measures like before-after, or difference across seemingly similar groups or the use of stand-alone anecdotes). There are other much more rigorous ways to measure the true impact of a program - randomized evaluations are one of those, especially in situations where a large amount of resources are to be committed to a new program or a change in policy.
So a big challenge is to have more implementers and donors focus on outcomes of their work instead of outputs, and try to measure these outcomes in a rigorous way (where possible or desirable - in some cases neither is true).
- notjanetjackson Iqbal Dhaliwal
Hi Iqbal, thank you very much for your informative reply. Do you think you could talk a little bit more about your last statement -- that sometimes, it is not possible or even desirable to measure the outcomes of a charity program? Why would this not be desirable?
Additionally, what proportion of charities would you say practice rigorous, controlled evaluations? Is it true that impact evaluation is becoming more widely regarded as a necessity in assessing the success or failure of different programs? How can we ensure that all charities incorporate it?
One last question -- are long-term assessments part of impact evaluations? Or is the time frame usually more short term, and if so, what implications does that have for the validity of the evaluation?
Thanks so much!
- Joel notjanetjackson
The challenge to wide scale standardized system of evaluations is that charities are engaged in very different activities and employing very different theories of change. Having agreed upon standards is one thing, but I'm not sure what a standardized system would look like. It would be impossible to create a one size fits all approach to evaluating the wide variety of activities, outputs and outcomes that charities are trying to measure. Impact evaluation is extremely challenging in most settings, as it is very difficult to attribute change to any organization's interventions. Often the best we can do is measure some contribution. Demonstrating impact often means being able to provide a control to your intervention. This is very difficult to do in any social science, but even more difficult in many of the contexts where charities work. I do agree though that there are some basic principles and perhaps even a list of standards that could become the industry norm when it comes to impact evaluations.
- Lilly Dimling
The title of this event is "How Do You Know a Good Charity When You see It?". I would love to panelists to answer that question. As an individual donor, what should I look for, what questions should be asked directly to the nonprofit that could inform the donation decision?
- Charity Navigator Lilly Dimling
The fact that this conversation is taking place is wonderful. Too many donors still give only with their heart. It is encouraging to see more donors interested in learning about the impact of their gift and willing to do a little work to decipher which charity is really delivering its mission.
We believe donors/ social investors should examine 3 aspects of a charity's performance before giving: the charity's Financial Health, Accountability & Transparency, Results Reporting. We offer more details on these tips here: http://ht.ly/nMiwp and http://ht.ly/nMiFu .
- amy_costello Charity Navigator
Thanks, Lily. I think our conversation has illuminated just how complicated it is to understand what makes charity effective. And it has also provided us with some clear guideposts for what to look for when assessing a charity we might want to get behind. You're right, it does indeed require us to use both our hearts... and our heads! Thank you again for being a part of this important conversation.
- amy_costello Lilly Dimling
Hi Lily. I think Dayna has posed in this discussion a number of great questions that you could ask to a potential charity, such as: "Do charities involve local people in coming up with options and ideas
for addressing the challenges and priorities they have identified? Do
local people have a voice in determining what programs or projects are
funded by the charity?"
Seems Dayna would say that if an org doesn't have good answers to those questions, you may want to look elsewhere..
- Lilly Dimling amy_costello
I understand what you are saying, why provide a something if the community has not identified it as a need? Whose responsibility it it to find out what it is the community thinks it needs? It seems that charities/aid orgs would not have the funds to do this AND the programming itself. And each aid org has a specialty, like poverty alleviation or education. One org can not address all the needs of a community I presume.
- amy_costello Lilly Dimling
Yes! And some critics contend that the very nature of non-profit orgs is problematic because it's a system that's ultimately structured to respond to the wishes/needs of donors (whom nonprofits need to please in order to get more funding) rather than being answerable and responsive to aid recipients themselves.
Seems donors must be aware of this tension...anyone experienced donors stepping back so that the nonprofits they fund can work unimpeded by donor expectations?
If you're commenting on behalf of an organization, please tell us your name and title. Thank you!
- Eric Hartman
Have the panelists seen impact measures and methods that gauge community self-efficacy or community sense of empowerment? My experience in community development suggests that the presence of new infrastructure or programming is far less interesting than the community's sense of ownership in relation to the development initiative. How do we know whether our interventions move beyond charity models and cultivate local capacity?
- Iqbal Dhaliwal Eric Hartman
Issues of community participation and empowerment are among the most fascinating questions in development. On the face of it, nothing seems more obvious - involving and empowering communities in local development programs should lead to better outcomes. But how effective such community involvement is in improving development outcomes can depend a lot on the context as well as the way such involvement is structured (details below).
Many of J-PAL’s affiliated professors have investigated the impact of greater community participation on the effectiveness of development programs, not just to learn whether it works, but to identify ways to improve the effectiveness of community participation and community ownership as strategies for making the programs more effective. So far the results are mixed: In Uganda, community participation in monitoring the quality of local healthcare both improved the quality of care and health, even reducing child mortality ( https://www.povertyactionlab.o... ). On the other hand, in India, mobilizing community members to monitor local schools did not encourage effective participation in school governance (https://www.povertyactionlab.o... ).
In Indonesia, top-down audits were more effective at reducing corruption in road construction projects than community participation, but this was likely because this particular type of corruption was difficult for community members to observe ( http://www.povertyactionlab.or... ). Yet in another evaluation in Indonesia, community participation led to substantially higher satisfaction with government social assistance programs (http://www.povertyactionlab.or... ).
So while we are beginning to understand the mechanisms that make community participation more effective, there is still more investigation that needs to be done, which is why we have so many ongoing impact evaluations on these very questions ( https://www.povertyactionlab.o... ).
- Holden Karnofsky Eric Hartman
I haven't seen any such measures that I consider to be convincing to an outsider. I agree that community ownership and empowerment are very important, but I haven't found a way to be confident that an organization is bringing them about. Improvements in health and wealth are easier to measure, and I think these capture forms of empowerment (albeit at the individual level).
- Dayna Brown Eric Hartman
ask them! There is no better way to guage community perceptions and changes than to ask people in the communities themselves!
- Eric Hartman Dayna Brown
Hi Dayna, I agree. And I / we (organizations I work with) do ask the people in the communities what initiatives they most desire, whether programs are working as they're implemented, how they'd like to evaluate, what success would look like, where they'd like to be in 5 years, etc., BUT - I'm still challenged by the question of cross-program, cross-project, cross-national evaluation of development initiatives and the ways in which those initiatives impact community self-efficacy. I wonder if this brings to mind any resources or research from you or other panelists?
- Dayna Brown Eric Hartman
We were looking at cumulative impact in our research and book, and there are several sector-wide evaluations, particualrly in the humanitarian field from the Tsunami Evaluation Coalition and of course in Haiti and other disasters. 3ie and others have done larger evaluations too, but with a very heavy focus on quantitive measures. I think one of the challenges is to look at cumulative, collective impact, rather than to measure effectiveness and impact of each project. There can be great projects and even great charities but too often their efforts are not followed up or linked to others which decreases their impacts or worse, may even do harm.
Here's a question/comment for the panel. There's tremendous pressure placed on charities to measure their impact, and to quantify the changes they are making in the communities in which they work. I agree that learning and accountability is crucial, but with many organizations impact measuring simply translates into the creative spinning of feel good stories donors want to hear (and selective metrics that support these narratives). When one digs a bit deeper at the community level (particularly in less accessible communities), the numbers often get a bit murky, and the tales become even taller.
Many have critiqued this tendency, but I'm wondering whether the panel thinks the world is ready for less flattering (and more honest) tales of success, and more conservative (and verifiable) numbers of impact. What about the stories of the many projects that were colossal flops, told by the organizations themselves, detailing what was learned from the process?
While on paper these organizations may look less impressive when compared to the impact (and numbers) of others, could these signs not be an indication of the maturity of a charity, or their commitment to realist goals and understandings of impact, and therefore make them potentially more worthy of an investment? Is the world ready not only to take this risk, but for a re-imagination of this feel good narrative of impact--and to understand clearly and honestly where their dollars go?
- Holden Karnofsky mike
Mike, I couldn't agree more. One of the things that GiveWell is trying to do is create a more mature conversation about charity, based on numbers and claims that hold up, even if they don't paint as rosy a picture as fundraising brochures. We write publicly about our reservations with our top charities; we post updates on their progress that include details on the good, the bad, and the ugly; and we publicly discuss our own mistakes as well. More:
* 2012 discussion of a former top-rated charity that had run into some major hurdles, and a guest post on our blog from the organization.
* Report on our current #1 charity that includes our reservations and uncertainty points.
* List of GiveWell's own shortcomings as an organization.
* We have also written about the need for an attitude change more generally - see, for example, our proposal to reward failure.
We're trying to flip the script so that charities that give a detailed, substantive, nuanced picture are more likely to be rewarded, whereas charities that stick to fundraising narratives that don't tell the whole story are not.
- Tom Murphy mike
I think you make a great point, Mike, on the shortcomings of measuring impact. To me it is less of an issue of what donors are ready for as opposed to what is best for the communities served. Transparent efforts will make sure that ineffective programs are cut and that donors can see what is actually making a difference (or not) for the lives of the poor.
There may be a level of discomfort as NGOs make the transition to more honest communications, but I have a hard time not seeing the benefits to the change. More realistic donors will demand impacts, rather than heartwarming takes of a child that does not really exist (yes, this is what NGO comms teams do).
- ZoFem Tom Murphy
Absolutely Tom, I think I agree with you there. Most at time, we
(speaking as someone working at the grassroots and a head of a local women's organization) get confuse between what the donor want and what the people on the want. I think, it is high time both donors and implementing institutions find a common ground to how best their support could transform and improved lives
- Dayna Brown mike
Hi Mike, I agree that we need a lot more honesty and open dialogue about what works and what doesn't--and for whom. This includes a need to have more honest discussions and debates with the public on what development and aid is all about, what can realistically be achieved, the role that local people must play in their own development, etc. Too often, donors and charities have not helped the cause by claiming that massive challenges and often systemic problems can be overcome with a small donation that is easily packaged into an easy amount for a donor to make to what is a huge undertaking.
- Charity Navigator mike
Our President & CEO's Bodacious Dream for the Social Sector is that "is that in perhaps ten or twenty years it will become standard practice for every charity of a certain size to not only conduct a financial audit, but also a results audit. Of course this will require the hard work of our developing agreed upon standards of reporting and measurement for every type of program out there. It is also expected that this results audit would be publicly reported." http://www.kenscommentary.org/...
- How Matters mike
There is a growing number of small NGOs and foundations that specialize in offering direct funding to grassroots leaders and small, often “informal” movements. They have the exact challenge you describe, Mike. "Impact" as it's currently defined in the philanthropic and aid sectors is difficult for them to prove. Evidence of learning is abundant but hard numbers are not.
I've found that these international small grantmakers are adept at keeping their minds, and perhaps more importantly their hearts, open to the possibility of results that build upon and unleash the common good in unimagined and unanticipated ways. And when compared with “old-school” donor-controlled, large-scale, risk-averse project-based international aid funding, small grants mechanisms can exhibit a profound shift in attitude and practice in working with impoverished and marginalized communities – from being passive recipients to active leaders of their own development.
As the role of grassroots initiatives as key drivers of social change enters the discourse in a more profound and imperative way, international small grantmakers’ experience are an untapped resource that has increasing relevance for the international aid and philanthropic sectors as a whole. We'll certainly find some great feel-good stories about people who are in the process of organizing at the grassroots level, but if we're honest, there remains quite a lot they and we cannot know about how social change occurs. I'm hedging my bets with people and organizations willing to admit that.
- ZoFem How Matters
Very true! Absolutely and most at times, these small scale network either lack the visibility or aren't in away accessible for global coverage.
I think, this effort is a global one, a collective one as well and at each and everyone's end, we should do our best to establish what works where and doesnt work where.
I hear you, Barry. There are many relevant orgs and individuals we could've invited to our panel. Charity Navigator is an obvious choice and I'm hoping they'll join us here. In fact, I will extend a personal invitation to them. That's the beauty of a forum like this. Absolutely anyone can contribute!
I think it will enrich this discussion to have the perspective of both orgs. As you point out, Charity Navigator has produced such an impressive breadth of work, allowing us to gain insights into thousands of aid groups. GiveWell, on the other hand, highlights the work of a relative handful of charities that it deems both outstanding and in need of funding. I think there's a role for both approaches to charity evaluation..and certainly room for both perspectives in this forum!
Of course, GiveWell and Charity Navigator can articulate their different approaches much more effectively than I can. I hope we'll hear from both of them!
- Holden Karnofsky amy_costello
Amy, that's an accurate characterization of GiveWell. We started as a group of donors trying to decide where to give, and we try to serveeffective altruists trying to do as much good as possible, so our focus is definitely on asking the question "Where should we give?" - and giving as thoroughly analyzed and clearly discussed an answer as possible - rather than on providing a large number of ratings. We hope that in the process of doing deep, thorough analysis in a transparent way, we'll help others think about the more general principles and approaches that could be used to evaluate a larger number of charities for people with different values.
- Barry Horwitz amy_costello
The in-depth approach is a great one, and very worthwhile - particularly if they go in-depth on charities focused on a cause one is interested in. And over time it will be interesting to compare the two approaches and see how closely the assessments align where there is overlap.
- amy_costello Barry Horwitz
Yes, absolutely, Barry. I'd venture to guess the more organizations doing independent assessments of nonprofits, the better. Ideally such evaluations help us become more informed and engaged citizens/donors.
- Barry Horwitz amy_costello
True ... generally... though we have to be careful, since a number of assessing organizations base their assessment on crowd-sourced responses, where the objectivity and "accuracy" can be suspect. (in the same way that there are ways for restaurants to manipulate Yelp ratings to work bad ones off and encourage more good ones...)
Also, it is not overly helpful to have in-depth assessments of charities in causes that I'm not interested in if I am looking for the best way to support a cause I care about .. which I believe is how many donors would go about their donations. If I am hoping to support organizations working to cure cancer or brain tumors, it isn't helpful to me to know that some charities in Africa are particularly efficient in delivering services to people there. That is a good and valid cause, but not one that I might be focused on in my charitable giving plans.
That said, the attention to effectiveness, and the very challenging effort of determining if a charity and its approach is having a positive impact or not, is a good thing. Accountability and a focus on thinking about impact is important even if it is difficult ... and is a good trend.
- Barry Horwitz
Interesting topic. I wonder, though, why you don't have Charity Navigator represented on this panel. They are the most significant evaluator of charities in the US - having rated over 6,500 charities (far more than anyone else) and involved in driving discussions on results reporting and other key topics...
- David Baron
I work with Amy Costello here at PRI's The World. Many thanks to our guests for taking questions! I'm curious to know: How can you evaluate if a charity -- even if it's effective, say, at saving lives -- is undermining local efforts to solve the same problem? For instance, if an outside charity steps in and provides obstetrical care to women in some rural part of Africa, doesn't that reduce the incentive for the local government to provide that service? How can a charity avoid this unintended outcome?
- Holden Karnofsky David Baron
David, I think it's usually possible to assess the risk on a case-by-case basis. Many of the charities that work on public health work *with* the government; for example, the Against Malaria Foundation consults with national malaria control programs to find areas that can't be covered (with insecticide-treated nets) with existing funds, and then steps in and funds distributions to fill the gaps. On a case-by-case basis, it's usually possible to look at what a government's budget and activities are in a particular area and determine whether the charity you're supporting is supplementing, adding on, or potentially getting in the way.
Each charity review we do has a "possible negative and offsetting impacts" section in which we discuss what we see as the most relevant risks.
- Iqbal Dhaliwal David Baron
David, this has been a great conversation, so thanks for organizing this. In response to your question, if the charity is a small organization with limited ambitions (will only work in one district or a few villages), then this is not really a problem because large NGOs or governments will mostly not change their general policies or efforts. But if this is a charity that has broader ambitions for expansion and feels its success may lead to pull back by others, then weaving in an impact evaluation into its roll-out plans will help answer your question. For instance by comparing the number of "institutional" deliveries or immunization rate for the ENTIRE population in the treatment villages vs. comparison villages (which continue to get existing services but no new programs), you will know what is the "net impact" of the charity that will account for the "pull-out" by existing organizations. The charity can then assess whether the cost and effort of the INCREMENTAL institutional deliveries or immunization rate is worth their effort.
But it does not need to even come to this calculus with a bit of planning - charities can work in partnership with other large agencies in that region to coordinate their efforts so that instead of crowding out efforts, charities can focus on addressing different parts of the puzzle - so one example would be for the charity to raise awareness of the importance of institutional delivery and mobilize women to go to pre-natal clinics while the government can focus on improving the staffing in those clinics instead of sending half the nurses into the field on awareness campaigns. This requires more coordination than just rolling out your own program but the right thing to do in places where your concern is a salient issue.
- Dayna Brown Iqbal Dhaliwal
David and Amy, thanks for organizing this and for facilitating an interesting discussion. In answer to your question, David, it is the responsibility of charities and any outside actor coming in to "do good" to be sure they are not undermining local capacities or disincentivizing local actors--whether they be the goverment, civil society or the private sector. We should "do no harm" in the long-term, even with humanitarian aid, and pay much more attention to the HOW, not just the WHAT we provide. As Iqbal points out, it requires and forward planning and coordination. Charities must also have an understanding of what already exists, why it is or isn't working and an agreed approach with local actors to strengthen what exists and to build capacities for what is needed. Aid agencies should be thinking from day one how they will work themselves out of a job, identifying with local actors what their roles and responsiblities are and setting clear benchmarks and indicators for success. I often question the effectiveness of organizations that have been working in the same place for decades, so ask they why they are still there. What are they doing now that builds on what they and others have done in the past? What are they doing to work themselves out of a job?
- Brian Mittendorf
I would be curious to hear the panelists' views on the ideal role of financial statements in evaluating charities.
First, my two cents...
For some reason, the popular view now is to see things as a financial metric vs. impact metric comparison. I would say they are better viewed as complements in that prudent financial practices are a necessary but not sufficient condition for a charity to be "effective". Further, the current practice of plugging in a few numbers into a formula to evaluate financial statements may be quick and easy but is often counterproductive. A full understanding of the financial picture of a nonprofit requires just as much care as gaining a full understanding of their practices. For some reason, many seem to be unwilling or unable to make that investment so instead opt to discredit financials as a useful source of information (e.g., referring to the "overhead myth").
- Charity Navigator Brian Mittendorf
Since we signed the letter to the donors of America about the 'overhead myth,' I'm sure you can guess where we stand on this topic. We believe financial measures are part of the mix, but aren't the only thing donors should examine. Our President & CEO's blog explains it very well:http://ht.ly/nMjqi.
But as we endeavor to rate charities based on their results, we find that very few charities actually measure and publicly report on those results. We need funders of all kinds to provide far more resources for the charities themselves to measure and report on their results. Without such an investment, most charities will never be able to know for certain if their efforts are as effective as they should be and as a consequence precious resources will be wasted.
- Holden Karnofsky Brian Mittendorf
Brian, I agree, and I think this viewpoint is gaining currency (seewww.overheadmyth.com). For our part, we've never used financial statements as anything but a relatively small piece of the picture. To us, the important metric is something more like "good accomplished per dollar" than "percentage of money spent in a particular accounting category." Understanding the former means reviewing evidence of impact, seeing a charity's work in the field, etc. - much more than scanning the financial statement. (More at our criteria)
- Linda Peterson
I am pleased to see The Rotary Foundation with a good rating. Rotary International just put in place a huge change in how clubs and districts use TRF to fund projects and report on grants. Accountability and sustainability will be required to fund projects.
- amy_costello Linda Peterson
Interesting, Linda. I'd be interested to hear what changes Rotary's put into place. Also, what rating are you referring to? And what's TRF?
- Linda Peterson amy_costello
TRF = The Rotary Foundation. Charity Navigator rating. The Rotary Foundation now requires much more accountability, asks that projects have sustainability, and long-range impact, higher value grants, and may include Rotarian and experts to travel to the countries to work hands-on as Vocational Training Teams.
- Charity Navigator Linda Peterson
Since Charity Navigator launched the 2nd dimension of our rating system which measures Accountability & Transparency, 50%+ of the charities we rate made improvements to their governance policies and procedures.
- amy_costello Charity Navigator
Welcome to the discussion, Charity Navigator! Please tell us your name and title. Also, can you tell us, very succinctly, how you measure "accountability"?
- Charity Navigator amy_costello
- amy_costello Charity Navigator
Welcome, Sandra! Curious to see how you measure a non-profit's "accountability"?
- shamsa birik
Thank you Amy for hosting this great talk glad to be here for the first time. As we ask this question of ''How do you know a good charity'' the organization I work for is in the process of downscalling most of the support staff due to declining donor funding and shrinking internal financial reserves. The organization explained that there is less funding being allocated to organizations working in the Horn of Africa as donor attention is now more focused in Syria or Mali and the financial meltdown in Europe has really affected things so they can no longer retain most of the staff. The process of terminating contracts has been painful to both the organization and the staff most of whom have served the organization more than 5 years. The organization has a good management team, good in program implementation, has good relations with the community they work with, so what exactly are they doing wrong to have their proposals eliminated during call for proposals? Simple they have been doing the same thing they did 20 years ago no change, no innovation, no sustainability to even retain their staff. My worry was not the minimal funding being allocated by donors to the Horn or the crisis in Europe but rather my question was 'what is this organization doing to adjust to these realities'. To me there is real crisis when charities refuse to innovate they simply are applying same fundraising strategies they did 20 years ago, they been have caught up by time. Learning from this mistake is key, the organization should invest an actually spend more money instead of downscalling to reap the fruits later. For example they need to hire a fundraising officer, invest in a communications department, hire a quality assurance officer, use crowdsourcing strategies to fundraise, run hackathons to help look for effective solutions to social problems, have better internal audits, more strict financial systems, use participatory video communication as an M&E tool, conduct research and surveys that will better help in interventions, use social media and simply get a new website that is more interactive that demonstrate what they do. Also enhance better cooperation between academia and international development as well as they should find the link between social enterprise and development. to get enough financial reserves. Most charities are faced by the same shocks from time to time and get vulnerable, do charities safety nets? yes. To me a good charity is one which does most of these, uses new technologies and hires people who are passionate about development. I would use the example of Kenya Red Cross which was once a 'dying' organization like ours but now is one of the best Charities in Kenya (https://www.kenyaredcross.org/...
- How Matters
We can probably all agree that aid can be more effective when well-formed
questions and well-executed, applied research offers relevant clues. We all want to see aid organizations engaged in deeper thinking behind the doing.
Where I differ often differ from randomistas and others leading the charge for evidence is on some fundamental beliefs about what prevents this and what ails the aid industry overall. Is it a lack of information about “what works”? Or is it a lack of respect for citizen-led initiatives? Or is it complex power dynamics that impede authentic relationships among development partners? If it’s the latter two, aren’t randomized control trials just a band-aid on a deeper issue?
Here's an impact measurement we should put more effort into--the ability of organizations to operate in a responsive manner to local needs and priorities. The leaders of the over 1 million grassroots organizations and movements around the world (conservative estimate) are not getting what they could use from us to transform their communities. In working with large corporate aid agencies over the years, and then with over 300 grassroots organizations in southern and east Africa, I continually experienced the severe limitations of large-scale, donor-controlled, sectorally-myopic, risk-averse, conditionality-driven, bureaucratic project-based funding. From the community-level perspective, there is a profound need for something other than “development” plans cooked by policy wonks in capital cities.
Rather than having international aid agencies and organizations develop and measure “their” projects in “their” village or district, I want to discuss how we ensure that these organizations seek out existing community initiatives and local movements to offer them support. (See http://www.how-matters.org/201...
I’m unashamedly hopeful about the ability of humans to change their own situations and the events of the Arab Spring certainly reminded us that lasting change must come from within. Isn’t that ultimately the most important measurement of effectiveness we should be pursuing? I'd love for the randomistas to run an experiment on how much the level of community ownership facilitates or impedes results of aid projects.
- Iqbal Dhaliwal How Matters
Actually there are many completed and ongoing randomized evaluations that try to measure the impact of community participation and empowerment on improving development outcomes, and to understand what are the mechanisms that make some of these interventions more effective than others. On the face of it, nothing seems more obvious - involving and empowering communities in local development programs should lead to better outcomes. But how effective such community involvement is in improving development outcomes can depend a lot on the context as well as the way such involvement is structured (more details in the reply I just posted above in response to Eric's question).
The issue I've become the most concerned about is girls and young women being kidnapped and forced into sex slavery. I want very much to help but don't have a lot of money to donate. I would like to know what organization(s) are legitimate. And, beyond that, which are having the most impact. Thank you.
- R Shaw
I was in Tanzania recently, and was told by a local that he liked work being done in his home village done by Oikos Institute out of Milan, Italy. Any info on this organization?
- amy_costello R Shaw
I've not heard of them but it's always informative to hear the perspectives of locals when it come to aid programs. It's why I'm especially excited to hear from our panelist, Dayna Brown, who has conducted extensive research on the ways international aid programs are perceived and evaluated by aid recipients themselves. Stay tuned for more from Dayna!
I would suggest Wine to Water. They bring clean water to people in 9 countries with two projects. One is a water filter jugs that last 5 years for a family for $35 or 100 people for 10 years for $100. They have built a jug factory in Haiti to employee local people.
The other project is to repair broken wells or drill new ones using as much local parts and labor as possible. They teach several locals residents how to repair wells so they can make a living and also repair wells in other villages and towns.
What I like is they don't just come in and do the project with drilling equipment we would use in the US that might cost $40k to drill the well. They try to use local materials and labor so more projects can be completed after W2W leaves. A well in Cambodia costs $500.
- amy_costello whatnowdog
Access to clean drinking water is certainly a pressing need across the globe. And I've also learned that it's actually quite difficult to do right and to sustain over the long-run. I know nothing about Wine To Water and so I will not comment on their activities here. But what we're hoping to explore in this forum is not so much the particular achievements or shortcomings of one nonprofit or another. Instead, what we hope to foster is a broader discussion of the ways we can measure the success of projects that are carried out in far away places; well-intentioned programs we will likely never get to see with our own eyes and so must judge by what we're told. So where does that leave us? What barometers can we use to meaningfully assess the merits of one international charity over another?
Thank you so much for joining in this discussion!
- Joel amy_costello
Amy, thanks so much for hosting and organizing this important discussion. As the Senior Director for World Vision United States' Program Insight and Results team, I have to say that answering these questions is very challenging, but so critical. We have a moral obligation to the communities we are working with to know whether we are being effective, and we also have a fiduciary responsibility to transparently let our donors know whether the resources they are providing to our organization are in fact supporting the mission of the organization. We should always feel this healthy tension as charities, and be reminded of the promises we've made to both the communities we seek to serve and the donors who make it all possible with their generous gifts and donations. We have not always gotten it right as an organization, but this is a big concern of ours and we are making lots of efforts to improve the way in which we measure our results. It requires clarity of mission, purposeful definitions of success, a strong M&E framework that is universally understood by both community participants and the organization, and adequate capacity. The really difficult balance comes in maintaining rigor, while keeping things straightforward and simple. Ensuring that all of our programs have the capacity to measure the right things in the right way is a huge burden for many organizations including my own, and of course, even more importantly, keeping communities fully engaged and participating in the process.
- amy_costello Joel
Joel, it's great to hear from one of the largest relief and development agencies in the world! Thank you for joining in.
Curious how you ultimately manage that tension between pleasing donors and serving communities. How do we manage donor enthusiasm when results from the field are telling us that it may not be what the community wants? For instance, my report on TOMS Shoes' buy-one-give-one business model (I believe World Vision is an official TOMS Giving Partner) questions whether free shoes is what impoverished communities need. And my story questions whether it is a charitable initiative driven by the desires of consumers rather than one that answers the needs of impoverished communities. (TOMS story:http://bit.ly/zx4FKU)
- whatnowdog amy_costello
The reason I liked this charity was when I heard the hour long interview on my local Public Radio Station W2W seemed to get a lot of bang for the buck. That is what I look for in deciding. That may be they are small around $500k. They don't have a lot of bureaucracy. The CEO gets his hands dirty teaching how to repair the wells.
I see too many charities go in and do everything and install something high tech or an expensive motor and everything is great until the motor breaks or the electricity only works part of the day. Living here you would not think getting a simple hand pump fixed is a step up in technology but in many places that is the case. If you teach some locals to to keep the pump in working order and go to other villages you have greatly increased the standard of living in a small rural village.
Iqbal Dhaliwa's MIT Action Lab gave the example of building schools was great but deworming the kids so they were more likely to be able to attend school was money better spent.
- Louise Rafkin
I had an amazing experience with Plan USA, supporting programs which helped build infrastructure in communities in rural Java. As the 'voice' of the program, kids from the Indonesian community exchanged letters with kids at the my martial arts school. The super interesting letters taught the mainly middle class Berkeley, CA children how well off they were - they couldn't write about video games and ipads...
After a few years of letters I decided to visit the families - a super out of the way journey, as you might expect, into a very untouristy part of Indonesia.
When I met the families in the pictures I had seen over the years I was blown away - the situation on the ground was way worse than I saw from the letters. One day I addressed the staff and volunteers there, who drove me around and translated, on what it was like to be a 'sponsor.' Turns out only one other person had visited, someone from Europe, in what I believe to be the 20 year history of the program.
Several years later that community project was completed, we were told. I always wondered what that meant; I was told the community had become better off and that the program was no longer needed.
I listened to an incredibly powerful TED talk recently by Dan Pallotta, founder of the AIDS Ride. It opened my eyes to a whole new way to think about charity and measuring it's effectiveness. Turned my head around fast and made me realize that a lot of what we've been told about the way to evaluate charities is both wrong and very damaging to the causes we'd most like to succeed. A great listen!!
- Charity Navigator Cai