This is the last post in a three-piece series of blogs summarising our Journal of Development Effectiveness paper titled ‘Walking the last mile on the long road to evidence-informed development: building capacity to use research evidence’. For people interested to read the full paper, it has just been published online and is open-access—follow the link here.
In blog 1, we outlined our overall theory of change of evidence-informed development, explaining how the use of scientific knowledge in the formulation, design, and implementation of development policies and programmes could potentially lead to enhanced capabilities and livelihoods of people living in poverty. We claimed that evidence-informed development would come about as a result of an institutionalised culture of evidence use practiced by all actors within the development system. This claim was then unpacked in blog 2, which showed how an institutionalised culture of evidence use could allow for the arrival of irreversible policy changes in development, which in return seem to present a first step towards building a learning development ecosystem. However, as explained in more detail in the paper, it seems that an institutionalised culture of evidence use also requires the development of evidence-literacy across all levels of development actors. This last blog then is concerned with illustrating the frontier of evidence-literacy as well as to relate this to current debates on evidence use in development.
Evidence-literacy is closely related to an institutionalised use of evidence. While the second refers to individual and organisational habits, routines, and processes, evidence-literacy refers to the skills required to access, understand, and produce evidence. The concept holds that evidence-informed decision-making is more likely to be valued in communities and societies if people understand what evidence is. To appreciate evidence’s importance during decision-making processes, it is argued, people need to be familiar with evidence, know where to find it, and understand how it is produced (if not produce it themselves). The concept of literacy emphasises that the need to be evidence-literate applies equally to each member of society. Early work on the importance of evidence-literacy in evidence-informed development has argued that it might be linked to a wider ‘culture of enquiry in society‘ arguably to be communicated through the formal education system. For the purpose of the paper, however, we treat evidence-literacy on a conceptual rather than a practical level.
To illustrate the role of evidence-literacy in evidence-informed development take the example of a vehicle. One could understand institutional frameworks that actively foster evidence use during decision-making as the engine, whereas evidence-literacy is the energy that makes the engine work. An evidence-literate public might demand public programmes to be based on good evidence of their likely effects, which in return might influence elected public officials to design programmes taking into account the available evidence-base. Evidence-literacy would allow a greater space for individuals directly affected by public programmes—nurses, farmers, unemployed, etc.— to express and review the impact of these programmes on their lives and professions. In a perfect world, this could lead to a development eco-system in which evidence in form of knowledge from research, practice, and experience is generated at the roots through learning in practice and then fed back into the decision-making processes at policy level and communicated to the wider community of practice.
One might say this is a very utopian view, but there now seems to be good reason to believe that such an evidence-informed approach to development is possible and indeed shows success in ending poverty. Over the course of 2015, BRAC ‘s Ultra Poor Graduation programme has made high-profile headlines (see eg here, here, and here). While the headlines celebrating the BRAC graduation programme certainly remind one of the many magic bullets that have been fired into the development universe, the actual programme does not. BRAC’s graduation approach is the outcome of 13 years of piloting, testing, and iterating different approaches to poverty alleviation. The NGO first started deploying different sets of interventions encompassing a range of social services and benefits to women living in extreme poverty in Bangladesh in 2002. The NGO then carefully evaluated these different interventions (eg micro-finance; direct provision of assets; health care recommendations) using mixed-methods designs to capture programme mechanisms, contexts, perceptions, and conflicts. The results of the evaluations were fed back to the local communities in which BRAC works and informed the design of the next programme iteration. Through this process of experimentation, evaluation and iteration the graduation approach was build incrementally through adapting unsuccessful programme features and carrying through effective components.
Evaluations of BRAC’s ultra-poor approach in Bangladesh were highly convincing and demonstrated eg ‘that 95 percent of participants not only improved their economic welfare through increased income from livelihood-generating activities and access to savings and social services, among other factors, but also continued that upward trajectory after “graduating” from the program.’ However, the NGO was quite clear that the specific interventions employed in the graduation approach in Bangladesh could not serve as a blue print or magic bullet to poverty reduction in other contexts; rather, it was the methodology of the graduation approach that could be transferred across contexts. In 2010, with support from the Ford Foundation and the World Bank, BRAC tried just that and implement ten pilots of the approach in eight different countries. This year the results of the external evaluation led by Abhijit Banerjee and Esther Duflo were made public showing that one year after completion (three years after initiation) in Ethiopia, Ghana, Haiti, Honduras, India and Peru the graduation approach works—and it works across contexts, and it can be scaled up by policymakers in government and the private sector alike. In the evaluation report, it was stressed that it was not one model of the approach that worked in all contexts but rather it were the basic concepts of the approach, which, adapted to contexts, were effective. As Lauren Whitehead explains:
‘the solution is neither cash nor cows, but a broader question of purpose and scope of the chosen intervention in elevating ultra-poor households. We have to first ask what we hope to gain from implementing an intervention in X region of Y country to reach Z population of the ultra-poor. If the goal is to provide income support and consumption smoothing in an environment with widely used digital financial services such as Kenya’s M-PESA (where appropriate delivery channels do not present a liability), then cash may be a more effective means of delivering assistance. If the aim is holistic improvement in welfare with prolonged impact over time, the scales might tip in favor of a graduation approach.’
To us, having written our paper a couple of months before the Duflo and colleagues study was published, the story of BRAC’s graduation approach is fascinating and seems to resonate with what we hoped would be the last step in our theory of change of evidence-informed development. It is very encouraging to see that there is indeed some good empirical evidence for the positive potential of the use of evidence in the design of development programmes and policies (the secondary evidence-base is less clear cut in this regard, see here).
It is not all blue skies and sunshine in the world of evidence-informed development though. Coinciding with the publication of the BRAC evaluations was the release of a new book by Rosalind Eyben and colleagues with the provocative title ‘The Politics of Results and Evidence in International Development—playing the game to change the rules’. There is a helpful summary of the book here but to give you a feel of the authors’ perceptions of the use of evidence in development consider the below extracts:
Development organizations increasingly seek to understand better what works for who and why – or why not. However, disputes arise around the power dynamics that determine who decides what gets measured, how and why. The cases in this book bear witness to the experiences of development practitioners who have felt frustrated by the results and evidence protocols and practices that have constrained their ability to pursue transformational development. Such development seeks to change power relations and structures that create and reproduce inequality, injustice and the non-fulfillment of human rights.
Those seeking to create or maintain space for transformational development can use the results and evidence agenda to better advantage, while minimising problematic consequences.
What is most stinking from the above is the perception that transformative and evidence-informed development are two contrasting approaches to development; and that the first needs to be safeguarded from the second. This is surprising, as we would have put evidence-informed development firmly into the transformative development corner (have a look at Kirsty Newman’s blog where she does this literally!). As we have outlined in the paper, and in the last couple of blogs, using evidence is about learning, about listening to people’s voices, about programme relevance and effectiveness, and about constant communication and exchange. Consider the story of the graduation approach: this is a story about evidence-informed development, but it is not about RCTs vs other research designs or about a top-down dictator-type of demand for evidence vs a bottom-up struggle and pressure to comply. BRAC’s graduation approach is transformational and so is the principle of evidence-informed development.
It seems the strong criticisms against evidence-informed decision-making in development are taking the concept out of scope. Evidence-informed development lays no claims to be another new ‘grand narrative’ in international development, akin to the ideas of the Sachs and Easterlys of this world. Rather, the term evidence-informed refers to a principle to guide one’s work akin to working in a sustainable manner or to be accountable. It seems that precisely this is often overlooked. We advocate the use of evidence to inform development policies and programmes. This hardly swings a sword at the goal of transformative development.
The world of development enjoys big debates and controversies, and to be fair, they make life more fun. We conclude in the paper though, as in this blog now, that in development policies—as in all social policy arguably—there are few big bangs out there. Change is incremental and transformation rather than revolution is the pathway to success. Evidence-informed development as a principle, not as new narrative of what will end global poverty, embraces incrementalism. It is about identifying little successes, learning from mistakes, stop doing harm, and slow but constant progress—how powerful these principles can be shows the transformation in the lives of the women part of BRAC’s graduation programme.