On the perils of embedded experiments

There is growing interest in ‘embedded experiments’, conducted by researchers and policymakers as a team. Aside from their potential scale, the main attraction of these experiments is that they seem to facilitate speedy translation of research into policy. Discussing a case study from Bihar, Jean Drèze argues that this approach carries a danger of distorting both policy and research. 

Evidence-based policy is the rage, to the extent that even village folk in Jharkhand (where I live) sometimes hold forth about the importance of ‘ebhidens’, as they call it. No one, of course, would deny the value of bringing evidence to bear on public policy, as long as evidence is understood in a broad sense and does not become the sole arbiter of decision-making. However, sometimes evidence-based policy gets reduced to an odd method that consists of using randomised controlled trials (RCTs) to find out ‘what works’, and then ‘scale up’ whatever works. That makes short shrift of the long bridge that separates evidence from policy. Sound policy requires not only evidence – broadly understood – but also a good understanding of the issues, considered value judgements, and inclusive deliberation (Drèze 2018a, 2020a).

Enormous energy has been spent on the quest for rigorous evidence, much less on the integrity of the process that leads from evidence to policy. As illustrated in an earlier contribution to Ideas for India (Drèze et al. 2020), it is not uncommon for the scientific findings of an RCT to be embellished in the process. This follow-up post presents another case study that may help to convey the problem. It also illustrates a related danger – casual jumps from evidence to policy advice. The risk of a short-circuit is particularly serious in ‘embedded experiments’, where the research team works ‘from within’ a partner government in direct collaboration with policymakers.

The case study pertains to an experiment conducted in Bihar in 2012-2013 and reported in Banerjee, Duflo, Imbert, Mathew and Pande (2020)1. This is a large-scale, influential experiment by some of the leading lights of the RCT movement – indeed, a formidable quartet of first-rate economists reinforced by one of India’s brightest civil servants, Santhosh Mathew. The high technical standards of the study are not in doubt, and nor is the integrity of the authors. And yet, I would argue that something is amiss in their accounts of the findings and policy implications of this study.

Read More »

Abolition Will Not Be Randomized

28476745294_cf7a695ec4_o

By Anastasia Wilson and Casey Buchholz

In the wake of the current uprising in support of Black Lives Matter, there has been increasing interest in the use of mainstream empirical methods in economics — like randomized control trials (RCTs) and administrative data evaluation — to address issues of racism and violence in the institution of policing. These interests are well intentioned, but similar to prior debates, we are reminded that “there is reason for concern” about the relevance of these approaches amidst a mass movement calling for deep structural and institutional change. In just two weeks, mass protests have sprung up across the U.S. and the world calling for the defunding, disbanding, and abolition of police as well as the dismantling of white supremacy. This moment has the potential to bring about an institutional and structural shift in our politics, society, and economy. Given this, we will echo many of the concerns shared by economists about the limits of some empirical methods, the biases embedded in administrative data, and the relevancy of these approaches to the current moment calling for immediate change. Read More »

Misunderstanding the average impact of microcredit?

6886186408_5de7def831_o.jpg
Photo: Peter Haden. Microfinance center leaders tally the week’s loan payments in India.

By Milford Bateman and Maren Duvendack

A recent article on the “average impact of microcredit” by Dr. Rachel Meager (LSE) has received much praise over the past few weeks. Meager deploys Bayesian hierarchical modelling to provide a new take on the argument in favour of a reformed system of microcredit. Her work builds on the data provided by six randomized control trials (RCTs) conducted by Abhijit Banerjee and colleagues (see Banerjee, Karlan and Zinman, 2015). Meager makes an attempt to exculpate the microcredit model from the awkward fact that its impact on the poor has been very much less than originally envisaged. She also claims to show that the critics have overstated the negative impact of microcredit. Microcredit should therefore continue to be a policy intervention, she goes on to say, but there need to be changes in the operating methodology for a more meaningful development impact to be possible in the future.

While seemingly a well-meaning attempt to explore the impact of microcredit, we were struck by the way that her overall argument appears to seriously misunderstand, and it definitely misrepresents, the existing research on microcredit as a development instrument. Read More »

Why positive thinking won’t get you out of poverty

attitude-be-positive-draw-262532.jpgBy Farwa Sial and Carolina Alves

In a recent article in the New York Times, the development economist Seema Jayachandran discusses three studies that used Randomised Controlled Trials (or RCTs) to understand the benefits of enhancing the self-worth of poor people. Despite wide differences in context, all the cases explore the viability of ‘modest interventions’ to ‘instill hope’ in marginalised communities, concluding that ‘remarkable improvements’ in the quest for poverty reduction are possible.

One of the studies from Uganda, for example, argues that “a role model can have significant effects on students’ educational attainment,” so the suggestion for policy-makers might be “to place more emphasis on motivation and inspiration through example.” Another case study of sex workers in Kolkata Brothels argues that “psychological barriers impede such disadvantaged groups from breaking the vicious circle and achieving better outcomes in life,” so small but effective changes that address these psychological constraints can alleviate the effects of poverty and social exclusion.

The underlying theme of these studies is that individuals can surmount the structural challenges of poverty through their own efforts using tools like ‘effective role models,’ the generation of ‘more hope,’ and the ‘improvement of their mental health.’ Positive psychology of this kind and an emphasis on behavior change to meet the goals of individuals have been around at least since the 1950s, first in the popular literature of self-help books and now in academia, where they form part of an increasingly fashionable trend to ‘do poverty reduction differently.’Read More »

Poor Behavior, Good Behavioral Policies? Double Standards for the North and South

behavioral.jpg

Behavioral approaches to development economics and policy have gained momentum in recent years. A growing number of papers studying behavior of people in poor countries have been published in top journals, accompanied by the rise of randomized controlled trials (RCTs). In 2015, the World Development Report was dedicated to behavioral and cognitive research and policy. Papers studying how to nudge farmers to use fertilizers or increase savings have become classics in the field. Lots of hope has been placed into social experiments and behavioral policies to fight global poverty.

Behavioral policies are of course not reserved for policy-making in poor countries. In fact, nudges became famous with a US-American savings plan. Many behavioral instruments have been discussed and tested in and for rich countries. But there has been an important difference as compared to the debates in development economics: when debating behavioral policies in rich countries, scholars have also devoted lots of time to consider normative and ethical concerns. For example, following Thaler and Sunstein’s exposition of Libertarian Paternalism (see also here), a debate unfolded on whether nudges could be anti-libertarian (here, here, here, or here). Implications of the use of nudges as a new form of government policy have been analyzed, for example, from a Foucauldian perspective, or with a focus on institutional change. Books have been written about ethical concerns. The debate has reached a great level of differentiation, e.g. when authors argue that so-called social nudges (these are nudges that seek to stimulate voluntary cooperation in social dilemma situations) may be justified for different reasons than those targeting individual welfare. Overall, the debate has become really sophisticated, and the autonomy, welfare, and dignity of citizens in rich countries as well as consequences of the use of behavioral policies for these countries’ modes of government have received lots of careful scrutiny (recently again here).Read More »