Our impact evaluation is not asking the important question

I have the habit of glancing at my department’s printer to see what my colleagues are reading these days. Today I caught an article written by Lant Pritchett, an international development expert, on the fact that most development ideas in RCTs do not pass his four smell tests of being important.

The upshot is summarized in the following table. The columns list his four smell tests for whether an idea is important in helping countries develop.

Lant Pritchett is trying to raise the point that our development ideas weren’t present in historically successful cases of development. But that point is obviously true! We have long known that the most important factor in economic development is not these micro-programs, but the macro political institutions of the countries, be it the British limited government pre-Industrial Revolution, the East Asian developmental states pursuing industrial policy post-WW2, or the Chinese reformist Communist Party in the 1980s.

These political institutions are omnipresent in all corners of a country’s economic life, shaping such fundamental pre-requisites such as price signal, contract enforcement, ease of doing business, transparent and predictable regulation. These factors singularly dwarfed the effects of a de-worming program or a teacher’s performance-related payscheme, which are insular to specific sectors and circumsribed within certain population groups. Furthermore, even if these micro-programs are enormously effective and deserve to be scaled-up, the dirty implementation of a nice idea will once again rest in the hand of the state (and that did not turn out well).

However, even though our micro development programs are not very good, they are the best that we can do. Indeed, there’s not much chance for development practitioners to shape a country’s political economic institutions. Even if there were, the ethics would be up for a lengthy debate.

All that we can ever implement are micro-programs, and at least with RCTs we will learn which of them actually does (non-zero, but limited) good.