Evidence at USAID, part two
In my last blog post, I shared some thoughts on Sarah Rose’s excellent recent working paper on how USAID could become more evidence based. TLDR: I argue that USAID doesn’t necessarily need to generate evidence (since there are plenty of other orgs that do that) but it does need to the use the latest and best evidence. Unfortunately, this is not always the case.
Roses’ paper has several interesting recommendations for ensuring that USAID makes better use of evidence. She recommends that USAID a) consolidate the various entities responsible for evaluation, b) create a new cadre of “evidence brokers” responsible for digesting and communicating the latest evidence, and c) add rules to ensure greater attention to evidence in program design and procurement.
These are all solid recommendations, but I think that they don’t get at the heart of the problem. In my opinion, the two biggest barriers to greater use of evidence are a) programming inertia and b) staff awareness of the latest evidence. First, there is a lot of inertia in major programming decisions. Much (most?) USAID funding is routed through large 5-year projects and, from what I can tell, when a project ends, it is typically replaced by a new 5-year project very similar in its overall design. For example, while at Abt Associates, most of my work was on the SHOPS project, a large 5-year project which focused on private sector health providers. Prior to the version of SHOPS that I worked on, there were 4 or 5 predecessors to SHOPS all of which did more or less the same thing. When I checked the Abt Associates website just now, it looks like there is yet another SHOPS project with more or less the same overall mission (though I think they have added “plus” to the name).
Programming inertia like this means that there is little scope for evidence to inform many of the biggest funding decisions USAID makes. Some amount of inertia is probably a good thing: you wouldn’t want to make major changes to the overall distribution of funding every year. But USAID seems to err on the side of too much inertia. I don’t know what the best way to fix this is. Perhaps USAID should do a one-time overall review of all programming; perhaps it should change the way it decides which projects to pursue; or perhaps it should shift away from project-based funding overall. While the formal rules for how this stuff happens is well spelled out, I wasn’t involved in these decisions while at USAID and thus don’t know how these decisions are actually made in practice and thus don’t have too much insight into what should change. Nevertheless, it seems like something should change.
The second barrier to greater use of evidence, in my opinion, is that staff simply don’t have the time or incentive to stay on top of the latest evidence. Even given the constraints of the projects they work on, USAID staff typically have a lot of flexibility in what gets funded or not. For example, on the SHOPS project mentioned above, there were dozens of sub-components to the project each of which our USAID officer could approve or reject at her discretion. Yet our AOR was typically so slammed that she had little time to devote to these decisions much less to do a thorough review of the evidence prior to making a decision.
Unfortunately, this is a tough problem to fix. Adding a rule or procedure is easy. Changing an organization’s culture so that staff have more time and are more inclined to stay on top of the latest evidence is far more difficult. I don’t have a solution to this, but here are a few ways USAID could move in this direction:
1. Slash the rules Andrew Natsios and Dan Honig have written much more eloquently than I could about how USAID staff are overloaded with rules and oversight Ironically, the desired intent of many of these rules is to ensure that programs are evidence-based and aligned with overall country strategy yet the cumulative effect, as Natsios and Honig show, is that technical officers are constantly busy complying with these rules and have little free time for field visits or to read the latest paper related to their sector. Many of these rules are outside the control of USAID. Yet there is one area where USAID could do much to reduce what Natsios calls the “obsessive measurement disorder”: it could reduce the ADS, particularly the program cycle ADS which specifies the rules staff must comply with when designing and carrying out programs.
2. Increase the capacity of the Chief Economist’s office When I worked at USAID, I found that nearly all of my colleagues were nice, congenial people. Overall, this is a very good thing. But, every once in a while, I wished that there was a curmudgeonly economist to call BS in a presentation I attended. In theory, the chief economist’s office could play this role – that is, it could help other offices digest and synthesize the latest evidence and, if necessary, call BS when a contractor’s claims are not backed up by data. In Rose’s terminology, the chief economist could serve as an “evidence broker” for the bureaus. To play that role, it would probably need a bigger budget and more staff though.
3. Minimize senior leaders’ pet projects Senior USAID staff and politicians often have their own development pet projects which they are passionate about. Hilary Clinton was passionate about clean cook stoves. Ivanka Trump was passionate about women’s empowerment. These pet projects typically don’t take up a lot of money, but they take up a lot of staff time and foster the impression that evidence matters less than the whims of senior leaders/politicians.
4. Make hiring and promotion decisions more based on contributions to sector knowledge Staying on top of the latest evidence takes a lot of time. Currently, there isn’t much of a professional incentive for staff to invest this time. Considering contributions to sector knowledge – such as blog posts, conference presentations, or papers – in hiring and promotion decisions would help correct this.
