A recent Perspectives on Politics symposium on Michael Desch's essay "Technique Trumps Relevance" seems to take for granted the notion that policymakers value qualitative research and not research based on more "sophisticated" methods. This is because Desch and Paul Avey had previously published an article based on a survey of senior U.S. government officials arguing that the academy, to be useful, should aim to give policymakers what they want: more qualitative research. (Don't worry, this post is not at all about picking sides in a methods fight: it's about questioning the reasons that folks fight over these things.)
But, for the purposes of argument, let's stipulate that academics should aim to please these policymakers. Do we really know what they want? Policymakers may claim in the abstract to like qualitative research, but they don't seem to know what qualitative research is. Avey and Desch helpfully posted raw survey responses along with their study, meaning that we can look through the answers that policymakers provided when asked to discuss useful social science research.
Before you look at the image below, think of your favorite examples of relevant qualitative research in international relations. Then compare your answer to this sample of survey responses. In the immortal words of Louie Anderson, "Is it up there?"
I encourage you to view the entire set of responses. They might scare you, but they will entertain you. Here are some of my nominations for "funniest example of useful social science research provided by a senior U.S. policymaker":
- "That Used to Be Us: How America Fell Behind in the World It Invented and How We Can Come Back [Hardcover] / Thomas L. Friedman (Author)." (In all seriousness, this is an example of how most of the non-quantitative, non-formal work listed by respondents is not really qualitative research.)
- "Found Daniel Drezner’s work on the international politics of zombies to be a refreshing dash of the whimsical with the real." (Too late to get that on the book jacket?)
- "I am [redacted]"
In addition to overstating the value that policymakers see in actual qualitative research, the study does not emphasize survey results that weaken its main argument. Figure 2 in Avey and Desch shows that 51% of policymakers think that quantitative analysis is somewhat useful. Surprisingly, 35% of them think that formal models are in that second-highest category as well. As an aside, it's strange that several of the policymakers' favorite quantitative works happen to be flawed or at least controversial. Two respondents flag as useful Reinhart and Rogoff on debt-induced downturns and three list Pape’s work on suicide bombing. Note also that Avey and Desch’s favorite example of practitioner-cited quantitative work, Mansfield and Snyder on war-prone emerging democracies, was rebutted in International Organization by Narang and Nelson in 2009.
I close with an observation that I fear to be emblematic of the study. In a section of their paper meant to convey practitioner frustration about sophisticated methods, Avey and Desch relate the following policymaker response: "Many micro-economic models and fitting of history into larger theories [are] not very useful." The authors want us to focus on the first part of the sentence, but doesn’t that second part seem like a rejection of… qualitative research, if not social science altogether?
Policymakers, at their peril, seem to care not about the overarching goal of theory building and testing, but instead about the narratives sometimes employed in testing theories. This would seem to rescue the assertion that more case-based qualitative work means more relevance. Yet the survey also shows that policymakers tend to get their narrative fix from sources outside of political science, since pop historians, public intellectuals, journalists, and futurists tell stories, too, and political scientists will almost always lose the readability battle with them. In any case, we should think carefully about handing the keys to the discipline over to those who would scrap the car but for the parts they like.