Economist Daniel Klein recently wrote a piece in the Atlantic that shows just how important it is to ask the right questions in research. A colleague of Klein designed a survey to explore whether ideological differences stem more from people’s beliefs about how the world works or from differences in their basic values. Klein and the colleague then isolated a few of the questions to test a respondent’s objective knowledge about economic matters. An analysis of the responses led the pair to conclude in 2010 that left-leaning people are less enlightened about economic matters than right-leaning people. They published their conclusions in Econ Journal Watch and Mr. Klein wrote an op-ed in the Wall Street Journal. The WSJ opinion piece set off exactly the firestorm you’d expect from a claim that the left is dumber than the right.
Klein now admits that those on the left were not necessarily less enlightened; they just tended to stick to their views. In other words, they were likely to refute a true statement if it didn’t align with their beliefs. The correct answers to the economic questions were more closely aligned with conservative views, giving left-leaners a disadvantage in showing off their economic intelligence.
A follow-up survey in 2011 tested whether conservatives and libertarians were equally unenlightened about statements that challenged their views. He concluded that regardless of political leaning, “the more a statement challenged a group’s position, the worse the group did.” In other words, the correct conclusion is that confirmation bias exists, not that left-leaning people are unenlightened.
There’s a great deal we can learn from Mr. Klein’s screw-up that we can apply to our own research.
Questions must be valid for your research purpose. Mr. Klein’s biggest mistake was how he analyzed responses to questions that were not intended to serve his research purpose. The questions from the original survey related to economic issues but were not designed to test economic knowledge.
Avoid seeing something that’s not there. Mr. Klein went digging for information that didn’t exist in the in the original survey. He was looking for it, so he saw it. A good researcher constantly seeks as much meaning as possible, explores different theories and ideas, and analyzes the data from different angles. But it’s just as important to not see things that aren’t there.Study design is critical in isolating variables so that you can get rid of the clutter that may contribute to an incorrect conclusion.
Repurpose with caution. Repurposing data can be an attractive option, because it’s often less expensive than doing your own original research. It’s especially efficient when looking for basic information and basic relationships. But when you are asking a more complicated question, it’s probably best to design your own study around the research question you are trying to answer.
Klein’s experience is a great reminder of why skepticism and transparency are key when you are delving into the results and conclusions of a study. When a claim seems some combination of outlandish, controversial or counter-intuitive, it’s a good idea to check the methods, the intended use of the data, and the history of the organization doing the research.
In the end, Mr. Klein’s mistakes weren’t costly, besides some initial embarrassment and the energy it took to respond to critics. The criticism led him to correct the mistakes made in the first publication and ultimately give us insightful conclusions based on a well-designed study that matched a well-defined research question.
Kara Capelli is a graduate of The George Washington University’s Graduate School of Political Management, in Strategic Public Relations. She is currently employed as a Public Affairs Specialist for the U.S. Geological Survey.
