Why Methodology Is So Important

What is it that separates political science from political opinion? It's the science. Political opinion is about expressing one's thoughts and ideas; political science involves testing one's thoughts and ideas by developing hypotheses and collecting evidence to determine when those hypotheses hold up. Methodology is critical for identifying variables and predicting how those variables will change in response to certain inputs.

Which brings me to an op-ed from yesterday's New York Times. In a piece entitled "What They're Saying in Anbar Province," Gary Langer, the director of polling for ABC News, challenges the claims by President Bush and General Petraeus that the recent developments in Anbar province represent progress. Specifically, Bush and Petraeus pointed to Anbar, where Sunni chiefs have turned against al Qaeda and have begun cooperating with US and Iraqi troops against al Qaeda forces, as a hopeful predictor of the future. Langer challenges that claim, asking:

Do United States military alliances with Sunni tribal leaders truly reflect a turning of hearts and minds away from Anbar’s bitter anti-Americanism?

The data from our latest Iraq poll suggest not.

Al Qaeda, it should be said, is overwhelmingly — almost unanimously — unpopular in Anbar, as it is in the rest of Iraq. But our enemies’ enemies are not necessarily our friends. The United States, it turns out, is equally unpopular there.

In a survey conducted Aug. 17-24 for ABC News, the BBC and NHK, the Japanese broadcaster, among a random national sample of 2,212 Iraqis, 72 percent in Anbar expressed no confidence whatsoever in United States forces. Seventy-six percent said the United States should withdraw now — up from 49 percent when we polled there in March, and far above the national average.

Withdrawal timetable aside, every Anbar respondent in our survey opposed the presence of American forces in Iraq — 69 percent “strongly” so. Every Anbar respondent called attacks on coalition forces “acceptable,” far more than anywhere else in the country. All called the United States-led invasion wrong, including 68 percent who called it “absolutely wrong.” No wonder: Anbar, in western Iraq, is almost entirely populated by Sunni Arabs, long protected by Saddam Hussein and dispossessed by his overthrow.

But here is why methodology is so important. The question Langer is asking -- Do you oppose the American presence in Iraq -- is not linked to the events he's trying to explain. Bush and Petraeus are claiming that, as a result of the surge making it more difficult for al Qaeda to attack US or Iraqi government targets, al Qaeda has turned to attacking softer targets, which has turned Sunni chiefs, who had been previously cooperating or at least tolerating al Qaeda's present, against the foreign insurgents. Petraeus' report to Congress claimed that monthly attacks in Anbar have dropped from 1,350 in October 2006 to 200 in August 2007, and that thousands of Sunnis have signed up for the Iraqi army and police force. This change in attitude from the Sunnis, if it could spread to other areas of Iraq, could help create both the breathing space necessary for political progress as well as develop some sense of national unity. So what does the attitude of the Sunnis towards the US have to do with these developments?

Nothing that Langer tell us. Langer notes that while 38% of respondents rated local security positively -- up from 0% in March -- and that Anbar Iraqis have demonstrated "remarkable advances in confidence in the Iraqi Army and police", "nobody surveyed in Anbar last month gave the United States any credit." But so what? Why does it matter what they think of the US? What matters according to Bush and Petraeus is how the Sunnis in Anbar relate to the Iraqi government. So why does Langer think these data matter? He doesn't really tell us; there's no methodology here. No connection between his data and his hypothesis. Sunnis in Anbar don't like the US; thus, the claims of progress in Iraq can't be true.

Langer's point seems to be that the Sunnis have suspect motives for their cooperation: "Anbar’s tribal leaders may have any number of motivations for their alliance with the United States. It’s been reported that the United States government has provided them arms, matériel and money, as well as undertaking more than $700 million in reconstruction projects in the province." Again, so what?

The only thing approaching a relevant point is when Langer notes that "Just 23 percent in Anbar expressed confidence in their “local leaders”; 77 percent had little or none. That’s better than it was in March — but still nearly the lowest level of confidence in local leaders we measured anywhere in Iraq." But that echoes the claims of Bush and Petraeus. Things had been dismal in the Sunni provinces, but now they're getting better. But Langer offers no way to judge the progress so we only know that it's improving.

It's hard to know what Langer's point is. It seems to be that he's simply trying to rebut the claims of Bush and Petraeus but his argument is not only totally unconvincing, it doesn't even support his own claim.