Mark Blumenthal of the Huffington Post links toward a different survey of Egyptian public opinion, this one by the pew Global Attitudes Project, which tells a completely different story. In contrast to the BBC study that we linked to this morning, which found relatively favorable attitudes toward the United States, the Pew poll finds that just 17 percent of Egyptians have a favorable view of the U.S. against 82 percent unfavorable. It also shows, again in contrast to the BBC study, that views of the United States declined somewhat between 2009 and 2010.
When we examine surveys in the United States, we’re used to seeing differences of opinion of 5 or 10 or 15 points based on things like question wording and sample design. But this is of a completely different magnitude.
As Mr. Blumenthal notes, the BBC poll was confined to several large urban areas, which reflect about one-quarter of Egypt’s adult population. In contrast, only a few remote areas were theoretically off limits in the Pew poll. I would certainly agree that, all else being equal, it would be better to have a survey that covers a larger fraction of the population. On the other hand, this argument seems like something of a red herring in this particular case, since, as Mr. Blumenthal noted, the Pew survey showed no appreciable urban-rural split. (In fact, opinions about the United States were nominally more favorable in rural areas in the Pew poll.)
There is also a difference in question wording: the BBC survey asks whether the United States has had mainly a positive or negative influence on world affairs, whereas the Pew poll asks whether people have a positive or negative view of the United States. While we would expect these results to be correlated, these are different questions and might elicit different responses.
Two other factors might have come into play in explaining the differences — and I suspect, given the magnitude of the discrepancy, that one of the two is the culprit. One is language issues. Both surveys were conducted across many different countries and had to be translated into several different languages: Arabic, for instance, in Egypt’s case. It’s possible that either or both surveys lost something in translation — anything from a fairly subtle shade of meaning, to an outright poor or erroneous translation.
Another issue is cluster sampling, a technique used in the Pew poll and also I would suspect — although it discloses less about its methodology — in the BBC survey.
Cluster sampling is common when surveys are conducted in person, as both the BBC and the Pew polls were; it is also used, for instance, in exit polls conducted on Election Day in the United States. What it consists of is only conducting the survey in certain areas — say, in a handful of neighborhoods, or at a handful of polling places — and then trying to make it demographically representative, either through the way the clusters are selected, through post-hoc weighting, or both. Cluster sampling is subject to certain statistical properties that intrinsically give it a higher margin of error — which is one reason, for instance, that exit polls have been unreliable in some years. It’s also a necessary evil when you conduct an in-person survey; otherwise you’d literally have to be willing to charter a jet to speak to some random dude in the desert whose random number got picked.
Cluster sampling should work reasonably well if you’re doing everything right. The real risk is that you might pick a set of neighborhoods that — perhaps owing to your lack of knowledge about local demographics — aren’t terribly representative at all. Say that you pick a couple of clusters where a number of military families live, or one where many people are employed by the foreign branch of a large multinational corporation; that could skew the results fairly significantly.
Yet-worse problems might occur if, rather than going door-to-door, the pollster instead set up court at a shopping bazaar or at some other public location, the traffic to and from which might be highly nonrandom.
It’s tempting, of course, to categorize one of these surveys as being “right” and the other as being “wrong”, but — although I can understand having some prejudice against a survey that was conducted only in urban areas — I don’t think that’s responsible without knowing quite a bit more about how they handled the issues I brought up above. I would say that Pew has an impeccable reputation — and in my view, a well-deserved one — for conducting quality survey work. But it’s not as though the BBC doesn’t know its way around the world.
What’s especially vexing is that the surveys not only reveal different attitudes, but also show them moving in different directions. Oftentimes, even in the presence of large “house effects”, the trends between different surveys tend to mirror one another, even if the absolute numbers are different.
It does seem slightly more plausible to me that, as the BBC survey shows, Egyptian opinion of the United States would have moved upward rather than downward between early 2009 and early 2010, given that Barack Obama gave a very well-received speech there in the intervening months. At the same time, it worries me a bit — now having seen the Pew study as a point of comparison, which has been less volatile — that the BBC poll’s numbers have shifted so significantly over the course of the past four years. We will endeavor to contact both organizations and see if we can learn a bit more.