Nate’s takedown of “pollster” Strategic Visions reminded me of another questionable pollster that gets bigtime media exposure. I’m thinking of Prince & Associates, a Connecticut-based consulting firm that does surveys of the super-rich that get featured in the Wall Street Journal and in Slate. I didn’t dig as deep into Prince & Associates as Nate did with Strategic Visions, but what I did find [link fixed] made me suspicious. From an article by Robert Frank in the Wall Street Journal (from October, 2008):
According to a new survey by Prince & Associates, voters worth $1 million to $10 million are favoring Sen. John McCain, while voters worth $30 million or more are favoring Sen. Barack Obama. The survey of 493 families showed: More than three quarters of those worth $1 million to $10 million plan to vote for Sen. McCain. Only 15% plan to vote for Sen. Obama (the rest are undecided). Of those worth more than $30 million, two-thirds support Sen. Obama, while one third support Sen. McCain.
Do I believe this? Not really. My problem here is that I don’t know where the survey is coming from. How did Prince & Associates sample people making $30 million or more? Without knowing at least something about the sampling, it’s hard to say anything at all about these claims.
For example, a graph accompanying the article linked above gives estimates of about 0.1 million households with over $25 million and 9 million households with over $1 million. This ratio is about 1%; thus, in a simple random sample of 493 people worth over $1 million, you’d expect to see about a whopping 5 people in the survey worth over $30 million. Or maybe there were 6 such people in the sample; that would explain why the percentages of the super-rich cited in the linked article are 16% (1 in 6) and 67% (4 in 6).
The survey might have more than 6 super-rich people in it; I don’t know since no details are given. (I searched on the web for the survey but all I could find were links to the Robert Frank article discussed here.)
How do you take a sample of super-rich people? Prince & Associates is a Connecticut-based consulting company that describes itself as “the foremost empirical research firm in the realm of private wealth. . . Using purposive sampling methodologies, Prince & Associates, Inc. has created statistically valid single-study and panel samples providing detailed insights into the hard-to-reach and exceptionally private universe of the affluent.”
I respect that this sort of sampling is difficult but it’s hard for me to evaluate it when no description is provided of the sample. I emailed Russ Alan Prince to see if he could enlighten me on this, but he didn’t respond. Really, though, I’d think it would be the responsibility of a Wall Street Journal reporter to ask some questions here. (I guess it’s possible that Frank did ask some questions but for proprietary reasons did not want to describe the sampling methodology, but if so I would’ve appreciated just a sentence or two on it, to give me a little more confidence in the results. He also didn’t respond to email. Again, he is not obliged to do so–but, then again, I’m not obliged to believe what he writes, either.)