Zagat has its annual survey out on fast foot restaurants and there are some surprising results, at least to my tastes. Perhaps the most surprising is that Starbucks beat Berkeley, CA based chain Peet’s Coffee for the title of best cup’o’joe.
Mind you, I’m not a Starbucks hater. It’s fine. The stores are usually comfortable and well run, and it’s often a convenient spot to sit down between meetings (or even to take a meeting).
What Starbucks isn’t, however, is Peet’s, a chain that elicits a ridiculous amount of loyalty from those in California and the few other locations where they have a store. Poll any ten San Franciscans, and I’d assure you with a high degree of confidence that at least seven of them would tell you that Peet’s has the better coffee.
Perhaps this Zagat survey proves that San Francisco values are, true to the conservative refrain, are out of touch with those in the rest of the country. Except that, this isn’t necessarily the apples-to-apples comparison that you might think. There are something like 200 Peet’s Cofeee stores around the country; by comparison, there are something like 11,000 Starbuckses. Most of the people that Zagat surveyed have probably never even heard of Peet’s Coffee, let alone tasted it.
Zagat’s methodology is a little opaque, but from what I gather it’s in fact quite simple and works like this. Periodically, Zagat sends surveys to a number of diners in a given area; these reviewers rate each restaurant they’ve sampled from 0 to 3. The ratings are then averaged together and multiplied by 10, so if I give a restaurant a 1 and my two best friends give it a rating of 2, it will get a composite score of 17, which is 1 plus 2 plus 2 divided by three, multiplied by 10, and rounded to the nearest whole number.
The problem is that different restaurants are rated by different numbers of people — in a national survey of 6,000 adults, for instance, like the one that Zagat just conducted, Starbucks will get thousands of ratings and Peet’s will probably only get a few dozen. When something like that happens, you could potentially encounter a problem such as this:
In this (entirely hypothetical) example, Starbucks finishes with a slightly higher rating on average than Peet’s, even though from among the five people (Alice, Chloe, Felix, Gwyneth and Harry) that rated both Peet’s and Starbucks, Peet’s was rated as the superior option on three of the five ballots and was tied with Starbucks on the other two.
Perhaps this seems like a contrived example, but just for fun, I looked up the ratings on Yelp.com, another user-driven restaurant review service, for busy Starbucks and Peet’s stores located about a block away from another on California Street in the Laurel Heights neighborhood of San Francisco. Overall, the ratings were barely distinguishable; the Starbucks store finished with 3.6 stars on average (out of 5 possible) and the Peet’s finished with an average of 3.8. However, from among the 11 reviewers that rated both the Peet’s and the Starbucks, six rated Peet’s higher and just one preferred Starbucks, with the other four rating the two stores equally. What seems like a too-close-to-call verdict is actually a relatively clear one for Peet’s.
Maybe Zagat’s methodology is more sophisticated than I’m giving it credit for, but it seems to me that you have the appearance of a comparison when there really isn’t one; very few of the people that rated Starbucks also rated Peet’s, and probably some of the people that did rate Peet’s failed to rate Starbucks. There might be demographic differences between the Starbucks’ customers and the Peet’s ones and there are certainly geographic ones. Perhaps it’s also the case that Peet’s customers are a bunch of San Francisco snobs and harsher critics in general and give lower ratings to all restaurants; that would tend to bias downward the results. I’m working on a consulting project for a (non-political) client right now and we’re encountering a problem very much like this one.
People are generally much better at making comparisons between two things than they are taking any one thing as an abstraction. This is something to keep in mind when evaluating political polls. You can find lots and lots of surveys to suggest that people want to cut taxes, balance budgets, and have the government spend more on many key initiatives. Well, terrific: I’m sure they want to cure cancer, win the lottery, and increase the size of that certain part of the male anatomy too. But these economic goals are rather mutually exclusive; the question is what happens when people are asked to pick between them. Surveys generally reveal, for instance, that people prefer lower taxes to balanced budgets, and that they’re willing to accept a higher tax burden for universal health care. It’s those sorts of results that drive political calculations in Washington — who’s to say about their coffee preferences.