“It’s difficult for us to break down exactly how good an experience was. It’s not, a lot of the time, as mathy as people expect it to be.”
Fandango is the Lake Wobegon of movie websites: All the actors are stunning, all the directors are auteurs and all the ratings are above average.1 In a recent analysis for FiveThirtyEight, Walt Hickey found that Fandango regularly inflates users’ movie ratings by rounding up to the nearest half-score — so a 3.6 rating turns into 4 full stars. Moreover, there appears to be an inflated floor for Fandango’s ratings overall — 98 percent of the movies analyzed had a 3-star rating or higher and 75 percent had a 4-star rating or higher. Those inflated ratings make it easy to see that Fandango is an outlier compared to other movie rating sites.
None of this would be that big of a deal except for the fact that Fandango sells movie tickets. The company’s financial incentive to convince visitors to click “buy” raises a whole host of tricky questions — including larger ones about trust online, and what it means to rely on the wisdom of the crowd in entertainment and product reviews.
On this week’s episode of our podcast What’s The Point, Walt discusses his reporting and what it taught him about the world of user-generated advice.
Plus, a look at a few statistics sent to us as part of World Statistics Day, which was this past Tuesday.
Stream or download the full episode above, and find a partial transcript below — plus citations for the stats we mentioned on the program.
Audio extra: Walt Hickey discusses what’s at stake for movie studios when it comes to online reviews, and whether they are trying to actively game the system. Find the clip on Dropbox.
What’s the Point: Why does Fandango love movies so much?
The crowd vs. the critic.
Jody Avirgan: All sounds like a lot of work, right? If you have to first review in your head the review site, and then read the reviews … that just seems like it’s adding a whole other step. Isn’t the point of these review sites just to make it simpler?
Walt Hickey: The point of these sites is to make it simpler, but unfortunately, the act of rating something that is entirely subjective is a fundamentally difficult thing. And if it were easier to turn something subjective into an objective number, then we would have a much easier job here [at FiveThirtyEight].
It’s not always an easy thing to take an experience, whether it’s food, whether it’s a film, whether it’s a TV show, whether it’s any kind of thing, and articulate it into something from one to five or even one to 100. There’s a lot of nuance in that. Was this a 96 or a 97? Was this a 40 or a 50? It’s difficult for us to break down exactly how good an experience was. It’s not, a lot of the time, as mathy as people expect it to be. In that regard, you should probably be a little bit more skeptical about seeing anything boiled down to a single number or single value when it comes to a subjective experience.
Avirgan: You said that it’s really hard to articulate a subjective experience, but we have people in this world who do that for a living — they’re called critics. I wonder if you, in the process of reporting this out or thinking about this, have changed your level of trust for the A.O. Scotts of the world? [That’s] how we used to get our advice for how to go to a movie — read a critic and pick a critic that you trust. One individual one as opposed to the aggregate crowd.
Hickey: That’s the smart move. Some of the responses from readers were like, “I only trusted [Roger] Ebert.” And that’s a sensible position to take. He was a guy who had a wide variety of interests when it came to talking about films. He had a specific interest in comparing a film to the best version of itself. He had a solid methodology that was consistent across the board, and as a result he was really useful.
World Statistics Day Stats
The statistics we read on the show, as suggested by FiveThirtyEight Twitter followers.
https://twitter.com/jeffreysmithJRS/status/656601248209104898
https://twitter.com/SriramVenkit/status/656526220818010112
https://twitter.com/SriramVenkit/status/656521034867326976
If you’re a fan of What’s The Point, subscribe on Apple Podcasts, and please leave a rating/review — that helps spread the word to other listeners. And be sure to check out our sports show Hot Takedown as well. Have something to say about this episode, or have an idea for a future show? Get in touch by email, on Twitter, or in the comments.
What’s The Point’s music was composed by Hrishikesh Hirway, host of the “Song Exploder” podcast. Download our theme music.