Grading New York Restaurants: What’s in an ��A’?

Michael Appleton for The New York Times

Late last summer, the New York City Department of Health began releasing the results of new restaurant inspections, and assigning letter grades based on compliance with the city Health Code and the state Sanitary Code. The letter grades have become ubiquitous in restaurant windows throughout the city. But what do they really mean, and how seriously should we take them in choosing a place to eat?

The grades take into account three categories of violations that may be observed during a restaurant’s annual inspection:

— Public health violations, with the potential to cause an immediate health threat. Inspectors are authorized to shut the place down immediately if the restaurant is unable to address these.

— Critical violations, which do not pose an immediate public health threat but are generally unappetizing, like the presence of live roaches, evidence of rats or mice, and food held at improper temperatures.

— General violations. The list of these is long, and may include improper thawing techniques for frozen meats, improper trash receptacles and unsanitary bathroom facilities.

(The complete list of violations is available in the city health department’s Blue Book for restaurants; it is not for the faint of heart.)

Restaurants are assigned points for each violation the inspectors find. The rules call for assigning at least 7 for a public health violation, 5 for a critical violation, 2 for a general violation, and the guidelines in the Blue Book give inspectors authority to assign more than that based on the severity of the violation. These points are added up to yield a letter grade: 13 points or less earns an A, 14 to 27 gets a B, and 28 points or more means the restaurant gets a C (and should probably be avoided at all costs).

Since the Department of Health began using the new system last July, more than 12,000 restaurants (of the estimated 24,000 restaurants in the city) have been graded. More than half (about 56 percent) received an A; slightly fewer than one-third (31 percent) received a B; and about one out of eight (12 percent) got a C grade:

Providing consumers with easy-to-read, easy-to-access information on their choices is valuable, both in restaurant window signs and through the health department’s website, which allows users to search for specific restaurants and access information about their inspections.

As it currently stands, though, the grading system is too coarse, because it masks wide variations in the quality of restaurants receiving the same grade, making it hard to say quite what it means to be an A-rated restaurant in New York City. Closer inspection of the underlying data reveals a suspicious distribution of restaurants near the cut-off point between an A and a B.

A restaurant receiving any score from 0 to 13 points gets an A, but the difference from one end of that range to the other is substantial. A zero score means that inspectors found no violations at all, while 13 points means they found a host of concerns. A (hypothetical) restaurant where hot food items were not being held above the required temperature of 140 degrees, toxic chemicals were improperly labeled or stored in a way that contamination of the food might occur, and the restroom had no toilet paper or trash can would get 12 points, according to the inspection system outlined in the Blue Book — and qualify for an A.

The graph below shows the distribution of A and B-rated restaurants inspected between July 28, 2010, when the grading began, and Jan. 3, 2011, when we compiled the data for this post. The horizontal axis tracks the number of violation points, and the vertical axis tracks the number of restaurants scoring in that category. The blue bars are A-rated restaurants, and the green bars are B-rated.

During this period, only 381 restaurants scored a perfect zero, while 1,012 scored 12 points and 721 scored 13 — just barely squeaking in to the A category. The data suggests that when you visit an A-rated restaurant, the odds are that it barely made the grade.

A more pressing statistical problem arises from the sharp break in the distribution that occurs around the grade cut-off point. Keep in mind that the 13-point threshold is somewhat arbitrary; there is likely to be little difference between a restaurant with 13 points worth of violations and one with 14 points.

Take a look, though, at the number of restaurants whose scores barely qualified for an A, and those whose scores just missed. Four times as many restaurants scored 11, 12 or 13 than scored 14, 15, or 16.

There are at least two possible explanations for this unusual distribution. One has to do with a provision of the system that allows non-“A” restaurants to request a re-inspection in hopes of earning a better score, and to hang a “Grade Pending” sign in their windows in the meanwhile. The spike in restaurants with 11, 12, or 13 points could be the result of restaurants with initial scores in the B range cleaning up their acts just enough to qualify for an A.

An alternative explanation could be related to the inspectors themselves. Knowing that restaurants that get B grades are likely to appeal them, inspectors may be more likely to rate a restaurants on the cusp with an A-range score; after all, the difference between 13 points and 14 is marginal, but the difference between an A and a B is meaningful. Given the subjective nature of the inspection process and the discretion that inspectors have to assign scores, the data suggest that inspectors may be disproportionately likely to assign restaurants a just-made-it A score than a just-missed B.

Given the current distribution of scores, what would happen if the cut-off for an A grade were to be raised slightly, to 12 or perhaps 11 or 10 points? Substantively, a difference of one or two points in a restaurant’s violation score is probably very small, but the shift in the number of A ratings would be quite significant.

Lowering the threshold for an A grade by one point would cut the proportion of restaurants earning an A by six percentage points, from 56 percent to 50. If it were lowered by two points, the percentage of A-rated restaurants would fall to 42 percent of the total — meaning that one-quarter of all restaurants now displaying an A would be downgraded to B.

The point is not to quibble over the cut-off point, but to acknowledge that switching the (seemingly) arbitrary threshold slightly results in a major change in our perception of how well restaurants are doing.

As an alternative to the coarse three-letter system currently used by the Department of Health, the graph below puts the current scores on the more familiar academic scale by subtracting each point in violations from a restaurant’s initial score of 100 percent. For example, a restaurant with 13 points in violation would score an 87 (100 − 13 = 87). As in a school setting, a score between 90 and 92 receives an A−; between 93 and 96, an A; between 97 and 100 receive an A+, and the B and C grades are broken down similarly. In addition to being more recognizable, a more nuanced grade system provides more precise information to consumers, and addresses the reality that not all A’s are equal.

Although the current grading scheme is imperfect, it does offer some consumer protection and ought to be applauded. According the Department of Health, tainted restaurant food accounts for more than 10,000 visits to the emergency room and 5,000 hospitalizations each year; providing customers with easy-to-access information may help them avoid joining those totals.

In rethinking the grade system, we should keep in mind that any grading systems necessarily involves arbitrary cutoffs. Publicly available information does not provide any rationale for 13 points as the cutoff for an A grade, and requests for explanation went unanswered by the Department of Health. The distribution of scores around the 13-point cutoff raises questions, but these questions should not lead us to abandon the system of scoring restaurants completely.