Risk assessment tools are touted by liberals and conservatives alike as a data-driven way to reduce both incarceration and crime. But how well they achieve those goals is really a matter of policy, not statistical modeling.
That’s the point we tried to illustrate in the parole simulation that’s part of our story on how risk assessment is transforming the criminal justice system. In the simulation, a risk assessment tool “predicts” how likely fictional inmates are to commit another crime if they’re released on parole; it’s up to readers to decide who should be released and who should stay behind bars. The underlying statistics never change, but readers’ choices make a big difference in both incarceration and recidivism rates.
The simulation is meant to be a reasonable approximation of reality, but it’s greatly simplified and isn’t intended to represent an actual parole system. In the real world, risk assessment raises lots of questions that aren’t addressed in our model. But the fundamental choice in our simulation is the same one facing all parole boards: how to strike a balance between keeping behind bars people who pose little threat and releasing people who are likely to commit more crimes in the future.
In phase one, the simulation shows a group of prisoners who are eligible for parole. Because this is a simulation, we know ahead of time who will reoffend if released and who won’t — but the fictional “parole board” doesn’t know who’s who. What the parole board does know is the results of a risk assessment, which estimates each prisoner’s probability of recidivism. The assessment might say, for example, that Prisoner A has a 25 percent chance of reoffending while Prisoner B has a 50 percent chance.
In phase two, readers use the sliders to choose the cutoff points between the low-, medium- and high-risk categories. To one reader, a 50 percent chance of recidivism might sound like a big threat, and Prisoner B would go into the high-risk pool; another reader might think 50 percent should only qualify as medium risk. The simulation then sorts inmates into the appropriate categories.
In phase three, everyone deemed low risk is released on parole, and all the inmates deemed high risk are denied parole and stay in prison until their sentences are up. People in the middle category are assigned to prison or parole at random. But because the tool can only assess probabilities, not certainties, some “mistakes” are inevitable: Some people who are released go on to commit more crimes, and some people are kept in prison even though they wouldn’t have reoffended. Going back and changing the cutoff points will change the number of people who end up in each category, but there’s no way to avoid mistakes entirely. There is no right answer — different readers will be comfortable with different levels of recidivism and incarceration.
The simulation is loosely based on the Ohio Risk Assessment System’s Re-Entry Tool, which is intended to assess the probability of prisoners reoffending after they are released from prison. (Although the Ohio tool is used on inmates before their release, it isn’t used for parole decisions.) The data comes from a study published in 2009 in which researchers in Ohio gave the assessment to 277 inmates who were nearing release from prison. They then tracked those inmates for about a year after their release to see how many were arrested for new crimes (not including parole violations). The report breaks down the scores each inmate received and who went on to be arrested again. Our simulation uses the Ohio numbers to determine how many people get each risk score and how likely a person with a given risk score is to reoffend. (See below for more technical details.)
At the start of the simulation, each “inmate,” represented by a dot, is assigned a probability of reoffending. The computer then uses those probabilities to decide whether each dot will “actually” reoffend. A dot with a 75 percent risk will, on average, “reoffend” (be filled in, rather than hollow, in the simulation) three out of every four times.
There is one additional layer of complexity, however: No one knows a prisoner’s true chance of reoffending, so the risk assessment has to try to estimate it. Each dot is therefore assigned a second, “estimated” probability representing the risk score generated by the assessment tool. That number will generally be close to — but not exactly the same as — the “true” probability. It is this estimated probability, not the “true” probability, that determines whether a dot ends up in the high-, medium- or low-risk category in phase two.
Technical details
We used the Ohio data to ground our simulation in reality, but the assumptions are not meant to perfectly represent real-world outcomes.
The simulation assigns “true” recidivism risk based on a normal distribution, with the mean and standard deviation based on the Ohio data. The “estimated” probabilities are randomly assigned to each dot based on its “true” probability; the random number is based on a normal distribution, with the mean set at the true risk and a standard deviation of 0.15. (The standard deviation was chosen to approximately mirror the noise seen in the actual data.)
Ohio’s risk assessment tool doesn’t directly estimate the probability of recidivism; rather, it produces a numerical score. (Theoretically, scores can range from 0 to 28, but actual scores in the 2009 study ranged from 3 to 23. We took those scores as the practical range of possible scores.) We translated those scores into probabilities using a logistic regression, with recidivism as the dependent variable.
In the simulation, everyone considered low risk is released on parole, and everyone who’s high risk stays in prison. In reality, parole boards generally retain at least some discretion and don’t always follow the recommendations of a risk assessment tool. Our simulation also assumes that medium-risk inmates have a 50-50 chance of being released; in fact, states vary widely in how often they grant parole. (And, as The Marshall Project reported last month, there aren’t good national statistics on how often parole boards release prisoners.)