Before You Call Your Senator, Read This On How Our Trump Scores Work

On Monday, we launched a dashboard that tracks how often members of the House and Senate have voted in line with President Trump’s position on bills and nominations. We were pleasantly surprised at how many people¹ were interested in this feature — the level of political energy right now is about as high as I can remember in the almost 10 years that I’ve covered politics. But as a result, what we’d expected to be a fairly slow rollout suddenly occurred much faster. So I wanted to add a word of caution, along with a bit more methodological detail.

The caution is simply this: For the time being, these calculations aren’t based on very many votes. Therefore, they’re likely to bounce around over the next few weeks until more votes are taken. As of Monday, they included just four votes in the House and six votes in the Senate. It’s also important to note that we aren’t tracking all votes — only those on which Trump takes a clear position.² So they represent a small sample size, for now.

Another unique feature of our dashboard is the plus-minus scores. The basic idea is to compare how often a member of Congress voted with Trump against others where the 2016 presidential vote was similar. For instance, you’d expect members to support Trump most of the time if they come from a state or district that voted for Trump by 30 percentage points, but not very often if they’re from one where Hillary Clinton won by that margin.

These estimates are calculated on a bill-by-bill basis. Here, for instance, is a breakdown of votes on confirming Mike Pompeo as CIA director based on the 2016 vote in each senator’s state. Almost all (38 of 40) senators from states that Trump won by 10 or more percentage points voted to confirm Pompeo. Just five of 24 from states that Clinton won by 10 or more points voted to confirm him, however. The majority of senators from swing states voted to confirm him, however, which is why his nomination eventually passed by a fairly clear 66-32 margin.

Technically speaking, our projections are calculated by probit regression. For each bill, we run a regression in which the input (independent variable) is the presidential margin in each member’s state or district and the output (dependent variable) is the member’s probability of voting for the bill. On the confirmation of Pompeo, for instance, the model shows Democratic Sen. Mazie Hirono of Hawaii, the bluest state in the 2016 election, with an 11 percent chance of voting to confirm him, but Republican Sen. Mike Enzi of Wyoming, the reddest state, with greater than a 99 percent chance. To reiterate, the regression is essentially just looking at how a member’s colleagues in states or congressional districts of similar partisanship voted. If we say that Sen. Dick Durbin of Illinois has a 32 percent chance of voting for Pompeo, for instance, that just means that about one-third of his colleagues in states like Illinois (which voted for Clinton by 17 percentage points) did so. Note that the regression does not consider a member’s party or any other factor apart from the 2016 presidential vote. Thus, Democratic Sen. Jon Tester of Montana and Republican Sen. Steve Daines of Montana are each given the same likelihood of voting for Pompeo, for instance.

Here’s an example of how the calculations work for Kentucky Sen. Rand Paul, a Republican with some libertarian and “maverick” tendencies. Trump won Kentucky by almost 30 points, and most of the senators from states like that have voted with Trump almost all of the time so far. (In the long run, Democratic Sens. Joe Manchin of West Virginia and Heidi Heitkamp of North Dakota will sometimes be exceptions to this, although even they have supported Trump’s position in five of six votes to date.) The regression model, therefore, estimates that a senator from a state that Trump won by 30 points “should” have voted with Trump 98 percent of the time so far. Instead, Paul voted against Trump twice — first on a bill to allow Republicans to repeal certain provisions of the Affordable Care Act without having to overcome a filibuster and secondly on Pompeo. In both cases, Paul was the only Republican “no” vote.

Thus, Paul has supported Trump only 67 percent of the time, instead of the 98 percent of the time you’d expect based on Trump’s margin of victory in Kentucky. In our parlance, that means Paul has a substantially negative (-31) plus-minus score — he’s supported Trump less often than you’d expect. If Paul keeps that up, he could be a real thorn in the side of the Trump administration. Still, the score is based on just six votes for now.

A contrasting case is Democratic Sen. Dianne Feinstein of California. After missing the vote on the Affordable Care Act to have pacemaker surgery, she has so far voted with Trump five of five times despite hailing from one of the nation’s bluest states:

Some of these “yes” votes were to be expected. All but one Democrat voted to confirm James Mattis as secretary of defense, and all but three Democrats (and independent Bernie Sanders) voted to confirm Nikki Haley as U.N. ambassador. However, most Democrats voted against Pompeo as CIA director, when Feinstein voted for him. The regression model would especially expect a Democrat from a state as blue as California — which went for Clinton by 30 percentage points — to have voted against Pompeo. It also gave Feinstein about a 50-50 chance of voting against John Kelly as secretary of homeland security, since support for Kelly was spotty among senators from states as blue as California, but Feinstein voted to confirm him.

Overall, Feinstein has voted with Trump 100 percent of the time, compared with the 56 percent of the time the model expected. That means she has a highly positive (+44) plus-minus score — that is to say, more pro-Trump than you’d assume given that she comes from a state where Trump is so unpopular.

Feinstein’s score is still based on just five votes, however, so you probably want to see how she votes on a few more issues before calling her office. Her score is also somewhat affected by having missed the Obamacare vote, although — especially after a technical change we made³ — that matters less than you might think. Feinstein would presumably have voted “no” on the Obamacare measure, as every other Democrat did. However, the regression analysis would have assumed that she’d almost certainly have voted no, so it wouldn’t have affected her plus-minus score all that much. Under the plus-minus system, the votes that matter are the ones that defy expectations, such as Paul’s vote against Pompeo or Feinstein’s in favor of him.

Over the next few weeks, we’ll be adding some new features to the dashboard — for instance, pages allowing you to see a detailed list of votes for and against Trump for all members of Congress, as in the examples for Paul and Feinstein above. And by later this year, after we’ve collected several dozen votes, we may place them into different categories. Do some members of Congress support Trump on key votes but oppose him in cases when their votes don’t matter much — or vice versa? Do some agree with him on economic policy but oppose him on expanding the federal government’s powers? Without any exaggeration, this year should be among the most important in the history of the U.S. Congress, and we’ll be covering what they do closely.

FiveThirtyEight: Nate Silver on Trump’s immigration ban