Skip to main content
ABC News
NFL General Managers Still Love The Senior Bowl. But Do Those Players Pan Out?

Spring is a season of tumult for the NFL. Team rosters are in flux as free agency reshuffles veteran talent across the league. Schemes are torn down and rebuilt as coaches try to shore up weaknesses and better exploit their playmakers’ strengths. 

Even the NFL combine in Indianapolis — a constant on the NFL calendar for decades — has been buffeted by the winds of change. The Super Bowl champion Los Angeles Rams didn’t send their GM or head coach to Indy this year— and not for the first time. The Denver Broncos top brass joined L.A. in shunning the combine back in 2020. Yet amid all the upheaval, at least one institution has managed to maintain its cachet: the Senior Bowl in Mobile, Alabama.

Popular perception holds that the Senior Bowl is the first draft-related event of the year (its motto is “The Draft Starts in Mobile”).1 It’s also the first offseason convention where GMs and other decision makers can begin evaluating prospects. And unlike Indianapolis, teams don’t appear to be skipping Mobile.

“Most every team sends all their coaches, front office and all components’ of player personnel as part of the week’s evaluation process,” Andy Dengler, the former assistant director of player personnel for the Jacksonville Jaguars who is now with the Las Vegas Raiders, told SB Nation in 2021.

The Senior Bowl’s privileged position in the draft hierarchy is a little strange, though, and the reason is right there in its name: It’s limited to older players.2 Research has shown that younger players who have early success against older competition are better bets for NFL success. Prioritizing older players runs counter to that finding. The calculus is different for each team, but it seems likely that GMs aren’t ignorant of the value of young players — they simply see Senior Bowl players as lower-risk bets.

“We’ve taken our share of juniors like anybody has,” Colts general manager Chris Ballard told ESPN recently. “But guys that have [long] productive careers in college and then have a good week here usually end up being pretty good players in the league.”

Since we love testable claims here at FiveThirtyEight, Ballard’s comment sent us down a Senior Bowl rabbit hole in an attempt to find out if his impression is correct. Do players who get invited to Mobile tend to end up being productive NFL players?

To answer the question, we turned to Meaningful Snaps Over Expected (MSOE), a metric we introduced during the 2021 season. MSOE measures the number of snaps a player plays over (or under) expected in a season after adjusting for draft slot, position, the year (2021 had 17 regular-season games, for instance) and combine testing data.3 For our purposes, “meaningful” snaps are defined as those that occur when the game is still in doubt — between 10 and 90 percent win probability. After training and testing, we applied the model to NFL players drafted from 2012 through 2021 who also attended the Senior Bowl the year they were selected.

To help ground the analysis, let’s first take a quick step back and give a sense of MSOE’s shortcomings: It measures opportunity — the volume of important snaps played — not the value of a snap.4 This means injuries — which are not evenly distributed among position groups — have a significant impact on the metric. Also, players drafted onto teams with deep benches are disproportionately hurt by the metric, since they have to leapfrog a skilled veteran or two in order to earn meaningful snaps. Teams lacking bench depth are more likely to start their young players early and allow them to accrue meaningful snaps.

A final factor to keep in mind is that MSOE rewards teams that find starters from later rounds of the draft more than it does players taken early. This isn’t a flaw — players selected early should be expected to contribute more meaningful snaps to their team — but it’s worth noting.

So, do Senior Bowl players get more meaningful snaps than we would expect? The answer is yes. As a group, players drafted from 2012 through 2021 generated 35,542 MSOE, indicating that Ballard’s impression about Senior Bowl players developing into good players is supported by some evidence.

The effect remains when we break out MSOE by position. Since 2012, the NFL has picked Senior Bowl players who have returned aggregate positive MSOE at every position but quarterback and running back. And among the position groups with aggregate positive MSOE, no position had a higher player-season hit rate5 or total surplus value than defensive back.

Senior Bowl defensive backs tend to get meaningful snaps

Meaningful Snaps Over Expected for NFL players drafted after appearing in the Senior Bowl, by position, 2012-21

pos MSOE picks
Defensive back 16,697 143
Linebacker 8,575 122
Tight end 4,788 46
Offensive lineman 4,474 161
Defensive lineman 3,345 139
Wide receiver 1,903 95
Quarterback -1,519 47
Running back -2,721 63

A meaningful snap is defined as a play for which the pre-snap win probability is above or equal to 10 percent and under or equal to 90 percent. Special teams players are excluded for sample-size reasons.

Sources: Pro-Football-Reference.com, ESPN Stats & Information Group

The NFL used 143 picks on Senior Bowl defensive backs since 2012, and the selections have paid dividends. Senior Bowl DBs played 16,697 meaningful snaps over expected, and 49 percent of the player-seasons produced by those picks returned surplus snaps to their teams. 

Interestingly, when we look at MSOE by round, the model suggests that the NFL is finding most of the surplus meaningful snaps from Senior Bowl players in the first two rounds of the draft. Round 3 provides positive MSOE in the aggregate, but more Senior Bowl players have been picked in the third round than any other since 2012, lowering its per-player average.

Most of the total value of Senior Bowl picks comes in Round 2

Meaningful Snaps Over Expected for NFL players drafted after appearing in the Senior Bowl, by round, 2012-21

MSOE
Round per player total picks
1 177.4 11,711 66
2 152.2 18,418 121
3 60.1 11,186 186
4 9.1 1,542 170
5 -3.8 -430 114
6 -64.7 -6,150 95
7 -11.5 -736 64

A meaningful snap is defined as a play for which the pre-snap win probability is above or equal to 10 percent and under or equal to 90 percent.

Sources: Pro-Football-Reference.com, ESPN Stats & Information Group

This distribution of snaps should probably give us pause. No one is immune to the effects of confirmation bias, and it’s reasonable to assume that early round picks will get more chances to succeed than late-round picks. Former NFL player Ryan Riddle has argued that first-round draft picks must prove time and again that they can’t play, while late-round prospects are forced to prove that they can. The model attempts to control for this, but it’s possible that coaches and GMs whose futures are tied to the success of early draft picks lavish those players with opportunity in a way that continues to bias the data.6

It’s also true that if a team is interested enough to take a player early in the draft, there is more to its interest than the fact that the player attended the Senior Bowl. Getting an invite to Mobile is probably worth a bump in the rankings, but it’s unlikely that teams would move a player up from a mid or late-round pick to a first- or second-round selection on that factor alone.

None of this appears to be lost on Ballard: “Look, some of it’s probably random and timing of need and us liking a player,” he said. Like many advantages in the NFL, a Senior Bowl edge may exist, but it’s small. And in a league where even an award-winning GM’s career is short, you don’t get many shots at getting lucky.

Footnotes

  1. The event has been held in Mobile almost every year since its inception in 1950.

  2. Although not actually only college seniors — graduating juniors may also be invited to participate in the game.

  3. The model used in this analysis is a generalized mixed-effects model with a zero-inflated Poisson error distribution. The variables for player ID and season were treated as random effects. For fixed effects, I used draft year, draft slot, player position, speed score, a binary variable indicating whether a speed score was imputed or not, and the interaction between speed score and the binary variable. Roughly 20 percent of speed score data was imputed. Data was from 2007-21 and was split into 70/30 train and test sets. Out-of-sample r-squared is 0.35.

  4. Though the two are linked: Better players tend to get more meaningful snaps.

  5. In this analysis, position groups are fairly broad and exclude special teams for sample-size purposes. A player’s hit rate here is conservative: It’s defined as a player season (of which every NFL player will have at least one, starting his rookie year) in which MSOE remained positive after subtracting two times the standard error for the player’s expected snaps.

  6. We should also probably be skeptical of the model’s results for the seventh round. A disproportionate amount of speed score imputation needed to be done for players picked in that last round, and that could affect the accuracy of the estimates.

Josh Hermsmeyer is a football writer and analyst.

Comments

Latest Interactives