Weekly Commentary & Review #5

A détente in the nerd-coach war? An improved metric for pass rushing pressure. And the shifting tides of NFL insiders' quarterback rankings

This post looks at a handful of relevant articles, analyses and podcasts from the week that provide useful insight to be absorbed, or have missing context to be added. I’ll add my takes on the material, while heavily quoting the relevant passages.

COACHES AND NERDS LIVING IN HARMONY?

Last’s week’s commentary and review on contracts extensions dropped a little early, so it didn’t includes an article from last Friday by The Athletic’s Ted Nguyen, “NFL’s nerds vs. coaches battle is over: How both sides are helping each other win”.

It wasn’t long ago that veteran NFL head coaches would bristle at the notion of advanced analytics being used in decision-making. To them, gut instincts were more tried and true than any chart of probabilities and they didn’t want to deviate from what they knew. But fast forward a few years that have included the likes of John Harbaugh and Doug Pederson showing success by embracing analytics, and you’ll witness a sea change in head coaches’ attitudes toward using the numbers.

Teams are passing more, getting more aggressive on fourth down, and player evaluation has evolved. Some teams have invested more into their analytics department than others, but overall, investment has been trending upward.

The last paragraph encompasses the story with analytics in the NFL pretty well. The trend is the friend of numbers geeks, but I’d stop way short of calling the battle between coaches and nerds “over”. Looking at the percentage of “go-for” rate from different distances in the most obvious field position to try (long field-goal range, between the opponents 30 and midfield), there has been a strong upward trend.

I’d connect the rise in the adoption of fourth downs to the Eagles Super Bowl in the 2017 postseason (highlighted by the dashed line on the plot). In 2017, fourth down tries from any distance hadn’t surpassed previous seasonal highs. By 2019, the rate of go-for adoption had hits highs from all distances, with the biggest relative gains for those longer than one yard to go. This shows a higher level of trust in analytics among coaches, since the value of model-based decision-making increases in less intuitive situations, and weighing conversion rates versus win probability gains becomes a more difficult mental task the further you have to gain.

But there is still a majority of the league that’s relatively conservative on fourth downs, according to the numbers from Ben Baldwin’s rbsdm.com. Nine of 32 NFL teams last year went for it on fourth down more than half of the time they gained at least 1% in win probability by doing so, according to his model. The model can’t incorporate every piece of context, but it’s got some conservatism built in with the 1% threshold.

Things have shifted a lot recently, but outside of a handful of teams, I think the battle is far from over, at least within a space we can judge from the outside, like fourth down decisions.

The learning goes both ways. There are data scientists with strong beliefs based on research who were challenged after talking to coaches about the nuance and intricacies of football. It’s difficult to make sweeping statements based on numbers without a deeper understanding of how schemes and the players contextualize them.

“My interactions with coaches are everything because I didn’t really know right from wrong before I entered football,” an NFL analytics staffer told me. “I just was a casual viewer, and I thought I knew a lot then once I got into a coaching staff. I realized I knew nothing, and I had to almost go in with a blank slate — take their opinions, learn from them. But then at the same point, you use those principles and my overall knowledge to come up with my own research.”

There’s obviously no comparison between the football knowledge of coaches and analytics staffers, especially when most of the latter are in their 20s and don’t have a ton of experience even dealing with football analytics, let alone the intricacies of the sport. But I’m not really sure the “sweeping statements” based on analytical research have been disproven as we’ve moved forward.

I’m sure internal research on the specifics of certain types of plays and players has needed major revision with coaching input, but the core insights of football analytics (more early down passing, more fourth down tries, more focus on spending and draft picks at premium positions, etc) have been proven correct in the NFL the same way they were for other sports. The league has moved almost universally in the direction of what an analytical thinker like Brian Burke was writing about in his blog Advanced Football Analytics from 10-15 years ago.

The area “analytics” has gotten most widespread buying is in game preparation, though I’m not sure it’s more than automating and speeding up the previous process.

The analytics team presents coaches with their reports early in the week. These reports will differ from team to team, but commonly, coaches tell their analytics department what they want to see and know about the opposing team.

Closer to the end of the week, the analytics team reviews coaching decisions or situational plays that happened around the league with the coaches and, sometimes, players.

Getting information into the coaches hands quicker and more accurately is highly valuable, but not really analytics in the way that I think of it.

Giants offensive coordinator Mike Kafka and the staff in New York have a great understanding of analytics, and they are leveraging their analytics department to gain edges in multiple ways.

“The (analytics) guys here do an awesome job of, ‘Hey, here’s things that happened around the league. Here are things we can learn from. Here are things that build in the database of situations that we got to talk about,'” Kafka said. “And for me as a play caller, ‘Here are things that you need to be prepared for and take a peek at this game or this clip, and what would you call in this situation.’ And then kind of giving you a multiple choice question and go, ‘Alright, if you chose this, your win percentage increased by two percent. If you did this, it increased by 12. If you did this, increase by 20 percent.'”

I like this much more as part of analytical insights, explicitly referring to win probability and the right/wrong decisions to make. At the same me, I bet Mike Kafka isn’t a typically offensive coordinator in terms of receptivity to this information. This type of pre-game planning is much more valuable, in my opinion, than having an analytical coordinator in the box talking to coaches on game day. The latter helps for unique situations, but more gains can be made from having coaches build lots of reps with these decisions outside of the game, developing an intuition based on analytics. it makes for less time spent going back and forth in the high-pressure and time-sensitive actual games, and more buy-in to incorporate insights on the front end.

More from Kafka:

“You can kind of play the game a little bit and have a little bit more of an edge when you’re kind of playing with the numbers a little bit and understanding like, ‘this decision right here could help us, could give us a better chance to win by X percentage,’ and in some cases, 12 percent might not sound like a lot, but in the grand scheme of it when you’re really looking at the whole season or the number of times that situation comes up, that’s going to help you win,” Kafka said. “It’s going to help you win a game. It’s going to help you win a crucial situation to help put yourself in place.”

This is a good illustration of how quantifying things can give us a better understanding of what is and isn’t significant, and then knowing how better to allocate our time and effort. To me, 12% sounds absolutely massive, but that’s because I’ve probably done more thinking about how much different factors actually influence win probability. A good example is looking at moneyline movement for games when non-quarterbacks get injured and are declared out. Even several injuries for non-quarterbacks will only move win probabilities by less than 10%, and individual players perhaps 1-2% each. Analytical decisions won’t make-or-break the results of a game or a season, but all the factors of NFL success (outside of maybe quarterback play/availability) are a collection of thousands of tiny edges gained. Teams spend massive amounts of time of effort to keep players healthy and ready to play (and the media focuses heavily on it), yet a few sound decisions can have as big of an impact, without nearly the costs.

Nguyen does a good job dispelling the common objections that “analytics” can’t factor in weather or quality of opponent, but then turns to the mysterious momentum.

One factor that analytics can’t quantify is momentum, so it doesn’t get baked into these models. Momentum is a controversial subject — some analytics experts don’t believe it exists, but others won’t ignore its existence because of the testimonials of players and coaches. The mental toll of coming away from a possession without points is another human factor that a coach has to consider. Coaches can still have these human factors in mind when making a decision, but the model helps to reduce the many factors that a coach has to weigh in real time.

This is the thing about stuff like momentum: its existence doesn’t matter if you can’t determine the correct action to enhance its positive effects, or mitigate its negative effects. I think the only fair way to look at something like momentum is to say that we can’t rule out its existence, but we also don’t have any solution for how it should affect our decision-making. There’s undoubtedly a feeling of disappointment when you fail to score. But that feelings, or the “mental toll” in this example, doesn’t come with a recommendation for the best action, other than doing what you know will maximize your win probability.

No one - to my knowledge - has be able to measure momentums positive/negative effects after turnovers, big plays, or fourth down conversions/failures. Team’s perform, on average, as well as you’d expect going forward after the “momentum-shifting” plays. In this circumstance, I put the “mental toll” as regret, and I’ve discussed before that the way to lower regret aversion is to understand the decision and accept the variance that naturally comes with its risks and rewards.

“GM hires might be the next wave with (Vikings GM) Kwesi (Adofo-Mensah) and with (Browns GM) Andrew Berry,” an NFL analytics manager said. “I think with (Eagles GM) Howie’s (Roseman) success, there’s going to be teams that look at, ‘Well, maybe we just need a guy that aggregates the information rather than an old-school traditional scout’ — like everyone for years just saw your GM needs to be … our best scout. Well, now it’s like, maybe he needs to be smart at the (salary) cap and be creative with knowing how to foster value and how to allocate resources to the premium positions.”

I think this is right, in that the change will come from others imitating success (like with the Eagles and fourth downs after 2017), more than understanding and buying into analytics. You would have that that Howie Roseman alone would have shown many that the super-scout model as GM doesn’t have to be the case, but I don’t think he even got proper credit until the last 12-18 months.

Change will come, but the battle isn’t over between nerds and coaches. If it was, that' would mean nerds don’t have much value to add versus consensus, and we’re far from that right now.

STRAINING TO MEASURE PASS RUSH EFFECT

A new academic study caught my eye this week, from friend of the Substack Ron Yurko and his Carnegie Mellon colleague Quang Nguyen and Loyola University professor Greg Matthews. They developed a new pass rushing metric called STRAIN. This is how they describe it in the paper’s abstract: