Week 6 Adjusted Quarterback Efficiency (AQE)

Normalizing quarterback efficiency for luck-based elements, now including YAC over expectation

Starting this week I’m gong back to labeling this analysis “Adjusted” rather than “Luck-Based” Quarterback Efficiency due to one element I added to the formula: yards after the catch over expectation (YACOE).

I still don’t have the tracking-data-based receiver “open” scores that ESPN produced for the last two seasons, but I think incorporating YACOE is necessary to get a fuller picture of quarterback play versus results. Plus, it’s incorporates a combination of coaching scheme advantages (or disadvantages), difficulty of throws (normally longer air-yard throws are harder) and a big element of receiver skill (producing YAC via avoided tackles). Research has also shown that YACOE looks extremely noisy on the receiver level, meaning there is a large degree of luck in how much is generated on individual plays. I’m not fully incorporating YACOE in the formula, with the idea that some of the benefit/cost of good YAC production is a quarterback skill that shouldn’t be stripped from the passer’s value.

All that being said, I think the results here will serve as another reminder of the value of the exercise. Making explicit adjustments to quarterback play, using an open and clear methodology, allows not only an understanding of the process behind the numbers, but empowers the individual reader to think about how they might rather alter the process, and be able to form a better approximation of how that would affect the concrete results.

A perfect illustration of how my process allows better understanding, interpretation and discussion (in my humble opinion) is the controversy over the various media rankings of Brock Purdy, specifically that of The Ringer. Steven Ruiz has Purdy ranked 22nd overall, below many quarterbacks who have struggled by efficiency metrics this season, like Ryan Tanehill (16th), Derek Carr (18th), and Daniel Jones (20th).

It’s certainly possible that Purdy is the 22nd best quarterback in the NFL (we should probably be saying 21st since Ruiz includes Tom Brady in the rankings as some sort of joke), but when he’s lapping the field in quarterback efficiency metrics, it becomes more difficult to make that case. Ruiz wrote up an explainer on Purdy’s ranking in an attempt to persuade:

The most straightforward explanation is that these rankings aren’t based solely on production, and maybe you see that as a flaw in the methodology, but there are plenty of sortable stat pages on the internet if that’s how you want to stack quarterbacks. Instead, the aim here is to rank NFL passers based on their skill sets, using uniform grading criteria for each quarterback. That means you won’t see wild swings after a good or bad game—that’s a feature, not a bug. Changes are meant to be more gradual because we don’t see quarterbacks improve or regress that quickly. 

Couple interesting points here. First, everyone knows that it isn’t solely based on production, so that’s a weak strawman being knocked down. The problem most have with the rankings isn’t that Purdy should exactly match his efficiency ranking of No. 1, but that he’s so far away from his production. If Purdy was even ranked in the teens I don’t think there would have been nearly the same degree of pushback.

Second, I agree with the idea that you should move rankings too quickly, a philosophy I try to concretely incorporate into my Bayesian quarterback rankings, that move in conjunction with sample size. The problem with controlling ranking moves by intuition instead of a rigorous method is opening the process up to the anchoring effect and confirmation bias. I believe my rankings are correct to move much more quickly in response to Purdy’s performance because of a lack of these biases.

His accuracy is merely good, not great. His timing has improved this season, but he still has room for further development. His decision-making is still mediocre, but playing in Shanahan’s system helps mitigate that issue. That’s what I see on film, and there’s statistical evidence to back it up. His turnover-worthy throw rate—as charted by Pro Football Focus—is identical to those of Jordan Love and Sam Howell. Those two have thrown 12 interceptions combined; Purdy has yet to be picked off. His accuracy rate ranks a mediocre 16th, per PFF, where he finds himself as the meat in a Baker Mayfield–Justin Fields sandwich. The perception that Purdy is uniquely accurate and making great decisions isn’t backed by the metrics designed to track those things specifically, and it also doesn’t match what I see when I watch him play.

Couple things in this section of the explainer to address. I agree with the notion that Purdy has been lucky with bad throws not turning into as many interceptions as others. But I also put that factor explicitly into the numbers below. Knowing how much to weigh that luck is just as important as knowing it exists. I can’t even find the PFF 16th accuracy rate for Purdy Ruiz states here. I’m sure it exists somewhere, but there are also contradictory indicators like the tracking-data-based NFL’s Next Gen Stats measure of completion percentage over expected (CPOE) that has Purdy ranked 8th (+4.1%).

Models are worse than the eye-test at judging individual throws, but they have incorporate and systematically weigh all the throw much better than a film-based analysis making rankings for every NFL quarterback. Ruiz gets another strawman into this section saying that there’s a perception Purdy is “uniquely accurate.” Maybe this is a misunderstanding of the definition of the word unique, but I don’t think anyone sees Purdy’s accuracy as “being the only one of its kind; unlike anything else.”

Going into the Dallas game, Purdy ranked 22nd in passing grade, presumably for many of the same reasons I had him at 25th before this week’s update. While our methodologies are not at all similar—PFF’s grading system is more scientific—the overarching goal is the same: to evaluate a player’s performance independent of factors like play-calling, scheme, and surrounding talent

This is probably my least favorite section in terms of using stats to fit your opinion rather than the other way around. Ruiz uses the rankings for PFF grading and his own on Purdy going into last week, when both are available updated for Purdy’s best game of the season. The gap between them was more narrow a week ago than it is now, which is why I presume that choice was made. Purdy’s passing grade through Week 5 at PFF ranks 12th (minimum of 80 dropbacks) versus Ruiz’s 22nd ranking. I also think the choice of using passing grade rather than offensive grade isn’t totally intellectually honest, as I assume things like Lamar Jackson getting a 120 score for “Creativity” in Ruiz’s 1-to-100 scale has something to do with his non-passing abilities. Purdy’s PFF offensive grade ranking is materially better through Week 5 at 8th.

Maybe it was an oversight, but the 22nd PFF passing grade ranking for Purdy going into Week 5 also shouldn’t have been used, as it includes every passer without a minimum dropback threshold. Purdy would have ranked 20th with any reasonable threshold, which would exclude higher graded quarterbacks Mike White (two dropbacks) and Bailey Zappe (10 dropbacks).

Using the Ringer’s process with Purdy’s rankings was really a longwinded way of me saying that there’s little value of most media rankings other than the specific ranker’s opinion, and the appearance of having a strong process melts away with any degree of scrutiny. What’s I’m doing here is giving you explicit data about how you adjust for quarterbacks who have been better/worse than their baseline efficiencies. Just proving that Purdy have been worse than his first ranked efficiency doesn’t justify any lower ranking. You’ll see below I do drop Purdy in per-play ranking (slightly) and even more in terms of total EPA generated.

Check out the first the first installment of the Luck-Adjusted Efficiency numbers (free post) to get more details on the methodology below. I also have lots of information in past posts on Adjusted Quarterback Efficiency (AQE):

• AQE introduction

• Update to 2.0

• Updated to 3.0

2023 LUCK-ADJUSTED RESULTS

The plot below shows each quarterback who has dropped back to pass 80 times this season. There are two points for each quarterback: 1) The team-colored dot for the actual EPA per play the quarterback has this season and 2) The quarterback headshot representing the adjusted quarterback efficiency (EPA/play). There is also a team-colored line linking the two on each row.