Luck-Adjusted Quarterback Efficiency
Documenting which quarterbacks have had the best/worst runs of high-variance outcomes
We’re three weeks into the NFL season, with impression on quarterbacks, young and old, forming with increasing certainty. The old #QBWinz argument is a big issue already (How would we view Jordan Love at 1-2 not 2-1 without a easy missed field-goal from the Saints?). But there are a number of other elements of quarterback efficiency - directly affecting team performance - that are more dependent on luck (variance) than skill.
I aggregated many of those luck-based elements, with additional factors like receiver skill and scheme ability to generate yards-after-catch, in my Adjusted Quarterback Efficiency (AQE) metric. The “Open” metric used to measure receiver talent and its affect on quarterbacks isn’t available yet (Hopefully soon: get on it ESPN!), but I can use EPA data from nflscrapR and charting data from FTNData to measure the effects of other elements.
The measures that I believe are most luck-based and part of this analysis:
Interceptions: FTNData tracks “interception-worthy throws”, which I compare to actual inteceptions on a play-by-play basis, and also adjust for expect interception return. Longer INT returns have a dramatic effect on the EPA, whether the quarterback’s fault or not.
Drops: I calculated the expected drop rate for throws, based on location, and compared them to actual drops and determined the EPA lost/gained.
Fumbles: Whether a fumble is recovered by the quarterback’s own team or not can turn a slightly negative play into massive loss. I look at expectations for recovery based on different types of fumbles, and whether the quarterback himself recovers the fumble or a teammate (luckier).
Defensive Pass Interference: There are so many underthrown balls that turn into big DPI gains that need to be recognized as partially luck. By their very nature, DPIs are not “open” receivers with the coverage defender close enough to affect the receiver.
Strength-of-Schedule: This is the one element that is most difficult to judge this early in the season. With three games played, a great offense can have a bigger affect making the defenses they played look “bad”, and vice versa, than might be the truth. But it still matters.
Weather: Based on expected EPA gains/loss versus average in different elements, based on wind, humidity and temperature.
If you want more details on many of the calculations, check out the Adjusted Quarterback Efficiency (AQE) primers from last season.
2023 LUCK-ADJUSTED RESULTS
The plot below shows each quarterback who has dropped back to pass 50 times this season. There are two points for each quarterback: 1) The team-colored dot for the actual EPA per play the quarterback has this season and 2) The quarterback headshot representing the adjusted quarterback efficiency (EPA/play). There is also a team-colored line linking the two on each row.
At the top of the efficiency ranks, Tua Tagovailoa still has a gap between his efficiency and everyone else, even after a large negative adjustment. You’ll see in the table below that details each specific adjustment, Tua has benefited from drop, SOS and weather luck versus the rest of the league. As I mentioned earlier, his SOS might not have been as easy as the model says, because the Dolphins passing offense was so good and drove down opponent numbers. Now, I use models that account for adjustments based on all combinations of scheduled opponents, but I still down-weighted the effect at this point.
Patrick Mahomes benefits by not having a lot of negative adjustments. It makes sense that the quarterbacks at the top of the actual efficiency numbers will generally have more good luck than those on the bottom, but that hasn’t been the case for Mahomes. While his SOS is big negative adjustment (Lions, Jags, Bears), he’s had bad luck with drops and interceptions, notably in Week 1 with Kadarius Toney singlehandedly torpedoing Mahomes’ efficiency.
Brock Purdy and Jordan Love fall into a similar bucket with big negative adjustments. For Purdy in particular, the idea might be that he benefits a ton from Kyle Shanahan’s scheme and his receivers’ ability to generate YAC, but those factors aren’t part of this adjustment. He’s been lucky outside of the surroundings in San Francisco, with the second most negative interception adjustment (four INT-worthy throws and zero INTs), second most negative drop adjustment (only one drop this season), most fumble recovery luck (four but one lost) and also an easier schedule (Steelers, Rams, Giants). Love has been close to as lucky by this measure, with big negative adjustments for INT luck, somewhat for fumble luck, a soft schedule and big gains via defensive pass interference.
The adjustments for Purdy and Love drop their EPA per play from second and third without adjustments to 10th and 7th, respectively, with them taken into account. Both have played well, but not as well as the headline numbers would imply.
Justin Herbert is the other quarterback who loses at least 10 EPA to adjustments, with three INT-worthy throws and zero INTs, suffered a relatively low drop rate (3.2%) and has played in domes or good weather. The adjusts drop his efficiency from fourth to eighth.
The biggest beneficiary from the luck-based adjustments is Matthew Stafford, who jumps all the way from 14th to second in total EPA. Stafford has had the worst luck with interceptions and interception returns (four INTs vs two charted worthy), his receivers have dropped a lot of balls (seven) and tougher schedule (Seahawks, 49ers, Bengals).
The other bigger positive adjustment quarterback moved from being lower in the table to close to the middle: Trevor Lawrence and Daniel Jones. Lawrence jumps seven spots to 20th, and Jones five to 23rd. It isn’t part of the adjustment that Lawrence has come very close to a few touchdowns on the back of the endzone, and the Jags extremely poor performances on higher leverage late downs (worst conversion vs expectation on 3rd and 4th downs). Lawrence has been closer to good than his bad EPA/play, but it might be early to declare elite.
I'm not a huge fan of EPA/P as a QB stat alone, but I think these adjustments, as the other commenter noted are hot fire. You can see bad luck on film and the adjustments numerically work. Interesting dive as usual Kevin!
🔥🔥🔥