This article comes to us from Danny Lanier, Jr. Enjoy a great read. If you want to check in with Danny on the post, e-mail him at dlanier698@gmail.com.
Abstract: Motivated by a recent debate on NFL Network’s “NFL GameDay Morning” program, this study applies scientific method to determine which position on the offensive line is most vital to a quarterback’s success. While conventional wisdom suggests that the left tackle is the obvious answer, some argue that other positions (e.g., center) are increasing in importance as defensives attempt to exploit weaknesses across the offensive line. Using player grade data obtained from ProFootballFocus.com (PFF), I use a regression model to determine the extent to which each OL position’s performance incrementally impacts the performance of the QB, and then test which position’s performance has the strongest impact. Results find that for one NFL QB, Matt Ryan, the LT and RT are the only two positions that contribute incrementally to the QB’s success, but that neither position is more important than the other. This study contributes by showing how rigorous data analysis can be used to provide evidence (or counterevidence) of claims made by football experts. It also provides a template for using PFF data to understand how certain players’ performances can impact the performance of another player.
Earlier this season, NFL Network's "NFL GameDay Morning" aired a segment discussing whether the left tackle (LT) was still the most important position on the offensive line (you can watch the clip here http://www.nfl.com/videos/nfl-videos/0ap2000000099838/Is-left-tackle-still-the-most-important-O-line-position). According to the narrator, conventional wisdom suggests that the LT is most important because he protects the QB's blind side against (in most cases) the opposing team's best pass rushing defensive end. As such, teams usually invest high draft picks and big money into the position. However, the segment goes on to argue, as defenses attempt to exploit mismatches - e.g., lining the best pass rusher against the right tackle (RT) or sending a DE up the middle against a guard - the other offensive line positions are gaining in importance. In fact, Ravens DE Terrell Suggs is interviewed in the segment and argues that the center position (C) has become most vital to a QB's success because it is imperative that the center be able to make the proper calls at the line of scrimmage. Despite these competing arguments, the segment does not provide much evidence to settle the debate. However, as someone who enjoys applying scientific methods to address empirical questions, this argument got me to thinking that such a question was actually testable using player performance data from ProFootballFocus.com (PFF).
Each week of the regular season and post season, PFF rates the performances of all offensive, defensive and special teams players using a scoring system that assigns grades ranging from -2 (worst) to +2 (best) for each play [If you are not familiar with PFF's grading system, I encourage you to read more about it here http://www.profootballfocus.com/about/grading/]. This data presents a unique opportunity to empirically test 1) the extent to which the performances of each of the five OL positions are associated with the performance of the QB; and 2) whether the performance of a single OL position is significantly greater than the other four. The latter test will provide some insight as to whether, consistent with conventional wisdom, the LT is indeed the most important position to the QB's success. If the LT is no longer the most important, then which one is?
Approach and Sample Selection
To conduct my analyses, I use PFF data pertaining to the Atlanta Falcons over the period beginning Week 1 of the 2008 season through Week 17 of the 2012 season.[i] This period marks the beginning of PFF's player grade data and also coincides with the beginning of Matt Ryan's tenure as the Falcons starting QB. Besides being the QB of the team I follow most intently, Matt Ryan presents an interesting case study for a couple of reasons. First, after selecting Ryan with the third overall pick in the 2008 draft, the Falcons traded back into the first round to pick LT Sam Baker, who has seen his share of struggles protecting Ryan's blind side. Second, several commentators, such as ESPN's Ron Jaworski and Tedy Bruschi, have noted that Ryan tends to struggle with pressure in his face, which would suggest that the performances of the interior offensive lineman are more crucial to Ryan's success. Thus, focusing on Ryan and the Falcons OL is a reasonable empirical setting for this investigation.
The Falcons have played 83 games (including playoffs) during the sample period, of which Ryan has started 81 of them. I eliminate one of these 81 starts from the sample - Week 12 of the 2009 season - because Ryan played only nine snaps before suffering before suffering a turf toe injury. Chris Redman would go on to finish that game and start the next two versus Philadelphia and New Orleans. Thus, my final sample consists of 80 games in which Ryan played the majority (if not all) of the snaps.
Variable Measurement
For each game in the sample, I collect PFF player grades for Matt Ryan and each the five OL positions (LT, LG, C, RG, RT). PFF grades QBs on a number of attributes (e.g., passing and rushing) which culminate into an overall performance rating. However, for the purposes of this analysis, I focus on Ryan's passing performance (PASS) only.[ii] PFF also grades each offensive line position on a number of attributes including run blocking, pass blocking and blocking on screen plays. Given that the primary focus of this study is to examine the impact of the OL's performance on the QB's success, I focus specifically on each OL position's pass block rating (PBLK) . Measurement of PBLK for each O-lineman is complicated in instances where the player who started the game did not finish (either due to injury or coach's decision). Since my analysis focuses on the OL positions and not specific OL players, I aggregate the PBLK grades for each player who played at a particular position for each game. For example, if Sam Baker started at LT and received a pass blocking grade of 1.0 for his snaps and was relieved by Will Svitek, who received a grade of 0.3, I code the PBLK grade for the LT position as 1.3 (1.0 + 0.3) for the game.[iii]
Average PFF Grades for Matt Ryan and Five OL Positions by Season
The table below presents the average PFF grades for Matt Ryan (PASS) and the five OL positions (PBLK) over the five year sample period. The third column of Table 1 shows that after a sophomore slump in 2009 (average rating of -0.08), Ryan's PFF rating improved to 1.67 in 2010 and 1.69 in 2011. Moreover, the 2012 is Ryan's finest season yet, with an average rating of 2.34 to date.
Table 1 - Descriptive Statistics by Year
Season
Games
Average PFF Passer Rating for QB Matt Ryan
Mean PFF Pass Blocking Grades by OL Position
LT
LG
C
RG
RT
2008
17
1.07
-0.42
-0.19
0.45
0.43
0.60
2009
13
-0.08
-0.04
-0.03
0.27
0.43
0.74
2010
17
1.67
-0.11
0.10
-0.45
0.45
0.76
2011
17
1.69
-0.60
0.55
0.40
-0.39
1.08
2012
16
2.34
0.43
0.08
-0.06
-0.62
0.73
Overall
80
1.40
-0.16
0.11
0.12
0.05
0.78
*Matt Ryan missed two full games and the majority of one other during the 2009 season due to injury.
Interestingly, after four years of sub-par pass-blocking grade averages, the LT position (usually manned by the embattled Sam Baker) also saw sharp improvement in 2012. This appears to justify the Falcons' decision to stick with Baker this season. The OL position with the best pass blocking performance over the five year period has been the RT, where Tyson Clabo has been a fixture during Ryan's tenure.
Assessing How QB Matt Ryan's PFF Rating Varies with the Rating of Each OL Position
While not conclusive, the results from Table 1 above suggest a pattern between Ryan's steady improvement over time and that of the RT position, while a spike in the LT's performance from 2011 to 2012 corresponds with Ryan's this season. However, a better approach is to evaluate how Ryan's performance varies with the performance of each OL position. Specifically, I rank each OL position's performances from lowest to highest and then evaluate the whether the trend in Ryan's passing performances follows suit. For the sake of brevity, I sort each OL position's pass blocking grades into the bottom 24 (30%) performances, middle 32 (40%) performances and top 24 performances. I then compute Matt Ryan's mean PFF passer rating within each performance group for each OL position. The results appear in Tables 2a and 2b below.
Table 2a - Mean Pass Blocking Grade for Each OL Position by Performance Rank
Performance Rank
Number of Games
LT
LG
C
RG
RT
Bottom 30%
24
-2.33
-1.52
-1.08
-1.60
-1.29
Middle 40%
32
0.14
0.34
0.29
0.28
1.04
Top 30%
24
1.61
1.43
1.08
1.39
2.52
Table 2a shows the distribution of pass blocking performances for each OL position. The third column of Table 2a indicates that the 24 worst-graded performances by the LT position received and average score of -2.33, while the top-graded performance by this position received an average score of 1.61. Also of note, the RT position's 24 top-graded performances earned an average grade of 2.52, substantially higher than the top performances of any other position.
Table 2b - Mean PFF Passer Rating for QB Matt Ryan by OL Performance Rank
Performance Rank
Number of Games
LT
LG
C
RG
RT
Bottom 30%
24
0.10
1.90
1.90
0.99
0.79
Middle 40%
32
1.90
0.58
0.97
1.44
0.82
Top 30%
24
2.01
1.98
1.46
1.73
2.77
Table 2b provides a clearer picture of how Matt Ryan's average passer rating varies with the performances of each OL position. For instance, in the games corresponding to the LT's lowest 30% performances, Ryan's average PFF passer rating was 0.10, while improving to 1.90 and 2.01 for the games representing the LT's middle 32 performances and top 24 performances, respectively. A similar pattern is revealed when examining the RT position. As the RT improves from the its bottom 24 performances to its top 24 performances, Ryan's mean passer rating increases from 0.79 to 2.77. Unlike to monotonic pattern of Ryan's performance with respect to both tackle positions, U-shaped patterns emerge when evaluating Ryan's performance in conjunction with both the LG and C positions. For instance, Ryan's average passer rating is 1.90 for the games corresponding to the bottom 24 performances by the LG, and then decreases to 0.58 in the middle 32 performances, followed by an increase to 1.98 for the top 24 performances. A similar pattern is reflected for the C position. The patterns exhibited with respect to the LG and C positions do not suggest a strong correlation between the performances of these positions and Ryan's performance - that is, Ryan appears to perform about as well when these positions have struggled with pass blocking as he does when they have excelled. Finally, Ryan's passing performances exhibit an increasing trend with respect to the RG's performances; however, this trend is far less dramatic when compared to the patterns associated with the LT and RT.
Using Regression Analysis to Estimate Each OL Position's Incremental Contribution to Ryan's Success
A limitation of the results from Table 2 is that they do not provide information about the incremental impact of each OL position's performance on Ryan's performance. That is, the dramatic improvement in Ryan's performance that corresponds with the movement from the bottom 30 percent to the upper 30 percent of the LT's performances does not take into consideration how the other OL positions performed in those games. Thus, it remains unclear how much of an impact the LT position's performance alone contributes to Matt Ryan's success. Evaluating this requires the use of a linear regression model, which estimates Ryan's PFF passer rating as a function of the performances of each OL position. This will allow us to evaluate the incremental contribution each OL position has on Ryan's success.[iv]
Using the sample of 80 game observations, I estimate the following regression model to test the association between each OL position's performance and the passing performance of QB Matt Ryan:
PASS = b0 + b1(LT_PBLK) + b2(LG_PBLK) + b3(C_PBLK) + b4(RG_PBLK) + b5(RT_PBLK) + e
where PASS is Matt Ryan's PFF pass rating as defined above; LT_PBLK is the pass block rating decile for the LT position, LG_PBLK is the pass block rating decile for the LG position, and so on.[v] The model coefficients - b1, b2 ...b5 - measure the incremental effect (if any) each OL position's performance has on Ryan's passing performance. If Ryan's passing performance is positively associated with an individual lineman's performance then the coefficient corresponding to that lineman's performance will be positive. The results are presented in Table 3 below:
Table 3: Regression results
Variable
Coefficient Estimate
t-statistic
Intercept
-0.24
-0.25
LT_PBLK
1.79*
1.77
LG_PBLK
-0.99
-1.02
C_PBLK
-0.65
-0.65
RG_PBLK
1.03
1.10
RT_PBLK
2.11**
2.01
Adjusted R2: 0.09[vi]
** and * denote statistical significance at the 0.05 and 0.10 levels, respectively.
Table 3 shows that the coefficient on LT_PBLK positive (1.79) and statistically significant at the 10% level (t-statistic of 1.77).[vii] This evidence suggests that the performance of the LT is incrementally associated with Matt Ryan's performance in the passing game. Interpretation of the coefficient suggests that, after holding constant the performance of the other 4 OL positions, the difference between the LT's worst performance to his best performance is corresponds to a 1.79 point improvement in Ryan's PFF passer rating. The opposite is also true - i.e., a transition from the LT's best performance to worst performance would correspond with a 1.79 point decrease in Ryan's PFF rating, all else equal.
The coefficient on RT_PBLK is also positive (2.11) and statistically significant at the 5% level, suggesting that the RT's performance is also incrementally important to Ryan's success. The coefficient on C_PBLK is not statistically significant, meaning that the C position's performance provides no incremental contribution to Ryan's passing success, after controlling for the other four positions. This contradicts claims from the "NFL GameDay Morning" segment that suggested the C has become most vital to the QB's success. Table 3 also finds that the coefficients on LG_PBLK and RG_PBLK are not statistically significant. I would caution, however, that one should not interpret this finding as suggesting that performances of these positions are unimportant; rather, they simply do not provide any incremental value to Ryan's performance after controlling for the performances of the other OL positions.
The overall takeaway is that, consistent with conventional wisdom, the LT position is incrementally important as it pertains to the QB's success. However, considering that the coefficient on RT_PBLK is slightly larger than the 1.79 coefficient on LT_PBLK, a question still remains as to which OL position is most important for Matt Ryan's success. To settle this, I use an F-test to determine whether the difference between the coefficients on LT_PBLK and RT_PBLK (b1 and b5) is significantly different from zero. If so, then we could conclude that the RT is actually the most important position on the Falcons OL as it relates to Ryan's success. The results from the F-test (not reported), however, suggest that the difference between b1 and b5 is indistinguishable from zero. That is, both the LT and RT positions are (statistically) equally vital to Matt Ryan's success. Nevertheless, both are incrementally more important to QB Matt Ryan's success than the other three OL positions.
Before concluding, I wish to point out several limitations to my study. First, I only use one team (Falcons) in my analysis, focusing specifically on QB Matt Ryan. Thus, these findings cannot be generalized to all NFL teams. Second, the grading scheme used by PFF may not necessarily reflect the grades used internally by the Falcons (or any other team). However, those data are not publicly available. Moreover, PFF is not the only site that provides player grades and it is therefore possible that different results may arise from using those data. However, I only subscribe to PFF and leave further investigation using other player grade data to a future study. Finally, by using a simple least squares regression, I make the implicit assumption of a linear association between the pass blocking performances of the OL positions and the passing performance of the QB, which may not necessarily be the case. Nevertheless, the study's findings shed some light on the dynamics of how one player's performance incrementally contributes to the success of another's. Moreover, it shows how PFF data can be used to address some of the debate issues the experts debate on morning pregame shows.
[i] Ideally, a cross-section data for all NFL teams during this period would provide a richer study if the researcher is interested in generalizing across all teams and all QBs. In this case, I am interested in one particular team (Falcons) and their QB during the sample period (Matt Ryan). As a result, a limitation of my findings is that they cannot be generalized to all other NFL teams.
[ii] I acknowledge that Ryan's (or any QB's) rushing performance can be tied to the performance (good or bad) of the OL, however, I argue that including rushing performance in the measure would add unnecessary noise to the analysis.
[iii] I did not use this approach when measuring PASS for the QB position because Matt Ryan's performance, not the QB position in general, is the central focus of my analysis. Moreover, in the few instances where Ryan was relieved by the backup QB, it was generally at the end of blowout victories.] This differs from my handling of the PASS variable for QB Matt Ryan because his performance is central to the study.
[iv] It should be noted here that there are obviously other factors that would be associated with the QB's performance - e.g., success of the running game, the performance of the receivers, quality of the opponent, coaching, practice reps, etc. - many of which are not observable or quantifiable. However, the development of a model to capture all of these elements is beyond the scope of this inquiry, which is motivated by a question about the relative importance of each OL position.
[v] I rank each OL position's PBLK grade into deciles to mitigate the effects of extreme observations. The deciles are numbered 0 (lowest) through 9 (highest). I scale each decile rank by 9 so that the values range from 0 to 1 to facilitate interpretation of the model parameters. That is, the coefficients (i.e., b1, b2 ...b5) can be interpreted as indicating the impact on Ryan's performance as the OL position moves from the lowest decile to the highest decile.
[vi] The adjusted R2 of 0.09 indicates that only about 9% of the variation in Ryan's passing performance is explained by the model. As indicated in note "iv" above, an attempt to explain more of this variation by adding variables beyond the OL is beyond the scope of this study.
[vii] Using a 2-tailed test, a t-statistic of 1.65 or higher indicates statistical significance at the 10% level. In regression results the t-statistic essentially represents a test of probability of incorrectly concluding that the coefficient (b1, b2, etc.) differs from zero (note: the higher the t-statistic, the lower the probability of incorrectly rejecting the hypothesis that the coefficient is zero). As a general rule, probabilities greater than 10% (or t-statistics smaller than 1.65) are considered to be statistically insignificant.