(This article was first published on Fantasy Football Analytics » R | Fantasy Football Analytics, and kindly contributed to R-bloggers)
In prior posts, I demonstrated how to download, calculate, and compare fantasy football projections from ESPN, CBS, NFL.com, and FantasyPros, which aggregates projections from many different sources to increase prediction accuracy. Last year, I compared fantasy projections from ESPN, CBS, NFL.com, and FantasyPros, including our average and latent projections to determine who had the best fantasy football projections. In this post, I compare fantasy football projections over the last two years to see if the best projections in 2012 are still the best in 2013.
The R Script
The R Script for comparing the projections from different sources is located at:
https://github.com/dadrivr/FantasyFootballAnalyticsR/blob/master/R%20Scripts/Evaluate%20Projections.R
To compare the accuracy of the projections, I use the following metrics:
R-squared (R2) – higher is better
Mean absolute scaled error (MASE) – lower is better
For a discussion of these metrics, see here and here.
Whose Predictions Were the Best?
The results are in the table below. The rows represent the different sources of predictions (e.g., ESPN, CBS) and the columns represent the different measures of accuracy for the last two years and the average across years. The “average” variable represents the mean of projections from ESPN, CBS, NFL.com, and FantasyPros. The “latent” variable represents the common variance among projections from ESPN, CBS, NFL.com, and FantasyPros. The source with the best measure for each metric is in blue.
Source
2012
2013
Average
R-squared
MASE
R-squared
MASE
R-squared
MASE
ESPN
.528
.577
.393
.684
.460
.630
CBS
.619
.501
.388
.676
.504
.589
NFL.com
.510
.642
.419
.595
.465
.618
FantasyPros
.674
.411
.500
.520
.587
.466
Average
.671
.422
.503
.520
.587
.471
Latent
.654
.427
.485
.531
.569
.479
Here is how the projections ranked over the last two years:
FantasyPros
Average
Latent
CBS
NFL.com
ESPN
Here is a scatterplot of the FantasyPros projections in relation to players’ actual fantasy points scored in 2013:
Interesting Observations
Projections that combined multiple sources of projections (FantasyPros, Average, Latent) were more accurate than single projections (CBS, NFL.com, ESPN).
The R-squared of the most accurate projection source was .67 in 2012 and .50 in 2013. This suggests that players are more predictable in some years than others. It also indicates that 1/3 to 1/2 of the variance in actual points is unexplained by projections, so there is much room for improvement in terms of prediction accuracy.
There was little consistency in performance across time among sites that used single projections (CBS, NFL.com, ESPN). In 2012, CBS was the most accurate single source of projection but they were the least accurate in 2013. Moreover, the least accurate in 2012 was NFL.com, but they were the most accurate in 2013. This suggests that no single source reliably outperforms the others. While some sites may do better than others in any given year (because of fairly random variability–i.e., chance), it is unlikely that they will continue to outperform the other sites. This may be similar to the finding that there is little consistency in the performance of mutual fund managers over time. The following charts are from Leonard Mlodinow’s book, “The Drunkard’s Walk: How Randomness Rules Our Lives“. The chart below depicts the performance of the top mutual funds from 1991 to 1995.
The next chart depicts the performance of the same funds in the same order over the subsequent 5 years (1996 to 2000).
In other words, the best funds from 1991-1995 weren’t necessarily the best funds from 1996-2000. That’s likely why a cat beat investors in a stock market challenge. This suggests that mutual fund managers differ in great part because of luck or chance rather than reliable skill. Although our sample size is much smaller with fantasy football projections, there appears to be little consistency in fantasy football sites’ rank-order in accuracy over time.
Conclusion
In conclusion, the most accurate projections were from FantasyPros.com, which combines many sources of projections. FantasyPros had the most accurate projections each of the last two years. No single projection (CBS, NFL.com, ESPN) reliably outperformed the others, suggesting that differences between them are likely due in large part to chance. I plan to calculate projections from more sites (Accuscore, Yahoo, FantasySharks) so we can compare even more projections.
The post Which Site Has the Best Fantasy Football Projections? 2014 Update appeared first on Fantasy Football Analytics.
To leave a comment for the author, please follow the link and comment on his blog: Fantasy Football Analytics » R | Fantasy Football Analytics.
R-bloggers.com offers daily e-mail updates about R news and tutorials on topics such as: visualization (ggplot2, Boxplots, maps, animation), programming (RStudio, Sweave, LaTeX, SQL, Eclipse, git, hadoop, Web Scraping) statistics (regression, PCA, time series, trading) and more...