Back in February, I posted an article to the NY Times pro football blog, The Fifth Down, about the accuracy of fantasy football experts when it comes to preseason overall rankings. Since then, I have tweaked my methodology for looking at the expert rankings and added a site (Rotoworld) to the analysis. And it's a good thing I added Rotoworld, too, since they led the way on almost every angle of this analysis. But before I get too deep into the results, I just want to put a big warning that these results have flaws; it is impossible (as far as I can tell!) to do a perfect analysis of this sort of thing since every site has a different methodology and likely different scoring systems that they're working off of. And perhaps more importantly, this is merely one year of analysis that I'm looking at; anyone who works with stats can tell you that one year of results doesn't hold very much weight...which is why I plan on continuing this (and hopefully adjusting my ranking process until I'm more comfortable with it) for years to come. Additionally, please keep in mind that I assume you have a brain, that you aren't going to just look at this list and never ever look at SI again; that would be ill-informed of you. This is just meant to be a supplementary tool as you start looking at various experts out there.
Also, there are some sites (like fantasyguru.com) that I only had positional rankings for, not overall top 100 rankings. This makes a difference only because I am attempting to compare apples to apples as much as possible. Perhaps this coming year I'll switch my analysis around by looking at these positional rankings the sites churn out rather than the overall Top 100 types of rankings (and pulling out the positional piece from the overall). See below for more on the positional issues that I've tried to address.
So I welcome your comments and if anyone has a better way of looking at this (keeping in mind my limitations of using just Excel and of course having a full-time job), by all means, please let me know.
So without further ado, here are the results:
1. Expert rankings from an overall perspective (see this article for an explanation on how I did these calculations):
1. Rotoworld (6005)
2. Ask The Commish (6,031)
3. FootballGuys.com (6,142)
4. FF Cheatsheets (6,157)
5. NFL.com (6,167)
6. FFToolbox (6,211)
7. CBS Sports (6,221)
8. Yahoo! Sports (6,253)
9. Fantasy Football Xtreme (6,301)
10. Fox Sports (6,345)
11. Pigskin Addiction (6,370)
12. SI (6,419)
13. ESPN (6,447)
14. ProFantasySports (6,557)
A few months ago many of you felt that looking at sites from an overall ranking and not a positional ranking was flawed (see the comments section of that article linked above for details). And I agree that positional analysis needs to be taken into account, but I am also very interested in how well a site can predict overall rankings as well. That still has value to me and in my book, it adds to what makes an expert an expert. But as requested, I took a stab at looking at the rankings within positions from the info in Top 100 rankings, so that I wasn't just looking at these from an overall rankings perspective. So this next list is an attempt to address some of the issues related to how overall rankings are created and taking positions into account a little more - but again, it's likely these results are flawed to some extent since I think that's a really tall order. But hey, read on anyway...
2. Expert rankings from a positional perspective (the number in parentheses is the total number of spots off a site's preseason ranking was from the final season outcome):
QB rankings
1. FootballGuys (192)
2. FFToolbox (196) and NFL.com (196)
4. Ask The Commish (197)
5. Rotoworld (200)
6. FF Cheatsheets (203)
7. Pigskin Addiction (211)
8. CBS Sportsline (213)
9. FoxSports/Rotter (215)
10. Yahoo! (221)
11. SI (225)
12. ESPN (229)
13. ProFantasySports (246)
14. FFXtreme (261)
RB rankings
1. Rotoworld (984)
2. FoxSports/Rotter (1000)
3. FF Cheatsheets (1007)
4. FFToolbox (1010)
5. Yahoo! (1013)
6. Pigskin Addiction (1020)
7. Football Guys (1021)
8. ESPN (1023) and NFL.com (1023)
10. ProFantasySports (1032)
11. SI (1037)
12 Ask The Commish (1039)
13. FFXtreme (1045)
14. CBS Sportsline (1060)
WR rankings
1. Rotoworld (1075)
2. FFXtreme (1078)
3. FF Cheatsheets (1082)
4. CBS Sportsline (1122)
5. Football Guys (1124)
6. FFToolbox (1127)
7. Pigskin Addiction (1150)
8. ESPN (1160) and Yahoo! (1160)
10. FoxSports/Rotter (1170)
11. ProFantasySports (1182)
12. Ask the Commish (1201)
13. SI (1237)
14. NFL.com (1245)
TE rankings
1. Football Guys (94)
2. NFL.com (95)
3. Rotoworld (96) and Yahoo! (96)
5. Ask the Commish (98)
6. FFXtreme (99) and FoxSports/Rotter (99)
8. Pigskin Addiction (100)
9. CBS Sportsline (104) and FF Cheatsheets (104) and FFToolbox (104)
12. ProFantasySports (105)
13. ESPN (107) and SI (107)
Now if you take those numbers and just do a quick averaging of the positional rankings (so for Football Guys, I'd average together their rankings of and 1 (QB), 7 (RB), 5 (WR), and 1 (TE) for a positional average of 3.5), you get the following:
1. Rotoworld (avg 2.5)
2. Football Guys (avg 3.5)
3. FF Cheatsheets (avg 5.25) tied with FF Toolbox (avg 5.25)
5. NFL.com (avg 6.5) tied with Yahoo! (avg 6.5)
7. FoxSports/Rotter (avg 6.75)
8. Pigskin Addiction (avg 7)
9. Ask the Commish (avg 8.25)
10. CBS Sportsline (avg 8.75) tied with FFXtreme (avg 8.75)
12. ESPN (avg 10.25)
13. ProFantasySports (avg 11.5)
14. SI (avg 12)
One final note, after a lot of reflection I did not include fan favorite Football Outsiders in this analysis since they have such a unique way of looking at rankings (which I love, by the way) - additionally, their rankings can be adjusted based on the scoring system you use, which is awesome, but it therefore means they don't really have one overall top 100 ranking; they have many, many overall rankings you can choose from. So it's unclear how they did in comparison; I can certainly pull out some of the players that FO did a good or bad job predicting though, so if that's of interest, let me know.
If you want all the gory details, see this Google spreadsheet. If you have trouble with that spreadsheet though and just want me to e-mail you the excel version, I can do that, too. Just contact me.
Enjoy!
Also, there are some sites (like fantasyguru.com) that I only had positional rankings for, not overall top 100 rankings. This makes a difference only because I am attempting to compare apples to apples as much as possible. Perhaps this coming year I'll switch my analysis around by looking at these positional rankings the sites churn out rather than the overall Top 100 types of rankings (and pulling out the positional piece from the overall). See below for more on the positional issues that I've tried to address.
So I welcome your comments and if anyone has a better way of looking at this (keeping in mind my limitations of using just Excel and of course having a full-time job), by all means, please let me know.
So without further ado, here are the results:
1. Expert rankings from an overall perspective (see this article for an explanation on how I did these calculations):
1. Rotoworld (6005)
2. Ask The Commish (6,031)
3. FootballGuys.com (6,142)
4. FF Cheatsheets (6,157)
5. NFL.com (6,167)
6. FFToolbox (6,211)
7. CBS Sports (6,221)
8. Yahoo! Sports (6,253)
9. Fantasy Football Xtreme (6,301)
10. Fox Sports (6,345)
11. Pigskin Addiction (6,370)
12. SI (6,419)
13. ESPN (6,447)
14. ProFantasySports (6,557)
A few months ago many of you felt that looking at sites from an overall ranking and not a positional ranking was flawed (see the comments section of that article linked above for details). And I agree that positional analysis needs to be taken into account, but I am also very interested in how well a site can predict overall rankings as well. That still has value to me and in my book, it adds to what makes an expert an expert. But as requested, I took a stab at looking at the rankings within positions from the info in Top 100 rankings, so that I wasn't just looking at these from an overall rankings perspective. So this next list is an attempt to address some of the issues related to how overall rankings are created and taking positions into account a little more - but again, it's likely these results are flawed to some extent since I think that's a really tall order. But hey, read on anyway...
2. Expert rankings from a positional perspective (the number in parentheses is the total number of spots off a site's preseason ranking was from the final season outcome):
QB rankings
1. FootballGuys (192)
2. FFToolbox (196) and NFL.com (196)
4. Ask The Commish (197)
5. Rotoworld (200)
6. FF Cheatsheets (203)
7. Pigskin Addiction (211)
8. CBS Sportsline (213)
9. FoxSports/Rotter (215)
10. Yahoo! (221)
11. SI (225)
12. ESPN (229)
13. ProFantasySports (246)
14. FFXtreme (261)
RB rankings
1. Rotoworld (984)
2. FoxSports/Rotter (1000)
3. FF Cheatsheets (1007)
4. FFToolbox (1010)
5. Yahoo! (1013)
6. Pigskin Addiction (1020)
7. Football Guys (1021)
8. ESPN (1023) and NFL.com (1023)
10. ProFantasySports (1032)
11. SI (1037)
12 Ask The Commish (1039)
13. FFXtreme (1045)
14. CBS Sportsline (1060)
WR rankings
1. Rotoworld (1075)
2. FFXtreme (1078)
3. FF Cheatsheets (1082)
4. CBS Sportsline (1122)
5. Football Guys (1124)
6. FFToolbox (1127)
7. Pigskin Addiction (1150)
8. ESPN (1160) and Yahoo! (1160)
10. FoxSports/Rotter (1170)
11. ProFantasySports (1182)
12. Ask the Commish (1201)
13. SI (1237)
14. NFL.com (1245)
TE rankings
1. Football Guys (94)
2. NFL.com (95)
3. Rotoworld (96) and Yahoo! (96)
5. Ask the Commish (98)
6. FFXtreme (99) and FoxSports/Rotter (99)
8. Pigskin Addiction (100)
9. CBS Sportsline (104) and FF Cheatsheets (104) and FFToolbox (104)
12. ProFantasySports (105)
13. ESPN (107) and SI (107)
Now if you take those numbers and just do a quick averaging of the positional rankings (so for Football Guys, I'd average together their rankings of and 1 (QB), 7 (RB), 5 (WR), and 1 (TE) for a positional average of 3.5), you get the following:
1. Rotoworld (avg 2.5)
2. Football Guys (avg 3.5)
3. FF Cheatsheets (avg 5.25) tied with FF Toolbox (avg 5.25)
5. NFL.com (avg 6.5) tied with Yahoo! (avg 6.5)
7. FoxSports/Rotter (avg 6.75)
8. Pigskin Addiction (avg 7)
9. Ask the Commish (avg 8.25)
10. CBS Sportsline (avg 8.75) tied with FFXtreme (avg 8.75)
12. ESPN (avg 10.25)
13. ProFantasySports (avg 11.5)
14. SI (avg 12)
One final note, after a lot of reflection I did not include fan favorite Football Outsiders in this analysis since they have such a unique way of looking at rankings (which I love, by the way) - additionally, their rankings can be adjusted based on the scoring system you use, which is awesome, but it therefore means they don't really have one overall top 100 ranking; they have many, many overall rankings you can choose from. So it's unclear how they did in comparison; I can certainly pull out some of the players that FO did a good or bad job predicting though, so if that's of interest, let me know.
If you want all the gory details, see this Google spreadsheet. If you have trouble with that spreadsheet though and just want me to e-mail you the excel version, I can do that, too. Just contact me.
Enjoy!
Comments
And anonymous, I will get in touch with fantasyfootball.com for their 2007 rankings - I'm open to adding any other sites as long as I can get my hands on their rankings. Problem is, a lot of them aren't freely available and now many of the 2007 rankings are hard to track down. But thanks for the suggestion - I'll see what I can do.
Anyone know what's up with fantasytruth.net? That was a good resource to use throughout the season - hope it returns.
I know that not all sites actually provide estimated points per player, but the rankings list don't accurately reflect which errors in rankings actually mattered. Being off a lot in the later rounds is not as big of an impact as being off a lot in the early rounds due to the relative magnitude of the contribution later round players contribute.
I think it'd be another interesting way to measure the error for sites that divulge enough information to make it possible.
with a BBA in statistical analysis and an MBA in Finance, i will leave you with 3 words Random Walk theory.
The random walk hypothesis is a financial theory stating that stock market prices evolve according to a random walk and thus the prices of the stock market cannot be predicted.
The random walk hypothesis was also applied to NBA basketball. Psychologists made a detailed study of every shot the Philadelphia 76ers made over one and one-half seasons of basketball. The psychologists found no positive correlation between the previous shots and the outcomes of the shots afterwards. Economists and believers in the random walk hypothesis apply this to the stock market. The actual lack of correlation of past and present can be easily seen. If a stock goes up one day, no stock market participant can accurately predict that it will rise again the next. Just as a basketball player with the “hot hand” can miss his or her next shot, the stock that seems to be on the rise can fall at any time, making it completely random.
Same for football, just my humble opinion
PS ask BIG BROWN
Ethan - I'd rather not touch the injuries, to be honest. While Harrison's injury was unexpected, there are plenty of injury-prone players and I don't want to be in the position of judging which players' injuries could have been anticipated. For those who are injury-prone, I think you get what you pay for if that player gets injured and you ranked them highly. And for those players that aren't injury-prone (like Harrison), most sites are punished equally for that kind of situation - more or less.
Dan R - I had considered squaring rather than abs valuing the results for the exact reasons you pointed out - and will try that out to see what kind of a difference that makes. Thanks!
Highlander - you make some good points but I don't think there's any harm in looking at rankings. To base your entire draft on a site's rankings without interjecting your own gut or external knowledge would be less than ideal; but the rankings are there as a supplement, not a drafting bible. Though I do think most FFers realize that; am I being naive to make that assumption?
And anonymous, you're right, this was an off year for Football Outsiders and their KUBIAK rankings; which is exactly why these results really need to be taken fairly lightly - it's just one year of results.
Thanks for the input, keep it coming.
Also, I don't see rotowire- that would really be useful, because I paid for them, and they gave me great stuff during the year. I thought it was worth the investment- I think I did a lot better because of their info.
-Doug
http://www.footballburrito.com/