A great question was brought up by Aaron in a recent comment about my Average Ranking Positions post. Aaron asked if I had any of the Top 100 lists from this same group of experts from 2006 and if so, how accurate the rankings were - so I've created a new spreadsheet, a 2006 Predicted Rankings vs Actual Fantasy Results spreadsheet. To find all of these 2006 rankings, I'm using the Internet Archive Wayback Machine, a website that has archived much of the Internet. Type in the website address you're interested and voila, a dynamic November 2005 version of that page is at your fingertips (or whatever other date you're looking for). Pretty cool, huh?
So I found ten of the same rankings I used this year for 2006 - of the ones I couldn't find, FF Toolbox had only the top 60 for 2006 so I had to throw it out; NFL.com didn't provide an 2006 overall ranking, just positional; and Fantasy Football Xtreme's 2006 version was not free. Otherwise, a ranking from each of the other sites was acquired and I did the same averaging thing for these rankings as I did for the 2007 rankings. Then I compiled the fantasy point totals for each of these players based on the standard scoring of one of my leagues (players who have retired or are not on active rosters were not included, like Tiki or Corey Dillon). The scoring system makes all the difference here, so the scoring rules I derived my fantasy points from are:
Passing Yards (25 yards per point)
Passing Touchdowns (6)
Interceptions (-2)
Rushing Yards (10 yards per point)
Rushing Touchdowns (6)
Reception Yards (10 yards per point)
Reception Touchdowns (6)
2-Point Conversions (2)
Fumbles Lost (-2)
Offensive Fumble Return TD (6)
Once I had all of the fantasy point totals for the season, I ranked all of the players based on fantasy points scored in the season. I compared this ranking of fantasy points scored to the 2006 pre-season rankings the experts provided. Then I took the absolute value of the difference between the two for each player and summed it up for each ranking. So the ranking that is the most accurate should have the smallest number. See the fuzzy chart below - I know it's hard to read the ranking names along the x axis, but the ranking with the smallest number, or best accuracy, was SI. So in order of greatest accuracy, the list goes:
SI
CBS Sportsline
ESPN
FootballGuys.com
Fox Sports
Pigskin Addiction
Ask The Commish
ProFantasySports.com
Yahoo Sports
FF Cheat Sheets
If you look at the spreadsheet (or above hard-to-read chart) you can see that ultimately they are all really close together - the span is not nearly as great as I thought it might be. Nor do any of them really seem to be all that accurate. This is also just one year's worth of data, so perhaps when time permits I will look into finding 2005 and 2004 data to solidify these findings.
I found it interesting to see which players had the most errant and accurate predictions from the expert rankings. Most errant included Jerry Porter, Chris Perry, Chris Brown, and Koren Robinson. The most accurate rankings were (not surprisingly) for LT, Larry Johnson, Reggie Wayne, and Steven Jackson.
Hope that was enlightening for some of you and I promise to provide posts with more timely fantasy information soon.
So I found ten of the same rankings I used this year for 2006 - of the ones I couldn't find, FF Toolbox had only the top 60 for 2006 so I had to throw it out; NFL.com didn't provide an 2006 overall ranking, just positional; and Fantasy Football Xtreme's 2006 version was not free. Otherwise, a ranking from each of the other sites was acquired and I did the same averaging thing for these rankings as I did for the 2007 rankings. Then I compiled the fantasy point totals for each of these players based on the standard scoring of one of my leagues (players who have retired or are not on active rosters were not included, like Tiki or Corey Dillon). The scoring system makes all the difference here, so the scoring rules I derived my fantasy points from are:
Passing Yards (25 yards per point)
Passing Touchdowns (6)
Interceptions (-2)
Rushing Yards (10 yards per point)
Rushing Touchdowns (6)
Reception Yards (10 yards per point)
Reception Touchdowns (6)
2-Point Conversions (2)
Fumbles Lost (-2)
Offensive Fumble Return TD (6)
Once I had all of the fantasy point totals for the season, I ranked all of the players based on fantasy points scored in the season. I compared this ranking of fantasy points scored to the 2006 pre-season rankings the experts provided. Then I took the absolute value of the difference between the two for each player and summed it up for each ranking. So the ranking that is the most accurate should have the smallest number. See the fuzzy chart below - I know it's hard to read the ranking names along the x axis, but the ranking with the smallest number, or best accuracy, was SI. So in order of greatest accuracy, the list goes:
SI
CBS Sportsline
ESPN
FootballGuys.com
Fox Sports
Pigskin Addiction
Ask The Commish
ProFantasySports.com
Yahoo Sports
FF Cheat Sheets
If you look at the spreadsheet (or above hard-to-read chart) you can see that ultimately they are all really close together - the span is not nearly as great as I thought it might be. Nor do any of them really seem to be all that accurate. This is also just one year's worth of data, so perhaps when time permits I will look into finding 2005 and 2004 data to solidify these findings.
I found it interesting to see which players had the most errant and accurate predictions from the expert rankings. Most errant included Jerry Porter, Chris Perry, Chris Brown, and Koren Robinson. The most accurate rankings were (not surprisingly) for LT, Larry Johnson, Reggie Wayne, and Steven Jackson.
Hope that was enlightening for some of you and I promise to provide posts with more timely fantasy information soon.
Comments
Does anyone have links to any other attempts to judge football projections? (Baseball projections is a big interest of mine.)
s= 1/r_max * Sigma [(r_max + 1 - r_a)/r_max * (1 - |r_a - r_p|/(r_max - 1)]
where r_max is the highest examined ranking (top 10 qbs, top 20 rbs), r_a is the actual rank based on yahoo point total for that player, and r_p is the expert's predicted rank for that player. If the expert didn't include a player ranked player with his top n players for that position, they received a score of zero for that prediction. This score is an ad hoc measurement of agreement score, since I did want to include the weighting.
For running backs, the best experts were (scores are normalized so 1.0 is perfect ranking):
1. ask the commish(.4107), 2. fantasy football cheat sheet(.4043), 3. football guys(.3786), 4. yahoo(.3655), 5. pro fantasy sports(.3720), 6.si(.3595), 7. fox(.3583), 8.cbs(.3506), 9. pigskin addiction (.3400), 10. espn(.3263)
For qbs:
1. si(.5980), 2. ask the commish(.5798), 3. yahoo(.5535), 4. espn(.5535), 5. football guys(.5434), 6. fox(.4989), 7. cbs(.4949),8. pro fantasy sports(.4829), 9. pigskin addiction (.4284), 10. ff cheat sheet(.4102)
Given one sample year, we really don't have any statistical power to say anything, but this is a start. And on a less technical note, let me just point out that none of the experts properly ranked LT #1, which was huge given the 97 point differential between him and Larry Johnson.
Quick question - WRs? Any reason you chose not to do them other than lack of time?
I had also noticed that no one ranked LT as #1 last year - stunning, isn't it? So whose to say that Frank Gore or Steven Jackson won't outscore LT this year. In fact, I've read a few things recently about LT's decline - or perhaps his return to normalcy is a better way of phrasing it. Interesting stuff.
I realized last night I actually calculated the weighting factor slightly different then I described above. Here are the scores for the method described in the post:
WR:yahoo(.6036), football guys(.5929), FF cheat sheet(0.5792), espn(0.5750), fox(0.5679), Ask the commmish(0.5465), cbs(0.5447), si(0.5387), pfs(0.5346), pigskin(0.5310)
QB:
si(.4889), commish(.4788), football guys(.4707), pfs(.4586), yahoo, (.4485), espn(.4404), cbs(.4222), fox(.3939), ffcs(.3899), pigskin(.3636)
RB:
commish(.5072), football guys(0.4947), pfs(0.4875), pigskin(.4453), yahoo(0.4417), ffcs(.4393), si(.4232), cbs(0.4066), fox(.3798), espn(.3667)
If you can arrange these posts into something coherent, you are more than welcome to do so. Good luck with your draft.