Happy summer! To celebrate the new season and my return to a land where I can get e-mail and cell phone reception (had a great time this weekend but I do love the creature comforts of civilization...), I have a little something fun for everyone: the start of the aggregate accuracy analysis results. For those who are new to the blog, I've spent the past several years analyzing the accuracy of preseason expert rankings to find out which sites did the best job at predicting position rankings for the season. I started in 2007 and looked at 14 websites; after joining up with the Fantasy Sports Trade Association the following year it grew to 24 websites, and then a whopping 48 sites in 2009. At this point it has become almost entirely a submission process so if your favorite site isn't included, it's likely because they either chose not to submit or just do not know about the contest. You can read all of the details about how I analyze the rankings each year here and if you're a site that's interested in participating this year, stay tuned for more info on that front in August. Also, I want to note that one important condition of participation is that I keep all of the results of the contest confidential except for the top performers, so I unfortunately can't share the full results of each year's contest with you all. Besides I'd really prefer to focus on those who performed well year in and year out versus those who didn't do such a strong job with their rankings.
So I've had a plan to combine the results of the contest each year to create a multi-year assessment of each site, which is far more informative than just one year of results, and I finally had the time to do the analysis. Today we will take a look at quarterback rankings to see which site has been the most accurate with their preseason QB rankings from 2007-2009. Over the past 3 years I've included 54 different sites in my analysis, however only 10 sites have been included all 3 years, and and additional 13 sites have been tracked for 2 years, so for the multi-year analysis I'm focusing on sites that have participated in at least 2 years of the contest. Honorable mentions for those that did well in their 1 year of participation are noted below, too, though.
As you can see above, the contest has grown enormously and with the variation in the number of sites ranked each year, I decided to calculate a percentile ranking for each site each year. For example, Football Guys had the best QB ranking of the 14 sites I analyzed in 2007 so they were in the 7th percentile (1/14) - keep in mind this is not like the SATs; in this case you actually want a low percentile. After calculating the percentile for each year, I averaged each site's ranking from each year it was included in the FFLibrarian accuracy contest. The lower the number in the average column, the better the site did compared to their peers. Ask The Commish's 0.30 percentile means they averaged being in the 30th percentile in QB pre-season rankings over the last three years - that's the best of any of the sites I have data on for all three years. For comparison's sake, the site with the worst percentile ranking for QBs was averaging 0.86 for all three years (0.81 for sites with 2 years of data; 0.98 for those with 1 year).
I'm sure there are important things I'm forgetting to mention or notes that I should be making here, so please feel free to ask questions or make comments. As always, this is a work in progress so if you have a better method, please let me know but in the meantime a big kudos to my husband for talking all of this out with me. Just one of the many times it pays to have fallen in love with an economist/stats dude.
I'll be posting the results for RBs, WRs, and TEs over the next week or so in addition to my normal linking. Enjoy and a hearty congrats to Ask the Commish!
So I've had a plan to combine the results of the contest each year to create a multi-year assessment of each site, which is far more informative than just one year of results, and I finally had the time to do the analysis. Today we will take a look at quarterback rankings to see which site has been the most accurate with their preseason QB rankings from 2007-2009. Over the past 3 years I've included 54 different sites in my analysis, however only 10 sites have been included all 3 years, and and additional 13 sites have been tracked for 2 years, so for the multi-year analysis I'm focusing on sites that have participated in at least 2 years of the contest. Honorable mentions for those that did well in their 1 year of participation are noted below, too, though.
As you can see above, the contest has grown enormously and with the variation in the number of sites ranked each year, I decided to calculate a percentile ranking for each site each year. For example, Football Guys had the best QB ranking of the 14 sites I analyzed in 2007 so they were in the 7th percentile (1/14) - keep in mind this is not like the SATs; in this case you actually want a low percentile. After calculating the percentile for each year, I averaged each site's ranking from each year it was included in the FFLibrarian accuracy contest. The lower the number in the average column, the better the site did compared to their peers. Ask The Commish's 0.30 percentile means they averaged being in the 30th percentile in QB pre-season rankings over the last three years - that's the best of any of the sites I have data on for all three years. For comparison's sake, the site with the worst percentile ranking for QBs was averaging 0.86 for all three years (0.81 for sites with 2 years of data; 0.98 for those with 1 year).
| Sites that have 3 years of participation | ||
| Ranking | Site | Average |
| 1 | Ask The Commish | 0.30 |
| 2 | Rotoworld | 0.31 |
| 3 | CBSSports.com | 0.38 |
| 4 | ESPN | 0.44 |
| 5 | FFToolbox | 0.45 |
| Sites that have 2 years of participation | ||
| Ranking | Site | Average |
| 1 | Football Guys | 0.16 |
| 2 | Fantasy Hulks | 0.27 |
| 3 | KFFL | 0.34 |
| 4 | NFL.com | 0.43 |
| 5 | Fantazzle | 0.46 |
| Sites that have 1 year of participation | ||
| Ranking | Site | Average |
| 1 | WhatIfSports | 0.04 |
| 2 | Junkyard Jake | 0.06 |
| 3 | FantasySharks.com | 0.08 |
I'm sure there are important things I'm forgetting to mention or notes that I should be making here, so please feel free to ask questions or make comments. As always, this is a work in progress so if you have a better method, please let me know but in the meantime a big kudos to my husband for talking all of this out with me. Just one of the many times it pays to have fallen in love with an economist/stats dude.
I'll be posting the results for RBs, WRs, and TEs over the next week or so in addition to my normal linking. Enjoy and a hearty congrats to Ask the Commish!
Comments