As the NFL season loomed on the horizon back in August, there was a lot of fantasy football talk around the CrowdFlower office. Naturally, we decided to crowdsource a killer fantasy football team.
We asked CrowdFlower workers to help us build a ranked list of the Top 75 players to guide our fantasy football draft. We used pair-wise comparisons to determine rankings. Specifically, we presented workers with two players, asking them to pick which player they thought would be more valuable.
We used the Top 75 players from ESPN’s 2010 Fantasy Football Draft Kit1, matching each player with his 74 counterparts, giving us a total of 2,775 player pairs. We also included each player’s position and team, as well as a link to more detailed statistics.
After the job finished, we ordered the players according to number of head-to-head victories.
(Download the full results here.)
We found that workers choose the player on the left 53 percent of the time, even though each match-up appears twice, once with Player A on the left and again with Player A on the right. With 5,500 data points, this is significant bias toward the player on the left. However, the final ranking doesn’t change even after accounting for this bias.
We settled on two likely explanations for the bias:
- The anchoring effect of putting something on the left (and as the first response option) may have caused more workers to select the first player.
- Our Gold was slightly biased toward the player on the left, which we have previously seen2 to have an effect on the overall distribution of the answers.
As a first comparison of our crowd of football fans with ESPN’s Fantasy Football brain trust, we plotted each player by the difference between his two rankings.
Will Chad Ochocinco and Matt Ryan vindicate the crowd? Stay tuned to find out.