At CrowdFlower we continuously evaluate our partner channels to make sure our customers are successful using our platform. In December, we announced that Mechanical Turk would no longer be a partner channel. One of the primary reasons we did this was because we could no longer ensure that our customers would have the best experience using our platform and get the best results possible using the Mechanical Turk channel.
That said, we understand that some of our customers may be concerned with this change. To help assure customers that this would not adversely affect their jobs, but in fact improve them, we ran an experiment before we dropped the Mechanical Turk contributor channel. The experiment was designed to compare accuracy and throughput — the time it takes to complete a job.
How did we do it? We ran a job using the Mechanical Turk contributor channel only, and we ran the exact same job using all other partner channels and excluded Mechanical Turk contributors. The results were astounding: The non-Mechanical Turk contributors outperformed in both accuracy and throughput. We also ran the job a second time to the two different sets of contributors and added advanced quality control mechanisms. This time the job was limited to better-performing contributors in both channels, which we call Level 2 contributors, and it incorporated variable judgments. (Note: Selecting variable judgments means you can collect more judgments when contributors do not agree). Here are the results:
Job: Find Official Websites for a List of 1200 Companies
You will notice that there is no data for the job that we ran exclusively to Mechanical Turk Level 2 contributors. That’s because we were unable to get a single contributor to complete a task on the job.
The Bottom Line
Our customers will get better results, faster, using our other partner channels whose performance we continuously monitor. If you still have concerns about not being able to access Mechanical Turk contributors through our platform, please contact us so we can better understand and address your needs.