Research & Insights

By John Le, April 23, 2010

Task Localization

When I think of a job that needs to be done, like laundry or dish washing, I always look for the best person for the job. My roommate comes readily to mind. I wouldn’t trust anyone else with my dirty work. But for a job where workers are located all over the globe, we cannot always be sure of their trustworthiness. So where in the world are our workers coming from? And how good are they? The following is a graph for a job sent through CrowdFlower to Amazon Mechanical Turk (amt), Gambit (gambit), and Samasource (sama).

Though we cannot reveal the complete details of the job, can you guess what kind of task it was, who the task was made for? Each circle represents workers from a particular location, the size of the circle representing the number of judgments those workers made. Colors range from red to yellow to green, indicating how much we trusted the workers.

Note the large and mostly green circles in France. This indicates we received many trusted judgments from these locations. The yellow circles in India indicates we did not trust the work from these workers in India. Of course that is not to say we never trust their work, as cultural biases can affect worker performance on a task. Particularly for this task, the questions were in French, a significant cultural bias. I cannot read French but the completed work shows quality in all the right places. The results may be confounded by the particular worker pool (amt, gambit, or sama), but we can still channel work per these pools to the right places for better accuracy.

John