Millions of people have participated in our tasks over the last few years, and tens of thousands of people are active at any given moment. However, crowdsourcing is not a traditional engagement model. Tasks are elective, which means people are free to come and go as they please. It’s a fair question, then, to ask whether they keep coming back.
Do people perform tasks only fleetingly, or has crowdsourcing become more of a long-term engagement? Furthermore, just how important is contributor retention in the world of crowdsourcing?
While a majority of people fall into the “one-and-done” camp, many of the most productive contributors tend to have participated in previous jobs. Within any single job, these seasoned veteran contributors also provide far more work than their less experienced counterparts.
Over a period of two months, we ran a series of five very similar jobs, retrieving ratings information for businesses throughout North America. In total, we collected over half a million judgments from a total of 2,901 unique contributors,1 representing multiple labor channels and 101 countries. As a first test to whether seasoned veterans are common, we looked at how many people participated in more than one job. In total, 2,389 people participated in one job only, meaning that First-Timers accounted for over 82 percent of all contributors.
But this may not be the best number to look at. We’re really interested in whether certain people recognize and seek out specific types of tasks after having worked on them before. To look for this behavior, we analyzed the most recent job, counting how many people participated in at least one prior iteration of the task.
For a recent job, a total of 906 people participated, 247 of whom had done the task previously. By this measure, Seasoned Vets constitute approximately 27% of the workforce. While the impact of returning contributors is greater under this methodology, the fact remains that these “loyal” contributors are firmly in the minority on this series of jobs.
However, as we’ve seen before, the individual impact of contributors varies widely, with a minority of people often providing the vast majority of work. With this in mind, we looked at the contributions of First-Timers and Seasoned Vets and found some striking differences.
Seasoned Vets, while constituting only 27% of the workforce, provided 47% of the total work completed. On average, each Seasoned Veteran provided 2.5 times more judgments than their less experienced counterparts. It’s also interesting to note that there was no significant difference between the quality of work provided by First-Timers and Seasoned Vets, no doubt due to our suite of quality control measures.
Given that the people who stick around tend to be far more productive, improving retention is a useful consideration (for these jobs, at least). We’re now interested in how best to attract people to return to the types of jobs they’ve already seen. That, however, is a work in progress and a story for another day.
1. Note that this analysis only considers Trusted workers.