Research & Insights

By John Horton, January 14, 2010

The Labor Economics of Paid Crowdsourcing

This is the first in (what we hope will be) a series of guest posts from John Horton, a Doctoral Candidate in Public Policy at the Harvard Kennedy School. John and Aaron Shaw are collaborating on some research projects and we were both introduced to Dolores Labs around the time of last year’s Mechanical Turk Meetup.

Since then, John’s been busy establishing himself as a Crowdsourcing research pioneer by designing a suite of online data collection tools as well as running numerous experimental and observational studies on several different Crowdsourcing labor markets. We really admire his work, which tends to involve well-designed methods and cut straight to big, interesting questions. In this post, John discusses a recent experiment he ran on Amazon Mechanical Turk that looks at worker motivations in the context of labor economics and theories of the “reservation wage.”

Hi – this is my first post here (though I’ve commented a bit). I work with Aaron Shaw and got to know Lukas at the last meet-up he hosted. Anyway, I’m interested in crowdsourcing and online labor more generally and Lukas was kind enough to let me write about some of my research here.

It’s pretty clear that many Amazon Mechanical Turk (AMT) workers are motivated primarily by money, which suggests economics is the best tool for understanding worker decision-making.Research by Winter Mason and Duncan Watts shows that workers behave in a way consistent with economic rationality: when they were paid more, workers produced more output. Although any sensible model predicts that workers will work more when paid more, standard labor economics models make several other predictions (some might call them assumptions): workers should make decisions based solely on the real wage offered — payment divided by time spent. They should compare this offered wage to their reservation wage for a particular task.

Because it drives decision-making, the reservation wage is the key parameter in labor supply models, but it is hard to estimate in practice; when we observe someone working, even if we know their wage we don’t get to observe their reservation wage parameter — we just know that their wage is above the reservation wage. In a new paper (joint with Lydia Chilton), we use a unique method that allows us to estimate reservation wages for AMT workers. Although we find some agreement with the predictions of the simple rational model, we also find some evidence that workers are “target earners,” meaning that the work until they reacg certain salient earnings targets (e.g., the maximum amount available). This kind of behavior has been found in other contexts, but it runs counter to the rational model.

The Task

For our task, subjects clicked back and forth between two vertical bars in a Flash game (screen shot below). A block of 10 back-and-forth clicks made up one unit of output, and subjects could decide how many blocks to complete. The amount paid per-block was constantly decreasing. This constantly decreasing rate allowed us to esimate a worker’s reservation wage, by looking at the implied wage when they “quit.” A live demo of the task is available here.

task.png

Results

Subjects were randomly assigned to either a HIGH or LOW group. The HIGH group was paid 3 times more than LOW for every task. The figure below shows output in both groups. One striking feature of the data is how bimodal output is: some workers produced lots of output and some produced very little. For this bimodality to be consistent with rationality, the distribution of reservation wages themselves would have to be very bimodal, which seems unlikely.We found that the imputed reservation wage distributions were quite different across groups. Because of randomization, the distributions should have been indistinguishable. In particular, we found that the reservation wages in LOW were too low, suggesting that workers in LOW, on average, worked more than they should have. Why?

output.png

Target earning

One possible explanation for why there is too much output in LOW is that at least some workers try to earn the maximum amount possible, regardless of the “wage” associated with this strategy. Having an earnings target may sound rational, but can lead to some perverse results. For example, workers might work longer when wages are low (because they still want to meet their target) than when they are higher (though there are other reasons this can happen, namely income effects). It is an open controversy in economics whether employees with “real” jobs are target earners (see this work by Henry Farber as well as Colin Camerer’s work), but we find several pieces of evidence for target earning in our data.

The strongest evidence we find for target earning is that some workers show a preference for earning total amounts divisible by 5. In the figure below, the earnings of workers in HIGH are plotted as a histogram, with horizontal panels for the whole cents earned. E.g., earnings amounts 29.2, 29.5 and 29.9 would all be in the same “29 cent” panel. The height of the bars show how many subjects earned that amount of money. Panels where the whole cents are divisible by 5 have black histogram bars; the others have white bars.We can see that several subjects earn the smallest amounts available (e.g., 2, 3 and 5 cents). Because these low earners quit very early, they presumably do not have a target or would not need a target. However, we see clear output spikes at 15, 20 and 25 cents. The probability of this happening by chance is about 3 in 1000 (see the paper for details).

Conclusion

We find some agreement with the rational model, as well as important anomolies consistent with some ideas from behavioral economics. While it’s probably too early to offer much practical design advice, it does seem that designers should give workers natural targets, as they seem to help at least some workers. The paper is called “The Labor Economics of Crowdsourcing” and is available here.