By Justin Tenuto, January 14, 2015

A Few Changes to the CrowdFlower Platform

We’re always making improvements to our platform but when the experience will change for our users, we think it’s important to let you know what’s changing and why we’re changing it. Our release today is focused on making our platform more intuitive and making the language around our product more consistent. You’ll still be able to run all the jobs you’re used to running; we just wanted to keep you informed.

Here’s what we’ve updated:

Consolidated Settings

This is the change you might notice first off. When you’re creating a new job and click “Job Settings” in the left-hand navigation menu, you’ll see that the menu is cleaner now:


We’ve consolidated the most important settings in this basic view. You’ll be able to select your contributor levels with a slider (more on that in a moment), set your judgements per row and rows per page, and choose how much you’d like to pay per page. Since these settings are essential to every job, we wanted to group them all in a single place.

If there’s a setting you’re used to seeing that isn’t on that page, you’ll find it in Advanced Settings. Just click the “Show Advanced Settings” text link below the Save button. If you can’t find what you’re looking for, please contact us so we can help.

Performance Slider

On the Job Settings page, you’ll see a performance slider. This refers to the contributors your job will be exposed to and replaced the “Levels” interface. Our internal quality controls haven’t changed and contributors are still ranked on levels based on their performance, but this slider more accurately reflects what choosing a particular level means.


“Highest speed” means that more contributors will see and work on your job. This was formerly called “Level 1.” “Highest quality” means that only our top rated contributors are exposed to your data. The latter is great for more difficult work but because there are less of these contributors, it could take a bit longer to complete your job.


We’re making our terminology consistent across all our pages. We feel these terms are more descriptive and will be easier for new users to understand. For example, “tasks” are now uniformly referred to as “pages.” You’ll see that consistency across all settings menus and across our Success Center.

A few important ones to call out:

Test Questions: In the past, a test question was sometimes referred to as “gold.” Test questions are used to make sure you get the best contributors for your job.

Rows: In the past, a row was sometimes referred to as a “unit.”

Dynamic Judgement: This term replaced “variable judgement.” You can read more about Dynamic Judgements here.

Pages: You’ll notice this on the settings menu. Instead of calling a group of rows a “task,” it’s now a “page.” No functionality has changed.

Test Questions

When you’re running your next job, you might notice that the “Skip Randomly” button isn’t there anymore. That’s because that action is now automated.

Simply put, skipping randomly through your data to create test questions ensures that contributors see a wider breadth of data when they’re in quiz mode. That means you can more easily test contributors on a variety of rows and get most qualified people for your job.

If you have any trouble finding anything you’re used to or just have a question and want to get in touch, please email us at We’re here to help.