By Tim Matthews, May 29, 2013

Recap: Builder Pro Training Day NYC

Thanks to everyone who came to our first Builder Pro Training Day on the East Coast. Builder Pro is the new advanced version of our crowd microtasking platform.

We had a full room in midtown and lots of great interaction. Our NHL sentiment analysis job was popular with the Rangers fans in the room.

Builder Pro Day NYC

Nathan Zukoff, our Solution Engineer, and Ari Klein, our Director of Customer Success, took the lead in covering the advanced features of Builder Pro. Skilled crowds were the most popular topic of the day, with folks thinking about what kind of skills they could use in their business. The new advanced quality control features were also a hit. Everyone wanted access to their Builder Pro account to try them out.

A number of questions came up during the day that I thought I would share:

Q: Can you target particular workers or types of workers?

A: Yes. Today you can target workers by location, i.e. country, or skill, such as language proficiency. CrowdFlower is working on building up more and more “skilled crowds.” We are also working to allow more granular demographic targeting.

Q: How do you balance between machine learning and crowd microtasks?

A: This depends quite a bit on the goal of the job. In general, there are two approaches. First, crowd microtasks can be used to train machine learning to allow more and more future tasks to be done algorithmically. Alternatively, tasks that were not effectively handled algorithmically can be routed to the crowd. More complex jobs can combine both of these at various steps in a workflow. Sometimes we use machine learning to help recommend correct answers, saving our contributors from navigating through large taxonomies or possible answers.

Q: Can Builder Pro be accessed via API, for data upload and job control?

A: Yes, Builder Pro has a complete set of REST APIs to upload data and control jobs on the platform. The documentation is available here

Q: How you determine how many “gold” units to create?

A: A conservative approach is create gold units that represent 10 to 20 percent of the total number of job units for smaller data sets. This ensures there are enough units to train workers and prevent repeats of the gold training set. The larger the number of units, the smaller the number of units as a percentage. In other words, there is no need to create 50,000 gold units for a 500,000 unit job. For larger data sets, 100-300 gold units are generally appropriate.

We’re thinking about bringing Builder Pro training to Chicago, Austin and Washington D.C. this year. Let us know if you want us to come, or if you live or work near another large city we should visit. Also, if you are interested in a Builder Pro trial license, let me know.