By David Lacy, November 20, 2013

Search Quality and the Crowd

Here on the CrowdFlower Customer Success team, we keep an eye out for customers who use crowdsourcing to employ innovative and creative approaches to solving problems.™(1) is one company that engages in such work in an effort to increase the quality of their search results. I had an opportunity to chat with Steven Rosiers, who leads the crowdsourcing effort at, to better understand the challenges and victories they have experienced while working on the CrowdFlower Platform.

Company description:™, the largest digital media company in Belgium, helps all Belgian companies in growing their business. The company operates through a comprehensive range of online, mobile and print channels. As a versatile digital marketing agency, assists entrepreneurs by offering them a full set of flexible and effective marketing and communication solutions, tailored to their specific needs.
The complementary services (mobile, online and print) allow potential clients to quickly and easily find the businesses and suppliers they are looking for in their region.

CrowdFlower (CF): What was the problem that led you to look into crowdsourcing? (gp): We were struggling to find a proper methodology to measure search quality. On the one hand we were making all kinds of improvements to our search engine, but on the other hand there was no framework to confirm the success of our changes. Compare it to a chocolate factory that changes the recipe slightly, but it has no clue whether its consumers like it or not. Our two main conditions, scalability and being un-biased, seemed very hard to combine. CrowdFlower turned out to be the perfect solution, providing access to a big pool of unbiased workers. Our search quality index was born.

CF: How has your experience been with CrowdFlower’s Platform? What have you learned?

gp: The user-friendly CrowdFlower interface offers us enough flexibility to create our tasks. A big advantage is that once you create a good template, you can reuse it later and just change the variables. We ask the contributors to execute a number of predefined searches, and then they have to give the first 10 results a relevancy score from 0 to 3. The template is fixed. We upload a new set of searches every month. Also, the results are easy to export, which gives us the ability to semi-automate the analysis process.

(1) is owned by Truvo Belgium. For more information go to

CF: What advantages have you gained by using CrowdFlower’s Platform?

gp: We met our initial requirement, which was to create a framework to measure search quality. Every month we now have a search quality index, indicating how happy our users are with the relevancy of the results. For every change we make to our search engine, we can tell if our users like it or not. But we went even further. By analyzing the low-rated searches, we were able to identify the main gaps in our search algorithm. So, the index is not only telling us if we are doing a good job, it is also informing us what we need to do to improve it.

CF: What advice would you give to others who are thinking about crowdsourcing?

gp: I really can’t think of an easier, faster and cheaper way to reach out to such a big pool of contributors. Outsourcing used to be a complicated process with quotes, contracts and invoices. With CrowdFlower it is nothing more than clicking a button. However, there is a big “but.” Be aware that you have to invest some time yourself to produce the best quality.

The platform not only attracts contributors with good intentions, but there are also lazy contributors and even cheaters. CrowdFlower’s Platform Basic offers enough possibilities to eliminate the cheaters (known-answer questions, qualification mode, etc.), but this is a learning process you have to go through. Invest time in quality, otherwise your results will be useless and the cheaters will become richer.

CF: In conclusion, we can see how crowdsourcing can provide data-centric organizations with a strong advantage through use of what we refer to as the Human API. CrowdFlower itself is the by-product of search algorithm improvement, and I would like to thank Steven Rosiers and his team for shedding light on their experience with the CrowdFlower Platform Basic.

You can also visit the search relevance article for more information on successful application of crowdsourcing to search relevance endeavors.