By Vaughn Hester, April 26, 2011

Announcing the TREC 2011 Crowdsourcing Track, co-sponsored by CrowdFlower

We’re proud to announce CrowdFlower’s participation in one of the most prestigious academic conferences worldwide.

The 20th Annual Text REtrieval Conference (TREC 2011), organized by the National Institute of Standards (NIST), will take place Nov 15-18, 2011, at NIST’s campus in Gaithersburg, MD. TREC 2011 will feature a new Crowdsourcing Track that will investigate:

  • How to obtain high-quality relevance judgments from individual crowd workers;
  • How to effectively compute consensus judgments from individual judgments;
  • Interaction between these (i.e., worker accuracy vs. subsequent consensus accuracy achieved).

Teams can choose to participate in either or both tasks: (1) collecting judgments and (2) computing consensus. We plan to evaluate using both ranking and classification metrics, giving teams the flexibility to focus on either one or both.

We welcome participation from those who have never before participated in TREC; note there is an agreement form to submit in advance; the form is available from the track website.

View the guidelines here.

Tentative Schedule

May 6, 2011: Initial deadline for participating teams to email organizers with team name and
intent to participate (will affect the partition of data distributed to all participants)

May 13: Finalized guidelines released, Stage 1 training sets distributed to participants

May 20: Stage 2 training set distributed to participants

August 1: Release of Stage 1 test data; submission system opens immediately

August 8: Stage 1 submissions due

August 15: Stage 2 test topics distributed

August 22: Stage 2 submissions due

~September 15: Preliminary results announced to participants

~October 15: Team papers due to NIST for inclusion in TREC conference

November 15-18: Text REtrieval Conference (TREC 2011) in Gaithersburg, MD

~February 1, 2012: Final team papers due to NIST for inclusion in TREC conference proceedings

For more information, please contact Track Organizers Gabriella Kazai (Microsoft Research) or Matthew Lease (University of Texas at Austin).