We’re proud to announce CrowdFlower’s participation in one of the most prestigious academic conferences worldwide.
The 20th Annual Text REtrieval Conference (TREC 2011), organized by the National Institute of Standards (NIST), will take place Nov 15-18, 2011, at NIST’s campus in Gaithersburg, MD. TREC 2011 will feature a new Crowdsourcing Track that will investigate:
- How to obtain high-quality relevance judgments from individual crowd workers;
- How to effectively compute consensus judgments from individual judgments;
- Interaction between these (i.e., worker accuracy vs. subsequent consensus accuracy achieved).
Teams can choose to participate in either or both tasks: (1) collecting judgments and (2) computing consensus. We plan to evaluate using both ranking and classification metrics, giving teams the flexibility to focus on either one or both.
We welcome participation from those who have never before participated in TREC; note there is an agreement form to submit in advance; the form is available from the track website.
View the guidelines here.
May 6, 2011: Initial deadline for participating teams to email organizers with team name and
intent to participate (will affect the partition of data distributed to all participants)
May 13: Finalized guidelines released, Stage 1 training sets distributed to participants
May 20: Stage 2 training set distributed to participants
August 1: Release of Stage 1 test data; submission system opens immediately
August 8: Stage 1 submissions due
August 15: Stage 2 test topics distributed
August 22: Stage 2 submissions due
~September 15: Preliminary results announced to participants
~October 15: Team papers due to NIST for inclusion in TREC conference
November 15-18: Text REtrieval Conference (TREC 2011) in Gaithersburg, MD
~February 1, 2012: Final team papers due to NIST for inclusion in TREC conference proceedings