Research & Insights

By Jodie Ellis, November 14, 2011

Did you say “Great!”, or “Oh Great!”?

 

Being tapped to write a blog post here at CrowdFlower is usually left to the experts. So with that, let me begin by making the disclaimer that I am neither a political analyst nor a data scientist. But I do have a personal fervor for politics and access to some impressive tools, thanks to my job here at CrowdFlower.

For those who aren’t familiar with CrowdFlower, we specialize in tapping human contributors worldwide to do massive amounts of simple, repetitive tasks (especially tasks that are hard for computers to do by themselves). Here’s a quick how-it-works animation.

I had been reading some old blog posts on the CrowdFlower blog when I came across an interesting 2008 post on election media bias.

I determined that this could be a great opportunity to revisit sentiment analysis, and specifically set out to see if automated sentiment detection tools vs. human assessments could yield any blog-worthy findings.

To see how far the automated sentiment tools have come, I began by using an enterprise-grade social media monitoring tool that provides sentiment analysis.

I ran a few quick monitoring searches of my own to see how the current Republican Primary election was tracking — it seemed a topical place that would be chock full of good commentary.

The instant access to well-organized data from blogs, news sources, and a variety of social media sources was outstanding.

However, I was surprised to find that for each search I conducted, the automated sentiment detection tool consistently returned an overwhelming proportion of “Neutral” ratings (frequently exceeding 90%). This seemed funny to me, given the typically emotive nature of politics.

It’s important to note that this particular tool uses a default value of “Neutral” for any post it cannot interpret.

A particularly interesting subset of the data was several thousand tweets about Herman Cain immediately following the news of alleged sexual harassment by Cain during his time as leader of the National Restaurant Association. Surely this would yield some sentiment-rich commentary that even machines couldn’t resist tagging.

For the posts about Herman Cain on Oct 31st, here is what the machine detected on just under 3,000 posts:

Naturally, I took to the CrowdFlower platform and decided I would run the same data through a simple sentiment analysis workflow. With the help of our team of crowdsourcing gurus, I utilized some simple, but effective best practices to control for quality (you can get a good overview here). Here is what the CrowdFlower contributors detected:

Here are just a couple of posts marked “Neutral” by the machine and “Negative” and “Positive”, respectively, by CrowdFlower contributors:

Takeaways

A spot check of the results on the automated set confirmed that when the machine actually tagged a post as positive or negative, it was usually very accurate (good precision).

However, the large amount of data that the machine was unable to make a determination on suggests that the pervasive problem of ‘recall’ is still the big challenge with automated sentiment detection.

This graph illustrates the recall difference a bit more clearly. The need for human analysis when dealing with the subtleties of language could not be more apparent.

Automated Tool: Good precision. Poor recall.

CrowdFlower Tool: Good precision. Good recall.

Sentiment Analysis is Insightful AND Entertaining

In addition to the Herman Cain Twitter data, I looked at headlines, blogs, and a broad swath of social media commentary on all the candidates. The conclusion I can draw from my effort is that sentiment detection, is indeed, still a very challenging problem to solve through automation.

This is consistent with what I see here at CrowdFlower daily — in today’s data-wealthy world, there are countless tasks that require human attention (good to know if my blogging career never gets off the ground).

Hopefully I’ll get the chance to continue exploring the sentiment about topical news as it breaks, and will look forward to sharing future findings.

Have experience monitoring sentiment? Let us know if this is consistent with what you’ve seen. Leave a comment.

***

To find out more about how CrowdFlower technology is used for sentiment analysis and a wide range of other human powered projects, visit the CrowdFlower products page.