A few months back, we ran a Twitter sentiment analysis job on the first GOP Debate. Those were the halcyon days when everything was all Trump, all the time, and, well, now that we mention it, not much has really changed. Sure, you hear a bit more about Ted Cruz (leading in Iowa!) and Ben Carson (kinda stabbed a guy!) but Trump still gets 27% of all campaign coverage, which is kind of insane when you consider he’s a nightmarish clownman.
At any rate, we took the data from that job (and an additional, near identical job we ran on one of the subsequent GOP debates) and decided to create a machine learning model in CrowdFlower. This evening, we’re hoping that model will allow us to run thousands of tweets in real-time to get instant judgments about the debate. What, particularly, is resonating with viewers? Who’s getting the most positive feedback? What subjects do people care most about?
Since our new machine learning tool takes human judgments from CrowdFlower job(s) and creates models from those, its important to note that both the original job and the model look at the the same four categories. Those are:
- Relevance: Is the tweet actually about the GOP Debate?
- Sentiment: Broken simply into positive, negative, and neutral
- Candidate: Which candidate does the Tweet primarily concern?
- Subject matter: What’s the substance of the tweet? Healthcare? Foreign policy? Gun control?
With all that in mind, we wanted to lift the hood on the model so we could show you what we’re working with. The best way to do that is with a set of predictors and anti-predictors. If you’re not familiar with what that means, basically, those are just certain words or bigrams that appear most frequently in certain categories of data. For a simple example, a tweet that uses the Twitter account @randpaul is almost certainly going to be about, well, Rand Paul. Our model looks at judgments our contributors made about 20,000 tweets and finds patterns between those judgments and the words in the Tweets themselves.
Since we just added visualizations to CrowdFlower AI, we thought we’d show a few off. Let’s start with sentiment, since it’s rather interesting in its own right:
In case it’s not clear from the screen shot, the green bubbles are showing predictors for positive sentiment while the gray bubbles are showing anti-predictors. A few interesting whatnots:
- “Loser” is anti-predictive for positive sentiment. Stop the presses.
- The candidates chosen hashtags make for positive predictors. Makes sense, as nobody who loathes Mike Huckabee is going to join his cheering section
- “Network” and “Hollywood” both show up as negative predictors. People really do love complaining about media coverage and fancy movie know-it-alls, apparently.
Also, while some subjects are quite easy to classify:
…others were, well, less easy. The predictors for “abortion”, for example, had a lot of noise, but that was mainly because, in the debate upon which we built the job, the topic wasn’t explicitly broached. Without much data (and tweets about abortion constituted just a fraction of a percent of the entire 20,000 row sample), a classifier can’t find any meaningful patterns past obvious ones like “pro-life” or “abortion.” We talk about this time and time again, but here’s further proof: a machine learning algorithm is only as good as the data you put into it. And the more rich data you put in, the smarter it gets.
In fact, on that point, CrowdFlower AI lets us build four separate models based on the same data and the confidence values of those four models are similar–if not identical–across multiple categories. The more data we feed into these models, the better that confidence gets.
So tonight, in addition to pushing tweets through our model for instant feedback, we’ll be marking those low confidence value rows and sending those back to the crowd for judgment. This practice, human-in-the-loop machine learning, lets machines judge the easy stuff and asks humans to judge the hard stuff. You then take those hard judgments and train the machine on the trickier data rows and outliers, which in turn creates a kind of virtuous circle where it learns more and more and humans are required for the most challenging parts.
Stay tuned this evening for more.