Research & Insights

By Renette Youssef, December 8, 2014

Don’t Judge a Tweet by its 140 Characters: How One App is Using Machine-Learning to Tackle Credibility on Twitter

 

When you use Twitter, how do you know when you are being presented with something credible instead of something totally bogus? The answer is, unless you spend a lot of time researching each tweet, you probably don’t. However, one thing is for certain, we rely on what we read on Twitter to be true.

Twitter is one of the fastest and most effective ways we disseminate news across our world. If this world of news and information were run by trolls and pranksters, we would of given up on it long ago, (or, worse, we would have believed it and become a society like the one portrayed in Mike Judge’s film “Idiocracy”). In short, Twitter and credibility must go hand in hand.

The Research and the App That’s Making it Happen:

Bad information being spread on Twitter may become a thing of the past. The app/chrome-extension, appropriately named, TweetCred may have found us a data-driven answer to the problem that is dubious tweeting. What does this answer look like? It looks like a hotel rating of 1-5 stars, except instead of stars they use 1-7 little blue round amoeba looking things (that sit right next to the tweets you are viewing, see below for an example).

 

tweetcred1.png

The idea is that this rating can tell you the credibility-level of content posted on Twitter.

The researchers behind the project are Aditi Gupta and Ponnurangam Kumaraguru from the Indraprastha Institute of Information Technology and Carlos Castillo along with Patrick Meier from the Qatar Computing Research Institute. This team proved that they could use machine-learning to develop algorithms that would correctly be able to judge a tweet’s believability.

Tweets related to disaster information was their focus, and their research points out a big problem; rumors and fake news surrounding disasters, such as the Boston Marathon Bombing, affect thousands of people and need to be thwarted before they take hold. To test their system, they used 500 tweets each from these six disasters:

  • The Boston Marathon blasts in the US

  • Typhoon Haiyan/Yolanda in the Philippines

  • The Cyclone Phailin in India

  • The shootings in the Washington Navy Yard in the US

  • The polar vortex cold wave in North America

  • The tornado season in Oklahoma, US “built a training set”

(On a positive note, the researchers found that disaster related tweets, in general, had a higher credibility rating than random tweets.)

How should we be thinking about credibility on Twitter?

When you think of credibility, what comes to mind? Humans are hardwired to look for signs of illegitimacy whether it’s a crooked smile, an insincere tone of voice, or in case of the web a cheesy, poorly done, company logo or links to news that seems too good to be true. Translate this to Twitter, and the focus moves to URL’s, who’s mentioned who, retweets and even tweet length. It’s also shown that location information, overlapping networks, and the profile picture play a big role in our quick assessments of tweets.

How can machine-learning tackle something as complex as trustworthiness?

The answer is it can’t, at least not alone. CrowdFlower played a large role in cleansing and enriching the data used to train the system. This means some of our 5 million online contributors helped TweetCred build it’s ‘sense of credibility’ by providing tons of examples, examples that passed the test of human judgement. We call this people-powered data enrichment. Once they learned from the input of these contributors, the algorithms looked for:

  • Tweet Meta-Data: Number of seconds since the tweet, Source (mobile / web/ etc), Geo-Coordinates

  • Tweet Content: Number of characters, Number of words, Number of URL’s, Number of hash-tags, Number of unique characters, Presence of stock symbol; Presence of happy smiley; Presence of sad smiley; Contains ‘via’; Presence of a colon

  • Linguistic Clues: Swear words, Negative emotion words; Positive emotion words, Pronouns; Mention of ‘self words’ in tweet (I; my; mine)

  • User’s Numbers: Follower count, Time since the user was on Twitter, Number of retweets, Number of mentions, If the tweet is a reply or a retweet, The link’s WOT score for the URL, Ratio of likes / dislikes for a YouTube video

From a more broad point of view of the technology’s workflow, here is a good visual diagram of how the information flows through TweetCred, by PreCog@IIIT:

tweetcredflowchart.png

Lovely idea for an app, but does it work?

Despite the clever use of algorithms and help from the crowd, a system that can detect the true credibility level of any tweet 100% of the time may still be out of reach. However, TweetCred’s analysis of disaster-related Twitter-data is a fascinating step in that direction.

According to the research, 63% of users either agreed with their automatically-generated credibility scores (or disagreed by 1 or 2 points). This could either mean that it is operating at 40% efficiency, or the number could be higher if the users are experiencing some kind of bias in their rating (or simply voting things down because they do not match their expectations).

tweetcred5.png

Whichever scenario is the case, the research reminds us that “social media users are starting to expect technologies that help them evaluate the credibility of the content they read”, and they say, “TweetCred is a first step towards fulfiling this expectation.”

Dig into the data:

Take a deeper look at the data behind their study here: TweetCred: Real-Time Credibility Assessment of Content on Twitter

Have your own Twitter data project in the works? Let us know in the comments.

Bonus:

Are you interested in data being used to leverage Twitter? Check out this article on a company using network science to help you choose who to follow on Twitter based on their reach of influence in a particular topic.