Last week, I wrote about the most Clintonian and Trumpesque moments of the first presidential debate. In this post, I’m going to ask the same question of the vice presidential debate: what moments did Tim Kaine and Mike Pence sound most like the candidates at the top of their ticket? We’ll then dissect the results in order to see how different assumptions about the data would affect results of this kind of linguistic style analysis.
To do this, I take the previous debates and train machine learning models to classify new utterances as being one of the 13 biggest talkers in past debates. So among Republicans the model knows about Bush, Carson, Christie, Cruz, Kasich, Rubio, and Trump. For the Democrats, it’s Sanders, O’Malley and Clinton. Finally, three moderators spoke enough to be included: Blitzer, Cooper and Tapper. I then apply these models to what Kaine and Pence said in the VP debate. The models don’t know anything about Kaine and Pence so they will always predict one of the other speakers. If a VP candidate always talked like his running mate, the model would tell us. Or who he is most like other than his running mate.
Here is the most Clintonesque moment of the debate–the one that models are most confident about to the exclusion of all other possible speakers. It’s by Tim Kaine:
The terrorist threat has decreased in some ways, because bin Laden is dead. The terrorist threat has decreased in some ways because an Iranian nuclear weapons program has been stopped. The terrorist threat to United States troops has been decreased in some ways because there’s not 175,000 in a dangerous part of the world. There’s only 15,000.
And here’s the most Trumpian moment of the debate, which was voiced by his running mate, Mike Pence:
Look, to get to your question about trustworthiness, Donald Trump has built a business through hard times and through good times. He’s brought an extraordinary business acumen. He’s employed tens of thousands of people in this country.
In a moment, I’m going to go deeper into methodology and assumptions but for the moment, those two quotes will come out on top whether you use multinomial Naive Bayes or logistic regression. For right now, let’s take a look at the phrases that most separate Kaine and Pence from each other. But to understand a bit more about what our models are doing, let’s filter these down to only phrases that are overwhelmingly and confidently classified by both our ML models.
In doing this, we’re going to situate the VP debate with all the past debates. It’s going to drop out phrases like “Governor Pence” and “Senator Kaine”. These are really important phrases for distinguishing the VP candidates (you use someone else’s title/name, not your own). But they have never been said in any of the previous debates by anyone else. If you like naming as much as I do, see this post about no-naming in the conventions.
You can see is that the models are sensitive to topics (with Russia) but especially to rhetoric (we’re going, American people). I would probably put I mean in the category of ‘discourse markers’ that help structure conversational turns–other examples would be you know, sentence-initial look or well. Starting a sentence with look, by the way, is characteristic of Mike Pence–as well as Donald Trump, Hillary Clinton, and Bernie Sanders. Other people don’t use it nearly as much.
If you listened to the debate, you know that topics like Vladimir Putin, nuclear weapons, and law enforcement came up a lot. And the models do know about these, however they disagree about who to associate them with and how strongly.
For example, Kaine brought up Vladimir Putin 16 times, Pence said that name twice. In earlier debates, Carly Fiorina used the name nine times but she’s not in either model. Tapper and Rubio used Vladimir Putin four times.
Both multinomial Naive Bayes and logistic regression are most confident that if you see Vladimir Putin, you should guess Tapper and not Rubio. But they come to different conclusions about how much more confident you should be. They also propose rather different “second best guesses”–in the case of MNB, it’s Christie, for LR it’s Trump. This has to do with the fact that they are also thinking about the two individual words that make up the phrase Vladimir Putin. For example, Trump says Putin (without Vladimir) 11 times–he only says the full name once.
That may seem like a lot of detail, but such details matter for making sure your models are doing what you intended them to do. It’s probably worth mentioning that people trust the confidences from logistic regression more. But we also value overall algorithm performance. In this data set, it’s impossible to be correct (neither Pence nor Kaine are in the training data). However, in the earlier problem, logistic regression was noticeably worse at predicting actual Clinton/Trump speech than MNB.
In this case, I essentially used an ensemble method to make sure that my claims were substantiated by models with different assumptions. But we are ready–or maybe overdue–to say who Tim Kaine and Mike Pence are, generally speaking most like. Both the MNB and LR models agree on the best classification for 54% of the data. They swap the “best” and “second-best” guess for another 29%. About 17% of the 547 conversational turns get very different classifications by the two models. These are also, for both models, the items that have the lowest confidence.
However we slice the data, a curious thing happens: both Tim Kaine and Mike Pence sound the most like…Donald Trump. Among the 294 unanimous classifications, 40% of both Pence and Kaine’s conversational turns sound like Trump.
Models can have problems with imbalances. In this case, our training data is made up of 58,622 unigrams of Donald Trump but 70,869 of Hillary Clinton. In other words, a baseline model would just guess “Clinton” all the time, not Trump.
Instead, I think three things are happening. One is just the fact that we’re enlisting models to do something they are very specifically not able to do since Pence and Kaine aren’t in the training data. The other is that Tim Kaine was repeatedly trying to use Donald Trump’s words against him, which is one very good reason he sounds like him.
Finally, Donald Trump uses fewer unique words. In our training data, Clinton uses 121% more total words than Trump but she uses 143% more word types. So that means that the average Clinton word is only going to be said 15 times, while the average Trump word is going to be said 18 times. That means that for the average word or phrase, there’s more support for Trump having said it because Clinton probably used a synonym or hitting an additional topic area he wasn’t.
Next week, I’ll be back with data from the second presidential debate where we can get back to having right and wrong answers. And when we may know enough to make some predictions about the third debate.