(This article was first published on R – Curtis Miller's Personal Website, and kindly contributed to R-bloggers)
Naturally, both Hillary Clinton and Donald Trump have been accused of lying; if I had told you in 2012 that both candidates from both political parties were being accused of lies, you would likely have given me a blank, disinterested stare; this alone is not shocking. What is shocking, though, is the level of deceit and how central a theme it was to this campaign season.
Some accuse the other side of being the liars, the other side counters with a similar accusation, and those not committed to a side like to lazily declare both sides to be equally guilty of lying. It does not take much thought, though, to realize there is no reason why both sides should be equally guilty of lying, and that is especially true for this election.
Donald Trump takes lying to a new level, living in his own invented reality and inviting the rest of America to participate in his nightmarish hallucination that only he can save us from. The media has struggled to handle this. They worry about appearing unfairly biased, and in the past, perhaps behaving as if both sides were equally guilty of lying seemed a good enough proxy to reality to avoid coming across as biased. But Donald Trump lies so much it’s thrown them off their toes (if you don’t believe me, read on; I have evidence later), and his candidacy has sparked a conversation in the media world about how to handle a candidate so casually dishonest he himself may not know what is true and what is not.
Here, I’m going to use R to dig deeper into the question about how honest are our politicians, and whether one party lies more than another. All of my data was scraped from PolitiFact’s website, a popular and well-known fact checker with an excellent categorization system that makes analyzing their data easier. I present various graphics and tables showing who lies more, and what they lie about.
Data Extraction and Analysis
Before scraping, I use MySQL to create a database that will hold the data I scrape from PolitiFact. The SQL statements that define the tables in the database politifactscraper are shown below.
I wrote a Python program to scrape the data from PolitiFact and save the data in the database politifactscraper. The code for the program is listed below:
Now that the data is in a database, it’s easy to access and process it using either R or Python. I will be using both languages for analyzing this data.
In R, I first assess how honest individuals who associate with a party are (I also include the “Unaffiliated” group, which does not include politicians but the unending stream of bloggers and Facebook/e-mail spam that we see on a daily basis). PolitiFact ratings can be seen as ordinal data, which means that metrics based on order, such as the median are well-defined. I base judgement of honesty on the median first, then break ties using the mean (which is not well-defined for this data). In the data base, I conveniently assigned the values in the aid column of the rating table so that it reflects the order of “honesty” for each possible rating, with 0 for “Pants on Fire!” and 5 for “True” (the flip-flopping scores have values 6 for “Full Flop” to 8 for “No Flip”, but are excluded in further analysis). The result is shown below:
Party Honesty
Median Honesty
Average Honesty Score
Rated Statements
Independent
Mostly True
3.25
181
Democrat
Half-True
3.07
4026
Libertarian
Half-True
2.92
147
Republican
Half-True
2.57
5480
Unaffiliated
Pants on Fire!
1.06
350
Data source: PolitiFact
It seems that Independents, in the aggregate, tell the truth the most, and Republicans the least. While Democrats, Libertarians, and Republicans all have median honesty scores of “Half-True”, the tie-breaker suggests Democrats are the most honest, Republicans the least. Interestingly, PolitiFact rates Republicans more often than Democrats.
As for the Unaffiliated group, take anything you see on Facebook, on some random dude’s blog, or in a chain e-mail with a grain of salt; their median honesty is “Pants on Fire!”
Okay, so that’s the parties. What about individuals? I repeat the above pipeline to see the the top 20 most honest individuals on PolitiFact. Because some people have only a couple ratings in file, I require that PolitiFact have rated at least 15 statements made by the individual to be included in the following lists.
Tope 20 Honest Entites
Party
Median Honesty
Average Honesty Score
Rated Statements
Alex Sink
Democrat
True
4
19
Dennis Kucinich
Democrat
Mostly True
3.84
25
Sheldon Whitehouse
Democrat
Mostly True
3.79
24
Cory Booker
Democrat
Mostly True
3.6
20
Mark Warner
Democrat
Mostly True
3.6
20
Gina Raimondo
Democrat
Mostly True
3.59
17
Sherrod Brown
Democrat
Mostly True
3.59
34
Rob Portman
Republican
Mostly True
3.57
47
Bill Nelson
Democrat
Mostly True
3.48
25
David Axelrod
Democrat
Mostly True
3.39
18
Hillary Clinton
Democrat
Mostly True
3.34
292
Bernie Sanders
Independent
Mostly True
3.25
107
John Kasich
Republican
Mostly True
3.25
64
Fred Thompson
Republican
Mostly True
3.25
16
Bill Richardson
Democrat
Mostly True
3.24
17
Alan Grayson
Democrat
Mostly True
3.18
34
Bill White
Democrat
Mostly True
3.08
26
Tim Kaine
Democrat
Half-True
3.38
50
Nathan Deal
Republican
Half-True
3.37
49
Bill Clinton
Democrat
Half-True
3.29
41
Data source: PolitiFact
Alex Sink, a Florida Democrat who ran for governor and the House of Representatives (and lost both races) has the highest rating. Rob Portman, the junior Senator from Ohio, is the most honest Republican, and Bernie Sanders the most honest Independent. Of those who ran for President in the 2016 election, Hillary Clinton, according to PolitiFact’s ratings, was the most honest candidate (Bernie Sanders was a close second), and John Kasich was the most honest Republican (in fact, tied with Bernie Sanders). Barack Obama, interestingly, does not appear on this list.
And now the list of shame: The most dishonest individuals.
Tope 20 Dishonest Entities
Party
Median Honesty
Average Honesty Score
Rated Statements
Chain email
Unaffiliated
Pants on Fire!
0.78
178
Bloggers
Unaffiliated
Pants on Fire!
0.92
72
Democratic Party of Wisconsin
Democrat
False
1.5
24
Ben Carson
Republican
False
1.54
28
Michele Bachmann
Republican
False
1.59
61
Facebook posts
Unaffiliated
False
1.65
100
Herman Cain
Republican
False
1.77
26
Donald Trump
Republican
False
1.82
327
Ken Cuccinelli
Republican
False
2.2
20
Democratic Congressional Campaign Committee
Democrat
Mostly False
1.44
34
National Republican Senatorial Committee
Republican
Mostly False
1.87
30
Allen West
Republican
Mostly False
1.88
26
Paul Broun
Republican
Mostly False
2
19
National Republican Congressional Committee
Republican
Mostly False
2.08
53
Reince Priebus
Republican
Mostly False
2.12
24
Ted Cruz
Republican
Mostly False
2.2
116
Tommy Thompson
Republican
Mostly False
2.26
27
Newt Gingrich
Republican
Mostly False
2.27
77
Republican Party of Florida
Republican
Mostly False
2.29
34
Rick Santorum
Republican
Mostly False
2.32
59
Data source: PolitiFact
I allowed chain e-mails, bloggers, and Facebook posts to appear in this list just to make the following point: they’re full of shit. Go to legitimate news sources to get your information. (In my defense as a blogger, I try to be pretty transparent; judge my honesty as you will.) The two “Democrats” that appear on this list are organizations, the Democratic Party of Wisconsin (is this why Scott Walker is a thing?) and the DCCC. Ben Carson is the most dishonest individual on this list and thus the most dishonest person who ran for President in the 2016 election season, according to PolitiFact. (In Dr. Carson’s defense, though, I don’t know if it’s “dishonesty” per se or just ignorance/stupidity.) Donald Trump, according to PolitiFact, is extremely dishonest, yet somehow Hillary is the corrupt liar.
98 people in PolitiFact’s data had at least 15 ratings, so here is the full list, with rankings provided:
Honesty of Politically Active Entities
Party
Rank
Median Honesty
Average Honesty Score
Rated Statements
Alex Sink
Democrat
1
True
4
19
Dennis Kucinich
Democrat
2
Mostly True
3.84
25
Sheldon Whitehouse
Democrat
3
Mostly True
3.79
24
Cory Booker
Democrat
4
Mostly True
3.6
20
Mark Warner
Democrat
5
Mostly True
3.6
20
Gina Raimondo
Democrat
6
Mostly True
3.59
17
Sherrod Brown
Democrat
7
Mostly True
3.59
34
Rob Portman
Republican
8
Mostly True
3.57
47
Bill Nelson
Democrat
9
Mostly True
3.48
25
David Axelrod
Democrat
10
Mostly True
3.39
18
Hillary Clinton
Democrat
11
Mostly True
3.34
292
Bernie Sanders
Independent
12
Mostly True
3.25
107
John Kasich
Republican
13
Mostly True
3.25
64
Fred Thompson
Republican
14
Mostly True
3.25
16
Bill Richardson
Democrat
15
Mostly True
3.24
17
Alan Grayson
Democrat
16
Mostly True
3.18
34
Bill White
Democrat
17
Mostly True
3.08
26
Tim Kaine
Democrat
18
Half-True
3.38
50
Nathan Deal
Republican
19
Half-True
3.37
49
Bill Clinton
Democrat
20
Half-True
3.29
41
Barack Obama
Democrat
21
Half-True
3.28
596
Jeb Bush
Republican
22
Half-True
3.24
79
John Cornyn
Republican
23
Half-True
3.23
26
Barbara Buono
Democrat
24
Half-True
3.12
16
Charlie Crist
Democrat
25
Half-True
3.12
80
George LeMieux
Republican
26
Half-True
3.12
17
Wendy Davis
Democrat
27
Half-True
3.07
27
David Cicilline
Democrat
28
Half-True
3.07
29
Gary Johnson
Libertarian
29
Half-True
3.06
51
Kay Bailey Hutchison
Republican
30
Half-True
3.06
17
Rand Paul
Republican
31
Half-True
3.04
51
Joe Biden
Democrat
32
Half-True
2.99
75
George Allen
Republican
33
Half-True
2.96
26
Paul Ryan
Republican
34
Half-True
2.95
65
Martin O’Malley
Democrat
35
Half-True
2.94
18
Tim Pawlenty
Republican
36
Half-True
2.94
17
Chris Christie
Republican
37
Half-True
2.93
102
Bob McDonnell
Republican
38
Half-True
2.91
35
Jon Huntsman
Republican
39
Half-True
2.89
18
Tammy Baldwin
Democrat
40
Half-True
2.88
25
John McCain
Republican
41
Half-True
2.88
183
Greg Abbott
Republican
42
Half-True
2.86
43
Ron Paul
Republican
43
Half-True
2.85
40
Marco Rubio
Republican
44
Half-True
2.84
148
Gwen Moore
Democrat
45
Half-True
2.84
19
Kendrick Meek
Democrat
46
Half-True
2.84
19
Lincoln Chafee
Democrat
47
Half-True
2.83
18
Russ Feingold
Democrat
48
Half-True
2.81
21
Debbie Wasserman Schultz
Democrat
49
Half-True
2.81
47
Mitch McConnel
Republican
50
Half-True
2.79
28
Rick Scott
Republican
51
Half-True
2.77
142
Karl Rove
Republican
52
Half-True
2.76
17
Ron Johnson
Republican
53
Half-True
2.73
44
Mitt Romney
Republican
54
Half-True
2.7
206
David Perdue
Republican
55
Half-True
2.69
16
Republican National Committee
Republican
56
Half-True
2.68
34
Scott Walker
Republican
57
Half-True
2.67
172
Mary Burke
Democrat
58
Half-True
2.65
34
Florida Democratic Party
Democrat
59
Half-True
2.64
25
Rudy Giuliani
Republican
60
Half-True
2.6
47
Rick Perry
Republican
61
Half-True
2.59
169
Mike Pence
Republican
62
Half-True
2.58
38
Tom Barrett
Democrat
63
Half-True
2.52
25
Republican Governors Association
Republican
64
Half-True
2.5
18
Harry Reid
Democrat
65
Half-True
2.5
24
Ted Strickland
Democrat
66
Half-True
2.48
21
Nancy Pelosi
Democrat
67
Half-True
2.38
29
John Boehner
Republican
68
Mostly False
2.64
69
Mike Huckabee
Republican
69
Mostly False
2.63
41
Eric Cantor
Republican
70
Mostly False
2.56
34
Dick Cheney
Republican
71
Mostly False
2.53
17
Carly Fiorina
Republican
72
Mostly False
2.45
22
Dan Patrick
Republican
73
Mostly False
2.41
22
Terry McAuliffe
Democrat
74
Mostly False
2.4
30
Josh Mandel
Republican
75
Mostly False
2.39
28
Crossroads GPS
Republican
76
Mostly False
2.37
19
David Dewhurst
Republican
77
Mostly False
2.35
40
Sarah Palin
Republican
78
Mostly False
2.33
39
Rick Santorum
Republican
79
Mostly False
2.32
59
Republican Party of Florida
Republican
80
Mostly False
2.29
34
Newt Gingrich
Republican
81
Mostly False
2.27
77
Tommy Thompson
Republican
82
Mostly False
2.26
27
Ted Cruz
Republican
83
Mostly False
2.2
116
Reince Priebus
Republican
84
Mostly False
2.12
24
National Republican Congressional Committee
Republican
85
Mostly False
2.08
53
Paul Broun
Republican
86
Mostly False
2
19
Allen West
Republican
87
Mostly False
1.88
26
National Republican Senatorial Committee
Republican
88
Mostly False
1.87
30
Democratic Congressional Campaign Committee
Democrat
89
Mostly False
1.44
34
Ken Cuccinelli
Republican
90
False
2.2
20
Donald Trump
Republican
91
False
1.82
327
Herman Cain
Republican
92
False
1.77
26
Facebook posts
Unaffiliated
93
False
1.65
100
Michele Bachmann
Republican
94
False
1.59
61
Ben Carson
Republican
95
False
1.54
28
Democratic Party of Wisconsin
Democrat
96
False
1.5
24
Bloggers
Unaffiliated
97
Pants on Fire!
0.92
72
Chain email
Unaffiliated
98
Pants on Fire!
0.78
178
Data source: PolitiFact
Barack Obama appears 21st on this list followed by Vice President Joe Biden (32), Speaker of the House Paul Ryan (34), Senate Majority Leader Mitch McConnel (50), Senate Minority Leader Harry Reid (65), and