In this article, we examine the relationship between metrics documenting politics-related Twitter activity with election results and trends in opinion polls. Various studies have proposed the possibility of inferring public opinion based on digital trace data collected on Twitter and even the possibility to predict election results based on aggregates of mentions of political actors. Yet, a systematic attempt at a validation of Twitter as an indicator for political support is lacking. In this article, building on social science methodology, we test the validity of the relationship between various Twitter-based metrics of public attention toward politics with election results and opinion polls. All indicators tested in this article suggest caution in the attempt to infer public opinion or predict election results based on Twitter messages. In all tested metrics, indicators based on Twitter mentions of political parties differed strongly from parties’ results in elections or opinion polls. This leads us to question the power of Twitter to infer levels of political support of political actors. Instead, Twitter appears to promise insights into temporal dynamics of public attention toward politics.
AdcockR.CollierD. (2001). Measurement validity: A shared standard for qualitatitve and quantitative research. The American Political Science Review, 95, 529–546.
2.
AsurS.HubermanB. A. (2010). Predicting the future with social media. In HuangX. J.KingI.RaghavanV.RuegerS. (Eds.), WI-IAT 2010: Proceedings of the 2010 IEEE/WIC/ACM international conference on web intelligence and intelligent agent technology (pp. 492–499). Washington, DC: IEEE.
BerminghamA.SmeatonA. F. (2011). On using Twitter to monitor political sentiment and predict election results. In BandyopadhyayS.OkumuraM., SAAIP: Proceedings of the workshop sentiment analysis where AI meets psychology at the international joint conference for natural language processing (IJCNLP) (pp. 2–10). Chiang Mai, Thailand: Asian Federation of Natural Language Processing.
5.
BollenJ.MaoH.ZengX. J. (2011). Twitter mood predicts the stock market. Journal of Computational Science, 2, 1–8.
6.
CarminesE. G.ZellerR. A. (1979). Reliability and validity assessment. Beverly Hills, CA: Sage.
7.
CeronA.CuriniL.IacusS. M. (2014). Every tweet counts? How sentiment analysis of social media can improve our knowledge of citizens’ political preferences with an application to Italy and France. New Media & Society, 16, 340–358.
8.
CeronA.CuriniL.IacusS. M. (2015). Using sentiment analysis to monitor electoral campaigns: Method matters—evidence from the United States and Italy. Social Science Computer Review, 33, 3–20.
9.
ChaffeeS. H.MetzgerM. J. (2001). The end of mass communication?Mass Communication & Society, 4, 365–379.
10.
ChakrabartiD.PuneraK. (2011). Event summarization using tweets. In ICWSM 2011: Proceedings of the 5th international AAAI conference on weblogs and social media (pp. 66–73). Menlo Park, CA: Association for the Advancement of Artificial Intelligence (AAAI).
ContractorD.FaruquieT. A. (2013). Understanding election candidate approval ratings using social media data. In SchwabeD.AlmeidaV.GlaserH.Baeza-YatesR.MoonS. (Eds.), WWW 2013: Proceedings of the 22nd international conference on World Wide Web (pp. 189–190). Geneva, Switzerland: International World Wide Web Conferences Steering Committee.
13.
CronbachL. J.MeehlP. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52, 281–302.
DiGraziaJ.McKelveyK.BollenJ.RojasF. (2013). More tweets, more votes: Social media as a quantitative indicator of political behavior. PLoS One, 8, e79449.
FinkC.BosN.PerroneA.LiuE.KopeckyJ. (2013). Twitter, public opinion, and the 2011 Nigerian Presidential election. In BilofR. (Ed.), SocialCom 2013: The 5th IEEE international conference on social computing (pp. 311–320). Washington, DC: IEEE.
18.
FranchF. (2013). (Wisdom of the crowds)2: 2010 UK election prediction with social media. Journal of Information Technology & Politics, 10, 57–71.
19.
GauravM.SrivastavaA.KumarA.MillerS. (2013). Leveraging candidate popularity on Twitter to predict election outcome. In GrossmanR. L.UthurusamyR. (Eds.), SNAKDD 2013: Proceedings of the 7th workshop on social network mining and analysis (Article no. 7). New York, NY: ACM.
20.
Gayo-AvelloD. (2011). Don’t turn social media into another ‘Literary Digest’ poll. Communications of the ACM, 54, 121–128.
21.
Gayo-AvelloD. (2012). No, you cannot predict elections with Twitter. IEEE Internet Computing, 16, 91–94.
22.
Gayo-AvelloD. (2013). A meta-analysis of state-of-the-art electoral prediction from Twitter data. Social Science Computer Review, 31, 649–679.
23.
GerringJ. (2012). Social science methodology (2nd ed.). New York, NY: Cambridge University Press.
González-BailónS.PaltoglouG. (2015). Signals of public opinion in online communication: A comparison of methods and data sources. The ANNALS of the American Academy of Political and Social Science, 659, 95–107.
26.
HandD. J. (2004). Measurement theory and practice. London, UK: Wiley.
27.
HopkinsD.KingG. (2010). A method of automated nonparametric content analysis for social science. American Journal of Political Science, 54, 229–247.
28.
HowisonJ.WigginsA.CrowstonK. (2011). Validity issues in the use of social network analysis with digital trace data. Journal of the Association for Information Systems, 12, 767–797.
29.
HubertyM. (2015). Can we vote with our tweet? On the perennial difficulty of election forecasting with social media. International Journal of Forecasting, 31, 992–1007.
30.
HyndmanR. J.KoehlerA. B. (2006). Another look at measures of forecast accuracy. International Journal of Forecasting, 22, 679–688.
31.
JensenM. J.AnsteadN. (2013). Psephological investigations: Tweets, votes, and unknown unknowns in the republican nomination process. Policy & Internet, 5, 161–182.
32.
JungherrA. (2013). Tweets and votes, a special relationship: The 2009 federal election in Germany. In WeberI.PopescA. M.PennacchiottiM. (Eds.), PLEAD 2013: Proceedings of the 2nd workshop politics, elections and data (pp. 5–14). New York, NY: ACM.
33.
JungherrA. (2014). The logic of political coverage on Twitter: Temporal dynamics and content. Journal of Communication, 64, 239–259.
34.
JungherrA. (2015). Analyzing political communication with digital trace data: The role of Twitter messages in social science research. Heidelberg, Germany: Springer.
35.
JungherrA.JürgensP.SchoenH. (2012). Why the pirate party won the German election of 2009 or the trouble with predictions: A response to Tumasjan, A., Sprenger, T.O., Sander, P.G. & Welpe, I.M. “Predicting elections with Twitter: what 140 characters reveal about political sentiment.””Social Science Computer Review, 30, 229–234.
36.
JungherrA.SchoenH.JürgensP. (2016). The mediation of politics through Twitter: An analysis of messages posted during the campaigns for the German federal elections 2009 and 2013. Journal of Computer-mediated Communication, 21, 50–68.
37.
KaneM. T. (1992). An argument-based approach to validity. Psychological Bulletin, 112, 527–535.
38.
LazerD.KennedyR.KingG.VespignaniA. (2014). The parable of Google flu: Traps in big data analysis. Science, 343, 1203–1205.
39.
LazerD.PentlandA.AdamicL.AralS.BarabásiA. L.BrewerD.…AlstyneM. V. (2009). Computational social science. Science, 323, 721–723.
40.
LinY. R.MargolinD.KeeganB.LazerD. (2013). Voices of victory: A computational focus group framework for tracking opinion shift in real time. In SchwabeD.AlmeidaV.GlaserH.Baeza-YatesR.MoonS. (Eds.), WWW 2013: Proceedings of the 22nd international world wide web conference (pp. 737–748). Geneva, Switzerland: International World Wide Web Conferences Steering Committee.
41.
MacKuenM. B.EriksonR. S.StimsonJ. A. (1989). Macropartisanship. American Political Science Review, 83, 1125–1142.
42.
Marchetti-BowickM.ChambersN. (2012). Learning for microblogs with distant supervision: Political forecasting with Twitter. In DaelemansW. (Ed.), EACL ‘12: Proceedings of the 13th conference of the European chapter of the association for computational linguistics (pp. 603–612). Stroudsburg, PA: Association for Computational Linguistics.
43.
MargolisM.ResnickD. (2000). Politics as usual: The Cyberspace “revolution.”Thousand Oaks, CA: Sage.
44.
McKelveyK.DiGraziaJ.RojasF. (2014). Twitter publics: How online political communities signal electoral outcomes in the 2010 U.S. House election. Information, Communication & Society, 17, 436–450.
45.
MessickS. (1989). Meaning and values in test validation: The science and ethics of assessment. Educational Researcher, 18, 5–11.
46.
MetaxasP. T.MustafarajE.Gayo-AvelloD. (2011). How (not) to predict elections. In VinciarelliA.PanticM.BertinoE.ZhanJ. (Eds.), SocialCom 2011: The 3rd IEEE international conference on social computing (pp. 165–171). Washington, DC: IEEE.
47.
MurphyJ.LinkM. W.ChildsJ. H.TesfayeC. L.DeanE.SternM.…HarwoodP. (2014). Social media in public opinion research: Executive summary of the aapor task force on emerging technologies in public opinion research. Public Opinion Quarterly, 78, 788–794.
48.
O’ConnorB.BalasubramanyanR.RoutledgeB. R.SmithN. A. (2010). From tweets to polls: Linking text sentiment to public opinion time series. In HearstM.CohenW.GoslingS. (Eds.), ICWSM 2010: Proceedings of the 4th international AAAI conference on weblogs and social media. Menlo Park, CA: Association for the Advancement of Artificial Intelligence (AAAI).
R Core Team. (2014). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. Retrieved fromhttp://www.R-project.org
51.
RuthsD.PfefferJ. (2014). Social media for large studies of behavior. Science, 346, 1063–1064.
52.
SakakiT.OkazakiM.MatsuoY. (2010). Earthquake shakes twitter users: Real-time event detection by social sensors. In RappaM.JonesP.FreireJ.ChakrabartiS. (Eds.), WWW 2010: Proceedings of the 19th international conference on the World Wide Web (pp. 851–860). New York, NY: ACM.
53.
SartoriG. (1970). Concept misformation in comparative politics. The American Political Science Review, 64, 1033–1053.
54.
SchoenH.Gayo-AvelloD.MetaxasP. T.StrohmaierM.GloorP. (2013). The power of prediction with social media. Internet Research, 23, 528–543.
55.
SkoricM.PoorN.AchananuparpP.LimE. P.JiangJ. (2012). Tweets and votes: A study of the 2011 Singapore general election. In SpragueR. H.Jr (Ed.), HICSS 2012: Proceedings of the 45th Hawaii international conference on system science (pp. 2583–2591). Washington, DC: IEEE Computer Society.
56.
SolerJ. M.CuarteroF.RoblizoM. (2012). Twitter as a tool for predicting elections results. In KarrayF.PolatF. (Eds.), ASONAM 2012: IEEE/ACM international conference on advances in social networks analysis and mining (pp. 1194–1200). Los Alamitos, CA: IEEE Press.
57.
ThapenN. A.GhanemM. M. (2013). Towards passive political opinion polling using Twitter. In CoceaM.GabeM.WiratungaN.GökerA. (Eds.), Proceedings of the BCS SGAI workshop on social media analysis 2013 (pp. 19–34). Aberdeen, UK: School of Computing Science and Digital Media, Robert Gordon University.
58.
TjongE.SangK.BosJ. (2012). Predicting the 2011 Dutch Senate election results with Twitter. In Proceedings of the workshop on semantic analysis in social media (pp. 53–60). Stroudsburg, PA: Association for Computational Linguistics.
59.
TourangeauR.RipsL. J.RasinskiK. (2000). The psychology of survey response. Cambridge, England: Cambridge University Press.
60.
TsakalidisA.PapadopoulosS.CristeaA. I.KompatsiarisY. (2015). Predicting elections for multiple countries using Twitter and polls. IEEE Intelligent Systems, 30, 10–17.
61.
TumasjanA.SprengerT. O.SandnerP. G.WelpeI. M. (2010). Predicting elections with Twitter: What 140 characters reveal about political sentiment. In HearstM.CohenW.GoslingS. (Eds.), ICWSM 2010: Proceedings of the 4th international AAAI conference on weblogs and social media (pp. 178–185). Menlo Park, CA: Association for the Advancement of Artificial Intelligence (AAAI).
62.
VaccariC.ValerianiA.BarberáP.BonneauR.JostJ. T.NaglerJ.TuckerJ. (2013). Social media and political communication: A survey of Twitter users during the 2013 Italian general election.”Rivista Italiana di Scienza, 43, 325–55.
WhiteleyP.ClarkeH. D.SandersD.StewartM. C. (2013). Affluence, austerity and electoral change in Britain. Cambridge, England: Cambridge University Press.
65.
WickhamH. (2009). ggplot2: Elegant graphics for data analysis. New York, NY: Springer.
66.
All links were accessible on October 18, 2015.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
