Civil unrest, encompassing protests and riots, is an increasing global concern, with incidents rising at an alarming rate, a trend that has been observed in South Africa over the years. This issue is particularly pronounced in today’s social media era, where platforms like ‘X’ (formerly Twitter) serve as powerful tools for mobilization. This raises the question: What factors drive civil unrest, and how can machine learning, using social media data, be employed to forecast such events? In response, this study had as objective to develop a hybrid machine learning model to forecast protest and riot events in South Africa using Twitter data. Employing the CRISP-DM methodology, data was collected from Twitter for the period between 2019 and 2024, resulting in 18,487 curated tweets, with associated ground truth data extracted from the ACLED database. Using this data, a hybrid model combining Bidirectional LSTM (Bi-LSTM) networks with eXtreme Gradient Boosting (XGBoost) for classification and regression tasks was developed to forecast civil unrest in South Africa. Additionally, SHapley Additive exPlanations (SHAP) were used for model explainability. The proposed model outperformed the base model, achieving an R-squared value of 33% for protests and 23% for riots in regression, along with classification accuracies of 92% for protests and 86.2% for riots. SHAP results indicated that the key predictors of unrest included sentiment-related features, tweet engagement features, regional factors, the day of the week, public holidays, and the topics being discussed. This study demonstrates the value of a hybrid model in forecasting civil unrest events and identifies key features that stakeholders can use to target their efforts more precisely in addressing civil unrest, ensuring resources are allocated where they are needed most. The study concludes with a discussion of valuable insights for stakeholders on how to leverage social media data to predict and mitigate civil unrest.
ApicellaA.IsgròF.PreveteR. (2024). Don’t push the button! Exploring data leakage risks in machine learning and transfer learning. SSRN. https://doi.org/10.2139/ssrn.4733889
3.
BahramiM.FindikY.BozkayaB.BalcisoyS. (2018). Twitter reveals: Using twitter analytics to predict public protests. arXiv. https://arxiv.org/abs/1805.00358
4.
BakermanJ.PazdernikK.KorkmazG.WilsonA. G. (2022). Dynamic logistic regression and variable selection: Forecasting and contextualizing civil unrest. International Journal of Forecasting, 38(2), 648–661. https://doi.org/10.1016/j.ijforecast.2021.07.003
5.
BasedauM.RustadS. A.MustE. (2018). Do expectations on oil discoveries affect civil unrest? Micro-Level evidence from Mali. Cogent Social Sciences, 4(1), 1470132. https://doi.org/10.1080/23311886.2018.1470132
6.
BekkerM. (2022). Better, faster, stronger: Using machine learning to analyse South African police-recorded protest data. South African Review of Sociology, 52(1), 4–23. https://doi.org/10.1080/21528586.2021.1982762
7.
BertholdM.BorgeltC.HöppnerF.KlawonnF.SilipoR. (2020). Data understanding. In Orit Hazzan, Frank Maurer (Eds.). Guide to Intelligent Data Science. Texts in Computer Science (pp. 33–83). Springer. https://doi.org/10.1007/978-3-030-45574-3_4
ChintaA.ZhangJ.DeLuciaA.DredzeM.BuczakA. L. (2021). Study of manifestation of civil unrest on twitter. In Proceedings of the seventh workshop on noisy user-generated text (W-NUT 2021) (pp. 396–409). Association for Computational Linguistics. https://doi.org/10.18653/v1/2021.wnut-1.44
10.
CollellG.PrelecD.PatilK. R. (2018). A simple plug-in bagging ensemble based on threshold-moving for classifying binary and multiclass imbalanced data. Neurocomputing, 275(31), 330–340. https://doi.org/10.1016/j.neucom.2017.08.035
11.
ComptonR.LeeC.XuJ.Artieda-MoncadaL.LuT.-C.SilvaL. D.MacyM. (2014). Using publicly visible social media to build detailed forecasts of civil unrest. Security Informatics, 3(1), 4. https://doi.org/10.1186/s13388-014-0004-6
12.
DaiY.ZhouQ.LengM.YangX.WangY. (2022). Improving the Bi-LSTM model with XGBoost and attention mechanism: A combined approach for short-term power load prediction. Applied Soft Computing, 130, 109632. https://doi.org/10.1016/j.asoc.2022.109632
13.
Da SilvaD. G.MenesesA. A. D. M. (2023). Comparing Long Short-Term Memory (LSTM) and bidirectional LSTM deep neural networks for power consumption prediction. Energy Reports, 10, 3315–3334. https://doi.org/10.1016/j.egyr.2023.09.175
14.
De JuanA.WegnerE. (2017). Social inequality, state-centered grievances, and protest: Evidence from South Africa. Journal of Conflict Resolution, 63(1), 31–58. https://doi.org/10.1177/0022002717723136
15.
DengS.NingY. (2021). A Survey on societal event Forecasting with deep learning(No. arXiv:2112.06345). arXiv. https://arxiv.org/abs/2112.06345
16.
DinR. U.AhmedS.KhanS. H. (2024). A novel decision ensemble framework: Customized attention-BiLSTM and XGBoost for speculative stock price forecasting. ArXiv, abs/2401.11621. https://api.semanticscholar.org/CorpusID:267068521
17.
DuncanJ. (2020). South Africa’s doctrinal decline on the right to protest: Notification requirements and the shift from fundamental right to national security threat. Constitutional Court Review, 10(1), 227–249. https://doi.org/10.2989/CCR.2020.0009
18.
EchriguiR.HamicheM. (2023). Optimizing LSTM models for EUR/USD prediction in the context of reducing energy consumption: An analysis of mean squared error, mean absolute error and R-squared. E3S Web of Conferences, 412, 01069. https://doi.org/10.1051/e3sconf/202341201069
19.
ErtugrulA. M.LinY.-R.ChungW.-T.YanM.LiA. (2019). Activism via attention: Interpretable spatiotemporal learning to forecast protest activities. EPJ Data Science, 8(1), 5. https://doi.org/10.1140/epjds/s13688-019-0183-y
Gomez-CraviotoD. A.Diaz-RamosR. E.Hernandez-GressN.PreciadoJ. L.CeballosH. G. (2022). Supervised machine learning predictive analytics for alumni income. Journal of Big Data, 9(1), 11. https://doi.org/10.1186/s40537-022-00559-6
22.
GregoruttiB.MichelB.Saint-PierreP. (2017). Correlation and variable importance in random forests. Statistics and Computing, 27(3), 659–678. https://doi.org/10.1007/s11222-016-9646-1
23.
GuoM.WangY.YangQ.LiR.ZhaoY.LiC.ZhuM.CuiY.JiangX.ShengS.LiQ.GaoR. (2023). Normal workflow and key strategies for data cleaning toward real-world data: Viewpoint. Interactive Journal of Medical Research, 12, Article e44310. https://doi.org/10.2196/44310
24.
HodsonT. O. (2022). Root-mean-square error (RMSE) or mean absolute error (MAE): When to use them or not. Geoscientific Model Development, 15(14), 5481–5487. https://doi.org/10.5194/gmd-15-5481-2022
25.
HossainK. S. M. T.HarutyunyanH.NingY.KennedyB.RamakrishnanN.GalstyanA. (2022). Identifying geopolitical event precursors using attention-based LSTMs. Frontiers in Artificial Intelligence, 5, 893875. https://doi.org/10.3389/frai.2022.893875
26.
HuangN.LuG.XuD. (2016). A permutation importance-based feature selection method for short-term electricity load forecasting using random forest. Energies, 9(10), 767. https://doi.org/10.3390/en9100767
27.
HuttoC.GilbertE. (2014). VADER: A parsimonious rule-based model for sentiment analysis of social media text. Proceedings of the International AAAI Conference on Web and Social Media, 8(1), 216–225. https://doi.org/10.1609/icwsm.v8i1.14550
28.
IydaJ. J.GeethaP. (2022). An improved deep belief neural network based civil unrest event forecasting in twitter. Applied Intelligence, 53, 5714–5731. https://doi.org/10.1007/s10489-022-03746-3
29.
JohnsonJ. M.KhoshgoftaarT. M. (2019). Deep learning and thresholding with class-imbalanced big data. In 2019 18th IEEE International Conference on Machine Learning and Applications (ICMLA), Boca Raton, FL, USA, 16–19 December 2019, pp. 755–762. https://doi.org/10.1109/ICMLA.2019.00134
30.
KaritaS.ChenN.HayashiT.HoriT.InagumaH.JiangZ.SomekiM.SoplinN. E. Y.YamamotoR.WangX.WatanabeS.YoshimuraT.ZhangW. (2019). A comparative study on transformer vs RNN in speech applications. In 2019 IEEE Automatic Speech Recognition and Understanding workshop (ASRU), Singapore, 14–18 December 2019, pp. 449–456. https://doi.org/10.1109/ASRU46091.2019.9003750
31.
KorkmazG.CadenaJ.KuhlmanC. J.MaratheA.VullikantiA.RamakrishnanN. (2015). Combining heterogeneous data sources for civil unrest forecasting. In Proceedings of the 2015 IEEE/ACM international conference on advances in social networks analysis and mining 2015, Paris, France, 25–28 August 2015, pp. 258–265. https://doi.org/10.1145/2808797.2808847
32.
KorkmazG.CadenaJ.KuhlmanC. J.MaratheA.VullikantiA.RamakrishnanN. (2016). Multi-source models for civil unrest forecasting. Social Network Analysis and Mining, 6(1), 50. https://doi.org/10.1007/s13278-016-0355-8
33.
KorolovR.LuD.WangJ.ZhouG.BonialC.VossC.KaplanL.WallaceW.HanJ.JiH. (2016). On predicting social unrest using social media. In 2016 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM), San Francisco, CA, USA, 18–21 August 2016, pp. 89–95. https://doi.org/10.1109/ASONAM.2016.7752218
34.
KumarA. (2021). Predicting future, past, and misinterpreted COVID-19 cases using bidirectional LSTM model for proper health monitoring: Advances in data analytics. In MalikH.FatemaN.AlzubiJ. A. (Eds.), AI and machine learning paradigms for health monitoring system: Intelligent data analytics (pp. 155–168). Springer. https://doi.org/10.1007/978-981-33-4412-9_9
35.
LundbergS. M.LeeS.-I. (2017). A unified approach to interpreting model predictions. In Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS'17) (pp. 4768–4777). Curran Associates Inc. https://doi.org/10.5555/3295222.3295230
36.
LuoQ.ZengW.ChenM.PengG.YuanX.YinQ. (2023). Self-attention and transformers: Driving the evolution of large language models. In 2023 IEEE 6th International Conference on Electronic Information and Communication Technology (ICEICT), Qingdao, China, 21–24 July 2023, pp. 401–405. https://doi.org/10.1109/ICEICT57916.2023.10245906
37.
MaarifM. R.SalehA. R.HabibiM.FitriyaniN. L.SyafrudinM. (2023). Energy usage forecasting model based on long short-term memory (LSTM) and eXplainable Artificial Intelligence (XAI). Information (Basel), 14(5), 265. https://doi.org/10.3390/info14050265
MongaleC. O. (2022). Social discontent or criminality? Navigating the nexus between urban riots and criminal activities in Gauteng and KwaZulu-natal provinces, South Africa (2021). Frontiers in Sustainable Cities, 4, 865255. https://doi.org/10.3389/frsc.2022.865255
NahmF. S. (2022). Receiver operating characteristic curve: Overview and practical use for clinicians. Korean Journal of Anesthesiology, 75(1), 25–36. https://doi.org/10.4097/kja.21209
43.
NaidooK.LewisS.EssopH.KochG. G. V.KhozaT. E.PhahlamohlakaN. M.BadriparsadN. R. (2023). 2021 civil unrest: South African diagnostic radiography students’ experiences. Health SA Gesondheid, 28, a2253. https://doi.org/10.4102/hsag.v28i0.2253
44.
NarayanK. R.MookherjiS.OdeluV.PrasathR.TurlapatyA. C.DasA. K. (2023). IIDS: Design of intelligent intrusion detection system for internet-of-things applications. In 2023 IEEE 7th Conference on Information and Communication Technology (CICT), pp. 1–6. https://doi.org/10.1109/CICT59886.2023.10455720
45.
NodangalaN. Z.MasumbeP. S. (2023). A narrative of Marikana tragedy against right to life and right to human dignity under the South African constitution. E-Journal of Humanities, Arts and Social Sciences, 4(10), 1297–1306. https://doi.org/10.38159/ehass.202341012
46.
OladeleT. M.AyetiranE. F. (2023). Social unrest prediction through sentiment analysis on Twitter using support vector machine: Experimental Study on Nigeria’s #EndSARS. Open Information Science, 7(1), 20220141. https://doi.org/10.1515/opis-2022-0141
47.
OzsahinD. U.AmeenZ. S.HassanA. S.MubarakA. S. (2024). Enhancing explainable SARS-CoV-2 vaccine development leveraging bee colony optimised Bi-LSTM, Bi-GRU models and bioinformatic analysis. Scientific Reports, 14(1), 6737. https://doi.org/10.1038/s41598-024-55762-7
48.
PapageorgiouH.BaggottJ.BekkerM. (2024). Protest events have a “twitter signature”: Evidence from South Africa’s #FeesMustFall. South African Crime Quarterly, 73(2), 11–24. https://doi.org/10.17159/2413-3108/2024/vn73a16815
49.
PhungulaN. (2024). Understanding the dynamics of South Africa’s July 2021 social unrest. Journal of Nation-Building & Policy Studies, 8(1), 71–87. https://doi.org/10.31920/2516-3132/2023/v8n1a4
50.
PurwandariK.NurlailaI. (2022). Sequential topic modelling: A case study on one health conversation on twitter. In 2022 5th International Seminar on Research of Information Technology and Intelligent Systems (ISRITI), Yogyakarta, Indonesia, 08–09 December 2022, pp. 457–461. https://doi.org/10.1109/ISRITI56927.2022.10052987
51.
RaleighC.KishiR.LinkeA. (2023). Political instability patterns are obscured by conflict dataset scope conditions, sources, and coding choices. Humanities and Social Sciences Communications, 10(1), 74. https://doi.org/10.1057/s41599-023-01559-4
52.
RamakrishnanN.ButlerP.MuthiahS.SelfN.KhandpurR.SarafP.WangW.CadenaJ.VullikantiA.KorkmazG.KuhlmanC.MaratheA.ZhaoL.HuaT.ChenF.LuC. T.HuangB.SrinivasanA.TrinhK.GetoorL.KatzG.DoyleA.AckermannC.ZavorinI.FordJ.SummersK.FayedY.ArredondoJ.GuptaD.MaresD. (2014). ‘Beating the news' with EMBERS: Forecasting civil unrest using open source indicators. In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 1799–1808). Association for Computing Machinery. https://doi.org/10.1145/2623330.2623373
53.
ReyesM.MeierR.PereiraS.SilvaC. A.DahlweidF.-M.Tengg-KobligkH. V.SummersR. M.WiestR. (2020). On the interpretability of artificial intelligence in radiology: Challenges and opportunities. Radiology. Artificial Intelligence, 2(3), Article e190043. https://doi.org/10.1148/ryai.2020190043
54.
RomeoG. (2020). Chapter 13—data analysis for business and economics. In RomeoG. (Ed.), Elements of numerical mathematical economics with excel (pp. 695–761). Academic Press. https://doi.org/10.1016/B978-0-12-817648-1.00013-X
55.
RuncimanC.AlexanderP.RampediM.MolotoB.MarupingB.KhumaloE.SibandaS. (2016). Counting police-recorded protests: Based on South African police service data. South African Research Chair in Social Change Report #2. https://doi.org/10.13140/RG.2.1.1191.8962
56.
SaitoT.RehmsmeierM. (2015). The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets. PLoS One, 10(3), Article e0118432. https://doi.org/10.1371/journal.pone.0118432
57.
SherstinskyA. (2020). Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) network. Physica D: Nonlinear Phenomena, 404, 132306. https://doi.org/10.1016/j.physd.2019.132306
VaswaniA.ShazeerN.ParmarN.UszkoreitJ.JonesL.GomezA. N.KaiserL.PolosukhinI. (2023). Attention is all you need(No. arXiv:1706.03762). arXiv. https://doi.org/10.48550/arXiv.1706.03762
61.
VerdonckT.BaesensB.ÓskarsdóttirM.Vanden BrouckeS. (2024). Special issue on feature engineering editorial. Machine Learning, 113(7), 3917–3928. https://doi.org/10.1007/s10994-021-06042-2
62.
VujovicŽ. Ð. (2021). Classification model evaluation metrics. International Journal of Advanced Computer Science and Applications, 12(6), 599–606. https://doi.org/10.14569/IJACSA.2021.0120670
63.
WubetY. A.LianK.-Y. (2024). How can we detect news surrounding community safety crisis incidents in the internet? Experiments using attention-based Bi-LSTM models. International Journal of Information Management Data Insights, 4(1), 100227. https://doi.org/10.1016/j.jjimei.2024.100227
ZadehM. H.CicekliI. (2023). Protest event analysis: A new method based on Twitter’s user behaviors. Information Technology and Control, 52(2), 457–470. https://doi.org/10.5755/j01.itc.52.2.33077
66.
ZambeziS. (2021). Predicting social unrest events in South Africa using LSTM neural networks [Master’s thesis, University of Cape Town]. https://hdl.handle.net/11427/33986
67.
ZhaoL.GaoY.YeJ.ChenF.YeY.LuC.-T.RamakrishnanN. (2022). Spatio-temporal event forecasting using incremental multi-source feature learning. ACM Transactions on Knowledge Discovery from Data, 16(2), 1–28. https://doi.org/10.1145/3464976
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
