Organizations use intrusion detection systems (IDSes) to identify harmful activity among millions of computer network events. Cybersecurity analysts review IDS alarms to verify whether malicious activity occurred and to take remedial action. However, IDS systems exhibit high false alarm rates. This study examines the impact of IDS false alarm rate on human analyst sensitivity (probability of detection), precision (positive predictive value), and time on task when evaluating IDS alarms. A controlled experiment was conducted with participants divided into two treatment groups, 50% IDS false alarm rate and 86% false alarm rate, who classified whether simulated IDS alarms were true or false alarms. Results show statistically significant differences in precision and time on task. The median values for the 86% false alarm rate group were 47% lower precision and 40% slower time on task than the 50% false alarm rate group. No significant difference in analyst sensitivity was observed.
AxelssonS. (2000). The base-rate fallacy and the difficulty of intrusion detection. ACM Transactions on Information and System Security, 3(3), 186205. doi:10.1145/357830.357849
2.
Ben-AsherN.GonzalezC. (2015). Effects of cyber security knowledge on attack detection. Computers in Human Behavior, 48, 5161. doi:10.1016/j.chb.2015.01.039
3.
BlissJ. P. (2003). An Investigation of Extreme Alarm Response Patterns in Laboratory Experiments. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 47, 16831687. doi:10.1177/154193120304701319
4.
BlissJ. P.GilsonR. D.DeatonJ. E. (1995). Human probability matching behaviour in response to alarms of varying reliability. Ergonomics, 38, 2302312. doi:10.1080/00140139508925269
5.
BustamanteE. A.AndersonB. L.BlissJ. P. (2004). Effects of Varying the Threshold of Alarm Systems and Task Complexity on Human Performance and Perceived Workload. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 48, 19481952. doi:10.1177/154193120404801633
6.
DAmicoA.WhitleyK.TesoneD.OBrienB.RothE. (2005). Achieving Cyber Defense Situational Awareness: A Cognitive Task Analysis of Information Assurance Analysts. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 49, 229233. doi:10.1177/154193120504900304
7.
DurstR.ChampionT.WittenB.SpagnuoloL.MillerE. (1999). Testing and Evaluating Computer Intrusion Detection Systems. Communications of the ACM, 42, 5361. doi:10.1145/306549.306571
8.
DuttV.AhnY.-S.Ben-AsherN.GonzalezC. (2012). Modeling the effects of base-rates on cyber threat detection performance. Proceedings of the 11th International Conference on Cognitive Modeling (ICCM 2012), 8893.
GrardN.ManzeyD. (2010). Are false alarms not as bad as supposed after all? A study investigating operators responses to imperfect alarms. In de WaardD.AxelssonA.BerglundM.PetersB.WeikertC. (Eds.), Human Factors: A system view of human, technology and organisation (pp. 55-69). Shaker Publishing.
11.
GettyD. J.SwetsJ. A.PickettR. M.GonthierD. (1995). System Operator Response to Warnings of Danger: A Laboratory Investigation of the Effects of the Predictive Value of a Warning on Human Response Time. Journal of Experimental Psychology: Applied, 1, 1933. doi:10.1037/1076-898X.1.1.19
12.
GuenzlerT.ManzeyD. (2013). Asymmetries in human tolerance of uncertainty in interaction with alarm systems: Effects of risk perception or evidence for a general commission bias?Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 57, 13621366. doi:10.1177/1541931213571301
13.
HartS. G.StavelandL. E. (1988). Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research. Advances in Psychology, 52, 139183. doi:10.1016/S0166-4115(08)62386-9
14.
JulischK. (2003). Clustering intrusion detection alarms to support root cause analysis. ACM Transactions on Information and System Security, 6, 443471. doi:10.1145/950191.950192
15.
ManganarisS.ChristensenM.ZerkleD.HermizK. (2000). Data mining analysis of RTID alarms. Computer Networks, 34, 571577. doi:10.1016/S1389-1286(00)00138-9
16.
MeyerJ. (2001). Effects of Warning Validity and Proximity on Responses to Warnings. Human Factors, 43, 563572. doi:10.1518/001872001775870395
17.
RiceS. (2009). Examining single- and multiple-process theories of trust in automation. Journal of General Psychology, 136, 303322. doi:10.3200/GENP.136.3.303-322
18.
RodenWilliamLaymanL. (2020). Cry Wolf: Toward an Experimentation Platform and Dataset for Human Factors in Cyber Security Analysis. Proceedings of the 2020 ACM Southeast Conference (ACMSE 2020), 264267. doi:10.1145/3374135.3385301
19.
RodenWilliamLaymanL. (2019a). The Cry Wolf IDS Simulator - An environment for conducting controlled experiments of cyber security analysis tasks. https://uncw-hfcs.github.io/ids-simulator/
RyuY. U.RheeH. S. (2008). Evaluation of intrusion detection systems under a resource constraint. ACM Transactions on Information and System Security, 11, Article 20. doi:10.1145/1380564.1380566
22.
SawyerB. D.HancockP. A. (2018). Hacking the Human: The Prevalence Paradox in Cybersecurity. Human Factors, 60, 597609. doi:10.1177/0018720818780472
23.
SundaramurthyS. C.BardasA. G.CaseJ.OuX.WeschM.McHughJ.RajagopalanS. R. (2015). A Human Capital Model for Mitigating Security Analyst Burnout. Proceedings of Eleventh Symposium On Usable Privacy and Security (SOUPS 2015), 11, 347359.
24.
WendtH. W. (1972). Dealing with a common problem in Social science: A simplified rankbiserial coefficient of correlation based on the U statistic. European Journal of Social Psychology, 2, 463465. doi:10.1002/EJSP.2420020412
25.
WiczorekR.ManzeyD. (2010). Is Operators Compliance with Alarm Systems a Product of Rational Consideration?Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 54, 17221726. doi:10.1177/15419312100540197
26.
ZhongC.LinT.LiuP.YenJ.ChenK. (2018). A cyber security data triage operation retrieval system. Computers and Security, 76, 1231. doi:10.1016/J.COSE.2018.02.011
