Multi-agent systems are proposed as a solution to mitigate nowadays software requirements: open and distributed architectures with dynamic and adaptive behaviour. Like any other software, multi-agent systems development process is error-prone; thus testing is a key activity to ensure the quality of the developed product. This paper sheds light on agent testing as it is the primary artefact for any multi-agent system’s testing process. A framework called JADE Testing Framework (JTF) for JADE platform’s agent testing is proposed. JTF allows testing agents at two levels: unit (inner-components) and agent (agent interactions) levels. JTF is the result of the integration of two testing solutions: JAT a well-known framework for JADE’s agent’s interaction testing and UJade, a new solution that was developed for agent’s unit testing. UJade provides also a toolbox that allows for enhancing JAT capabilities. The evidence of JTF usability and effectiveness in JADE agent testing was supported by an empirical study conducted on seven multi-agent systems. The results of the study show that: when an agent’s code can be tested either at agent or unit levels UJade is less test’s effort consuming than JAT; JTF provides better testing capabilities and the developed tests are more effective than those developed using UJade or JAT alone.
BakarN.A. and SelamatA., Runtime Verification of Multi-agent Systems Interaction Quality, in: Intelligent Information and Database Systems, Berlin, Heidelberg, 2013, pp. 435–444. doi: 10.1007/978-3-642-36546-1_45.
2.
BakarN.A. and SelamatA., Agent systems verification: Systematic literature review and mapping, Appl. Intell.48(5) (May 2018), 1251–1274. doi: 10.1007/s10489-017-1112-z.
3.
BellifemineF.L.CaireG. and GreenwoodD., Developing multi-agent systems with JADE, vol. 7. John Wiley & Sons, 2007.
4.
Bharat KumarT.HarishN. and Sravan KumarV., An Catholic and Enhanced Study on Basis Path Testing to Avoid Infeasible Paths in CFG, in: Global Trends in Information Systems and Software Applications, vol. 270KrishnaP.V.BabuM.R. and AriwaE., Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2012, pp. 386–395. doi: 10.1007/978-3-642-29216-3_42.
5.
BringsJ.DaunM.KellerK.Aluko ObeP. and WeyerT., A systematic map on verification and validation of emergent behavior in software engineering research, Future Gener. Comput. Syst.112 (Nov. 2020), 1010–1037. doi: 10.1016/j.future.2020.06.049.
6.
BriolaD.MascardiV. and AnconaD., Distributed Runtime Verification of JADE Multiagent Systems, in: Intelligent Distributed Computing VIII, Cham, 2015, pp. 81–91. doi: 10.1007/978-3-319-10422-5_10.
7.
BruntinkM. and van DeursenA., An empirical study into class testability, J. Syst. Softw.79(9) (2006), 1219–1232. doi: 10.1016/j.jss.2006.02.036.
CaireG.CossentinoM.NegriA.PoggiA. and TurciP., Multi-Agent Systems Implementation and Testing, Vienna, Austria, 2004, 11.
10.
CarreraÁ.IglesiasC.A. and GarijoM., Beast methodology: An agile testing methodology for multi-agent systems based on behaviour driven development, Inf. Syst. Front.16(2) (Apr. 2014), 169–182. doi: 10.1007/s10796-013-9438-5.
11.
ChekamT.T.PapadakisM.Le TraonY. and HarmanM., An Empirical Study on Mutation, Statement and Branch Coverage Fault Revelation That Avoids the Unreliable Clean Program Assumption, in: 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE), Buenos Aires, May 2017, pp. 597–608. doi: 10.1109/ICSE.2017.61.
12.
ChellaA.CossentinoM.SabatucciL. and SeiditaV., From passi to agile passi: Tailoring a design process to meet new needs, in: Proceedings. IEEE/WIC/ACM International Conference on Intelligent Agent Technology, 2004. (IAT 2004)., 2004, pp. 471–474. doi: 10.1109/IAT.2004.1342998.
13.
CoelhoR.CiriloE.KuleszaU.von StaaA.RashidA. and LucenaC., JAT: A Test Automation Framework for Multi-Agent Systems, in: 2007 IEEE International Conference on Software Maintenance, 2007, pp. 425–434. doi: 10.1109/ICSM.2007.4362655.
14.
CoelhoR.KuleszaU.von StaaA. and LucenaC., Unit testing in multi-agent systems using mock agents and aspects, in: Proceedings of the 2006 International Workshop on Software Engineering for Large-Scale Multi-Agent Systems – SELMAS ’06, Shanghai, China, 2006, p. 83. doi: 10.1145/1138063.1138079.
15.
ColesH.LaurentT.HenardC.PapadakisM. and VentresqueA., PIT: a practical mutation testing tool for Java (demo), in: Proceedings of the 25th International Symposium on Software Testing and Analysis, New York, NY, USA, Jul. 2016, pp. 449–452. doi: 10.1145/2931037.2948707.
16.
CorteseE.CaireG. and BochicchioR., JADE Test Suite User Guide, TILab, 2005. [Online]. Available: https://jade.tilab.com/doc/tutorials/JADE_TestSuite.pdf.
17.
CossentinoM.SabatucciL. and ChellaA., Designing JADE systems with the support of CASE tools and patterns, Spec. Issue JADE Telecom Ital. J. EXP Sept., 2003.
18.
CunhaF.Diniz Da CostaA.VianaM. and Pereira De LucenaC.J., JAT4BDI: An Aspect-Based Approach for Testing BDI Agents, in: 2015 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT), vol. 2, Dec. 2015, pp. 186–189. doi: 10.1109/WI-IAT.2015.121.
19.
DehimiN.E.H.MokhatiF. and BadriM., Testing HMAS-based applications: An ASPECS-based approach, Eng. Appl. Artif. Intell.46 (Nov. 2015), 232–257. doi: 10.1016/j.engappai.2015.09.013.
20.
DeMilloR.A.LiptonR.J. and SaywardF.G., Hints on test data selection: Help for the practicing programmer, Computer11(4) (Apr. 1978), 34–41. doi: 10.1109/C-M.1978.218136.
21.
DixJ.HindriksK.LoganB. and WobckeW., Engineering Multi-Agent Systems (Dagstuhl Seminar 12342), Dagstuhl Rep.2(8) (2012), 74–98. doi: 10.4230/DagRep.2.8.74.
22.
EckM.PalombaF.CastelluccioM. and BacchelliA., Understanding Flaky Tests: The Developer’s Perspective, in: Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, New York, NY, USA, 2019, pp. 830–840. doi: 10.1145/3338906.3338945.
23.
FranklP.G.WeissS.N. and HuC., All-uses vs mutation testing: An experimental comparison of effectiveness, J. Syst. Softw.38(3) (Sep. 1997), 235–253. doi: 10.1016/S0164-1212(96)00154-9.
24.
GammieP. and van der MeydenR., MCK: Model Checking the Logic of Knowledge, in: Computer Aided Verification, Berlin, Heidelberg, 2004, pp. 479–483. doi: 10.1007/978-3-540-27813-9_41.
25.
Gómez-SanzJ.J.BotíaJ.SerranoE. and PavónJ., Testing and Debugging of MAS Interactions with INGENIAS, in: Agent-Oriented Software Engineering IX, Berlin, Heidelberg, 2009, pp. 199–212. doi: 10.1007/978-3-642-01338-6_15.
Henry ColesLaurentT.HenardC.PapadakisM. and VentresqueA., PIT: a practical mutation testing tool for Java (demo), in: Proceedings of the 25th International Symposium on Software Testing and Analysis, New York, NY, USA, Jul. 2016, pp. 449–452. doi: 10.1145/2931037.2948707.
28.
InozemtsevaL. and HolmesR., Coverage is not strongly correlated with test suite effectiveness, in: Proceedings of the 36th International Conference on Software Engineering, Hyderabad India, May 2014, pp. 435–445. doi: 10.1145/2568225.2568271.
29.
JustR.JalaliD.InozemtsevaL.ErnstM.D.HolmesR. and FraserG., Are mutants a valid substitute for real faults in software testing?, in: Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering – FSE 2014, Hong Kong, China, 2014, pp. 654–665. doi: 10.1145/2635868.2635929.
30.
KiczalesG.HilsdaleE.HuguninJ.KerstenM.PalmJ. and GriswoldW.G., An Overview of AspectJ, in: Proceedings of the 15th European Conference on Object-Oriented Programming, Berlin, Heidelberg, 2001, pp. 327–353.
31.
KintisM.PapadakisM.PapadopoulosA.ValvisE.MalevrisN. and Le TraonY., How effective are mutation testing tools? An empirical analysis of Java mutation testing tools with manual analysis and real faults, Empir. Softw. Eng.23(4) (Aug. 2018), 2426–2463. doi: 10.1007/s10664-017-9582-5.
32.
KravariK. and BassiliadesN., A survey of agent platforms, J. Artif. Soc. Soc. Simul.18(1) (2015), 11. doi: 10.18564/jasss.2661.
33.
LamW.MuşluK.SajnaniH. and ThummalapentaS., A study on the lifecycle of flaky tests, in: Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering, New York, NY, USA, Jun. 2020, pp. 1471–1482. doi: 10.1145/3377811.3381749.
34.
LeottaM.CerioliM.OlianasD. and RiccaF., Fluent vs Basic Assertions in Java: An Empirical Study, in: 2018 11th International Conference on the Quality of Information and Communications Technology (QUATIC), Sep. 2018, pp. 184–192. doi: 10.1109/QUATIC.2018.00036.
35.
LeottaM.CerioliM.OlianasD. and RiccaF., Two experiments for evaluating the impact of Hamcrest and AssertJ on assertion development, Softw. Qual. J.28(3) (Sep. 2020), 1113–1145. doi: 10.1007/s11219-020-09507-0.
36.
LeottaM.CerioliM.OlianasD. and RiccaF., Hamcrest vs AssertJ: An Empirical Assessment of Tester Productivity, in: Quality of Information and Communications Technology, Cham, 2019, pp. 161–176. doi: 10.1007/978-3-030-29238-6_12.
37.
LomuscioA.QuH. and RaimondiF., MCMAS: An open-source model checker for the verification of multi-agent systems, Int. J. Softw. Tools Technol. Transf.19(1) (Feb. 2017), 9–30. doi: 10.1007/s10009-015-0378-x.
38.
LomuscioA.QuH. and RaimondiF., MCMAS: A Model Checker for the Verification of Multi-Agent Systems, in: Computer Aided Verification, Berlin, Heidelberg, 2009, pp. 682–688. doi: 10.1007/978-3-642-02658-4_55.
39.
LuoQ.HaririF.EloussiL. and MarinovD., An empirical analysis of flaky tests, in: Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, New York, NY, USA, Nov. 2014, pp. 643–653. doi: 10.1145/2635868.2635920.
40.
MascardiV. et al., Engineering multi-agent systems: State of affairs and the road ahead, ACM SIGSOFT Softw. Eng. Notes44(1) (Mar. 2019), 18–28. doi: 10.1145/3310013.3322175.
41.
Massimo Cossentino, From Requirements to Code with PASSI Methodology, in: Agent-Oriented Methodologies, Brian Henderson-Sellers and Paolo Giorgini, Eds. Hershey, PA, USA: IGI Global, 2005, pp. 79–106. doi: 10.4018/978-1-59140-581-8.ch004.
NascimentoN.AlencarP.LucenaC. and CowanD., A metadata-driven approach for testing self-organizing multiagent systems, IEEE Access8 (2020), 204256–204267. doi: 10.1109/ACCESS.2020.3036668.
45.
NguyenC.D.PeriniA.BernonC.PavónJ. and ThangarajahJ., Testing in Multi-Agent Systems, in: Agent-Oriented Software Engineering X, Berlin, Heidelberg, May 2009, pp. 180–190. doi: 10.1007/978-3-642-19208-1_13.
46.
NguyenC.D.PeriniA. and TonellaP., A Goal-Oriented Software Testing Methodology, in: Agent-Oriented Software Engineering VIII: 8th International Workshop, AOSE 2007, Honolulu, HI, USA, May 14, 2007, Revised Selected Papers, Berlin, Heidelberg, 2008, pp. 58–72. doi: 10.1007/978-3-540-79488-2_5.
47.
NorthD., Introducing BDD, Dan North & Associates, 2006. https://dannorth.net/introducing-bdd/ (accessed May 20, 2021).
48.
PěchoučekM. and MaříkV., Industrial deployment of multi-agent technologies: Review and selected case studies, Auton. Agents Multi-Agent Syst.17(3) (Dec. 2008), 397–431. doi: 10.1007/s10458-008-9050-0.
49.
SaatyT.L., How to make a decision: The analytic hierarchy process, Eur. J. Oper. Res., vol. 48, no. 1, Art. no. 1, Sep. 1990. doi: 10.1016/0377-2217(90)90057-I.
50.
SaywardF.DeMilloR.BuddT.A. and LiptonR.J., The design of a prototype mutation system for program testing, in: Managing Requirements Knowledge, International Workshop on, Los Alamitos, CA, USA, 1978, p. 623. doi: 10.1109/AFIPS.1978.195.
51.
SirajS.MikhailovL. and KeaneJ.A., PriEsT: An interactive decision support tool to estimate priorities from pairwise comparison judgments: PriEsT: An interactive decision support tool to estimate priorities from pairwise comparison judgments, Int. Trans. Oper. Res.22(2) (Mar. 2015), 217–235. doi: 10.1111/itor.12054.
52.
TiryakiA.M.ÖztunaS.DikenelliO. and ErdurR.C., SUNIT: A Unit Testing Framework for Test Driven Development of Multi-Agent Systems, in: Agent-Oriented Software Engineering VII, Berlin, Heidelberg, 2007, pp. 156–173. doi: 10.1007/978-3-540-70945-9_10.
53.
Vera-PérezO.L.MonperrusM. and BaudryB., Descartes: A PITest Engine to Detect Pseudo-Tested Methods: Tool Demonstration, in: 2018 33rd IEEE/ACM International Conference on Automated Software Engineering (ASE), Sep. 2018, pp. 908–911. doi: 10.1145/3238147.3240474.
54.
WangS. and ZhuH., CATest: A Test Automation Framework for Multi-agent Systems, in: 2012 IEEE 36th Annual Computer Software and Applications Conference, Izmir, Turkey, Jul. 2012, pp. 148–157. doi: 10.1109/COMPSAC.2012.24.
55.
WatsonA.H.WallaceD.R. and McCabeT.J., Structured testing: A testing methodology using the cyclomatic complexity metric, vol. 500. US Department of Commerce, Technology Administration, National Institute of …, 1996.
56.
WinikoffM., Challenges and Directions for Engineering Multi-agent Systems, presented at the Dagstuhl Seminar 12342 (August 2012), Dagstuhl, 2012. [Online]. Available: http://arxiv.org/abs/1209.1428.
57.
WooldridgeM.FisherM.HugetM.-P. and ParsonsS., Model checking multi-agent systems with MABLE, in: Proceedings of the First International Joint Conference on Autonomous Agents and Multiagent Systems: Part 2, New York, NY, USA, Jul. 2002, pp. 952–959. doi: 10.1145/544862.544965.
58.
YeX.LiuW. and WangN., Runtime Verification of Multi-Agent Self-Adaptive System, in: 2021 IEEE 24th International Conference on Computer Supported Cooperative Work in Design (CSCWD), May 2021, pp. 12–17. doi: 10.1109/CSCWD49262.2021.9437643.
