Execution traces comprehension is an important topic in computer science since it allows software engineers to get a better understanding of the system behavior. However, traces are usually very large and hence they are difficult to interpret. Parallel, execution traces comprehension is a very important topic into the algorithms learning courses since it allows students to get a better understanding of the algorithm behavior. Therefore, there is a need to investigate ways to help students (and teachers) find and understand important information conveyed in a trace despite the trace being massive.
In this paper, we propose a new approximation for execution traces comprehension based on fuzzy linguistic descriptions. A new methodology and a data-driven architecture based on linguistic modelling of complex phenomenon are presented and explained. In particular, they are applied to automatically generate linguistic reports from execution traces generated during the execution of algorithm implemented by the students of an introductory course of artificial intelligence. To the best of our knowledge, it is the first time that linguistic modelling of complex phenomenon is applied to execution traces comprehension.
Throughout the article, it is shown how this kind of technology can be employed as a useful computer-assisted assessment tool that provides students and teachers with technical, immediate and personalised feedback about the algorithms that are being studied and implemented. At the same time, they provide us with two useful applications: they are an indispensable pedagogical resource for improving comprehension of execution traces, and they play an important role in the process of measuring and evaluating the “believability” of the agents implemented.
To show and explore the possibilities of this new technology, a web platform has been designed and implemented by one of the authors, and it has been incorporated into the process of assessment of an introductory artificial intelligence course. Finally, an empirical evaluation to confirm our hypothesis was performed and a survey directed to the students was carried out to measure the quality of the learning-teaching process by using this methodology enriched with fuzzy linguistic descriptions.
FaesslerL., HinterbergerH., DahindenM. and WyssM., Evaluating student motivation in constructivistic, problembased introductory computer science courses, in E-Learn: World Conference on E-Learning in Corporate, Government, Healthcare, and Higher Education, Association for the Advancement of Computing in Education (AACE), 2006, pp. 1178–1185.
2.
FinkL.D., Creating significant learning experiences: An integrated approach to designing college courses. John Wiley & Sons, 2013.
3.
HattieJ. and TimperleyH., The power of feedback, Review of Educational Research77(1) (2007), 81–112.
4.
NicolD.J. and MacFarlane-DickD., Formative assessment and self-regulated learning: A model and seven principles of good feedback practice, Studies in Higher Education31(2) (2006), 199–218.
5.
SarsarH.S., and F., Student and instructor responses to emotional motivational feedback messages in an online instructional environment, Turkish Online Journal of Educational Technology - TOJET16(1) (2017), 115–127.
6.
van der KleijF.M., EggenT.J., TimmersC.F. and VeldkampB.P., Effects of feedback in a computer-based assessment for learning, Computers & Education58(1) (2012), 263–272.
7.
HamiltonI.R., Automating formative and summative feedback for individualised assignments, Campus-Wide Information Systems26 (2009), 355–364.
8.
LavoletteE., The accuracy of computer-assisted feedback and students responses to it, 19 (2015).
9.
PirzadehH., Hamou-LhadjA. and ShahM., Exploiting text mining techniques in the analysis of execution traces, in Software Maintenance (ICSM), 2011 27th IEEE International Conference on, IEEE, 2011, pp. 223–232.
10.
NoorbehbahaniF. and KardanA.A., The automatic assessment of free text answers using a modified BLEU algorithm, Computers & Education56(2) (2011), 337–345.
11.
PérezD., GliozzoA.M., StrapparavaC., AlfonsecaE., RodríguezP. and MagniniB., Automatic assessment of students’ free-text answers underpinned by the combination of a bleu-inspired algorithm and latent semantic analysis, in Proceedings of the Eighteenth International Florida Artificial Intelligence Research Society Conference, Clearwater Beach, Florida, USA, 2005, pp. 358–363.
12.
LachnerA., BurkhartC. and NücklesM., Formative computer-based feedback in the university classroom: Specific concept maps scaffold students’ writing, Computers in Human Behavior72 (2017), 459–469.
13.
FyfeE.R., Providing feedback on computer-based algebra homework in middle-school classrooms, Computers in Human Behavior63 (2016), 568–574.
14.
DouceC., LivingstoneD. and OrwellJ., Automatic test-based assessment of programming: A review, Journal on Educational Resources in Computing (JERIC)5(3) (2005), 4.
15.
BurgosD., Van NimwegenC., Van OostendorpH. and KoperR., Game-based learning and the role of feedback. A case study, 2007.
16.
ZadehL.A., Fuzzy sets, Information and Control8(3) (1965), 338–353.
17.
StrobinL. and NiewiadomskiA., Linguistic summaries of graph datasets using ontologies: An application to semantic web, Journal of Intelligent & Fuzzy Systems32(2) (2017), 1193–1202.
18.
Ramos-SotoA., Vázquez-BarreirosB., BugarínA., GewercA. and BarroS., Evaluation of a data-to-text system for verbalizing a learning analytics dashboard, Int J Intell Syst32(2) (2017), 177–193.
19.
GkatziaD., HastieH.F., JanarthanamS. and LemonO., Generating student feedback from time-series data using reinforcement learning, in ENLG 2013 - Proceedings of the 14th EuropeanWorkshop on Natural Language Generation, Sofia, Bulgaria, 2013, pp. 115–124.
20.
Sánchez-TorrubiaM.G., Torres-BlancC. and TriviñoG., An approach to automatic learning assessment based on the computational theory of perceptions, Expert Syst Appl39(15) (2012), 12177–12191.
21.
Rubio-ManzanoC. and TriviñoG., Improving player experience in computer games by using players’ behavior analysis and linguistic descriptions, Int J Hum-Comput Stud95 (2016), 27–38.
22.
RiazM., ÇağmanN., ZareefI. and AslamM., N-soft topology and its applications to multi-criteria group decision making, Journal of Intelligent & Fuzzy Systems36(6) (2019), 6521–6536.
23.
AhoE., HaverinenH.-L., JuusoH., LaukkaS.J. and SutinenA., Teachers principles of decision-making and classroom management; a case study and a new observation method, Procedia-Social and Behavioral Sciences9 (2010), 395–402.
24.
RiazM., DavvazB., FirdousA. and FakharA., Novel concepts of soft rough set topology with applications, Journal of Intelligent & Fuzzy Systems36(4) (2019), 3579–3590.
25.
RiazM. and TehrimS.T., Cubic bipolar fuzzy ordered weighted geometric aggregation operators and their application using internal and external cubic bipolar fuzzy data, Computational and Applied Mathematics38(2) (2019), 87.
26.
RiazM. and HashmiM. R., Fuzzy parameterized fuzzy soft compact spaces with decision-making, Punjab Univ j Math50 (2018), 131–145.
27.
GoldbergE., DriedgerN. and KittredgeR.I., Using natural-language processing to produce weather forecasts, IEEE Expert9(2) (1994), 45–53.
28.
CochJ., System demonstration interactive generation and knowledge administration in multimeteo, Natural Language Generation (1998).
29.
PortetF., ReiterE., GattA., HunterJ., SripadaS., FreerY. and SykesC., Automatic generation of textual summaries from neonatal intensive care data, Artif Intell173(7-8) (2009), 789–816.
30.
WhiteM. and CaldwellT., EXEMPLARS: A practical, extensible framework for dynamic text generation, in Proceedings of the Ninth International Workshop on Natural Language Generation, INLG 1998, Niagara-on-the-Lake, Ontario, Canada, 1998, 1998.
31.
BusemannS. and HoracekH., Generating air quality reports from environmental data, in Proceedings of the DFKIWorkshop on Natural Language Generation, 1997, pp. 15–21.
32.
GattA. and KrahmerE., Survey of the state of the art in natural language generation: Core tasks, applications and evaluation, J Artif Intell Res61 (2018), 65–170.
33.
YagerR.R., Fuzzy summaries in database mining, in Artificial Intelligence for Applications, 1995 Proceedings, 11th Conference on, IEEE, 1995, pp. 265–269.
34.
MarínN. and SánchezD., Fuzzy sets and systems+ natural language generation: A step forward in the linguistic description of time series, Fuzzy Sets and Systems285 (2016), 1–5.
35.
Ramos-SotoA., BugarínA. and BarroS., On the role of linguistic descriptions of data in the building of natural language generation systems, Fuzzy Sets and Systems285 (2016), 31–51.
36.
ZadehL.A., The concept of a linguistic variable and its application to approximate reasoningi, Information Sciences8(3) (1975), 199–249.
37.
Ramos-SotoA., BugarínA., BarroS., GallegoN., RodríguezC., FragaI. and SaundersA., Automatic generation of air quality index textual forecasts using a data-to-text approach, in Advances in Artificial Intelligence - 16th Conference of the Spanish Association for Artificial Intelligence, CAEPIA 2015, Albacete, Spain, Proceedings, 2015, pp. 164–174.
38.
Ramos-SotoA., DizA.J.B., BarroS. and TaboadaJ., Linguistic descriptions for automatic generation of textual short-term weather forecasts on real prediction data, IEEE Trans Fuzzy Systems23(1) (2015), 44–57.
39.
TrivinoG. and SugenoM., Towards linguistic descriptions of phenomena, International Journal of Approximate Reasoning54(1) (2013), 22–34.
40.
Conde-ClementeP., AlonsoJ.M., NunesÉ.O., SanchezA. and TrivinoG., New types of computational perceptions: Linguistic descriptions in deforestation analysis, Expert Systems with Applications85 (2017), 46–60.
41.
Conde-ClementeP., TrivinoG. and AlonsoJ.M., Generating automatic linguistic descriptions with big data, Information Sciences380 (2017), 12–30.
42.
Conde-ClementeP., AlonsoJ.M. and TrivinoG., Toward automatic generation of linguistic advice for saving energy at home, Soft Computing22(2) (2018), 345–359.
43.
Sanchez-ValdesD. and TriviñoG., Linguistic and emotional feedback for self-tracking physical activity, Expert Syst Appl42(24) (2015), 9574–9586.
44.
Sanchez-ValdesD., Alvarez-AlvarezA. and TriviñoG., Linguistic description about circular structures of the mars’ surface, Appl Soft Comput13(12) (2013), 4738–4749.
45.
ArguellesL. and TrivinoG., I-struve: Automatic linguistic descriptions of visual double stars, Engineering Applications of Artificial Intelligence26(9) (2013), 2083–2092.
46.
EciolazaL., Pereira-FariñAM. and TrivinoG., Automatic linguistic reporting in driving simulation environments, Applied Soft Computing13(9) (2013), 3956–3967.
47.
ZadehL.A., From computing with numbers to computing with words from manipulation of measurements to manipulation of perceptions, IEEE Transactions on Circuits and Systems I: Fundamental Theory and Applications46(1) (1999), 105–119.
48.
BernardiM.L., CimitileM., MartinelliF. and MercaldoF., An ensemble fuzzy logic approach to game bot detection through behavioural features, in 2018 IEEE International Conference on Fuzzy Systems, FUZZ-IEEE 2018, Rio de Janeiro, Brazil, 2018, pp. 1–9.
49.
PachecoC., TokarchukL. and Pérez-LiébanaD., Studying believability assessment in racing games, in Proceedings of the 13th International Conference on the Foundations of Digital Games, FDG 2018, Malmö, Sweden, 2018, pp. 20:1–20:10.
50.
WangJ.-Y., Classification of humans and bots in two typical two-player computer games, in 2018 3rd International Conference on Computer and Communication Systems (ICCCS), IEEE 2018, pp. 502–505.
51.
Rubio-ManzanoC., Similarity Measure Between Linguistic Terms by Using Restricted Equivalence Functions and Its Application to Expert Systems, Springer International Publishing, Cham, 2019, pp. 97–102.
52.
BatyrshinI., CrossV., KreinovichV. and RifqiM., Special issue on similarity, correlation and association measures dedicated to the memory of lotfi zadeh, Journal of Intelligent & Fuzzy SystemsPreprint, 1–2.
53.
BustinceH., BarrenecheaE. and PagolaM., Restricted equivalence functions, Fuzzy Sets and Systems157(17) (2006), 2333–2346.
54.
AlonsoJ.M., Conde-ClementeP. and TriviñoG., Linguistic description of complex phenomena with the rldcp R package, in Proceedings of the 10th International Conference on Natural Language Generation, INLG 2017, Santiago de Compostela, Spain, 2017, pp. 243–244.
55.
SoniB. and HingstonP., Bots trained to play like a human are more fun, in Proceedings of the International Joint Conference on Neural Networks, IJCNN 2008, part of the IEEE World Congress on Computational Intelligence, WCCI 2008, Hong Kong, China, 2008, pp. 363–369.
56.
HingstonP., A turing test for computer game bots, IEEE Trans Comput Intellig and AI in Games1(3) (2009), 169–186.
57.
ManD.P., Automatic evaluation of users short essays by using statistical and shallow natural language processing techniques advanced studies diploma work, 2004.
