Assessing Computational Thinking: Development and Validation of the Algorithmic Thinking Test for Adults

Abstract

This study describes the development and validation process of a computational thinking (CT) test for adults. The team designed a set of items and explored a subset of those through two qualitative pilots. Then, in order to provide validity evidence based on the test content, a team of 11 subject-matter experts coded the initial pool of items using two different systems of categories based on CT components and contents. Next, the items were piloted on a sample of 289 participants, 137 experts in CT, and 152 novices. After a series of confirmatory factor analyses, a unidimensional model that represents algorithmic thinking was adopted. Further analyzing the psychometric quality of the 27 items, 20 of them with excellent reliability indices were selected for the test. Thus, this study provides a tool to evaluate adults’ CT: the Algorithmic Thinking Test for Adults (ATTA), which was developed according to psychometric standards. This article also reflects on the nature of CT as a construct.

Keywords

computational thinking algorithmic thinking algorithms higher education assessment evaluation

Get full access to this article

View all access options for this article.

References

American Educational Research Association, American Psychological Association, and National Council for Measurement in Education . (2014). Standards for Educational and Psychological Testing. American Psychological Association. Washington, DC.

Atmatzidou

Demetriadis

(2016). Advancing students’ computational thinking skills through educational robotics: A study on age and gender relevant differences. Robotics and Autonomous Systems, 75, 661–670. DOI: 10.1016/j.robot.2015.10.008 .

Brilliant (2021). Learn to Think. Brilliant. https://brilliant.org/.

Buitrago Flórez

Casallas

Hernández

Reyes

Restrepo

Danies

(2017). Changing a generation’s way of thinking: Teaching computational thinking through programming. Review of Educational Research, 87(4), 834–860. DOI: 10.3102/0034654317710096.

Caeli

E. N.

Yadav

(2020). Unplugged approaches to computational thinking: A historical perspective. Tech Trends, 64(1), 29–36. DOI: 10.1007/s11528-019-00410-5 .

Carneson

Delpierre

Masters

(2016). Designing and Managing Multiple Choice Questions. DOI: 10.13140/RG.2.2.22028.31369 .

(2017). Assessing elementary students’ computational thinking in everyday reasoning and robotics programming. Computers and Education, 109, 162–175. DOI: 10.1016/j.compedu.2017.03.001 .

Coursera (2021). Computational Thinking for Problem Solving. https://www.coursera.org/learn/computational-thinking-problem-solving.

Cronbach

L. J.

(1951). Coefficient alpha and the interval structure of tests. Psychometrika, 16, 297–334. DOI: 10.1007/BF02310555.

(2019). “Computational thinking,” In Fincher

Robbins

(Eds.), Cambridge Handbook of Computer Science Education Research (pp. 513–546). Cambridge University Press. DOI: 10.1017/9781108654555.018 .

(2017). Developing a two-dimensional categorization system for educational tasks in informatics. Informatica (Netherlands), 28(1), 23–44. DOI: 10.15388/Informatica.2017.119 .

(2016). A systematic mapping study on assessing computational thinking abilities. IEEE Frontiers in Education Conference (FIE) (pp. 1–9). IEEE. DOI: 10.1109/FIE.2016.7757678 .

(2012). Content validity evidences in test development: An applied perspective. International Journal of Clinical and Health Psychology España, 12(3), 449–460.

(2016). Exploration of computational thinking of software engineering novice students based on solving computer science tasks. International Journal of Engineering Education, 32(3), 1–10.

(2021). Computational thinking Through an empirical lens: A systematic review of literature. Journal of Educational Computing Research, 07356331211033158. DOI: 10.1177/07356331211033158 .

16.

Falchikov

Boud

(1989). Student self-assessment in higher education: A meta-analysis. Review of Educational Research, 59(4), 395–430. DOI: 10.3102/00346543059004395 .

(2010). El análisis factorial como técnica de investigación en psicología [Factor analysis as a research technique in psychology]. Papeles del Psicólogo, 31(1), 18–33.

(2004). The use of verbal protocols as data: An analysis of insight in the candle problem. Memory and Cognition, 32(6), 990–1006. DOI: 10.3758/BF03196876 .

(2013). First year student performance in a test for computational thinking Proceedings of the South African Institute for Computer Scientists and Information Technologists Conference (pp. 271-277). ACM DOI: 10.1145/2513456.2513484 .

(1977). The mode, median, and mean inequality. The American Statistician, 31(3), 120–121.

(1989). The relation between problem categorization and problem solving among experts and novices. Memory and Cognition, 17(5), 627–638. DOI: 10.3758/BF03197085 .

(Eds), (2011). Validity and Limitations of College Student Self-Report Data: New Directions for Institutional Research (Vol. 110). John Wiley & Sons.

(2018). How to learn and how to teach computational thinking: Suggestions based on a review of the literature. Computers and Education, 126, 296–310. DOI: 10.1016/j.compedu.2018.07.004 .

(1999). Cut-off criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1–55. DOI: 10.1080/10705519909540118 .

(2016). A framework for computational thinking based on a systematic research review. Baltic Journal of Modern Computing, 4(3), 583–596. http://www.bjmc.lu.lv/fileadmin/user_upload/lu_portal/projekti/bjmc/Contents/4_3_15_Kalelioglu.pdf.

26.

Korkmaz

Ö.

Çakir

Özden

M. Y.

(2017). A validity and reliability study of the computational thinking scales (CTS). Computers in Human Behavior, 72, 558–569. DOI: 10.1016/j.chb.2017.01.005.

27.

Laakman

(2015). Cracking the coding interview. CareerCup: 6thed.

28.

Lamprou

Repenning

(2016). Computational Thinking ≠ Programming. Magazine SwissInformatics Society’s New Online Platform. https://magazine.swissinformatics.org/en/computational-thinking-%E2%89%A0-programming/.

29.

Nelson

K. G.

Shell

D. F.

Husman

Fishman

E. J.

Soh

L. K.

(2015). Motivational and self‐regulated learning profiles of students taking a foundational engineering course. Journal of Engineering Education, 104(1), 74–100. DOI: 10.1002/jee.20066.

30.

Papert

(1980). Mindstorms: Children, Computers, and Powerful Ideas. Basic Books.

(2017). Which cognitive abilities underlie computational thinking? Criterion validity of the Computational Thinking Test. Computers in Human Behavior, 72, 678–691. DOI: 10.1016/j.chb.2016.08.047 .

32.

Romero

Lepage