Development and validation of a multidimensional AI literacy scale for higher education students: A mixed-method study

Abstract

This study developed and validated a comprehensive AI literacy scale for higher education students through a mixed-methods approach. The development process integrated literature review, expert interviews, and BERTopic modeling analysis. Following content validity assessment and pilot testing, the scale was validated with a sample of 400 students. Factor analyses supported a 17-item, four-dimensional structure comprising AI Fundamental Knowledge, AI Impact Assessment, AI Performance Evaluation, and AI Practical Application. The scale demonstrated adequate internal consistency and configural and metric invariance across academic disciplines. Group differences were observed according to academic major, AI course experience, and academic level. Structural equation modeling indicated that AI literacy is positively associated with academic self-efficacy, which is in turn related to multi-item measured creativity and students’ GPA. Mediation analysis indicated that academic self-efficacy mediated the association between AI literacy and creativity, while the indirect association between AI literacy and academic achievement via this pathway was offset by the negative link between creativity and GPA. However, the generalizability of these findings may be constrained by the specific cultural context and sample characteristics.