It's Not How They Talk,It's Who They Are: Learning Style,Not Interaction Modality,Predicts AI Newbies’ Engagement with Educational Chatbots

Abstract

Educational AI chatbot (EAIC) research has focused heavily on interaction modality, reflecting the assumption that modality drives engagement. This study challenges that assumption. We examined how students with no prior AI experience (“AI Newbies”, n = 96) engaged with an EAIC during a ten-week course. Contrary to expectations, interaction modality did not significantly affect engagement behaviors. Instead, learning style emerged as a stronger predictor. Verbal group students gravitated toward critical analysis questions. Solitary group toward conceptual exploration. Social group toward assessment guidance. Students also predominantly used mobile devices despite limited mobile support, and 77% migrated to general-purpose chatbots by the end of the course, preferring comprehensiveness over the controlled accuracy of curriculum-aligned systems. These findings suggest that designing for AI Newbies requires attending to individual learning differences rather than assuming a one-size-fits-all approach. We propose the Optimize, Personalize, Integrate (OPI) framework to guide EAIC development for this underserved population.

1. Introduction

Research on educational AI chatbots (EAICs) has focused heavily on interaction modality (e.g., voice, text, both), reflecting an implicit assumption that modality drives engagement. Our study challenges this assumption. Unlike general-purpose AI chatbots, such as ChatGPT (Wolf & Maier, 2024a), EAICs are specifically designed for educational purposes(Chaudhry & Kazim, 2022). While research has investigated EAICs in higher education (Park & Ahn, 2024) most studies assume students already possess baseline AI literacy.

This assumption overlooks a critical population. Some students possess little to no exposure to AI technologies (Alsobeh & Woodward, 2023; Savelka et al., 2023). We refer to this group as “AI Newbies”. As AI literacy becomes increasingly important for educational success (Nguyen et al., 2024a), students without it risk marginalization, contributing to the larger digital divide in education (Hornberger et al., 2023).

Understanding how AI Newbies engage with EAICs is critical for optimizing their learning experiences. By engagement, we refer to a construct of phenomena that includes the behavioral, cognitive and effective characteristics (Fredricks et al., 2016; Mansor & & Abdullah, 2025). In terms of theory, this definition is grounded in the self-determination theory (Ryan & Deci, 2000) Although there could be many ways to measure student engagement, we operationalize the engagement construct with data surrounding the chat between students and the EAIC. Engagement behaviors (Jansen et al., 2025), such as the number of sessions, messages, and feature usage, can affect learning outcomes directly or indirectly (Hornbæk & Oulasvirta, 2017). Beyond these system-level factors, individual differences matter: students differ by learning style, defined as preferences for learning activities that contribute to learning effectiveness (Pashler et al., 2008a). Studies have found that students with different learning styles approach course-related questions differently (Faria et al., 2015; Ilin, 2022). Motivation also influences what students ask and how often. How these factors emerge among AI Newbies interacting with an EAIC is not well understood. Against this backdrop, we pose three research questions:

• RQ1: How do AI Newbies engage with an EAIC under different interaction modalities?

• RQ2: What is the relationship between students’ learning styles and their question themes with an EAIC?

• RQ3: What factors encourage and discourage AI Newbies from engaging with an EAIC?

RQ1 tests the prevailing assumption that modality affects engagement by comparing text-only, voice-only, and mixed interaction conditions. RQ2 examines whether learning style offers a more predictive framework for understanding engagement patterns by qualitatively analyzing question themes. RQ3 identifies what motivates and discourages AI Newbies’ engagement through thematic analysis of student feedback. Overall, these questions address a critical gap: understanding how true beginners (i.e., not students with existing AI familiarity) engage with educational AI systems.

2. Related Literature

2.1 The Influence of AI Literacy on Student-Chatbot Engagement

AI literacy shapes how students engage with chatbots, yet research has largely ignored students who lack this literacy entirely or to a large extent. Prior studies demonstrate the importance of AI literacy among students, particularly in higher education (Iovine, 2026; Kavitha & Joshith, 2024; Labadze et al., 2023a, 2023b; Luckin & Cukurova, 2019; Walter, 2024). Nguyen et al. (2024b) found that students’ mindset and attitude toward AI significantly influence their use of AI in education. In AI-assisted educational contexts, engagement between students and EAICs is influenced by different levels of AI literacy (Li et al., 2026; Shah et al., 2024). This literacy gap manifests in how students interact (Chaaban et al., 2025): non-AI experts designing prompts for LLMs approached the task opportunistically rather than systematically, quickly finalizing outcomes without testing robustness (Zamfirescu-Pereira et al., 2023). This lack of systematic engagement persists even among students with positive attitudes toward AI chatbots (Lowenthal et al., 2020; Xue et al., 2025). The skills to effectively interact with AI chatbots remain a challenge for students in higher education.

2.2 Students’ Engagement with EAICs

While AI literacy shapes whether students can engage effectively, engagement itself is multidimensional. Engagement between humans and EAICs can be categorized in at least two key ways: the duration of engagement (long-term and short-term) and the communication control (user-initiated or chatbot-driven) (Følstad & Skjuve, 2019). Beyond these categories, engagement is a complex social and psychological construct encompassing behavioral, cognitive, and emotional dimensions (Brodie et al., 2013; Fredricks et al., 2004; Hollebeek et al., 2019). While these dimensions are interrelated and often overlap, their interplay adds complexity to understanding engagement as a construct (Fredricks et al., 2004; Malik et al., 2018; Peddibhotla & Jani, 2019).

Despite this complexity, engagement remains critical for improving learning outcomes (Donnermann et al., 2021; Yuan & Liu, 2025). Evidence confirms that AI chatbots can enhance engagement. Li and colleagues (2024) found that students in a ChatGPT-assisted group exhibited significantly higher engagement, which they attributed to the AI's contextual responses and support for self-directed learning.

One key advantage of EAICs for supporting student engagement is their capacity for immediate feedback. Such feedback systems enable students to reflect on their tasks and promote self-assessment, which may lead to improved motivation, self-learning, and higher academic performance (Abdelhalim, 2024; Ortega-Ochoa et al., 2024). These outcomes, however, depend on their learning styles (Kirschner, 2017; Pashler et al., 2008b).

2.3 Motivations to Engage with EAICs

Engagement does not occur in isolation: motivation drives it. High levels of motivation often lead to increased engagement in a focal activity (Martin et al., 2017). Research highlights the importance of motivation for academic success, as it strongly correlates with enhanced engagement and a positive attitude toward learning (Lee & Reeve, 2012; Ryan & Deci, 2020). Using self-determination theory, motivation can be categorized into three interrelated but distinct types: (1) intrinsic motivation, encompassing students’ internal desire and curiosity to learn; (2) extrinsic motivation, referring to external pressures and rewards; and (3) amotivation, a lack of willingness to act. Self-determination theory posits that fulfilling certain psychological needs—autonomy, competence, and relatedness—enhances students’ motivation to learn (Deci et al., 1991; Ryan & Deci, 2000).

Multiple factors enhance motivation to engage with EAICs, though their relative importance remains unclear. Teachers’ roles and students’ prior knowledge increase motivation by fulfilling key psychological needs (Chiu et al., 2023), though teachers’ impact on students’ sense of autonomy appears limited. Wolf and Maier (2024b) emphasized motivational factors such as novelty and ease of use, while Kobiella et al. (2024) noted that students’ motivation stems from a need for accomplishment and improved productivity. Collectively, these studies provide strong evidence that students find AI tools effective in encouraging motivation and engagement in learning.

The evidence on what drives EAIC engagement is contradictory. Zhen et al. (2020) identified four distinct engagement patterns, with sustained or increasing engagement linked to stronger learning determination, and declining engagement to weaker determination (He et al., 2024; Jansen et al., 2025; O’Brien & Toms, 2008). Yet some studies report no significant differences in motivation between EAICs and traditional classrooms (Kumar, 2021; Ortega-Ochoa et al., 2024; Wu et al., 2023). Compounding this uncertainty, these findings derive primarily from students with existing AI familiarity. How AI Newbies engage with EAICs and what motivates that engagement remains unexplored. This is the gap our study addresses.

3. Methodology

The research adopts a mixed-methods approach (Poth, 2023), combining both quantitative and qualitative approaches to examine students’ engagement with an EAIC in higher education. Regarding the timeline, this study was conducted in early 2024 with a group of vocational school students who had little to no knowledge about an EAIC and referred to in this research as AI Newbies. In the first question of the survey, ‘Did you know about EAICs before this study?’, AI Newbies chose “I have never heard of them” or “I have heard of them but never used them” as their answer (n = 96, 97%). We collected quantitative data through system logs (exported from Cipherbot) and self-evaluated metrics (collected from post-experience survey), while qualitative insights were gathered through student open-ended responses (also collected from the survey) and systematic analysis of chatbot conversation transcripts (the chat messages exported from Cipherbot chatlogs). This helps us compare engagement between different interaction modalities while getting a comprehensive perspective of the qualitative insights into students’ learning experience.

3.1 The EAIC: Cipherbot

Cipherbot is an EAIC powered by LLMs, designed to assist educators by responding to student queries using course materials pre-uploaded by teachers (Jung et al., 2025; Xuan et al., 2026). Built on OpenAI's ChatGPT-4 Turbo (at the time of the study), Cipherbot generates answers by drawing from relevant stored knowledge. Unlike general-purpose AI chatbots, such as ChatGPT or Gemini, Cipherbot is a specific-purpose AI chatbot, developed for educational purposes, addressing challenges like “hallucinated” responses (Dalalah & Dalalah, 2023), aimed at the use of reliable resources (Zou et al., 2022), and providing controls for class interactions. The solution that Cipherbot addresses helps both students and teachers. Students can avoid waiting time in questioning because Cipherbot responds anytime based on pre-uploaded materials. Teachers can ensure that all questions are answered faster when teaching a large group of students while still controlling the quality of responses. In this study, we set up our Cipherbot classes to support two primary interaction methods (see Appendix 1): (a) text-based, where students input questions via text and receive text-based responses derived from the uploaded materials, (b) voice-based, where students communicate verbally and receive a text-based answer in return, and (c) mixed, where both voice and text were enabled. This design provides a versatile learning tool that adapts to different learning styles (verbal, logical-mathematical, social, solitary, visual-spatial, and naturalistic (Iku-Silan et al., 2023) and class settings (see Appendix 2).

3.2 Research Design

We conducted the study during a 10-week digital marketing course with 215 enrolled students in 2024. Institutional Review Board (IRB) approval was obtained from the institute; 165 students (76.74%) voluntarily participated, with no additional incentives provided, and finalized with 99 students (46.04%) participating in the surveys’ data collection, and 96 students (44.65% of the class) belonging to AI Newbies. Students were divided into three groups:

Group 1 (Text-only) interacted with Cipherbot through text-only (n = 25, 33%),

Group 2 (Voice only) used voice-only (n = 32, 33.33%),

Group 3 (Mixed) had access to both text and voice inputs (n = 39, 40.62%).

Students were randomly assigned to one of the three experimental groups from the class list without any prior condition. The moderator set up similar learning experiences in classroom, lectures, and support for all participants, and no issues were reported during the study. Two lecturers were assigned to teach based on their expertise and maintain similar experiences across all weeks by teaching all classes at the same time throughout the study. For example, Lecturer A taught all classes in weeks 1, 4, 7,8 while Lecturer B taught the remainder of the course. Both lecturers are the researchers for this study and facilitated the participants with technical inquiries during the study. All participants received detailed pre-course guidelines, signed consent forms, and were given weekly instructions on using Cipherbot in their course. The guidelines included explanations of the definition, benefits and drawbacks, how to use, and the ethical aspects of EAICs (see Appendix 3).

3.3 Participants

Participants were students enrolled in a digital marketing course at a vocational school in early 2024. Students were provided with clear explanations of the study's purpose, assigned unique IDs, and assured anonymity throughout the process. All participants were aged between 19 and 22 (M = 19.67, SD = 0.87). Genders included Female (n = 49, 51.0%), Male (n = 43, 44.8%), and four participants chose not to disclose their gender (n = 4, 4.2%).

3.4 Data Collection

The data was collected at the end of the course, including one survey and Cipherbot chatlogs. Table 1 provides an overview of the data collected (see Appendix 4 for survey items). The survey was sent via email, and students had five days to complete it before the links were closed. The dataset included 96 survey responses and 1,760 recorded engagements with Cipherbot.

Table 1.

Variables and how They are Measured in Each RQ.

RQs	Variables	Measures	Sources
RQ1: How do AI Newbies engage with an EAIC?	Study groups	Interaction modalities
	Cipherbot Engagement metrics (Number of follow-up questions, number of recommended questions, devices, number of sessions, average session duration (minutes), and number of questions)	Numerical	Cipherbot data from the teacher account.
	Student Background Information	Multiple choices
RQ2: What is the relationship between students’ learning styles and how they ask questions from an EAIC?	Learning styles:Verbal Learner: Verbal learners excel in language-based activities, such as reading, writing, and discussions. They prefer to learn through written or spoken words.Logical-Mathematical Learner: These learners are adept at reasoning, critical thinking, and problem-solving. They excel in activities that involve logic, numbers, and mathematical concepts.Social Learner: Social learners thrive in group settings and collaborative environments. They learn best through interaction with others and enjoy group discussions, teamwork, and peer teaching.Solitary Learner: Solitary learners prefer to study and work independently. They are comfortable with self-paced learning and may find solitary activities such as reading or online courses more effective.Visual-Spatial Learner: Visual-spatial learners have a strong ability to perceive and understand spatial relationships, often excelling in tasks that involve visualizing and manipulating objects in space.Naturalistic Learner: Naturalistic learners have a strong affinity for the natural world and learn best when exposed to real-life examples from nature or when learning in natural settings	Multiple choices	(Fleming, 2012; Howard E, 1993)
RQ3: What factors encourage and discourage AI Newbies to engage with an EAIC?	Motivations to use Cipherbot	Open ended	(Chiu et al., 2023)
	Difficulties in using Cipherbot	Open ended
	Other AI chatbot usage after the course	Multiple choice and open-ended

3.5 Data Analysis

The study employed a mixed-methods approach to examine students’ engagement with Cipherbot. We combined quantitative analysis of survey data with qualitative analysis of open-ended responses and recorded Cipherbot chatlogs. Data cleaning involved removing incomplete responses and participants with high AI literacy. Statistical tests were selected based on the type of data collected, the data distribution, and the nature of each RQ. Non-parametric tests were favored where data were skewed or ordinal. For RQ1, we examined different engagement behavior metrics and how they are influenced by different interaction methods (see Figure 1). For RQ2, we conducted a thematic analysis (Braun & Clarke, 2006) the frequency of different question themes then analyze their relationship with different learning styles. For RQ3, we conducted two additional thematic analyses to categorize the open-ended questions about motivations to use and limitations of Cipherbot. For thematic analyses, the themes were coded manually by one of the authors and by an external coder. The coders took turns coding 10% of the answers separately, then compared and discussed any differences in coder perception of the categories before finalizing the code. This process is repeated until finished. By the end of the study, we analyzed the adoption impacts after learning about AI in the course by asking students if they used others AI systems and listing names.

Figure 1.

Cipherbot features. (a) Interaction Modalities: Voice only, Text only, and Mixed. Modalities and different questions are set by the educator from the ‘Teacher’ account (by turning on/off each mode). (b) Follow-up questions are extended questions tailored to the previous questions provided by students.

4. Findings

4.1 RQ1: How Do AI Newbies Engage with an EAIC Under Different Interaction Modalities?

Interaction modality did not significantly affect engagement behaviors. A Kruskal-Wallis H test showed no statistically significant difference in the number of questions posted by students across the three interaction modalities (text-only, voice-only, and mixed), H (2) = 3.13, p = .210. This null finding challenges the assumption that modality is a primary driver of engagement with EAICs.

AI Newbies rarely used advanced features (e.g., follow-up questions and recommended questions). Follow-up questions were used sparingly: approximately half of the students (n = 47, 49%) did not use them at all, and the interquartile range (IQR = 3) indicated that 75% used three or fewer follow-ups (M = 7.63, Mdn = 1.0). Similarly, recommended questions were underutilized: more than half of participants (n = 54, 56.3%) never used this feature, with a median of 0. Only 5.2% (n = 5) used recommended questions more than five times.

Despite this low feature adoption, AI Newbies did engage with the basic system. When they did, they predominantly chose mobile devices. Among students with recorded activity (n = 72), mobile devices accounted for 62.76% of all interactions compared to 37.24% for desktops. Device preferences were notably polarized: 40.62% of participants (n = 39) used exclusively mobile devices, while only 18.75% (n = 18) used only desktops.

Among those who engaged, session patterns varied considerably. Participants engaged in a mean of 5.54 sessions (SD = 7.82), ranging from 0 to 46. Session duration reached a maximum of 27.33 min (M = 2.50, SD = 4.21). A clear relationship emerged between session frequency and duration: participants with 1–5 sessions had shorter session durations (M = 1.31), while those with 20+ sessions had longer durations (M = 7.45). This pattern suggests that frequent users developed more sustained engagement.

This variability was even more pronounced in message frequency. Message activity was highly concentrated among a small subset of heavy users: the top 20% of participants accounted for 73.18% of all 1,760 messages recorded in the system. A quarter of participants (25.00%, n = 24) sent no messages at all, while 23.96% (n = 23) sent 20 or more. Heavy users reported using Cipherbot to catch up on missed classes: “I forgot the class schedule, and I used it to study to make up for it” (P176). This pattern reveals a substantial engagement disparity among AI Newbies.

Behavioral patterns showed remarkable consistency across all interaction modalities. Spearman's rank correlation analyses revealed stable relationships between engagement metrics regardless of learning conditions, with similarly strong correlations between message frequency and session count (rs > .95, p < .00001), message frequency and session duration (rs > .83, p < .00001), and message and follow-up interactions (rs > .77, p < .00001) across all modalities. This stability suggests that AI Newbies in all three conditions engaged with Cipherbot in fundamentally similar ways. The underlying structure of engagement appears to transcend modality differences. As illustrated in Figure 2a, the mixed group demonstrated substantially higher message counts and greater variability than the text-only and voice-only. The width of each violin represents the probability density of data points at different values. The embedded box plots indicate median and interquartile ranges. The y-axis displays the number of questions ranging from approximately −50 to 250. Results show that for AI Newbies, different interaction modalities do not have a statistically significant impact on their engagement; however, the mixed group showed a higher engagement level with Cipherbot. Figure 2b further shows that correlation patterns between message frequency and other engagement metrics remained consistent across all three modalities, strengthening the finding that AI Newbies engaged with Cipherbot in similar ways regardless of interaction condition.

Figure 2.

Behavioral differences. (a) Violin plot displaying the distribution of message frequency across three interaction modalities (text-only, voice-only, and mixed). (b) The results of Spearman's rank correlation analyses between each engagement metric and message frequency in all three interaction modalities and show the overall similarity in patterns.

4.2 RQ2: What is the Relationship Between Students’ Learning Style and how They Asked Questions from an EAIC?

While interaction modality did not differentiate engagement, learning style did. A Chi-square test revealed a statistically significant association between learning styles (see Table 2) and question themes (see Table 3), χ² (12, n = 1760) = 246.64, p < .0001. This stands in stark contrast to the null modality finding from RQ1. This section details how different learning styles engaged with Cipherbot in distinct ways.

Table 2.

Descriptive Distribution of Learner Style, Their Descriptions, Frequencies, and Corresponding Percentages.

Learning Style	Description	Number	Percentage
Social	Students who thrive in group settings and collaborative environments. They learn best through interaction with others and enjoy group discussions, teamwork, and peer teaching.	33	34.40%
Solitary	Students who prefer to study and work independently. They are comfortable with self-paced learning and may find solitary activities like reading or online courses more effective.	29	30.20%
Logical	These students are adept at reasoning, critical thinking, and problem-solving. They excel in activities that involve logic, numbers, and mathematical concepts.	15	15.60%
Verbal	Students who excel in language-based activities, such as reading, writing, and discussions. They prefer to learn through written or spoken words.	11	11.50%
Naturalistic	Students who have a strong affinity for the natural world and learn best when exposed to real-life examples from nature or when learning in natural settings.	8	8.30%
Visual-spatial	Students who have a strong ability to perceive and understand spatial relationships, often excelling in tasks that involve visualizing and manipulating objects in space.	0	0%
Total		96	100%

Table 3.

The Distribution of Question Themes, Their Descriptions, Frequencies, and Corresponding Percentages.

Question Themes	Description	Correspondence to Learning Theories	Frequency	Percentage
Conceptual Understanding and Exploration	Building deep knowledge through inquiry and exploration of fundamental concepts.	Constructivism (Cobern, 1993)	997	51.3%
Applied Learning and Problem-Solving	Applying knowledge to real-world situations and developing practical problem-solving skills.	Problem-Based Learning (Hamid et al., 2023)	540	27.8%
Assessment and Assignment Guidance	Providing clear instructions, criteria, and feedback to support successful completion of tasks	Feedback theory (Hattie & Timperley, 2007)	124	6.4%
Critical Analysis and Perspective Taking	Evaluating information critically while considering diverse perspectives and assumptions.	Transformative Learning (Fleming, 2018)	99	5.1%
Total		1760	100%

The participants’ learning style distribution was dominated by Social and Solitary learners, who together represented nearly two-thirds of students (64.6%). Social style constituted the largest group (n = 33, 34.4%), followed by Solitary (n = 29, 30.2%), Logical (n = 15, 15.6%), Verbal (n = 11, 11.5%), and Naturalistic (n = 8, 8.3%). This concentration enabled robust comparisons between Social and Solitary learners while limiting generalizability for smaller groups.

Students’ questions revealed a clear hierarchy of needs. The most common theme was Conceptual Understanding and Exploration (n = 997, 51.3%), focusing on main ideas and topic exploration. AI Newbies also actively sought to apply their learning: Applied Learning and Problem-Solving questions (n = 540, 27.8%) involved solving real-world problems. Less frequent were Assessment and Assignment Guidance (n = 124, 6.4%) and Critical Analysis and Perspective Taking (n = 99, 5.1%).

Learning style was associated with distinct questioning patterns (see Figure 3). Verbal learners showed a marked preference for Critical Analysis & Perspective Taking questions (n = 92), a theme notably underrepresented among other styles. All styles demonstrated interest in Conceptual Understanding & Exploration, with Solitary showing the highest frequency (n = 299), followed by Verbal (n = 240), Social (n = 221), Logical (n = 164), and Naturalistic (n = 73). For Applied Learning & Problem-Solving, Verbal (n = 156) and Solitary (n = 154) asked the most questions, while Naturalistic (n = 24) asked the fewest. Social learners demonstrated the highest engagement with Assessment & Assignment Guidance (n = 43). These patterns suggest that learning style significantly shapes not just how much students engage, but what they seek from an EAIC.

Figure 3.

Distribution of question themes by learning style. Each bar represents the frequency of questions asked within each theme.

4.3 RQ3: What Factors Encourage and Discourage AI Newbies from Engaging with an EAIC?

Speed, learning support, and ease of use motivated AI Newbies to engage with Cipherbot. Qualitative analysis of student responses (n = 28) revealed that fast response times (n = 7, 25%) and learning support features (n = 7, 25%) were most valued: “Provides information quickly within 5 s” (P14); “Responding to theory inquiries in the course” (P191). Ease of use (n = 6, 21.4%) also mattered: “Easy to use, quickly answers questions, no need to wait” (P67). Students additionally noted Cipherbot's performance (n = 5, 17.9%) and accuracy (n = 3, 10.7%) (see Table 4).

Table 4.

Cipherbot's Attributes That Support and Discourage Engagement among the Students (a Student can Give More Than one Attribute).

Attributes That Support Engagement	Description	Attributes That Discourage Engagement	Description
Fast responses(n = 7, 25%)	Students enjoyed the speed of information retrieval using Cipherbot	Difficulty in Usage / Unfamiliarity(n = 64, 58.2%)	Students expressed difficulty in using Cipherbot due to unfamiliarity
Learning Support(n = 7, 25%)	Cipherbot helps students for learning support.	Difficulty with Voice Input(n = 14, 12.7%)	The voice input was difficult to use
Ease of Use(n = 6, 21.4%)	Cipherbot is easy to use for studying	Performance Issues(n = 13, 11.8%)	Students experienced slow/laggy
Chatbot Performance(n = 5, 17.9%)	Students mentioned Cipherbot's effectiveness in searching for course information	Preference for Other Tools(n = 12, 10.9%)	Students have experienced and preferred others
Accuracy of Answers(n = 3, 10.7%)	Cipherbot's accurate responses that students can trust	Limited Usefulness(n = 7, 6.4%)	Cipherbot only answers from pre-upload files

However, technical and design barriers significantly hindered adoption. Analysis of barriers (n = 110) revealed that usability issues dominated (n = 64, 58.2%), including login difficulties (“When I use it, I always have to log in with my email before I can use it, and I feel like it wastes time,” P60), ineffective search functionality (“Hard to use, can't find what I’m looking for,” P136), and the lack of practical examples (“I find it quite difficult to use. And there are no practical examples to follow,” P155).

Voice input challenges were primarily reported by the voice-only group (n = 14, 12.7%): “My Cipherbot uses voice input, so every time I speak, it doesn't understand me correctly” (P103). Other issues included performance problems (n = 13, 11.8%), preference for alternative tools (n = 12, 10.9%), and limited usefulness due to Cipherbot's reliance on pre-uploaded materials (n = 7, 6.4%).

Perhaps the most striking finding: by course end, 77% of AI Newbies (n = 74) had started using other AI chatbots, primarily ChatGPT (52.7%, n = 39) and Gemini (36.5%, n = 27). This migration reveals a tension at the heart of EAIC design: students preferred the comprehensiveness of general-purpose chatbots over the controlled accuracy of curriculum-aligned systems. As one student noted about Cipherbot's limitations: “It's all theory without the necessary knowledge” (P172). Students who began the course with zero AI experience ended it preferring tools that offered breadth over pedagogical constraints.

5. Discussion

5.1 Research Contribution

Our findings challenge a prevailing assumption in EAIC research: that interaction modality is a primary driver of engagement. For AI Newbies, modality did not significantly affect engagement behaviors (p = .210)—but learning style did (p < .0001). This contrast has important implications for how educators design and deploy educational AI systems.

The null finding on modality challenges conventional EAIC design priorities. While the mixed group showed the highest engagement levels descriptively, the differences were not statistically significant. This suggests that designers may overemphasize modality as a design variable. More notable was the engagement disparity: the top 20% of users accounted for over 73% of all messages, while a quarter of students never engaged. AI Newbies also predominantly used mobile devices (62.76% of interactions), even without dedicated mobile support, indicating a mismatch between how students want to engage and how EAICs are typically designed.

Learning style predicted engagement patterns more reliably than modality, suggesting that personalization should prioritize learner characteristics over interface design. Verbal learners treated Cipherbot as a conversational partner, asking diverse questions, including those that required critical analysis. Solitary learners focused on deepening conceptual understanding, treating the EAIC as intellectual support. Social learners gravitated toward assessment guidance. These distinct patterns suggest that a one-size-fits-all EAIC design may fail to serve learners effectively.

Students valued speed, ease of use, and learning support, but technical barriers undermined these benefits. Usability issues (58.2%) and voice input problems significantly hindered adoption. These barriers help explain the most provocative finding. By course end, 77% of AI Newbies had started using ChatGPT, Gemini, or other tools, preferring their comprehensiveness over Cipherbot's controlled accuracy. This migration reveals a tension at the heart of EAIC design. Cipherbot was designed for high trust and a limited scope (Salminen et al., 2024), while general chatbots offer high comprehensiveness with variable trust (Boyd et al., 2022). Our findings suggest AI Newbies prioritize comprehensiveness over controlled accuracy. They may be willing to sacrifice reliability for breadth of information.

This creates a dilemma for educational institutions. Bounded systems are trustworthy but limited; open systems are comprehensive but potentially unreliable. Students’ preferences for broader responses suggest they may not fully appreciate the pedagogical value of constrained, curriculum-aligned information. Relying on comprehensive but potentially inaccurate information could undermine critical thinking skills, source evaluation abilities, and understanding of disciplinary boundaries. This finding challenges the assumption that student preferences align with optimal learning conditions.

5.2 Design Implications for EAICs

Based on our findings, we propose a framework to design EAICs for AI Newbies, summarized as Optimize, Personalize, Integrate, and Implement (OPI), to guide the development and application of AI technologies in educational contexts.

O ptimize for mobile. AI Newbies overwhelmingly preferred mobile devices (62.76% of interactions), yet most EAICs lack dedicated mobile support. This mismatch likely suppresses engagement. Designers should prioritize mobile-first interfaces with simplified functions for beginners.

P ersonalize based on learning style. One-size-fits-all EAIC design fails different learner types. Verbal learners sought critical analysis; Solitary learners sought conceptual depth; Social learners sought assessment guidance. Designers could detect these preferences through early interactions and adapt questioning strategies, accordingly, offering perspective-taking prompts to Verbal learners and self-directed exploration to Solitary learners.

I ntegrate features to encourage sustained engagement. Heavy users spent more time per session and found greater value in the system, yet most AI Newbies remained light users. Designers should build features that reward continued use, transforming short-session users into sustained learners while giving educators visibility into student engagement patterns.

5.3 Limitations and Future Research

These findings should be interpreted within three categories of constraints of sample, design, and measurement.

Sample limitations affect generalizability. We focused exclusively on AI Newbies in a single digital marketing course at one vocational institution. Future research should examine students across disciplines (STEM, humanities, professional programs), institution types, and geographical locations. Longitudinal studies could also track how engagement patterns evolve as students gain AI literacy.

Design limitations constrained what we could observe. We examined engagement from the student perspective only, without analyzing the full chatlog dynamics between AI Newbies and the EAIC. Additionally, Cipherbot's restriction to pre-uploaded course materials may have limited its perceived utility compared to general-purpose chatbots. Future research should investigate two-way conversational dynamics and explore hybrid approaches that maintain educational relevance while offering broader knowledge access.

Measurement limitations introduce potential bias. Survey data were self-reported, which could reflect participants’ perceptions rather than actual behaviors. The ten-week timeframe captured initial engagement but not long-term effects beyond the novelty period. Future studies should compare self-reported data with behavioral logs and extend observation periods. Finally, ethical questions remain about whether EAICs promote deeper learning or inadvertently encourage over-reliance on AI tools.

6. Conclusion

For AI Newbies encountering educational chatbots for the first time, how they interact with the system matters less than who they are as learners. This study reveals that engagement with EAICs is shaped more by learning styles than by interaction modalities. Modality, text, voice, or mixed, did not significantly affect engagement behaviors (p = .210), while learning style significantly predicted questioning patterns (p < .0001). Verbal, Solitary, and Social learners showed distinct engagement behaviors, suggesting that one-size-fits-all EAIC design fails to account for meaningful individual differences.

Students valued speed, learning support, and ease of use, but technical barriers hindered adoption. Most strikingly, 77% migrated to general-purpose chatbots by course end, preferring comprehensiveness over curriculum-aligned accuracy. This migration challenges assumptions about what students want versus what serves their learning.

The proposed OPI framework (Optimize, Personalize, Integrate) offers practical guidance for designing EAICs that account for these findings. As AI tools become increasingly embedded in education, designing systems that attend to individual learning differences—instead of assuming interaction modality is the key variable—may be critical for preventing the marginalization of students just beginning to navigate these technologies.