Principal Investigator: Christopher Chen, Lecturer in Chemical Engineering
Summary: Understanding how students interact with AI chatbots is critical to helping engineering educators better support both effective AI use and critical evaluation of AI output. However, student-AI interaction data is rarely accessible to instructors, as most commercial tools do not provide chatlog data. This project leverages a unique dataset of nearly 1000 student-AI interactions collected from a custom "novice" chatbot – intentionally designed to produce student-like errors – deployed in Material & Energy Balances (MEB) at Columbia University in Fall 2024. Because 75% of these interactions were voluntary study use rather than required course activities, this dataset offers a rare naturalistic window into how undergraduate chemical engineering students think with AI when learning course material. Using LLM-enabled Dialogue Act annotation validated against human coders, we will develop a taxonomy of student-AI interaction types from these chatlogs. We will then map these interaction categories against student performance on assignments requiring evaluation of AI-generated errors – a proxy for thinking about AI – to identify which interaction patterns correlate with stronger critical evaluation of AI output. Focus groups with student testers will provide think-aloud context to ground the chatlog-derived categories in student intent. This work will produce the first taxonomy of student-AI interactions for studying in chemical engineering education and demonstrate a scalable LLM-assisted methodology for chatlog analysis applicable across STEM disciplines.