2025-2026 Grants

Summary

During the 2024-2025 academic year and with the support of SOLER, I have conducted eleven data collection workshops (under IRB-approved protocol AAAU9821) with business students (both graduate and undergraduate), testing the efficacy of an early team-building design activity that uses generative-AI relative to one that utilizes physical materials. The final data collection occurred on April 14th, and data are currently being analyzed to be written up for publication (target: Journal of Management Education). Preliminary data analysis (from two pilot studies) show that while both activities are rated positively by students in terms of effectiveness and helping students understand different points of view, the AI design challenge uniquely enhances psychological safety (team members’ feeling safe to take risks within their team and their unique perspectives being valued by their team members) as well as teams’ ability to bring different ideas together to overcome challenges. 

With the current data, we have been able to assess the efficacy of the AI-mediated team building activity immediately following the exercise. In the next phase of the project, I will collect longitudinal data collection to see if this early team experience (i.e., AI design challenge) shapes group outcomes over the longer team cycle. I plan to collect longitudinal data in the 2025-2026 academic year during the EMBA student orientation (i.e., during the Learning Team Jumpstart sessions that I facilitate). Each semester, 20 to 25 EMBA learning teams are organized among the incoming EMBA student class. These learning teams complete coursework and projects together over the academic year, and I assess their team effectiveness during their time at CBS. In addition, I will run team-building sessions with professional teams and track their group functioning over time.

Summary

Continuing our work in Summer 2024 - Summer 2025, we plan to conduct the same controlled experiment for Simulation 4 in Summer 2025. In the Fall 2025 semester, we will again carry out the controlled experiments for Simulations 3 and 4. The purpose of repeating some experiments in different semesters is to increase the sample size of the experimental and the control groups. In the Spring 2026 semester, we will conduct controlled experiments for Simulations 5, 6, and 7. We will also return to the completed simulations and address the issues raised in focus groups. For both the Fall 2025 and Spring 2026 semesters, we will measure students’ attitudes toward econometrics separately at the beginning and the end of each semester, before and after using MetricsMentor. We are still evaluating the possible variations, such as omitting the actual lecture that teaches the material in the simulations and comparing the video (control) group and the simulation (experimental) group results. Since these videos will eventually be available with a click within each simulation, we are also considering a textbook group as the control. The textbook group will have access to the textbook between the pre- and post-quizzes. The textbook group results will be compared to the simulation group results using differences-in-differences methodology. Ultimately, our goal is to make MetricsMentor accessible to all instructors for use in their econometrics classes. After completing all seven simulations and conducting the experiments in those classes, we will present the results at the CTREE (Annual American Economic Association Conference on Teaching and Research in Economic Education).

Summary

Continuing our work with SOLER, we are currently exploring two potential directions for the next phase of the project:

1. Qualitative Expansion
A more qualitative approach would involve analyzing the discourse and interaction dynamics in the recorded feedback sessions. This would allow us to better understand how feedback is delivered—examining the linguistic features, tone, and techniques instructors use when providing feedback. We would also analyze how students engage (or disengage) during feedback sessions by documenting their verbal responses, questions, clarification requests, and non-verbal cues indicating comprehension or confusion. These observations would help identify whether students are passive recipients of information or active participants in a collaborative learning process. In particular, we aim to transcribe and code the feedback meetings to identify patterns of student participation and engagement, such as moments of resistance or acceptance.

2. Exploratory Technical Integration
Alternatively, we are considering an exploratory technical direction that adapts methods developed in related SOLER projects. For example, using machine vision software to track student attentiveness (Singh et al., 2023; Zhang et al., 2020) during feedback sessions or presentations. This could yield valuable insights about behavioral indicators of feedback uptake. While this direction is more technologically demanding, we are in preliminary conversations with possible collaborators (e.g., Prof. Alfredo Spagna) to assess feasibility.

Regardless of the direction chosen, we aim to complete analysis of:
• Student presentation videos to assess progress in targeted areas.
• Feedback session recordings to understand patterns of feedback delivery.
• Open-ended survey responses about perceived impact on speaking skills. 

We are also preparing to submit findings to Chinese Language Teachers Association Annual Conference—one of the leading international conferences in Chinese language pedagogy next year— where we believe our work on feedback dynamics will contribute meaningfully to the field.

Summary

To continue the iterative investigation of tailored behavioral nudges in the Mailman Core Quant course, this next phase will expand participation and enrich our descriptive analyses of student behavior following nudges. Instead of a randomized controlled trial, we will implement a staggered rollout model in which all students receive nudges but at different times, allowing for an effectiveness evaluation of integrating tailored nudges for all students. We also aim to explore when and how students respond to nudges: specifically, what actions they take in the course platform after receiving them. This approach will allow for sequential analyses of student behavior to better understand not just whether nudges are effective, but how they unfold in practice (Chen, Knight & Wise 2018).

This new phase of research will have two primary aims. Aim one of this phase will include all students in the Core. Because all students will be receiving the intervention, we do not need to randomize and therefore should be able to receive exempt status for this intervention. This aim will be referred to as effectiveness evaluation of integrating tailored nudges for all students. In the first aim, the effectiveness of tailored nudges will be assessed for all students, deploying nudges in rolling cycles across the semester. The effectiveness of nudges will focus specifically on video watching behaviors, which were found to be the most responsive to tailored nudges across all learning motivation groups in the previous study phase. By allowing opt out consent with IRB’s approval this will allow us to increase validity of the effectiveness of nudges among students who are less engaged but were screened out by the consent requirement. Aim two of this phase will be rich, descriptive sequential analysis. Up until this phase, our research has examined whether or not students responded to the nudges by engaging in a specific behavior. For this phase, we will use clickstream data from CourseWorks and Panopto to conduct sequential analysis to explore what behaviors follow the initial targeted behavior. It has been suggested in primary previous research that sequential analysis may be helpful in identifying what behaviors students take after being nudged that may increase their engagement but are not identical to the behaviors that the faculty have pre-specified.

Summary

In 2023, our research investigated how large language models (LLMs) can be effectively leveraged in a real estate finance course and in professional practice. We chose the Joint Venture (JV) Waterfall (an essential real estate finance concept) as a learning objective and used a mixed-methods approach to evaluate the effectiveness of ChatGPT as an instructional tool, assigning students to either a control group with traditional instruction or a treatment group using ChatGPT to independently complete a JV waterfall modeling assignment in Microsoft Excel.

In our 2023 iteration of the study, we assessed ChatGPT as a replacement for classroom instruction by comparing classroom instruction without ChatGPT vs. ChatGPT without classroom instruction. At the ARES conference, the discussant suggested a follow-up iteration of the study that would assess ChatGPT as a supplement to classroom instruction by introducing a new comparison group: ChatGPT WITH course instruction, again compared to classroom instruction without ChatGPT. Our finding that ChatGPT impaired learning provides an important counterpoint to a number of studies reporting enhanced student performance with AI assistance in professional schools (Kavadella et al., 2024). Our proposed second iteration examining ChatGPT WITH course instruction is required to draw a complete conclusion about whether LLMs are detrimental to learning or whether they can supplement but not replace classroom instruction (Altamimi et al., 2023).

Summary

Reflective writing is widely recognized as a pedagogical tool that encourages metacognition and student self-awareness. However, there is little empirical evidence directly linking the structural features of reflections to measurable learning outcomes. This project will investigate whether classical features of reflective writing, as described in educational theory, are predictive of success on subsequent exams. In tandem, it will also generalize machine learning and natural language processing techniques typical to the behavior change and cognitive science literatures to uncover novel critical features of reflective writing. This study will analyze reflections from a Spring 2025 introductory psychology course to identify textual features associated with improved exam performance. Ultimately, this work will build theoretical footing for understanding the mechanisms by which reflection impacts future behavior as well as lay the foundation for adaptive, data-driven tools to support reflection as a learning practice.

Summary

This project seeks to enhance student learning outcomes in a new online Master of Science in Business Analytics (MSBA) program at Columbia Engineering, to be launched in the Spring 2026 semester. Targeting adult, mid-career professionals with technical backgrounds, this cohort-based program will integrate real-time learning analytics to identify early signs of student disengagement and enable timely, data-driven support interventions. Our research aims to evaluate the impact of analytics-informed strategies on student persistence, performance, and satisfaction. We will develop a dashboard for instructional and support staff (student success team), collect and analyze behavioral data across multiple learning platforms (LMS, video platform, coding environments, AI tools), and gather qualitative feedback from students, instructors, and course assistants. This project contributes to the Scholarship of Teaching and Learning (SoTL) and Learning Analytics by examining how analytics can inform inclusive, responsive teaching in online graduate education. Our hypothesis is that early intervention enabled by behavioral data can significantly improve retention, engagement, and instructional quality in online, cohort-based programs.