Projects

Title: Nudges in public health education: Tailored nudges to increase engagement in quantitative analysis coursework

Principal Investigator: Christine Mauro, Assistant Professor of Biostatistics, Mailman School of Public Health (MSPH)

Award Date: Fall 2024 - Spring 2025

Status: 

Summary: To expand and enrich research activities of Phase 3 of this iterative project with increase participation and, in turn, statistical power. We will assess, in a randomized controlled trial, the impact of tailored educational nudges on student engagement and learning outcomes in an interdisciplinary biostatistics and epidemiology Core course. We hypothesize that tailored nudges, compared to generic nudges, will result in significantly higher 1) engagement with course materials and 2) learning outcomes. We further hypothesize that the effects of tailored nudges may differ by student characteristics, including baseline quantitative skills.

*********************************************************************************************************************

Title: Collaborative Art & Play as Practice: Learning Inclusive Leadership Skills with AI-generated Art

Principal Investigator: Dr. Ashli B. Carter, Lecturer, Management

Award Date: 2024 - 2025 

Status: 

Summary: Teaching inclusive leadership in business schools remains challenging in part because doing so requires developing students’ social-emotional skills—like getting curious about the self and others, and approaching, rather than avoiding, intergroup differences (Luckman, 2023). These skills cannot be learned through passive channels alone, that are often found in higher education. In the current project I utilize and test the efficacy of text-to-image generative-AI as a novel tool for business students and professionals to practice inclusive team leadership behaviors through collaboration and imaginative play exercises.

Art generative-AI is a promising tool for practicing inclusive behaviors as it 1) quickly makes what is abstract and theoretical, real and concrete; 2) elicits experiences of awe and amazement which can serve as a precursor to enacting prosocial behaviors (Piff et al., 2015); and 3) integrates arts-based (versus analytical) activities in the classroom which have been shown to enhance social-emotional skills (Yang et al., 2023)—all while allowing participants to develop their technical skills (i.e., prompt generaJon) as AI tools proliferate in day-to-day business functioning.

During the 2023-2024 academic year and with the support of SOLER, I have designed three team exercises utilizing text-to-image generative-AI for business students and professionals to engage in imaginative group play and practice inclusive leadership behaviors. The first team exercise created, a Design Challenge, was piloted in the MBA classroom with ~400 students. In addition, I have created and refined two additional exercises—Art Matching Task and Imaginative Story Illustration—with ~300 professionals. Preliminary data (both quantitative and qualitative) suggest these activities enhance interpersonal communication within teams, invite team members to get curious about their differences and unique backgrounds, and allow diverse teams to form stronger bonds early in their development and work trajectory.

With these pilots, I have also refined numerous exercise logistics, including staffing and technology requirements, appropriate assessment questionnaires and generative-AI software applications (i.e., testing Midjourney, Adobe Firefly, and ChatGPT4’s DALL-E). Importantly, I have also identified a comparison activity (the Marshmallow Design Challenge, a group exercise that does not utilize generative-AI) with which to compare the use of generative-AI in the development of inclusive team leadership behaviors. Finally, I have received IRB approval to conduct this comparison through an experimental design in the Lead classroom as well as in executive workshops with both student and professional populations. Next steps involve data collection and analysis across ten workshops (~500 participants) scheduled for the 2024-2025 academic year.

*********************************************************************************************************************

Title: Testing the Efficacy of Dialogic Feedback in Enhancing Public Speaking Skills

Principal Investigator: Zhongqi Shi, Senior Lecturer, East Asian Languages and Cultures

Award Date: Spring 2024 - Summer 2024

Status: 

Summary: Effective public speaking skills are essential for college students to express ideas clearly, persuade others, and present themselves confidently. However, traditional feedback methods often leave students feeling confused and disengaged. This study proposes using a novel "dialogic feedback" approach, defined as interactive exchanges where interpretations are shared, meanings negotiated and expectations clarified, to potentially enhance feedback effectiveness. We aim to assess the efficacy of dialogic feedback in improving students' presentation skills and perceptions compared to traditional didactic feedback. Dialogic feedback offers students opportunities for enhanced comprehension and engagement through an interactive process. Employing a longitudinal research design, this study seeks to empirically assess the cognitive, emotional, and motivational benefits of dialogic feedback versus didactic feedback. Participants will be recruited from Business Chinese classes that emphasize developing communication and public speaking skills. They will be randomly assigned to either an experimental group receiving dialogic feedback sessions or a control group receiving didactic (one-way) feedback sessions. We hypothesize that students receiving dialogic feedback will outperform the control group across measured outcomes related to feedback retention, perceptions, and motivational behaviors. Data will be collected through post-presentation student surveys and research assistant evaluations of performances. This study aims to contribute insights into effective feedback practices to improve language education. Successful findings will be shared to inspire broader adoption of dialogic feedback, ultimately enhancing student learning and preparing them for success.

*********************************************************************************************************************

Title: StreamLine

Principal Investigator: Talha Siddiqui, Lecturer, Chemistry

Award Date: Spring 2024 - Fall 2024

Status: 

Summary: The rise of digital media and technological advances has led to a significant use of instructional videos across educational platforms, particularly in flipped classrooms as well as public streaming services. An increase in these practices has been notably observed, for example, from 2010 to 2015 in K-12 education, and the trend continued through the COVID-19 pandemic, which necessitated a transition to digital instruction. Our proposal, StreamLine, seeks to optimize engagement by investigating the efficacy of “chunking” instructional content in videos used for bridge modules for organic chemistry I courses as a preamble to the content they will learn in their upcoming studies. Bridge modules are a series of recorded lectures that bridge concepts between courses from different semesters. The hypothesis poses that shorter, concept-driven videos will enhance student understanding and retention of information compared to traditional longer videos. The existing literature suggests that segmenting content into manageable portions can significantly reduce cognitive load; therefore, this is expected to improve attention and information retention. Previous methodologies, which largely relied on self-reporting and lacked interactive elements, which yielded inadequate results regarding true engagement accurately. StreamLine will utilize eye-tracking technology alongside traditional survey methods to include a comprehensive analysis of student engagement with supplemental video content. We will evaluate “chunked” versus full-length video content by using empirical methods to assess attention metrics, with the goal of refining video instructional design for enhanced educational outcomes.

*********************************************************************************************************************

Title: “FeedForward”: An Artificial Intelligence (AI) Tool for Creating Summative Feedback and Individualized Learning Plans (ILPs) for Surgery Clerkship Medical Students

Principal Investigator: Roman Nowygrod (MD), Department of Vascular Surgery, Columbia University Vagelos College of Physicians and Surgeons

Award Date: Summer 2024 - Summer 2025

Status: 

Summary: At the heart of medical education is the exchange of feedback, a critical tool in shaping a student's clinical and professional growth. Our project, "FeedForward," proposes the innovative use of artificial intelligence (AI) to improve the feedback process for medical students during their surgical clerkship at Columbia University's Vagelos College of Physicians and Surgeons. Currently, providing personalized and high-quality feedback is a time-consuming task that challenges educators, particularly in the context of formative (ongoing) and summative (conclusive) assessments. This project targets the inefficiencies in creating summative feedback and individualized learning plans (ILPs) for medical students, addressing the specific needs for scalability and quality enhancement in feedback mechanisms. The project introduces a custom-developed AI chatbot, leveraging Columbia's ChatGPT Enterprise, to automate the initial creation of summative feedback narratives and ILPs for medical students at the conclusion of their general surgery clerkship. This tool is designed to analyze compiled student performance evaluations (SPEs) and generate comprehensive, detailed, and actionable feedback that aligns closely with medical education performance objectives (MEPOs). Our project specifically aims to evaluate if AI-generated feedback, compared to human-generated can 1) match or exceed quality in terms of accuracy, relevance, and detail 2) match or exceed utility for various stakeholders (students, clerkship directors, program directors, and medical student performance evaluation (MSPE) writers), and 3) provide a novel tool to increase efficiency for those tasked with providing summative feedback. To assess these, we plan to use a combination of qualitative and quantitative methods to analyze the AI and human-generated feedback as well as direct feedback from a variety of stakeholders including students, clerkship directors, program directors, and MSPE writers on the clarity, relevance, and helpfulness of the AI-generated feedback. "FeedForward" aims to revolutionize the feedback process in surgical education by integrating AI, thereby improving both the quality of education and the operational efficiency for educators. With this project, we anticipate setting a benchmark for the future integration of AI across various educational settings, demonstrating substantial benefits for both learners and educational institutions.

*********************************************************************************************************************

Title: Preparing for college through COVID: impacts of disrupted and online high school learning on general chemistry readiness

Principal Investigator: Christopher Eckdahl, Lecturer, Chemistry

Award Date: Summer 2024 - Summer 2025

Status: 

Summary: Online and interrupted learning caused by the COVID-19 pandemic has led to decreases in high school students’ performance on standardized tests across all subjects and to decreases in their self-reported academic self-efficacy. Many students who experienced learning disruptions in high school are now in college, and future classes of college students will have experienced similar disruptions in earlier years. Higher education instructors, including those in chemistry, have felt that students struggle in ways that they did not use to and attribute this perceived change to COVID-related learning disruptions. However, no quantitative study has identified this effect in introductory chemistry, likelydue to the statistical challenge of isolating COVID-related discrepancies from other year-to-year changes in exam scores. This study proposes to surmount this statistical challenge by using Columbia’s large population of nontraditional students as an internal control. Specifically, the change in traditional vs. nontraditional exam performance will be compared for pre-2020 and post-2020 iterations of General Chemistry. Past years’ questions will be coded based on content, math or reading requirement, and complexity. Specific skills that have been strongly impacted by COVID-related learning disruption will then be identified and could be the targets of future curricular interventions.

*********************************************************************************************************************

Title: Practices and Implications around Concealment of Social Class Identities among Graduate Students in Engineering

Principal Investigator: Paul Ingram, Kravis Professor of Business, Columbia Business
School

Award Date: Summer 2024 - Summer 2025

Status: 

Summary: Processes of identity management are key to understanding the experiences of disadvantaged individuals in education institutions, and therefore, inequality. We plan to use a novel “whole identity” approach, combined with natural language processing, to examine concealment of identity elements associated with lower social class standing among graduate students in engineering at Columbia University. Evidence elsewhere suggests that students in elite professional programs are motivated to conceal lower social class identity elements to avoid penalties in the job market, out of social embarrassment, and to appear as more attractive partners for building professional networks. These arguments, however, have been developed among law and business school students, contexts where the importance of social/cultural status may be higher relative to technical skills than among engineering students. It matters whether engineering students do conceal lower social class identities, as identity concealment is associated with lower wellbeing, lower performance, and lower commitment to the context in which concealment occurs. We will also examine the intersection of identities around social class, race and gender to test, for example, whether social class concealment is more or less common among female students. If the data collection opportunity allows, we also intend to conduct experiments aimed at reducing the costs of concealment and helping the students achieve better outcomes in their programs and beyond.

*********************************************************************************************************************

Title: Evaluating the Impact of MetricsMentor - an interactive visual graphical platform for econometrics courses

Principal Investigator: Seyhan Erden, Lecturer, Economics 

Award Date: Summer 2024 - Summer 2025

Status: 

Summary: This project evaluates the hypothesis that a new innovative course design enhances students’ learning outcomes in econometrics. Currently, the course relies on traditional lecture-style teaching, which may not fully engage students or help them grasp complex concepts like endogeneity problems in regression analysis. To address this gap, we are developing MetricsMentor, an interactive online platform that provides a hands-on learning experience. This platform allows students to explore different scenarios and visually understand the effects of specific endogeneity problems and their solutions. We will conduct controlled experiments to compare the effectiveness of the platform with traditional lecturing methods. Students will be randomly assigned to either use MetricsMentor or receive traditional instruction, and their performance will be assessed through quizzes before and after each simulation. Additionally, we will measure students’ attitudes toward econometrics before and after using MetricsMentor. By using both quantitative and qualitative methods, we aim to provide a comprehensive assessment of the platform’s impact on student learning outcomes and attitudes. We expect MetricsMentor to improve both the immediate comprehension of contents and overall attitudes toward econometrics among students.

Title: Evaluating augmented reality for embodied learning in introductory biochemistry

Principal Investigator: Brent Stockwell, Professor, Biological Sciences

Award Date: Summer 2023

Status: Completed, manuscript in prep.

Summary:  In response to the COVID-19 pandemic, which prompted a shift to online learning, interest in virtual reality (VR) as a tool for teaching biochemistry emerged. Through SOLER-supported and CUIT-supported studies conducted from 2020-2022, the potential of VR in enhancing student engagement and understanding of disciplinary concepts was explored, revealing positive student perceptions despite technical challenges. Now, with instruction returning to classrooms, attention turns to augmented reality (AR) as a complement to in-person learning, leveraging advanced hardware and software like the Quest Pro headsets and Nanome program. Collaborating with the CTL, this iteration aims to integrate embodied learning principles and explore AR's impact on student outcomes, motivation, and interest in further study or professional opportunities. By incorporating qualitative research methods and innovative pedagogical approaches, the study seeks to elucidate AR's potential to create immersive learning environments that foster meaningful connections and deeper understanding of complex biochemistry concepts.

*********************************************************************************************************************

Title: Flexible neuroscience- and technology-driven frameworks for the study of classroom engagement

Principal Investigator: Alfredo Spagna, Lecturer, Psychology

Award Date: Summer 2023

Status: Ongoing, manuscript submitted

Summary: Classroom engagement has a crucial impact on academic success. However, the role of affective, cognitive, and behavioral components of engagement in classroom learning remains elusive. We conducted a study within a real-world undergraduate classroom using a non-invasive, research-based approach to clarify the role of these components. We employed portable EEG headsets to measure cognitive engagement via theta/beta ratio, in-class quizzes to assess content retention, and post-class subjective questionnaires to index affective engagement by measuring feelings of learning and attention. As expected, content retention was positively related to affective engagement, while cognitive engagement positively related to content retention. Interestingly, however, cognitive engagement was negatively related to subsequent affective engagement, while holding content retention constant. Crucially, accounting for both of these sources of variance in the classroom provided a much more robust model of affective engagement, underscoring the need for real-world, multimodal learning analytics. We discuss how these approaches enable real-time studies of classroom engagement and can be integrated to develop neurofeedback interventions.  

*********************************************************************************************************************

Title: Promoting engagement in quantitative data analysis coursework via tailored educational nudges

Principal Investigator: Samantha Garbers, Associate Professor, Population and Family Health 

Award Date: Summer 2023

Status: Ongoing, manuscript submitted

Summary: Nudge theory – the behavioral economics idea that subtle environmental cues influence decision making – shows promise in education in the form of reminders, social comparisons, gain/loss framing, or goal setting. However, prior research, including ours, suggests that nudges are effective only when they tap into motivations for learning and engagement. We propose to test the hypothesis that nudges tailored to students’ motivations will enhance engagement and learning outcomes in an interdisciplinary Public Health course. Mailman School of Public Health (MPH) students spend their first semester in a Core curriculum that provides essential interdisciplinary knowledge. Research Methods and Application is a Core course taught by Biostatistics and Epidemiology faculty in a hybrid format (“Core Quant”). It is the most challenging Core course: 3% of students in Fall 2022 did not successfully complete it despite the availability of tutoring. Furthermore, student engagement with supportive materials such as slide decks and answer keys is low. In two SOLER-supported studies (2021-2022), we investigated the use of educational nudges in Core Quant. In Study 1, a randomized controlled trial revealed that nudges linking behavioral engagement (e.g., accessing readings, watching lecture videos) with assessment grades did not significantly affect student actions. Our findings were published in March 2023. In Study 2, we implemented a mixed-methods study (manuscript in preparation) of students’ motivations and behaviors featuring interviews and forced-choice analyses comparing nudge messages that will inform the development of tailored nudges to be tested in our proposed third study. Persona creation: We will create three personas with narrative descriptions and visual avatars for students to select, aligned with motivation profiles identified in our qualitative analysis and corresponding to established features of nudges: learners (intrinsically motivated), doers (task-oriented), and performers (grade-motivated). Nudge finalization: Each persona will have visual, tailored nudges deployed throughout the semester; a set of text-only generic nudges (without appeal to motivation) will be the control condition. Experimental study design: Students will identify their learning persona at the start of the semester; within each persona, students will be randomized 1:1 to receive either a generic or persona-specific nudge. The generic nudge will be embedded in a weekly email to students that summarizes upcoming work across the Core courses. Tailored nudges will be deployed through Courseworks’ Group feature and Announcement function. For blinding, the groups will be named numerically rather than by persona label; group membership will not be visible to students. Students randomized to the generic group, or who opt out, will be assigned to a group but will not receive a nudge via the announcements. We will compare the effect of tailored vs. generic nudges within each persona group and across all groups on the following outcomes: engagement with course materials and learning supports, pre-post change in trust and confidence scales, post-course empirical knowledge score, and course grade. If tailored nudges enhance engagement learning outcomes in this study, they can be implemented in courses with similar challenges within and beyond MPH. 

*********************************************************************************************************************

Title: Assessing graduate student attitudes toward ChatGPT and its effectiveness as a teaching tool for real estate finance

Principal Investigator: Chris Munsell, Associate Professor of Professional Practice, GSAPP

Award Date: Summer 2023

Status: Completed, manuscript in prep.

Summary: In the master's level course Real Estate Finance II, one challenging learning objective requires students to apply the mechanics and theory of time value of money to the joint venture waterfall, an essential concept. Students struggle to achieve this learning because it requires an advanced knowledge of Excel and financial theory. We believe that this project will be innovative because it will allow students and ourselves to understand how effective or ineffective AI can be as an external teaching aid in real estate finance. We propose a mixed-methods study to test the effectiveness of large language model generative AI (such as ChatGPT) as an alternative instructional practice in this context, with students using the technology in one homework assignment. We will compare students who are randomly assigned to complete the homework with or without the use of AI. We predict that students in the AI group will exhibit enhanced performance on their homework assignment, but inferior performance on their final exam. The non AI group we predict that they will not perform as well as the AI group. We will also ask both student groups to respond to open-ended questions that address their learning process, academic integrity reflections, and ethical decision making in real estate finance. 

*********************************************************************************************************************

Title: Collaborative art & play as practice: learning inclusive leadership skills with AI-generated art

Principal Investigator: Ashli Carter, Lecturer, GSB

Co-Sponsor: Innovative Course Module Design Grant (CTL)

Award Date: Summer 2023

Status: Ongoing

Summary: More than ever, emerging leaders must learn how to create inclusive environments wherein individuals with diverse talents, backgrounds, and perspectives can thrive, and a culture of learning, growth, and experimentation can flourish. However, the social-emotional skills needed for inclusive leadership are learned by doing and not just knowing. When it comes to practicing inclusivity, there often exists a gap between what individuals understand on an abstract level versus their actual behaviors. This may occur in part because practicing inclusive behaviors – particularly as a novice – involves interpersonal risk-taking, which individuals may especially avoid in diverse settings. Group-based, imaginative play has recently been explored as a means for diverse teams to foster stronger team bonds and more effective collaboration. This play enhances team functioning by reducing intergroup anxiety and fostering creativity, trust, and a sense of psychological safety. We propose to evaluate a new tool for facilitating group-based, imaginative play and teaching inclusive leadership behaviors – art generative-AI. Art generative-AI is a promising tool for practicing inclusive behaviors as it allows students to engage in key social-emotional skills of inclusive leadership (i.e., interpersonal risk-taking and novel relationship building) while having fun together. Furthermore, generating art with AI tools may also aid learning inclusive behaviors by inspiring moments of collective awe and facilitating rapid and repeated experiences of collaboratively turning abstract ideas into reality. In the following proposal, we outline a study testing the effectiveness of a group-based, imaginative play exercise facilitated through art generative-AI for enhancing inclusive behaviors within MBA student teams.
 

*********************************************************************************************************************

Title: Sense of scale embodied: investigating the impact of object-based learning on undergraduate STEM education

Principal Investigator: Bradley Pitcher, Lecturer, Earth & Environmental Sciences

Award Date: Summer 2023

Status: Completed, manuscript in prep.

Summary: This research proposal aims to evaluate the hypothesis that incorporating physical objects and movement, using an embodied learning approach, enhances student learning and cultivates a better comprehension of scale. This is a major learning objective of Frontiers of Science (FoS), a core course for Columbia College first-year students. The traditional approach of teaching sense of scale via a written activity that involves sorting a list of objects inadequately addresses the learning objective. The innovative approach proposed in this study, sense of scale embodied (SSE), requires students to manipulate M&M candies to visualize large numbers, areas, and volumes, and to physically move around the room to estimate relative sizes and distances of planetary objects. The impact of the SSE will be compared against a traditional approach as measured by student performance on quizzes administered immediately after the activity and at the end of the semester. FoS is particularly well suited for this experiment because over 500 students are enrolled each semester and instructors each teach two sections, allowing them to teach one SSE and one traditional section. The results will be shared with FoS instructors and directors, and if SSE proves to be more effective, it will be permanently included in the curriculum. This proposed research is innovative as it would provide a unique direct comparison of the impact of an object-based learning activity on the development of students’ sense of scale, thereby greatly contributing to the growing body of research in favor of embodied pedagogical techniques in STEM higher education.

*********************************************************************************************************************

Title: Evaluating the effectiveness of an upstander training across modalities

Principal Investigator: Jean Marie Alves-Bradford, Associate Professor, Psychiatry (VP&S)

Co-Sponsor: Innovative Course Module Design Grant (CTL)

Award Date: Summer 2023

Status: In progress

Summary: Medical students experience bias incidents at alarmingly high rates of over 80% in some student samples. Allyship and Upstander skills are part of the AAMC Diversity Equity and Inclusion competencies, yet many schools have not yet implemented bias-response training. Students, faculty, and clinical team members can be trained to be “Upstanders,” those who act and intervene on behalf of themselves and others. Bias-response training has been shown to increase medical students’ self-assessed confidence responding to bias incidents. With prior Provost support, we developed trainings to facilitate skill uptake. Investigating causal relationships between prior experience, topic, and modality on effectiveness of learning has direct connections to the science of learning and implications for all teaching. However, this support was not designed to draw causal conclusions. Additionally, while many prior studies comparing asynchronous online versus offline teaching modalities have demonstrated equal effectiveness, online modules intended to increase confidence responding to microaggressions showed lowest effectiveness when compared to other topics within cultural competence. Furthermore, there is emerging evidence that prior experience with microaggressions impacts effectiveness of bias training. Therefore, this project has three core goals - to strengthen existing correlational findings with causal conclusions, to evaluate the effectiveness of in-person versus asynchronous online modality in teaching upstander skills, and to explore the relationship between existing experience with microaggressions and uptake of Upstander skills. Given that this project addresses a required medical education competency, this grant will be instrumental in determining how to move forward with efficiency and efficacy.

*********************************************************************************************************************

Title: Exploring the use of Large Language Models to generate text versions of neuroscience lectures

Principal Investigator: Darcy Kelley, Professor, Biological Sciences

Award Date: Summer 2023

Status: In progress

Summary: We propose a feasibility exploration of using LLMs (Large Language Models) to create a text version of the lectures comprising Developmental and Systems Neuroscience. The project addresses the following problem: the rapid pace of discovery renders commercial neuroscience texts obsolete upon publication. Undergraduates rely on texts to support learning, but what if those texts are wrong (out of date). Our goal is to pilot – using an LMM – a living, written text based on audio recorded in Spring 2023 lectures delivered by the PI. Each 2023 lecture was accompanied by illustrations (figures from the primary literature, reviews and text figures) in PowerPoint format. A secondary goal is to replace the 2023 figures (heterogeneous in style and source) with original illustrations in a uniform style, an important element for visual learners. The project is innovative because it will provide a means to replace a static, outdated source of information with a dynamic, current, written and visual resource that can be updated in real time. Ascertaining impact on 2024 students relative to 2023 will be a major challenge for this project; CTL Lead Teaching Fellows and other personnel will support the development of assessment frameworks. The value of original illustrations will be equally challenging to assess, but we will work closely with Nicoletta Barolini (CU Art Director, Office of Communications & Public Affairs) to develop and evaluate new illustrations. 

Title: Evaluating augmented reality (AR) using Microsoft Mesh and Hololens 2 for teaching deep expertise in introductory biochemistry

Principal Investigator: Brent Stockwell, Professor, Biological Sciences

Co-Sponsor: CUIT Emerging Technology Teaching and Learning Grant

Award Date: Summer 2022

Status: Deferred to 2023-2024

Summary: Learning biochemistry is challenging for students using chalkboards, PowerPoint or Zoom, due to 3D aspects of molecular conformations. AR/VR offer a solution—they can be more immersive and enhance attention due to the lack of distractions, and allow engagement with 3D objects, such as proteins, that cannot be directly experienced in the real world. We previously performed studies with support from the Provost’s Office and from SOLER using VR to teach biochemistry using Oculus Quest 1 headsets and Spatial.io, meeting with students to discuss 3D aspects of biochemistry, and Quest 2 headsets with the online platform Glue. We found that using VR was impactful in terms of engagement and learning improvements. We found that many students are highly engaged during the in- person experience as well as the VR experience, but that rendering detailed proteins and their motions is challenging with the limited processing power of the Quest 2 headsets. Most critically, we found that students need to be able to translate the insights from interacting with 3D models of proteins back to representing proteins in two dimensions in order to build on their knowledge with subsequent reading of papers, textbooks, and other 2D representations of 3D molecules. This translation from 3D back to 2D is difficult to do in VR because it is difficult to simultaneously view 3D and 2D representations of molecules with high fidelity. We hypothesize that Augmented Reality (AR) will provide the missing link in translating deep insights into the 3D structure-function relationship in protein molecules into 2D representations of these molecules that can be used outside of the AR/VR format.

*********************************************************************************************************************

Title: Neuro-DBER: A flexible neuroscience- and technology-driven framework for Discipline-Based Education Research

Principal Investigator: Alfredo Spagna, Lecturer, Psychology

Co-Sponsor: CUIT Emerging Technology Research Grant

Award Date: Summer 2022

Status: Ongoing, manuscript submitted

Summary: What predicts student success? The classroom is a dynamic setting where students and instructors of diverse backgrounds exchange ideas in the pursuit of learning. Many variables complicate this collaborative paradigm: composition changes due to student absence; content difficulty increases as the course deepens; student fatigue accumulates as the semester progresses. While the relative contribution of complicating variables has been studied in isolation, a comprehensive understanding of their synergistic effects remains elusive. What can be done to help instructors and students integrate solutions to these well-known challenges into a cohesive framework? This project aims to develop a flexible neuroscience and technology-driven framework for Discipline-Based Education Research (Neuro-DBER) to revolutionize teaching and learning in the classroom. Our system merges cognitive neuroscience, education, and innovative technology to enable investigations of the relationships between subjective experience, body language, , real-time learning, and neural measures of in-class activities. Neuro-DBER is a flexible technological framework suitable to any field of study to address critical questions about classroom experiences, such as the gap between a student's subjective feelings versus actual learning measured. Featuring integration of advanced research technology, our framework facilitates thoughtful design of in-class activities that will thereby transform the way we teach and learn, yielding an inclusive environment conducive to effective teaching and addressing students’ distinctive needs.

*********************************************************************************************************************

Title: Increasing student engagement with course content and learning supports through mixed-methods mapping of motivation and learning behaviors

Principal Investigator: Samantha Garbers, Associate Professor, Population and Family Health 

Award Date: Summer 2022

Status: Ongoing, manuscript submitted

Summary: This project aims to conceptualize, quantify, and explore qualitatively student engagement and motivations in an interdisciplinary biostatistics and epidemiology Core course. At the Mailman School of Public Health, all incoming students spend their first semester participating in an interdisciplinary Core curriculum. The most challenging course in the curriculum, Research Methods & Application - Quantitative Data Analysis, is taught in a hybrid format with synchronous and asynchronous content. Student supports have been deployed in recent years, but some students continue to face challenges in successful completion of this course. As revealed by our prior SOLER-funded project – a randomized trial of behavioral nudges – student engagement with Core course content is low and varies substantially by week; most troublingly, engagement with learning supports for task management declined steadily over the course of the semester. Academic engagement is a critical construct known to improve course persistence, predict student success (including grades, degree completion), educational attainment, and occupational attainment. Interventions are needed to increase student engagement across the board, for all students, but particularly for students whose baseline assessments indicate the need for academic support. These interventions should aim to increase student engagement with both course content and task management learning supports. Intervention strategies should be tailored to students’ academic needs, baseline competencies, and informed by evidence on how students of varying baseline competency levels engage with these course elements. More nuanced data on how students of different competency levels engage with course elements – including qualitative data on student motivations for engagement – are needed to guide the development and deployment of interventions that can be tested using rigorous designs. In Phase 1, we will replicate student engagement analyses using learning management system learning analytics data for the 2021 course, stratifying by baseline competency level. In Phase 2, we will use rigorous qualitative methods as one-on-one interviews to explore student learning behaviors and motivations that underlie engagement. Our prior study did not demonstrate efficacy of behavioral nudges most likely because the nudges, though evidence-based, were not informed by data on what motivates students. In the final phase of the project, we will quantitatively explore student perspectives of motivating behavioral nudges using forced-choice paired comparison methods that assess preferences across a range of dimensions, including messaging, placement/delivery, timing, and design. This proposed work is an essential step before deploying a larger-scale trial testing nudge interventions with more complex features, such as tailored nudges based on baseline characteristics.

*********************************************************************************************************************

Title: An exploration of the anti-oppressive practice of 'un-grading' and its effects on student learning

Principal Investigator: Amy Werman, Lecturer, Social Work

Award Date: Summer 2022

Status: Completed

Summary: A significant challenge facing instructors at Columbia School of Social Work is the fair, equitable, and accurate assessment of students. Instructors use letter grades to evaluate student outcomes, but there is little agreement as to what grades indicate. There is, however, a shared implicit assumption that grades measure learning. The current study interrogates that assumption by exploring the implementation of an "ungrading" pedagogy in a Program Evaluation course. Ungrading is an innovative practice in which students receive feedback on assignments and revise their work based on that feedback; however, their work is not graded. Rather, the student is asked to reflect on what they learned and on the process of learning (their motivation, effort, curiosity), and then give themselves a grade on the assignment. Anecdotal evidence suggests that ungrading fosters learning because students focus on the process of learning instead of the outcome of a grade. This study aims to examine objective and subjective student experiences with ungrading relative to traditional alternatives.   

*********************************************************************************************************************

Title: How teacher-student connection improves abstract thinking

Principal Investigator: Tugce Bilgin, Lecturer, Ecology, Evolution, and Environmental Biology

Award Date: Summer 2022

Status: Completed, manuscript submitted

Summary: The ‘social breathing’ and ‘interactive brain’ hypotheses as well as models of learning dynamics, joint action, and teacher-student interactions posit that low-level perceiving and signaling systems in nonverbal behavior scaffold and support interpersonal coordination and performance in joint tasks, such as teaching and learning. This study attempts to quantify this relationship. Teacher-student dyads recruited from various sections of the Columbia College Core course 'Frontiers of Science' will use a 1-on-1 online portal to collaboratively complete simple and complex tasks. We will examine the relationships between nonverbal behavior, task complexity and performance, and dyad familiarity (i.e., whether or not the student and instructor were drawn from the same section.) 

*********************************************************************************************************************

Title: Interactive simulations to support inquiry-based statistics instruction in Frontiers of Science

Principal Investigator: Nicholas Bock, Lecturer, Earth & Environmental Sciences

Co-Sponsor: Innovative Course Design Grant (CTL)

Award Date: Summer 2022

Status: Completed

Summary: Frontiers of Science is a required course for all first-year Columbia College students. The goal of the course is to provide an introduction to current research across four different scientific disciplines while helping students develop a framework for approaching, analyzing, and interpreting quantitative information (hereafter referred to as the scientific "habits of mind"). Students come to Frontiers of Science with a wide range of prior science coursework and attitudes/interests/experiences in science. Therefore, a central challenge in planning individual seminar sessions is identifying learning activities that address course objectives while serving the learning needs of all students in the classroom. This challenge can be particularly acute when introducing statistical concepts. Frontiers of Science students are introduced to basic statistical concepts (e.g., measures of central tendency and dispersion) at the start of the semester. During the first three weeks of the semester, students are expected to develop an understanding of how  these concepts can be used to formulate statistically supported inferences (e.g., the use of standard error to evaluate confidence intervals and to support hypothesis tests). However, based on an introductory survey completed by students in seminar sections during the spring 2022 semester, only 50 percent of students reported having completed previous coursework in statistics. Therefore, there is a clear need within Frontiers of Science for instructional tools that are appropriate and challenging for students both with and without prior experience in statistics. We will develop a series of interactive simulations to support inquiry-based learning of basic statistical concepts in Frontiers of Science seminar sessions. We additionally propose to work with SOLER Initiative faculty to investigate the impact of the proposed instructional materials on learning outcomes in Frontiers of Science seminar sections. Because all incoming first-year students enroll in Frontiers of Science, student interactions with these simulations will impact learning outcomes across Columbia College. Moreover, because the statistical concepts to be introduced are foundational to data analysis across many different fields of study, these simulations will be of use to faculty across the  university; for example, the simulations may be adapted to introduce Earth Science students to statistical analysis of environmental data. The simulations will additionally expand upon interactive simulations previously developed by faculty in the School of International and Public Affairs (SIPA) as part of the Stats Interactives website.

Access the project proposal here.

Title: Evaluating Oculus Quest 2 and Glue as a Virtual Reality (VR) Platform for Learning Biochemistry

Principal Investigator: Brent Stockwell, Professor, Biological Sciences

Co-Sponsor: CUIT Emerging Technology Grant

Award Date: Summer 2021

Status: Data collection complete; analysis in progress

Summary: Learning biochemistry is challenging for students using chalkboards, Powerpoint and Zoom, due to 3D aspects of molecular conformations. In all environments, learning can be compromised by a lack of attention. VR offers a solution to both issues—it can be more immersive and enhance attention due to the lack of distractions, and it allows engagement with 3D objects, such as proteins, that cannot be directly experienced in the real world. We previously performed a pilot study using VR to teach biochemistry using Oculus Quest 1 headsets and Spatial.io, meeting with students to discuss 3D aspects of biochemistry. In Summer 2021, we improved upon that experience by upgrading the tools: Quest 2 headsets allowed up to 40 people to meet together in the online platform Glue, whereas Quest 1 headsets only allowed 15 people to meet. Glue offers more realistic animated avatars that provide a better sense of presence—being together in a VR space—which may enhance student engagement. Finally, we explored the feasibility of generating animated FBX files to show molecular motion in VR, which may further aid in understanding biochemical mechanisms. To evaluate the impact of these tools, we performed a survey of students, administered a voluntary quiz, and compared performance on the normal course exams between students in the VR recitation and the Zoom recitation. The study yielded valuable practical insights about the effective use of these tools; furthermore, the data provided preliminary evidence that the VR experience enhances student learning and strong evidence that it is more appealing to students than the Zoom recitation format. In Fall 2021, we replicated the previous study with two key modifications: (1) the VR condition was now compared to an in-person, rather than Zoom-based, recitation section. This development reflects the evolving situation of the Covid-19 pandemic; with a return to in-person instruction now possible, it is crucial to evaluate the efficacy of VR experiences relative to the appropriate “default” setting. (2) Instead of a single VR recitation section led by Professor Stockwell, the course will now feature two or three separate VR recitation sections with each led by a teaching assistant who has been specially trained in VR instruction. We hypothesized that the advantages of VR relative to in-person recitations would be manifested in student performance on assessments and survey responses. Data analysis is ongoing.

Access the project proposal here.

*********************************************************************************************************************

Title: Measuring the Learning Outcomes of Self-Directed Rhetorical Grammar Modules for Multilingual Writers

Principal Investigator: Vanessa Guide Mesina, Lecturer, Undergraduate Writing Program and English & Comparative Literature

Award Date: Summer 2021

Status: In progress

Summary: This study aims to measure the efficacy and learning outcomes of a series of self-directed rhetorical grammar modules designed for multilingual students enrolled in International Student sections of the first-year writing course at Columbia, University Writing (UWIS). The six “mini-lesson” modules, originally created with the assistance of a Hybrid Learning Redesign grant from the Provost’s office, currently focus on first person pronoun use; reporting verbs; punctuation for complex sentences; passive vs. active voice; hedging words; and nominalizations. The modules are housed on each UWIS’s Canvas course site. Lessons center on specific English language “rules” and conventions that international students encounter in the American academic setting, and ask students to consider how context, convention, and intent inform writers’ choices. Students are required to complete three modules during semester, however, the online component of the course is semi-synchronous and self-paced. Students select which three modules to complete, allowing them to decide which topics are of the greatest interest and urgency to them. This project therefore accommodates students of varying levels of English language proficiency and frees up valuable class time for students to practices these skills with their peers and instructors. Each module consists of a preliminary intake questionnaire, and then a mix of screencasted “mini-lectures” on target forms, on-screen annotations of texts, readings, on-screen quizzes, and reflective writing exercises. Assessment of this project’s efficacy will be comprised of three components: 1) student “exit ticket” surveys; 2) rubric-based rating of anonymized participant essays; 3) rubric-based assessment of participant peer feedback letters. Participating students will be randomly assigned to complete modules in one of two forms: fully interactive modules (as described above) or passive modules (video components only); it is hypothesized that students completing fully interactive modules will exhibit the strongest outcomes.

Access the project proposal here.

*********************************************************************************************************************

Title: Nudges informed by past student data to increase current students’ engagement with course materials

Principal Investigator: Samantha Garbers, Associate Professor, Population and Family Health 

Award Date: Summer 2021

Status: Data collection complete; analysis in progress

Summary: This project will develop an evidence-based nudge strategy to make explicit to students the interconnection between course materials (lectures, readings, learning checks) and course assessments (homeworks, papers, group projects, tests). The objective of the intervention is to increase student engagement with course content, and in turn, increase and improve: 1) student experience with learning; 2) demonstration of achievement of learning outcomes on specific assignments, the course overall, and the entire Core; and perceived ability to demonstrate competencies; and 3) metacognition, specifically the awareness of links between course content and assessments. Using evidence of past course student behaviors to influence current course student behaviors is a novel approach to the application of actionable insights from learning analytics that is low-intensity, low-risk, and easily replicable in other courses. The impact of this intervention will be evaluated by comparing student engagement and learning outcomes between those who receive the nudge and those who do not.

Access the project proposal here.

*********************************************************************************************************************

Title: Comparing the effects of onsite and online simulation-based education
on the development of clinical reasoning in student physical therapists: a crossover study

Principal Investigator: Wing Fu, Assistant Professor, Rehabilitation and Regenerative Medicine

Award Date: Summer 2021

Status: Data collection complete; analysis in progress

Summary: The COVID-19 pandemic has forced many educators to replace onsite education with online education. As the end of the pandemic is expectantly approaching, it is imperative to compare the learning outcomes of onsite and online education in order to make informed decisions on selecting educational interventions appropriate for specific learning objectives in a post-confinement world. The overarching objective of this proposed crossover study is to compare the effects of onsite and online simulation-based education on the development of clinical reasoning (CR) in student physical therapists (SPTs). Clinical reasoning is a critical attribute that physical therapists must possess. Simulated patients (one form of simulated-based education) appear to have an effect on developing physical therapy clinical practice competencies including CR. They are typically used as onsite trainings. Despite the existence of virtual simulated patients, they are not in the form of recorded videos of simulated patient sessions, which the principal investigator used during the recent lockdown to replace the pre-lockdown onsite simulations in her course. To the principal investigator’s knowledge no studies have been done to compare the effects of the pre-lockdown educational intervention and the in-lockdown educational intervention on the development of SPTs’ CR. The effects will be assessed at levels I and II of the Kirkpatrick model, including SPTs’ clinical reasoning competency, self-perceived clinical reasoning gain and their satisfaction with the respective educational interventions. The proposed study can help promote the implementation of evidence-based education, which is much needed, as we are about to enter the transformed post-pandemic educational era.

Access the project proposal here.

*********************************************************************************************************************

Title: Investigating bias in standardized patient assessments of medical students 

Principal Investigator: Beth Barron, Associate Professor, Medicine

Award Date: Summer 2021

Co-Sponsor: Provost's Large Scale Teaching and Learning Grant

Status: In progress

Summary: This project will contribute to understanding how systemic racism in the form of implicit bias adversely impacts medical students of color. The Jaharis Simulation Center at Vagelos College of Physicians and Surgeons (VP&S) provides learner assessments utilizing standardized patient (SP) objective structured clinical examinations (OSCEs) throughout medical school and residency. We have noted that Underrepresented in Medicine (URiM) students are consistently rated 5-10% lower than white students on communication skills. This is consistent with literature demonstrating that students of color and women are given lower OSCE scores in the areas of communication and demonstrated empathy and raises concern for implicit bias in OSCE grading. We aim to contribute by probing how this disparity develops and to investigate methods that can ameliorate it. The project addresses two objectives: (1) investigate if and how SP biases may contribute to lower OSCE scores through review of prior data and implementation of a controlled comparison intervention; and (2) develop and evaluate interventions to reduce bias in SP grading. This project is of critical importance as assessment biases may unfairly damage URiM students’ long-term career goals, self-confidence, and trust in us. Furthermore, as medical educators, it is our responsibility to foster more accurate, unbiased assessments for all students. VP&S Dean of Education, Director of Equity and Justice in curricular affairs, simulation center leadership, and the Columbia Center for Educational Research and Evaluation (CERE) will support and assist with sustainability and academic dissemination of study results; furthermore, this project was awarded a 2021 Large-Scale Provost’s Teaching and Learning Grant and will receive support from the Columbia Center for Teaching and Learning (CTL) in instructional design, accessibility compliance, and more.

Access the project proposal here.

Title: Evaluation of virtual reality for learning biochemistry and enhancing student engagement

Principal Investigator: Brent Stockwell, Professor, Biological Sciences

Co-Sponsor: CUIT Emerging Technology Grant

Award Date: Summer 2020

Status: Completed (Fall 2020)

Summary: In Professor Stockwell's Biochemistry I course, students were recruited to participate in weekly 30-minute small group sessions, similar to office hours. Half of the students were randomly assigned to the virtual reality (VR) condition, and the other half were assigned to a Zoom-based control condition. The students in the VR condition used Oculus Quest headsets to convene on the Spatial.io virtual meeting platform, where they interacted with 3D models of molecules and biochemical pathways. The students in the Zoom condition interacted with similar materials in a 2D format. At the end of the semester, the two groups were compared with respect to student performance on assessments (e.g., exams) and responses to surveys that queried the students' engagement and attitudes about the suitability of the platform for learning biochemistry. It was predicted that students in the VR group would exhibit enhanced learning outcomes and more positive attitudes relative to the control group. The project provided an instructive foray into the use of VR for biochemistry education, with key insights centering around the effective implementation of 3D assets. Although no enhancements to student outcomes were detected, students provided many positive and constructive comments that will guide future iterations of this line of inquiry. 

Access the project proposal here.

Access the project final report here. 

*********************************************************************************************************************

Title: Imaging the brain activity of students studying brain imaging

Principal Investigators: 

• Alfredo Spagna, Lecturer & Director of Undergraduate Studies in Neuroscience & Behavior, Department of Psychology
• Xiaofu He, Assistant Professor of Clinical Neurobiology, Department of Psychiatry (CUIMC), Data Science Institute Affiliate

Co-Sponsor: Data Science Institute Collaboratory Grant

Award Date: Summer 2020

Status: In preparation (experiments planned for Fall 2021; see note below)

Summary: Portable electroencephalography (EEG) headsets will be used to record students’ neural activity during Fundamentals of Human Brain Imaging seminars. Students will process and analyze their own EEG data as part of the curriculum. The research design features two components: (1) an observational cognitive neuroscience study and (2) a comparison of student learning and attitude outcomes to a control course, Consciousness & Attention. For Component 1, classroom recordings will temporally align seminar content to EEG data. Brain wave frequencies associated with on-task as opposed to mind-wandering attentional states will be extracted from EEG data. The extent to which the same attentional states are indicated by EEG data, observed behavioral measures, and students' self reports will be determined. For each student, correctly and incorrectly answered questions on in-class assessments will be compared with respect to the attentional states (as inferred from EEG data and other measures) during the relevant lecture/discussions in class. It is predicted that correctly answered questions will be associated with on-task attentional states whereas incorrectly answered questions will be associated with mind-wandering states. For Component 2, the control course has similar learning objectives but involves no EEG. It is predicted that students in experimental course will exhibit similar or enhanced learning and attitude outcomes relative to the control course.

Note: due to the ongoing COVID-19 pandemic, the primary experiments originally planned for Spring 2021 were postponed until the resumption of fully in-person classes in Fall 2021. Data collection in Fall 2021 was successful; analysis is ongoing. 

Access the project proposal here.

Access the project final report [coming soon].