The Scholarship of Teaching and Learning (SoTL) is an evidence-based, academic community-facing enterprise in which faculty and colleagues use a scholarly lens to examine how learning happens in higher education. A major part of SOLER’s mission is to facilitate SoTL Research – SoTL that uses formal scientific approaches to investigate teaching and learning – in Columbia’s academic contexts. In other words, SoTL Research leverages scientific methods to formulate and address questions about the ways students gain knowledge, develop skills, and otherwise think, feel, and behave in higher ed.
Read below for a primer on SoTL Research experimental designs, topics, and papers.
Like the social (and natural) sciences in general, the various types of SoTL Research utilize a variety of designs including data mining, correlational quasi-experiments, and randomized controlled trials. Let’s consider a typical project that aims to compare two or more instructional approaches with respect to student outcomes. Designing a suitable study to achieve this aim can be summarized as a three-step process:
- Define factors and levels. DBER investigators compare different instructional approaches that constitute different levels of an experimental factor, also called an independent variable. The interventions should ideally align well with the research question and be derived from a relevant theoretical framework. The point of comparison for a novel instructional strategy should be the “gold standard,” i.e., a way of teaching and learning that is already accepted to be effective at the time of study; using bad teaching practices as a control condition will not only impair students' learning experiences but also yield little scientific insight.
- Define outcome variables. Student learning and attitude outcomes should be assessed by outcome measures with established validity and reliability. Researchers may administer a pre-intervention test of students’ knowledge or skills in order to establish comparability across groups (see below). Depending on project goals, assessment measures may be selected for generalizability to other courses, departments, or institutions. These measures are also called dependent variables.
- Devise research design. Participants (i.e., students) can be assigned to conditions via two schemes:
- In a between-subjects design, different individuals experience different conditions; each participant is exposed to a single level of the intervention.
- In a within-subjects (“crossover”) design, each participant experiences all the conditions, typically in a randomized order. Most studies with this design are "balanced": all participants receive the same number of treatments and participate for the same number of periods. Because each participant generates data for all conditions, this design can be especially useful when splitting the participants into groups (as in the between-subjects design) would yield very small sample sizes for each group.
For example, suppose a researcher is examining how learning in virtual reality (VR) affects academic performance. The researcher would need to first decide on the factors by clarifying the pedagogical interventions (e.g., comparing learning in a VR environment as opposed to learning in person). Then the researcher would choose the outcome variable (e.g., scores from a weekly quiz) and choose a research design (i.e., how students will be assigned to the different levels of the independent variable).
For studies featuring a pedagogical intervention, it is important to ensure equity, i.e., fair treatment of students. For example, if a researcher is comparing the effect of two instructional strategies on graded exam scores via a between-subjects design, the researcher should be aware that one group of students may be adversely affected in learning and grade achievement relative to the other. Equity can be enhanced through study features such as the following:
- Within-subjects (“crossover”) designs. As noted above, exposing participants to the same series of treatments throughout the study will ensure that every student has the same total experience.
- Between-subjects designs with non-graded assessments. Targeting the intervention to impact non-graded outcomes (e.g., a non-graded quiz that assesses learning with respect to “experimental” content) will reduce the impact of interventions on students’ grades
- Comparison of present and past course iterations. Compare learning outcomes to student performance in a previous semester means that students are not simultaneously compared to one another and presented with different experiences. However, this design is less controlled and therefore yields less clear interpretations of intervention impacts.
The list of topical areas in SoTL Research below was primarily informed by the NRC analysis of discipline-based education research.*
- Conceptual content understanding: misconceptions and preconceptions; learning progressions
- Cognitive domain: (interdisciplinary) problem-solving; quantitative/temporal/spatial reasoning
- Instructional strategies and technology: design, use, and evaluation of different instructional strategies and their effectiveness in various settings (e.g., large lecture, small seminar, lab) and modalities (e.g., online, hybrid learning, “metaverse”); community-based and cross-cultural learning
- Students' self-regulated learning and metacognition: encouraging students to reflect; learning assessment methods (e.g., minute papers, surveys); developing students' study skills
- The affective domain: attitudes, motivations, and values of students; attitudes of faculty towards students' career paths; attitudes of faculty towards inquiry-based teaching methods; science/math self-efficacy
*Singer, S., & Smith, K. A. (2013). Discipline-based education research: Understanding and improving learning in undergraduate science and engineering. Journal of Engineering Education, 102(4), 468-471.
Because SoTL Research strengthens one’s candidacy for tenure or promotion by documenting teaching excellence and highlighting active interest in improving pedagogy, SoTL Research project reports are excellent additions to teaching portfolios. But perhaps more importantly, SoTL Research expands one’s publishing opportunities; numerous interdisciplinary and discipline-specific journals provide relevant outlets. SoTL Research papers can be roughly organized into five categories:
- Empirical research papers report on data collection and analyses used to answer a specific SoTL Research question by testing a corresponding hypothesis
- Theoretical research papers describe new disciplinary education theories or models or learning processes
- Methodology papers describe new pedagogical materials or methods developed in the context of SoTL Research projects.
- Literature reviews synthesize and evaluate published literature on a particular SoTL Research topic, describing patterns and identifying gaps in order to make recommendations for future research directions.
- Commentary articles present a personal opinion to provide insight into an issue or trend of broad interest to the SoTL Research community.
Most SOLER-supported SoTL Research projects will yield empirical research papers, methodology papers, or commentary articles.
See Journals for a list of general and discipline-specific journals that publish SoTL Research papers of the various types listed above.