Faculty Guide

The SOLER team compiled this guide to inform Columbia faculty interested in applying for the Provost SOLER Seed Grant (PSSG). We aim for this guide – which should be especially valuable for PSSG awardees – to address four goals:

  1. To provide an introduction to Scholarship of Teaching and Learning (SoTL) and Discipline-Based Education Research (DBER)
  2. To distinguish between SoTL and DBER and to clarify the relationship between PSSG-supported projects and DBER
  3. To summarize best practices in grant writing, publishing, and intellectual community building in DBER 
  4. To provide faculty with additional resources for in-depth exploration of SoTL and DBER.

Section 1: Scholarship of Teaching and Learning (SoTL)

All of Section 1 was adapted from Nancy Chick’s blog A Scholarly Approach to Teaching.

The Scholarship of Teaching and Learning (SoTL) is a synthesis of teaching, learning, and research that uses a scholarly lens to examine how learning happens (or doesn’t) in higher education. SoTL is scholarly because it begins with intellectual curiosity, is conducted deliberately and systematically, is grounded in an analysis of relevant evidence, and results in findings shared with peers for critical review, the promotion of discourse, and the expansion of a knowledge base. 

SoTL can be outlined as a four-step process:

  1. Asking questions about student learning and teaching activities that are designed to facilitate it
  2. Answering those questions by devising methods to make relevant student learning visible and measurable, gathering evidence of thinking and learning, and then systematically analyzing the evidence
  3. Sharing the results of that analysis publicly to invite peer review and to contribute to broader bodies of knowledge on student learning
  4. Aiming to improve student learning by strengthening the practice of one’s own teaching and that of others.

While systematic research into teaching and learning has a rich history in education departments, such studies tend to focus on K-12 students and are conducted by education specialists. SoTL, in contrast, focuses on higher education and is conducted by disciplinary specialists. And though SoTL is often described as a “young” field, many disciplines have a long tradition of studying how students learn in their fields. 

The formal origins of SoTL date to 1990. In an effort to define the scholarship performed by professors in academia as more than just “teaching versus research,” Ernest Boyer, in his influential book Scholarship Reconsidered: Priorities of the Professoriate (1990), concluded that “the work of the professoriate might be thought of as having four separate, yet overlapping, functions. These are: the scholarship of discovery; the scholarship of integration [i.e., with ideas from other disciplines]; the scholarship of application [i.e., to practical concerns of individuals or society]; and the scholarship of teaching” (p. 16). Boyer’s conceptualization of scholarship elevates the traditional role of teaching from “a routine function, tacked on” to an essential component of a professor’s scholarly life. Furthermore, Boyer argued that the academy should recognize and reward all four components of scholarship, including the scholarship of teaching (see Fig. 1). 

Boyer's Scholarship Framework

Figure 1. The work of the professoriate, highlighting the scholarship of teaching. Adapted from Boyer (1990).

Building on Boyer’s work, Charles Glassick, Mary Taylor Huber, and Gene Maeroff, in their book Scholarship Assessed: Evaluation of the Professoriate (1997), identified six markers of scholarly work:

  1. Clear goals
  2. Adequate preparation
  3. Appropriate methods
  4. Significant results
  5. Effective presentation
  6. Reflective critique

These goals – familiar to faculty members from their evaluations of the scholarship of discovery (traditionally called simply “research”) – are also applicable as standards of the other three scholarships.

Historian David Pace acknowledges the increasing calls for Boyer’s conceptual combination of teaching and research: “A consensus has formed within growing circles in academia that there is scholarly research to be done on teaching and learning, that the systematic creation of rigorous knowledge about teaching and learning is a crucial prerequisite to responding to major challenges facing academia, that this knowledge must be shared publicly and should build cumulatively over time, and that the explorations of this area should be conducted by academics from all disciplines, not just those with appointments in schools of education” (2004, p. 1174).

The inaugural issue of Teaching & Learning Inquiry, the journal of the International Society for the Scholarship of Teaching and Learning (ISSOTL), features several essays that articulate the reasons for Pace’s “consensus,” including the need for and value of SoTL. Dan Bernstein, for instance, argues that those who participate in SoTL are “assets” to their institutions because they “generate visible analyses of student learning taking place in their institutions, provide excellent models of practice for local colleagues, generate high-quality evidence for internal and external assessment, and offer accessible examples of quality education to prospective students” (2013, p. 35).

Nancy Chick and Gary Poole explain that SoTL “offers an alternative and more faculty-friendly model of faculty development that replaces the pervasive model based on competition, outputs, performativity, and solitude with one that’s democratic, dialogic, process-focused, and collaborative” (2013, p.3).

Keith Trigwell (2013) offers a syllogism that accounts for SoTL’s impact on student learning: Previous studies have shown that certain approaches to teaching lead to deeper learning than other approaches; these more effective approaches are highly utilized by those who practice SoTL; therefore, practitioners of SoTL are more likely to foster students’ deep learning.

The merits of SoTL aren’t limited to its practitioners; SoTL’s goal of improving teaching and learning means that instructors and scholars across campus will benefit from it. SoTL is therefore notable for its pursuit of a multidisciplinary audience. SoTL is a specialized kind of research that delves deeply into how students learn within and across disciplines, so reading it and keeping up with new findings – even without conducting actual SoTL studies – are effective ways to maintain a scholarly approach to teaching. Then, as is the case within the disciplines, some scholars will choose SoTL as an area of their research. As Lee Shulman, president emeritus of the Carnegie Foundation for the Advancement of Teaching, explains: “Scholarly teaching is what every one of us should be engaged in every day that we are in a classroom, in our office with students, tutoring, lecturing, conducting discussions, all the roles we play pedagogically. Our work as teachers should meet the highest scholarly standards of groundedness, of openness, of clarity and complexity. But it is only when we step back and reflect systematically on the teaching we have done, in a form that can be publicly reviewed and built upon by our peers, that we have moved from scholarly teaching to the scholarship of teaching.” (2004, p. 166).

Figure 2 below provides a diagrammatic illustration of Shulman’s conceptual framework.

Shulman's SoTL Framework

Figure 2. Situating SoTL within a broader context. Adapted from Chick & Poole (2013).

Compelling reasons to engage in SoTL work include the following:

  1. SoTL can improve your teaching effectiveness: By its very nature, SoTL directly impacts your teaching and student learning. Engaging in SoTL will help you become a more effective teacher, as you become increasingly aware of your classroom practices and make strides toward systemic change.
  2. SoTL can impact your application for tenure or promotion: SoTL makes it easy to demonstrate your practices in the classroom, and highlights your active interest in improving your teaching. Outcomes from SoTL research make natural components of teaching portfolios, which are often used in tenure and promotion decisions.
  3. SoTL gives you expanded research and publishing opportunities. SoTL is quantifiable and rigorous research, and numerous interdisciplinary and discipline-specific journals are devoted to carrying SoTL research.

In 1995, Robert Barr and John Tagg published “From Teaching to Learning: A New Paradigm for Undergraduate Education,” an article that soon became part of the SoTL canon, arguing: “A paradigm shift is taking hold in American higher education. In its briefest form, the paradigm that has governed our colleges is this: A college is an institution that exists to provide instruction. Subtly but profoundly we are shifting to a new paradigm: A college is an institution that exists to produce learning. This shift changes everything. It is both needed and wanted.” (p. 13).

Recognizing the difficulty of change in focus – from teaching to learning – Barr and Tagg carefully outline the distinction as it influences the mission of the university, its criteria for success, the structures of teaching and learning, learning theory, measures of productivity, and the nature of the roles of various members of the campus.

In light of this change, the language “the scholarship of teaching” fell short of capturing the aims and assumptions of this research: improving teaching does not necessarily lead to student learning. The two processes must be understood and investigated together. Shulman and Pat Hutchings explain this kind of thinking about learning, claiming that SoTL “requires a kind of ‘going meta,’ in which faculty frame and systematically investigate questions related to student learning—the conditions under which it occurs, what it looks like, how to deepen it, and so forth—and do so with an eye not only to improving their own classroom but to advancing practice beyond it.” (1999, p. 10).

Thus, Boyer’s initial term “scholarship of teaching” was transformed into the “scholarship of teaching and learning.”

Section 2: SoTL vs. Discipline-Based Education Research (DBER)

Section 2 was adapted from three sources:

  • The National Association of Geoscience Teachers’s Geoscience Education Research website
  • The University of Central Florida’s Faculty Center Guide to SoTL and DBER
  • A 2015 Earth Educators’ Rendezvous workshop entitled “Getting Your Education Research Published” by Heather Petcovic (Western Michigan University) and Kristen St. John (James Madison University).

PSSG-supported projects at Columbia fall into a research-focused subset of SoTL called Discipline-Based Education Research (DBER). Understanding what distinguishes DBER from other forms of SoTL is crucial for defining a project, writing a manuscript, identifying a journal for submission, and communicating your work to others (including promotion and tenure committees).

As explained in Section 1, the goal of SoTL is to improve one's own teaching practice through innovations in pedagogy and curriculum and to serve as a model for others. SoTL studies are typically descriptive, and focus on innovations that address learning goals. Scholars systematically gather data that lead to self-reflection, improved teaching practices, and improved student learning. SoTL studies are often specific to a course and the instructor's personal context, but conclusions must be supported by evidence and have broader applications so as to serve as a potential model for other instructors and at other institutions. 

SoTL uses discovery, reflection, and evidence-based methods to research effective teaching and student learning in higher education. SoTL goes beyond teaching based on evidence to produce evidence for specific teaching and learning practices. These findings are peer reviewed and publicly disseminated in an ongoing cycle of systematic inquiry into classroom practices.

Different research methods and arguments can be used to demonstrate student learning, though they vary in the strength of evidence they can provide. Deductive arguments and experimental methods generally provide stronger evidence of learning than inductive arguments and case studies; combining methods can capitalize on the advantages of each. Any of these can be effective SoTL methods. See Section 3f for an in-depth discussion of strength of evidence in SoTL.

Some additional notes on SoTL:

  • SoTL is an opportunity to think deeply, gather data purposefully, and disseminate your teaching excellence.
  • Professional development workshops can provide ideas and tools to early career faculty on SoTL research approaches.
  • SoTL research is a good option as a first foray into education research (ER).
  • SoTL is also a sustainable research direction for those who plan to maintain their disciplinary research career while also branching into ER.
  • To publish successfully, think broadly about where to submit your work; the SoTL presence in disciplinary newsletters, conferences/meetings, and professional development workshops means that there are multiple outlets to disseminate your work.
  • Recognize that SoTL publications may not be highly cited, (i.e., that metric may not be the best measure of impact).

The goal of DBER is to test theory and produce generalizable findings focused on teaching, learning, and ways of thinking in a particular discipline, traditionally within the sciences. DBER can also include investigations into the development and nature of expertise in a discipline as well as strategies for making the discipline more inclusive. DBER scholars combine discipline expertise with knowledge of the science of teaching and learning. Together, they form a collection of related research fields such as biology education research (BER), physics education research (PER), chemistry education research (CER), and geoscience education research (GER), among others. DBER scholars use common methods and draw on common theories, but are each influenced by the practices and history of their parent discipline. While DBER may differ slightly between disciplines, all DBER researchers systematically gather data that leads to knowledge for improved teaching and student learning. DBER findings should be broadly applicable beyond a single course or instructional context and are usually published in peer-reviewed journals. 

The National Academies of Science, Engineering, and Medicine published a report in 2012 that defines DBER, synthesizes the DBER literature, and calls for the increased use of DBER findings and recognition of DBER as a valuable field of scholarly study. Free access to the report is available from the National Academies Press website.

DBER is typically more generalizable to other courses or instructional contexts than is SoTL. Rather than classroom-level studies that are characteristic of SoTL, DBER studies address broader goals such as understanding how people learn certain concepts in a discipline and how this knowledge can be translated into classroom practice. See Table 1 below for a summary of key differences.

Table 1. Comparison of SoTL and DBER. Adapted from Petcovic & St. John (2015). 

There is no strict dividing line between SoTL and DBER; it is perhaps best to conceptualize a SoTL-DBER continuum, as described by the National Association of Geoscience Teachers (NAGT). Alternatively, as suggested above and visualized in Figure 3 below, DBER can be conceptualized as a research-based subset of SoTL. In this formulation, SoTL is the intersection of disciplinary content (i.e., the knowledge and skills one acquires in the course of engaging with a particular academic field; the term “interdisciplinary content” may be appropriate in contexts of integration across fields) and the advancement of teaching and learning. DBER, in turn, is the intersection of SoTL and social science research (i.e., the theoretical, methodological, and analytic frameworks used to study the cognition and behavior of individuals and groups).

DBER Venn Diagram

Figure 3. DBER lies at the intersection of advancement of teaching and learning, social science research, and disciplinary content. In this formulation, DBER is a subset of SoTL. Adapted from Lukes, et al. (2015). 

To further illustrate the distinction between SoTL and DBER, it may be helpful to compare and contrast the evaluation-like perspective of SoTL with the research-focused nature of DBER. Table 2 below summarizes the similarities and differences between evaluation and research.

Table 2. Comparison of evaluation and research. SoTL often resembles evaluation, whereas DBER is a research process. Adapted from Petcovic & St. John (2015). 

Section 3: DBER

As explained in Section 2, the research-based subset of SoTL termed DBER is best established in the natural sciences. DBER’s advanced position in the natural sciences may be due to the combination of (1) the solid intellectual propensity and skills for conducting quantitative empirical research that natural sciences faculty possess and (2) the greater availability of funding for natural science education research relative to other fields, including the social sciences. Note that the perspective and skills of social scientists are even more applicable to DBER than those of natural scientists, as DBER can be considered a subfield of social science. Figure 4 below outlines various elements of SoTL/DBER and their alignment with the process of disciplinary research (i.e., scientific inquiry in the scholar’s area of expertise) practiced by natural and social science faculty. Crucially, control/experiment studies of pedagogical interventions are highlighted as most analogous to traditional scientific inquiry.

SoTL/DBER and Scientific Disciplinary Research

Figure 4. Schematic of relationship between SoTL/DBER and scientific disciplinary research. Adapted from Feig, 2013.

Topical areas of DBER are listed below, with an emphasis on the undergraduate level and research areas that can inform best practices of teaching and learning. This list was compiled by NAGT and was primarily informed by the NRC analysis of discipline-based education research (Singer et al., 2012) along with several other sources. Some of the topical areas listed below are intertwined and overlap, and some topics have multiple scales of investigation. The bullet points below each topic are examples that illustrate the scope of the topic rather than a complete list.

  • Students' conceptual understanding of content
    • Misconceptions and preconceptions
    • Learning progressions
    • Systems thinking / complexity
    • Concept inventories
  • Cognitive domain and problem solving
    • Problem-solving, including solving of interdisciplinary problems
    • Quantitative reasoning
    • Temporal and spatial reasoning
    • Using and understanding models, simulations, and visualizations
    • Expert-novice spectrum
    • Promoting transfer to new learning situations or settings
  • Instructional strategies to improve learning in different settings and with different technologies
    • Design, use, and evaluation of different instructional strategies and their effectiveness in various settings (e.g., large lecture, small seminar, lab)
    • Role of technology (e.g., on-line, hybrid learning, e-learning)
    • Teaching and learning with large complex online datasets
    • Community-based and cross-cultural learning
  • Students' self-regulated learning and metacognition
    • Encouraging students to think about their learning
    • Basic learning assessment methods: Minute papers, knowledge surveys, etc.
    • Developing students' study skills
  • The affective domain and the discipline
    • Attitudes, motivations, and values of students
    • Attitudes of faculty towards students' career paths (e.g., K-12 teaching career vs. research career)
    • Attitudes of faculty towards inquiry-based teaching methods
    • Science/math self-efficacy
    • Beliefs (e.g., about evolution, climate change)
    • Strategies to promote motivation or provide materials and opportunities that students value or find relevant
  • Access and success of under-represented groups in the discipline
    • Recruitment and retention of students and faculty
    • Socio-cultural influences on access and success in the discipline
    • The role of mentoring in the discipline
    • Broadening participation, diversity and inclusion
    • Access and success at different scales (e.g., courses, programs, community)
  • Nature of the discipline
    • Developing disciplinary expertise (knowledge and skills experts use when working on complex problems; the novice-expert continuum)
    • Teaching nature of the discipline throughout the curriculum (e.g., introductory to upper-level learning progressions)
    • Using research and research-like experiences
  • Elementary, middle, and secondary teacher education
    • Pre-service teacher preparation
    • In-service teacher professional development
  • Training and professional development of graduate students and college/university educators
    • Preparation and continuing professional development for faculty
    • Preparation of graduate students
    • Preparation and professional development of teaching assistants
    • Strategies for promoting changes in faculty practice at different scales (e.g., individuals, departments, programs, institutions, communities)
  • Disciplinary informal education and literacy (i.e., influence of informal education on public attitudes and understanding of relevant issues)
  • Teaching about the discipline in the context of social problems (e.g., sustainability, human health, globalization)
  • Development and validation of surveys and instruments to collect data on all of the topics listed above
    • Validation and adaptation of existing instruments originally developed for other contexts (e.g., other disciplines, general education student perceptions of course learning environment)
    • Development of new assessment instruments

A collection of surveys, classroom observation protocols, concept inventories, and other assessment tools compiled by NAGT can be found here. Although the site is geared towards ER in the field of geoscience, many of the tools (and the themes they raise) are broadly applicable. 

As PSSG-supported projects progress at Columbia, a repository of implemented assessment tools will be compiled on this webpage. 

This section was adapted from NAGT’s Writing GER Grant Proposals webpage. 

SOLER-sponsored projects will, by definition, receive monetary and in-kind support from the Provost’s Office. Ultimately, however, SOLER aims for partner faculty to apply for external grants that will provide financial support to extend DBER projects. The grant writing process is therefore important for the pursuit of such external opportunities; moreover, best practices in DBER grant writing will also greatly strengthen the proposals of applicants for PSSGs. The following tips, examples, writing templates, and links can help you be a more informed and effective proposal writer:

  1. Read the program announcement carefully. Talk with a program officer to make sure that your ideas fit the program. Contact the program officer via email and send her/him a one-page summary. Be patient.
  2. Work on projects you care deeply about. Let your commitment come through. Make a compelling case for the need and importance of the work to your institution and to others.
  3. Do your homework. Build on what others have done. Know the literature. Be current. Add to the body of knowledge. Discuss the value added of your project.
  4. Emphasize broader impacts. Your project must have more than just a local impact (i.e., beyond your students and your institution). Consider how others can build on your proposed work. Think about what is transferable and what can be sustained.
  5. Have measurable goals and objectives. Enhancing student learning, improving undergraduate education, and other similar things are lofty, but may be difficult to measure. Make sure you have measurable goals and objectives. Consider what activities will align with such goals, what will be delivered in the end, and what is needed to convince others that this will work and is worth supporting.
  6. Emphasize collaboration. Successful projects are team efforts, although individuals matter too. Your project team should be greater than the sum of the parts. Identify your support network (e.g., administrators, colleagues, collaborators, industry) and keep them engaged. Get a good group of internal and external advisors and an independent evaluator (or evaluation team).
  7. Practice good management. Develop a realistic timeline with milestones and key deliverables. Develop a strong management plan for the team and project, and be sure to engage and empower your team (with accountability built in).
  8. Evaluation is about "impact and effectiveness". Consider how you will know that the project goals are being met. You need evidence to know your project is making an impact and is effective. You can not evaluate yourself; external (independent) validation is key. Build in evaluation from the beginning.
  9. Spread the word. Have a proactive dissemination plan. Disseminate within your scholarly community and beyond (reach other disciplines). Share the materials developed widely.
  10. Serve as a reviewer. Offer to be a reviewer and to help others. Gain experience and learn from reviewing strong and weak proposals. Find programs for which you have expertise and that you think you want to submit to. Email the program officer and share your interest to be a reviewer, and include your resume/CV.

Much of this section was adapted from Grauerholz & Zipp (2008). 

There are five primary reasons that SoTL/DBER manuscripts are rejected by journals:

  1. Overall rationale is not appropriately grounded in literature
  2. Pedagogical interventions are not clearly defined or explained
  3. Pedagogical interventions are not clearly aligned with research question
  4. Assessment of student outcomes is simplistic or lacks validity and/or reliability. For example:
    • No pre-test of students’ knowledge/skills
    • No control group or lack of comparability in control groups (i.e., inability to account for other factors)
    • Small sample size
    • Lack of generalizability to other courses/departments/institutions
  5. Lack of human subjects IRB approval

The strongest SoTL/DBER manuscripts are characterized by: 

  1. Strong methodological and theoretical frameworks with citations of appropriate references. 
  2. Pre- and post-tests
  3. Control groups and appropriate control for other factors 
  4. Multiple courses or large sample size, even different institutions 
  5. Multiple and direct measures of learning outcomes

SoTL/DBER papers can be roughly organized into several categories:

  • Research Papers
    • Empirical Research: papers describe data collection and analyses to answer a specific disciplinary education research question or test a hypothesis.
    • Theoretical Research: papers describe new disciplinary education theories, including philosophies, developed to fill a theoretical or philosophical gap.
  • Curriculum & Instruction Papers
    • Curriculum Papers: describe new materials developed for disciplinary instruction and provide evidence of their effectiveness.
    • Instructional Approaches Papers: describe new teaching methods developed for disciplinary instruction and provide evidence of their effectiveness.
  • Literature Reviews: synthesize and evaluate the published literature on a particular topic within disciplinary education research or practice. Patterns and trends in the literature are described, and research gaps are identified and used to make recommendations on future directions.
  • Commentary Articles: seek to provide a critical or alternative viewpoint on a key issue or provide an insight into an important development that is of broad interest to disciplinary educators or researchers. A strong literature-based context is expected. Unlike a literature review article, the author gives their own opinions and perspectives. The Commentary category is not a venue for “data-weak” Curriculum & Instruction or Research papers, nor a venue for project reports or updates.

Note: there are journals both in and outside your discipline that may be appropriate venues for publishing your studies.

This section was adapted from St. John & McNeal (2017). 

Background:

The healthcare community has conceptualized and visualized “strength of evidence” as a pyramid diagram in order to help medical practitioners evaluate the validity of medical findings and claims (for example, see Murad et al., 2016). In the diagram below (Fig. 5), the healthcare community’s approach is applied to DBER in the form of a DBER strength of evidence pyramid (SEP). Such an application makes sense because DBER should ultimately inform education practice (i.e., teaching and learning) just as medical research informs medical practice.

DBER Strength-of-Evidence Pyramid

Figure 5. Discipline-Based Education Research “Strength of Evidence Pyramid” (DBER SEP). Adapted from St. John & McNeal (2017).

Description:

There are five levels to this DBER SEP. The bottom level (green) is Practitioner Wisdom/Expert Opinion. This knowledge base of "what we know" about education research in the discipline is also the interface where results of education research directly connect to teaching practice. Practitioner wisdom and expert knowledge is informed by DBER studies at higher levels of the pyramid, as well as personal practitioner experience. It is not, however, a level that directly involves scholarly peer-review publication or original research and analysis. At this level, wisdom about teaching and learning is often shared via professional development workshops and dissemination outlets such as commentaries published in relevant journals.

The next two levels of the pyramid represent original qualitative and quantitative studies that are peer-reviewed and published as primary literature. Most common are case studies (yellow) that focus on a single course or institution that is taught by the researcher using curriculum or instructional methods that they developed and are testing in their class. The methods of analysis often rely on a single instrument appropriate to that site (e.g., course or institution). The population depends on the location and scope of the study, but may be of limited diversity. Less common, but important for determining generalizability of study findings are cohort studies (orange). These typically address many of the same research questions as case studies, but they investigate a broader cross section of courses, institutions, and/or populations. The instruments therefore must be broadly applicable as well, and the research design typically uses a mixed-methods approach (i.e., a blend of quantitative and qualitative data collection). By increasing the depth and/or size of the study, researchers also reduce potential bias; most importantly, the investigators are not the instructors of all of the courses that are testing an intervention.

The uppermost levels of the DBER SEP – meta-analyses and systematic reviews – are the least common, in part because these depend on access to data, methods, and findings from previously published research. These types of studies result in increased data aggregation, syntheses, and generalizability; as such, they are powerful for elevating confidence in DBER community claims. Both types of studies need to be done carefully to minimize bias and avoid inclusion of data from poor-quality studies (e.g., weak methods used in previously published work) in their analyses. Meta-analyses (light blue) involve application of statistical methods to look at a broad suite of existing quantitative or qualitative data. The goal is to provide a more comprehensive description and analysis of a topic or question than could be addressed by smaller-scale case or cohort studies. Systematic reviews (dark blue) use transparent methods to identify, select, and evaluate relevant published literature on a particular topic or question (see Higgins & Green, 2011). Reviews encompass an extensive time frame – recent studies and historical research – to summarize the evolution of a topic and identify patterns, trends, and important questions in need of further investigation. Systematic reviews may incorporate meta-analyses when available.

The width of the pyramid at each level represents the relative number of studies (or available resources in the case of sharing practitioner wisdom). Note that the base of the pyramid relates primarily to the development and application of new teaching innovations and curricula rather than empirical hypothesis testing; as such, the base can be thought of as the non-DBER portion of SoTL (see Fig. 3). Both elements of SoTL are important for improving teaching practice, however, and both can yield peer-reviewed publications. For example, in the Journal of Geoscience Education, non-DBER SoTL typically results in Curriculum & Instruction papers, and DBER typically results in Research papers (see Section 3e above). 

Ways in which this conceptual diagram should be used:

  • To situate or contextualize DBER results from different types of studies. For example, authors of literature review papers could identify the types of studies their analysis draws from (e.g., are they largely case studies or are they cohort studies? Do they use meta-analysis as part of their review methods?).
  • To support discussions on ER needs.
  • To identify gaps in DBER, such as the need for literature reviews of certain topics.
  • To consider the generalizability of study findings.
  • To consider the strength and limitations of aggregate results.
  • To consider the strength and limitations of claims that influence education practitioner knowledge.

Ways in which this conceptual diagram should not be used:

  • To evaluate the particular impact of an original study in the primary literature.
  • To judge the value of qualitative vs. quantitative research methods used in individual studies.

This section was adapted from Kim Kastens’ essay “A Community of Practice for GER” on the NAGT website. 

What is a Community of Practice (CoP)?

One strategy for ramping up the accomplishments and impact of DBER is to foster a "DBER Community of Practice." This section provides an introduction to CoPs, proposes some reasons to think that fostering a DBER CoP would be useful, and introduces a systems model for thinking about what drives and sustains successful CoPs.

Here are two somewhat-overlapping definitions:

  1. "Communities of Practice are groups of people who share a concern or a passion for something they do, and learn how to do it better as they interact regularly" (Wenger-Taylor & Wenger-Taylor, 2015)
  2. "Communities of Practice are...persistent, sustained networks of individuals who share and develop an overlapping knowledge base, set of beliefs, values, history and experiences focused on a common practice and/or mutual enterprise" (Barab, et al., 2004)

The concept of CoP was first developed by anthropologists studying apprentices. They observed that apprentices were not just taught a skill set; they were inculcated into a community. They gradually moved from peripheral participation to core membership as they developed expertise and became conversant with the norms of the group. The concept has since been expanded to encompass groups in schools, businesses, and many other types of formal and informal organizations. But not every group is a "community of practice." To be a CoP, the essentials are:

  1. The Domain: an area of concern or passion (e.g., DBER)
  2. The Community: a group of connected people who interact regularly. 
  3. The Practice: ideas, frameworks, tools, documents, ways of doing things, narratives, exemplars, that the community shares. Building this “toolbox” is an effort to gather and share the artifacts of the practice.

The purpose of the Community is to build competence and capacity in the practice so as to advance/improve the domain of concern or passion. Learning and improving in the practice – by both individuals and the collective – is central to the CoP concept.

Within these broad bounds, effective CoPs show considerable variation in their characteristics and activities. This means there is considerable flexibility in the path forward. As far as characteristics, CoPs:

  • can be distributed or located with an institution
  • can be large or small
  • can grow organically or be purposefully constructed
  • occupy a continuum from formally institutionalized to unrecognized
  • can have formal leaders (or not).

Typical activities of a CoP could include:

  • sharing information
  • building relationships that enable members to learn from each other
  • teaching each other
  • developing a shared repertoire of resources (experiences, stories, tools, ways of addressing recurring problems)
  • mapping knowledge and identifying gaps
  • co-inventing new solutions/approaches/practices.

Why would a CoP be useful for advancing DBER?

  • All DBER subfields struggle with bringing research into practice and achieving uptake of their research findings in the classroom (Singer, et al, 2012). CoP events can establish a "trading zone" where researchers and practitioners can meet, where practitioners' challenges and insights can help to steer research efforts, and where researchers' findings can precipitate action.
  • A CoP can support individuals, by helping them overcome a sense of isolation, find collaborators and a network of like-minded colleagues, access targeted professional development opportunities, and build their career in this new discipline.
  • A CoP can accumulate and create community-wide capacities. A CoP can curate and disseminate the artifacts that embody these capacities, as exemplified by the building of this Toolbox. In addition to sharing existing tools and knowledge, a CoP can invent new and better ways of doing things.
  • Well-functioning CoPs energize their participants and engender deep loyalty. Participants feel good during CoP interactions and seek out more such interactions. As a result CoPs can facilitate amazing things relative to their available resources.

What drives successful CoPs?

Kastens (2016) proposes a conceptual systems dynamic model for what drives successful CoPs, summarized in Figure 6 below. 

Model of a community of practice as nested reinforcing feedback loops.

Figure 6. Model of a community of practice as nested reinforcing feedback loops. Adapted from Kastens (2016). 

The dynamic that this model seeks to capture is that both the individual's capacity and the community's capacity for the practice ratchet up over time as CoP activities play out. Such a ratcheting in one direction implies reinforcing (also called "positive") feedback loop(s). In fact, this model has three interlocking reinforcing feedback loops. The first, at the right of the diagram, involves the individual learning through participation in the CoPs activities. The second, circumnavigating the outside of the diagram, has the individual contributing more to the community as his/her capacity to contribute rises, and conversely has the community providing more to the individual as the community's capacity rises. The real driver of the system, though, is the loop at the left of the diagram, which concerns affect and emotion. Capacity to contribute to the CoP is useless without desire to contribute to the community, which in this model is driven by a warm collegial feeling of belonging and accomplishment.

This model is new and untested, but it highlights two important insights:

  1. Both the individual and the collective need to benefit from CoP activities; both need to be continually building up capacity for the practice. If the individuals aren't benefiting, they won't come back and they won't contribute. If the collective is not benefitting, the practice won't move forward and the CoP will have nothing new to offer to the individuals.
  2. That "warm collegial feeling of belonging and accomplishment" needs to be nurtured and fostered. It's not just a lovely side benefit; it's at the heart of what makes the CoP tick.

Note: Graduate students, postdocs, and faculty involved in DBER, who may have only a small DBER community in their institution, can benefit from getting involved in a virtual community across institutions. 

Section 4: References and Resources

  1. Barab, S. A., MaKinster, J. G., & Scheckler, R. (2003). Designing system dualities: Characterizing a web-supported professional development community. The Information Society, 19(3), 237-256.
  2. Barr, R. B., & Tagg, J. (1995). From teaching to learning—A new paradigm for undergraduate education. Change: The magazine of higher learning, 27(6), 12-26.
  3. Bernstein, D. (2013). How SoTL-active faculty members can be cosmopolitan assets to an institution. Teaching and Learning Inquiry, 1(1), 35-40.
  4. Boyer, E. L. (1990). Scholarship reconsidered: Priorities of the professoriate. Princeton University Press.
  5. Chick, N., & Poole, G. (2013). Launching TLI: SoTL's purposes, processes, and people. Teaching and Learning Inquiry, 1(1), 1-4.
  6. Feig, A. D. (2013). The allochthon of misfit toys. Journal of Geoscience Education, 61(3), 306-317.
  7. Glassick, C. E., Huber, M. T., & Maeroff, G. I. (1997). Scholarship Assessed: Evaluation of the Professoriate. Special Report. Jossey Bass Inc.
  8. Grauerholz, L., & Zipp, J. F. (2008). How to do the scholarship of teaching and learning. Teaching Sociology, 36(1), 87-94.
  9. Higgins, J. P. T., & Wells, G. A. (Eds.). (2011). Cochrane handbook for systematic reviews of interventions. Wiley. 
  10. Kastens, K., & Manduca, C. (2017). Leveraging the power of a community of practice to improve teaching and learning about the Earth. Change: The Magazine of Higher Learning, 49(6), 14-22.
  11. Lukes, L. A., LaDue, N. D., Cheek, K. A., Ryker, K., & St. John, K. (2015). Creating a community of practice around geoscience education research: NAGT-GER. Journal of Geoscience Education, 63(1), 1-6.
  12. Murad, M. H., Asi, N., Alsawas, M., & Alahdab, F. (2016). New evidence pyramid. BMJ Evidence-Based Medicine, 21(4), 125-127.
  13. Pace, D. (2004). The amateur in the operating room: History and the scholarship of teaching and learning. The American Historical Review, 109(4), 1171-1192.
  14. Shulman, L. S., & Hutchings, P. (2004). Teaching as community property: Essays on higher education. Jossey-Bass.
  15. Singer, S., & Smith, K. A. (2013). Discipline‐based education research: Understanding and improving learning in undergraduate science and engineering. Journal of Engineering Education, 102(4), 468-471.
  16. St. John, K., & McNeal, K. S. (2017). The strength of evidence pyramid: One approach for characterizing the strength of evidence of geoscience education research (GER) community claims. Journal of Geoscience Education, 65(4), 363-372.
  17. Trigwell, K. (2013). Evidence of the impact of scholarship of teaching and learning purposes. Teaching and Learning Inquiry, 1(1), 95-105.
  18. Wenger-Trayner, E., & Wenger-Trayner, B. (2015). Communities of practice: A brief introduction.
  • The SoTL Guide. This guide was developed by Nancy Chick, the current Director of the Endeavour Center for Faculty Development at Rollins College. The guide consists of two parts: Understanding SoTL and Doing SoTL. It is interspersed with videos of experienced SoTL Researchers talking about their approaches.
  • Carnegie SoTL Tutorial. This interactive tutorial is a compilation of materials from workshops and conferences sponsored by the American Association of Higher Education, the Carnegie Foundation for the Advancement of Teaching, the Professional and Organizational Development Network in Higher Education, the Lilly Conferences on College Teaching, and various colleges and universities. 
  • Annotated list of science education journals compiled by David Rudge, Western Michigan University.
  • Annotated list of SoTL/DBER journals maintained by Kennesaw State University Center for Excellence in Teaching and Learning. 
  • How to Publish in Scholarly Journals (Klingner et al., 2005). A succinct general primer for preparing manuscripts and choosing journals (not education research-specific). 
  • A list of predatory publishers and their journals, i.e., publishers whose business model is based upon soliciting publication fees from authors, is available here.
  • The Research Methods Knowledge Base is a digital textbook that covers the basics of social scientific research methods.
  • Recommended reading: