The Seeds of SEISMIC

By Nita Tarchinski and Tim McKay

Introduction

The SEISMIC project emerged from a series of conversations between March and June of 2018 between Tim and Elizabeth Boylan, who was then the Sloan Foundation’s Director of Programs on Diversity, Equity, and Inclusion in STEM Higher Education. The original focus is right there in the acronym: the Sloan Equity and Inclusion in SteM Introductory Courses project. The best short summary is probably provided by the original proposal’s cover sheet. The initial SEISMIC grant was awarded by the Sloan Foundation Board in November 2018.

When Nita started working as Project Manager at the end of 2018, the project really got off the ground. The first task was to recruit a set of partner institutions. To do this, we both spent the period from January to May 2019 visiting eight of our nine partner institutions, with our ninth visit in the fall. During these visits, we met with local leaders, ran SEISMIC workshops, and explored what everyone we talked to was most excited about doing within the SEISMIC community. This was one of the most fun and stimulating things we’ve ever done – an exciting sign for us that the SEISMIC idea was a good one.

At our first Summer Meeting, held in June 2019 in Ann Arbor, we gathered a handful of representatives from each institution, and launched a series of three working groups focused on Measurements of Equity and Inclusion, Classroom Experimentation, and Institutional Change. Meanwhile, the wider SEISMIC community began engaging with the collaboration through public presentations, book groups, campus events, and more. Among other things, these discussions led to the launch of a fourth ‘Constructs’ Working Group in February 2020.

Then in March, the COVID-19 pandemic began, unsettling and rearranging all of our plans. Remarkably, the shift to remote everything, with its ability to easily connect people on different campuses, led to a major expansion of interest in SEISMIC. After the murder of George Floyd, members of the Constructs Working Group helped our community come together to speak out and increase our commitment to action. Together, we released a “SEISMIC Statement on White Supremacist Violence and Anti-Blackness in the United States”, committing ourselves to antiracist action in policy, representation, research, and teaching and mentorship.

Since that launch, the SEISMIC community and its interests have grown in ways both anticipated and unexpected. Lately we’ve had a lot of discussions about what we hope to do together and how to do it best. As a contribution to these conversations, we thought it might be useful to remind everyone of where we started and recall the project’s original mission and goals. The two of us returned to the original SEISMIC proposal to remind ourselves of these “seeds of SEISMIC”, and wrote this blog post to convey what we found back to our members.

5 Key Points

To share the details, we’re posting the whole original proposal. To prevent you from needing to read it all, we’re providing pointers to places where you can find descriptions of some key elements. There are five main issues we thought it would be useful to address:

  1. The SEISMIC Mission
  2. Our Commitments to our Funder
  3. The Intent behind SEISMIC
  4. The Influence of Current Events
  5. The Role of the SEISMIC Task Force

We explain our vision for the next two years of the collaboration and hope that SEISMIC members continue to find intersections with their own goals and interests.

1. The SEISMIC Mission

From the beginning, SEISMIC’s main focus has been on transforming introductory STEM courses (Cover Sheet – Project Goal p. 1; Proposal p. 31). To accomplish this goal, we sought to establish a new standard for excellence for these introductory STEM courses that centered equity and inclusion (Proposal pp. 4, 19-20). We recognized that independent reform efforts were taking place on campuses across the country. SEISMIC was created to take advantage of these parallel efforts, drawing together a close, multifaceted collaboration of about 100 faculty, students, and staff in the STEM reform communities from a set of large research universities. Our ten SEISMIC institutions, each led by a local principal investigator, joined the collaboration because of this shared focus on introductory courses.

From the start, SEISMIC was envisioned as an open collaboration, inviting participation from everyone working on our member campuses. We aim to provide spaces within which individuals interested in equity and inclusion in STEM education can accomplish their own goals – doing things together that they find difficult to do alone. This has worked well, and in addition to our original goal of introductory course reform, we now have teams working on definitions of equity and inclusion, methods for encouraging and enabling reform, advancing critical approaches to our research methodologies, and more. This has been thrilling to see, and while the SEISMIC project retains a focus on introductory course reform, it is clear that the impact of the SEISMIC community will be much broader.

2. Our Commitments To Our Funder

When the grant was first awarded by the Sloan Foundation, we worked with Dr. Boylan to establish a few key metrics for success. You can read those here. In the proposal, we committed to conducting parallel analyses of institutional data across our SEISMIC institutions. We also agreed to conduct coordinated classroom interventions in STEM courses across the collaboration. Other commitments included building a community of people dedicated to equity and inclusion in STEM, seeking and obtaining additional funding to support our interventions and community, and setting a new standard for metrics of equity and inclusion as well as for research and development for STEM reform (Cover Sheet – Expected Products p. 1; Cover Sheet – Expected Outcomes p. 1; Proposal p. 14). More specifically, we planned to build tools to help researchers explore the individual student experience without relying on “simple characterizations of identity” (Proposal p. 52). In addition, we pledged to investigate new forms of student data, such as student performance and behavior in learning management systems, classroom observations, and student survey data (Proposal pp. 52-53).

While we have, as a community, expanded our activities well beyond these goals, these commitments remain essential for the central SEISMIC project. Much work remains to be done on them during the remaining years of the grant.

3. The Intent Behind SEISMIC

A key design goal of SEISMIC is to help participants interested in equity and inclusion do things together that they could never do alone. In many cases, this is work they’re already trying to do, often work that is central to their jobs and career ambitions. The advantage of joining the collaboration is having access to a community of people interested in related work and who might be interested to collaborate on specific projects. Thus, individuals will be able to do more in SEISMIC and have a broader impact than they might have had working alone (Proposal pp. 12-13). Additionally, SEISMIC Central provides organizational support to assist collaborative projects that align with the goals of SEISMIC. Creating this community and providing this organizational support is what we hoped to do and what we committed to the Sloan Foundation (Cover Sheet – Objectives p. 1; Proposal p. 18). We have been quite successful in these efforts, building a community of more than 600 individuals who connect with the collaboration through research projects, collaboration events, our monthly newsletters, and community activities such as our summer book groups.

In many ways, SEISMIC shares similarities with professional societies or conference communities. We are a group of people interested in similar topics who come together to talk about their work, form collaborations, and learn from each other. One benefit, for example, is that being part of SEISMIC can be seen as an advantage to funding agencies looking to support research projects with broader impacts (Proposal pp. 15-16). Several SEISMIC members have had success with their grant proposals after including details on the SEISMIC infrastructure.

4. The Influence of Current Events

It comes as no surprise that global events over the last two years have had an impact on SEISMIC. When SEISMIC began in 2019, we had a clear focus on transforming introductory STEM courses. When COVID-19 shut down our universities in March 2020, our introductory courses did transform: they shifted online, making instructors rethink many of their longstanding teaching practices. Inequities always present in the classroom began to be acknowledged in the new learning environment.  COVID-19 also highlighted many longstanding racial disparities.

After the murder of George Floyd, long-standing demands for racial justice reached a new peak. Conversations previously about DEI (Diversity, Equity, and Inclusion) began to be about justice and antiracism. There were new calls for proposals related to racial equity. Academic movements on Twitter like #ShutDownSTEM, #ShutDownAcademia, and #Strike4BlackLives gained visibility.

Understandably, the focus of many SEISMIC members and higher education in general shifted. Members of the SEISMIC collaboration wrote a public statement on white supremacy and antiracism that was quickly endorsed by the project and organized a GoFundMe for the NAACP Legal Defense and Educational Fund. We also coordinated book groups on antiracism and related topics and organized affinity groups for our BIPOC members, graduate students, and LGBTQIA members. We collaborated with the Accelerating Systemic Change Network (ASCN) to host events on structural equity and our organizations’ roles in promoting justice.

Many of these activities were possible because of the community we built around SEISMIC and the flexibility of our collaboration to adapt to the community’s goals. While we encourage our community members to follow their calls for disrupting institutional structures and taking bigger steps for racial equity in higher education, and will continue to provide organizational support for these efforts to the extent we are able, our commitment to SEISMIC’s first goal of transforming introductory STEM courses remains at the heart of what we do. This is one sphere of influence our collaboration has that can contribute to racial equity and institutional change.

5. The Role of the SEISMIC Task Force

In parallel with the global challenges of the last two years, SEISMIC has also been undergoing internal challenges related to the inclusivity of our collaboration. Members, particularly women of color and junior scholars, have been harmed in SEISMIC spaces. In light of this information and subsequent discussions, SEISMIC leadership established a Task Force to examine current collaboration structures and recommend new or revised structures to make our collaboration more equitable and inclusive. More information on the Task Force and the conversations leading up to its establishment are here. The SEISMIC Task Force will support our community as we continue to work towards our collective goals.

Call to Action

As we start our third academic year together, we are looking for SEISMIC members eager to join us as we refocus on the original goal for SEISMIC: to make measurements of equity in introductory STEM courses. This goal is what initially brought our institutions together, and we believe many in our community are ready to return to it in earnest. Looking to the next two years of SEISMIC (the time we have left before our current funding runs out), we hope to share models for using data to probe equity and inspire transformation in introductory STEM courses. We aim to share code to run these analyses on institutional data sets and to share the results of analyses run on our institutions. If you are interested in joining these efforts, please let us know here.

At the same time, we recognize that our community has grown beyond the initial participants interested in our original SEISMIC goal. Our equity measures are in many ways the “pilot project” of SEISMIC, not our only project. Our SEISMIC Task Force will help to strengthen our community over the next few months and open the door to additional projects of interest to our members. We are thrilled there are new directions for this community. Knowing the strong ties between us makes us hopeful for how the SEISMIC community can be sustained even if funding runs out. Whether you joined SEISMIC when we began in 2019 or more recently, there is space for you and your ideas in this collaboration. We hope to see you at our next event!

 


 

 

Nita Tarchinski

Nita Tarchinski is the Project Manager for the Sloan Equity and Inclusion in STEM Introductory Courses (SEISMIC) Collaboration, coordinating multi-institutional and multidisciplinary research and teaching projects focused on making introductory STEM courses more equitable and inclusive.

 

 

 

Tim McKay, Ph.D.

Timothy A. McKay is the Director of the Sloan Equity and Inclusion in STEM Introductory Courses (SEISMIC) Collaboration. He is also the Associate Dean for Undergraduate Education in the College of Literature, Science and the Arts at the University of Michigan. He is a data scientist, currently using data to understand and improve teaching and learning.

 


 

Building Structural Equity: What do our organizations need?

By Nita Kedharnath and Rachel Renbarger

Background and Context

ASCN and SEISMIC began collaborating in spring 2021 to understand how both organizations could synergistically support STEM education transformation. These collaborative meetings included the project coordinators of both organizations and the authors of this piece, Nita Kedharnath (project manager of SEISMIC) and Rachel Renbarger (research director of ASCN). We started meeting monthly to discuss equity challenges within our organizations, to problem-solve together, and to share resources to ensure equity progress continues for both our groups even while we target different priorities.

ASCN: The Accelerating Systemic Change Network (ASCN) is a community of change agents who create events, community, and resources to improve instruction in higher education. The network, sponsored by grant funders, is free for anyone to join.

In March, the murder of Asian American women working at spas in Atlanta amplified calls for anti-racist actions from institutions in the United States, including institutions of higher education. During one of our collaborative meetings, we asked each other what steps our respective organizations have taken toward racial justice. While both ASCN and SEISMIC had been working on diversity, equity, inclusion, and social justice in STEM education since the organizations’ inceptions, this work was outward-facing and dedicated to helping university faculty members change their institutional settings. What were our organizations’ roles in tackling racial justice? What should we be doing as leaders? This lack of a clear path forward led us to ask these questions to experts who have been doing this reflective work within STEM higher education.

Panel Event

We first began to answer this question through examining the latest report by the American Council on Education (ACE) and talking to experts who were openly leading events centered around racial equity. The ACE report, titled Shared Equity Leadership: Making Equity Everyone’s Work, documented how multiple campus leaders promoted equity by convincing more campus stakeholders to be involved in making change on campus. This report encouraged us to examine our own personal reasons for doing this work, but we felt a disconnect since we were part of organizations rather than campuses. This led us to ask organizational leaders to speak on a panel and describe how they helped their organization do this reflective, internal work as well as the outward, external work.

This panel was titled, “Building Structural Equity: National Networks’ Role in Promoting Justice,” and included one of the writers of the ACE report, Dr. Elizabeth Holcombe, as well as Dr. Shirley Malcom, Director of the STEM Equity Achievement (SEA) Change initiative at the American Association for the Advancement of Science; Dr. Tabbye Chavous, Director for the National Center for Institutional Diversity; and Dr. Stanley Lo, President-Elect for the Society for the Advancement of Biology Education Research (SABER). From the panel event, we learned there were consistent challenges to doing this equity work within organizations such as limited mental and emotional bandwidth, difficulty convincing colleagues to join in the work, frustration with changes that are band-aids rather than systemic changes, and limited resources or support for doing justice-oriented work. However, we also learned what we could do to improve equity within organizations: collect data to understand challenges and successes, listen to those impacted by racial injustice, write grants to fund these initiatives, recognize the work already being done, and create groups of dedicated people to do the work together. These challenges and actions led us to work with our members to look internally to see what each organization should focus on for the next year.

After the Panel

Following the panel event, ASCN and SEISMIC co-hosted a working session for ASCN and SEISMIC members to gather and discuss how the lessons from the panel could be applied to our organizations. We asked members to consider what barriers stood in the way of our organizations’ promotion of justice and what actions they’d like us to take to address those barriers. Some barriers that were shared related to interactions between members – questions of power structures, institutional roles, and whether the ideas of STEM scholars or social science scholars were more valued in the organizations. Another key barrier was the lack of a clear vision for the organizations that centered equity and justice. The fact that most of our members are not paid to do the work of our organizations presented another challenge for our organizations’ ability to act. For actions we could take, several members suggested we hold town hall meetings or other open forums for members to develop shared goals and values for our organizations. Members were also interested in offering invitations to self-work, organizing seminar series, applying for external funding, and creating space for members to get what they need from the organizations, instead of assuming all members need the same thing. ASCN and SEISMIC then conducted individual meetings to create organization-specific equity plans.

During the SEISMIC Summer Meeting, SEISMIC leaders took initial steps towards these actions by hosting a discussion on the vision and goals of SEISMIC. The conversation quickly turned inward, with members asking how SEISMIC can achieve its outward-facing goals while internal structures hinder full participation of members. SEISMIC created an anonymous space for members to highlight structural issues with the collaboration, and then facilitated a discussion to hear from members on specific actions SEISMIC needs to stop and specific ones to start to make the collaboration more inclusive. In response to these difficult conversations, SEISMIC leadership sent out a message articulating how SEISMIC would examine its own structures and make changes, set up a facilitated open forum to hear feedback on the plan, and is now in the process of creating a SEISMIC Task Force to examine existing collaboration structures and recommend new ones. In addition, SEISMIC has created spaces for members to come together around shared experiences, including a Black, Indigenous, People of Color (BIPOC) group, a Graduate Students group, and a Lesbian, Gay, Bisexual, Transgender, Queer/Questioning, Intersex, Asexual (LGBTQIA) group.

Since the working session, ASCN has been in discussions with many leadership groups to understand how we can best implement the action items. Leaders included the PIs and the working group leaders along with specific working groups implementing summer programming. Each group discussed how they would want to move equity forward as each working group functions differently. For example, our working group 6 that focuses on aligning incentives with systemic change decided to update the purpose of their working group to weave inclusivity and equity within all of their activities. This differed from our working group focused on change leaders that decided to partner with equity scholars to collaborate on events. The ASCN leadership also created an equity meeting open to all members to receive feedback on the plan and received comments that have thus been incorporated into the fall plan. We created communication and leadership expectation guidelines, reviewed and edited by leadership, to ensure our norms are clearly stated. Lastly, we asked for collaborators in racial equity and wrote a grant this summer regarding a research project to create an institute for institutional teams to create change on their campus using systemic change and racial equity principles. This project will also provide an open access, racial equity framework for systemic change that will be available on ASCN’s website if funded.

Together, ASCN and SEISMIC have proposed strategies for organizational equity within the broader STEM education community and committed to regular accountability meetings. We proposed this conversation as a session at the AAC&U Transforming STEM conference to bring together other networks and share how we can build authentic communities and bridges between communities to focus on common challenges to equity within our organizations. We also plan to continue to meet monthly to keep both organizations accountable toward our racial equity plans.

SEISMIC This Fall

Following summer activities, the SEISMIC Task Force will begin meeting this fall over the course of 12 weeks to examine existing collaboration structures and propose new ones to make SEISMIC more inclusive. SEISMIC will also be co-sponsoring the Society for the Advancement of Biology Education Research (SABER) “Striving towards inclusion in academic biology” seminar series for the second year. SEISMIC leadership will be working to publicize the original vision and goals for SEISMIC and outline paths for members to continue pushing forward its mission. At the same time, leadership has recognized that some members, in part due to the shifting focus brought on by the COVID-19 pandemic and the renewed calls for racial justice, are looking for SEISMIC to have broader impacts in higher education beyond the original vision and goals. This fall will be important for the collaboration to reestablish its scope and priorities while maintaining a community where members can get what they need.

ASCN This Fall

For ASCN, our priorities are to collect data to understand gaps to address, clarify our commitment to equity, increase leadership representation, and commit to recognizing the work of internal and external leaders. This fall we will be collecting data in a member survey, adding questions to previous member surveys to understand demographics (i.e., race, gender, role) of our membership and leadership as well as members’ experiences within ASCN. In clarifying our commitment to equity, our working group leaders suggested equity be a part of ASCN’s values so that equity is part of everything ASCN does, not just as a separate working group. To do this, we will create an ad-hoc committee to draft ASCN values to be ratified at our annual meeting in January. As we continue to make ASCN leadership more representative of our membership, we are seeking nominations for PIs of color to provide a much-need perspective as ASCN moves forward. Finally, we will commit to recognize the leaders of equity. We will highlight the work of ASCN leaders within regular meetings and also continue to advocate for equity work as important within the broader higher education community.

 

 

This is the first in a series of posts called Doing Organizational Equity Work in STEM meant to increase transparency and accountability regarding our organizations’ diversity, equity, inclusion, and social justice goals. The next update will be posted in January 2022 to discuss progress towards the goals mentioned in this post and new directions in our organizations’ internal work to promote equity.

 


 

Nita Kedharnath

Nita Kedharnath is the Project Manager of the SEISMIC Collaboration. She coordinates the implementation and evaluation of SEISMIC activities and co-leads the Student Perspectives project in the Constructs Working Group.

 

Rachel Renbarger, Ph.D.

Dr. Rachel Renbarger is the Research Director of ASCN where she helps support the inclusion and dissemination of research-based educational strategies. With her PhD in Educational Psychology, her research works to address issues of access and equity in postsecondary education. This includes undergraduate as well as graduate education. Follow her on Twitter: @Dr_Rachel_Ren


Introducing SEISMIC Themes

By Linda Adler-Kassner & Nita Kedharnath

Why are we adding Themes to SEISMIC?

At our 2020 SEISMIC Summer Meeting, we heard from participants a desire to more formally connect the work of our Working Groups. In many ways, “measurement,” “experiments,” “implementing change,” and “constructs” are parts of larger processes for institutional change. For instance, to recommend policies, we need to experiment with changes; we need to define “constructs” to assess efforts toward equity and inclusion. With this in mind, while still maintaining the communities and projects of the Working Groups, we decided to add a new organizational layer to our collaboration called Themes.

Themes will tie together the knowledge gained through SEISMIC projects within Working Groups to focus on specific SEISMIC goals and to make SEISMIC-level recommendations. To achieve our ambitions for large-scale improvement in equity and inclusion in introductory STEM courses, Themes will enable us to leverage the expertise across the collaboration, intentionally bringing together experts in DEI scholarship, STEM teaching faculty, campus leaders, and institutional researchers.

What are the Themes?

Each Theme is defined by a short phrase and guiding question. The Theme Leaders (more on that below) will identify specific deliverables for each Theme. Each Theme is intentionally defined to include people and projects from across the four SEISMIC Working Groups (measurement, experiments, implementing change, and constructs); they are not meant to align with a single Working Group or Key Project.

The three Themes that we have identified and the questions at their core are:

Policies, Practices, and Assessments for Change

What are the best data-driven policies, practices, and assessment strategies to promote STEM classroom equity, and how can departments incorporate these strategies across their introductory courses?

Presenting Data at Multiple Levels for Change

What is the best way to operationalize equity and inclusion and present usable and compelling data to different audiences (faculty, department, administration) to motivate change and promote asset-based thinking regarding students at multiple levels?

Capacity-Building for Introductory STEM Change Makers

How can SEISMIC support introductory STEM change makers in leveraging their resources and developing structures so all students who have been historically “othered” can feel they belong in STEM?

Okay, but how are these different from Working Groups?

Good question. SEISMIC’s Working Groups bring together people and/or projects around specific foci under a wider umbrella. All are related to the Working Group topic. For example, our Experiments Working Group brings together many discipline-based education researchers and instructors to run specific experiments in STEM courses. These experiments may or may not be related to each other, but are connected in that they are all experiments in STEM courses. The Working Group provides the community to brainstorm with and try out these many projects.

Our Themes, on the other hand, seek to bring together the knowledge from multiple Working Groups into an aligned set of recommendations or activities. A Theme’s focus is not on running projects, but rather is focused on finding the connections between different SEISMIC activities. The ultimate goal of each Theme is to produce something actionable on behalf of the collaboration. The figure below describes how the different pieces of SEISMIC fit together.

 

What will the Themes actually do?

Based on the core questions (above), Theme Leaders will determine appropriate deliverables for each Theme. This could involve developing a set of recommendations, running workshops for the collaboration, publishing papers on their findings, or other activities. Once this is established, the Themes will connect with Working Groups and project teams as needed to learn what the collaboration has done related to these Themes, and what is missing.

Each Theme is responsible for organizing at least two SEISMIC-wide events in 2021. Events the Themes might hold include facilitated discussions, workshops, talk presentations, and more. These events will be an opportunity for the Themes to share what they are learning, get feedback, and stay connected to the collaboration. Each Theme will also provide a progress report to the SEISMIC Collaboration Council twice a year, including details on their most recent event, how they are making progress on their set of goals and deliverables, and upcoming plans to apply for funding. Finally, each Theme will present an update on their work at the 2021 SEISMIC Summer Meeting.

This sounds like a lot of work. How is this really going to happen?

Each Theme will be guided by two leaders, a Director and a Fellow (6 positions total for three Themes). All leaders will receive a $4,000 stipend for the 2020-2021 year. Applications for these positions close 10/30/2020 (apply here). Our Collaboration Council will review applications and select the leaders by mid-November 2020.

Anyone affiliated with a SEISMIC Institution is encouraged to apply for either the Director or Fellow positions. We especially encourage members with grant-writing experience to apply to be a Theme Director. The Director will take on a bigger picture role around activities of the Theme such as writing grant proposals or planning a SEISMIC-wide Theme event. They will look out for funding opportunities for the Theme. 

We especially encourage graduate students and postdoctoral fellows looking for leadership experience to apply to be a Theme Fellow. The Fellow will be responsible for connecting directly with the SEISMIC projects and staying updated on SEISMIC activities.

Theme Leaders are Expected To:

  • Apply for cross-institutional funding to support the work of the Theme
  • Lead organization of Theme events at least twice per year
  • Pull together Theme progress reports to send to the Collaboration Council twice per year
  • Facilitate collaboration within their Theme and between their Theme and the Working Groups
  • Identify clear goals and deliverables for their Theme and move their team towards them. Deliverables could include developing a set of recommendations, running workshops, publishing, etc.

 

All SEISMIC members are encouraged to join any Theme they are interested in. Participation in these Themes will look different for each one, but will generally involve supporting the Theme to achieve its (to be determined) deliverables. Once our Theme Directors and Fellows are selected, we will provide more information on what it means to be a Theme participant for each Theme. For now, feel free to email Nita (nitaked@umich.edu) if you would like to indicate interest in a Theme or if you have any questions.


 

Linda Adler-Kassner, Ph.D.

Linda Adler-Kassner, Ph.D. is the Faculty Director of the Center for Innovative Teaching, Research, and Learning at the University of California Santa Barbara as well as a Professor of Writing Studies and the Associate Dean of Undergraduate Education. Linda is a member of the SEISMIC Collaboration Council and co-leads the Access to Practice project with Vanessa.

 

Nita Kedharnath

Nita Kedharnath is the Project Manager of the SEISMIC Collaboration. She coordinates the implementation and evaluation of SEISMIC activities and co-leads the Stakeholder Perspectives project in the Constructs Working Group.


Fostering Student Success with Access to Practice

By Vanessa Woods, Maggie Safronova, and Linda Adler-Kassner

 

As we develop courses that introduce students to practices and content within our respective disciplines, we must recognize that the students who enter our classrooms come with different social and academic experiences that inform their learning. In large introductory courses where the majority of participants can be categorized as disciplinary novices, this lack of access can lead to a disconnect that discourages students and frustrates professors.  As educators, we strive to design courses that provide students with access to our discipline regardless of how their prior academic experience shaped students’ process of learning.  Access to Practice (ATP) investigates how the development and implementation of highly structured peer review activities contribute to student academic experience.

Through ATP, UC Santa Barbara instructors work with the Center for Innovative Teaching, Research, and Learning (CITRAL) to design highly structured writing and peer review prompts focusing on challenging course concepts or challenging ways of writing. Over the two+ years CITRAL has sponsored ATP, faculty from Classics to Psychological and Brain Sciences, Political Science to Ecology, Evolutionary, and Marine Biology have developed ATP prompts and peer reviews, often in very large classes. Students have written and provided feedback about subjects as wide-ranging as the calculations involved in stoichiometry and the current-day applications of the Oedipus myth.

Structured peer review activities provide students with an environment in which they can develop ways of practicing within a discipline. As the students practice the application process several times in a guided environment they have the opportunity to develop their own approach that can be transferred to different contexts. The goal of this process helps students figure out the disciplinary norms including what type of questions to ask and how to answer them, writing conventions, and appropriate types of evidence. As the students get the opportunity to actively engage with disciplinary practices we expect to see an increase in student’s metacognitive strategies as well as help them feel like true members of their discipline. Preliminary data from UCSB suggests that by writing and providing feedback aimed at practicing disciplinary norms, students gain a deeper understanding of challenging concepts. This promising data is the basis for further exploration of the mechanism by which peer review activities affect elements that contribute to student success. The possible mechanisms being explored are metacognitive strategies, a sense of disciplinary membership, and disciplinary practices. We are currently looking for collaborators. Contact Vanessa Woods at vewoods@ucsb.edu for more information.


 

Vanessa Woods, Ph.D.

Vanessa Woods, Ph.D. is a Lecturer with Potential Security of Employment (LPSOE) which is a tenure track lecturer position (teaching professor) at the University of California Santa Barbara. She is an expert in effective teaching practices and student success research. Vanessa is active in the SEISMIC collaboration and leads the Access to Practice Project (Experiments Working Group, Project 4).

 

Maggie Safronova, Ph.D.

Maggie Safronova, Ph.D. is the Associate Director at the Center for Innovative Teaching, Research, and Learning at the University of California Santa Barbara. Maggie is involved in projects that explore the role of pedagogical innovations on students’ sense of belonging in large universities. Maggie is also the project director for UCSB’s ECoach project.

 

Linda Adler-Kassner, Ph.D.

Linda Adler-Kassner, Ph.D. is the Faculty Director of the Center for Innovative Teaching, Research, and Learning at the University of California Santa Barbara as well as a Professor of Writing Studies and the Associate Dean of Undergraduate Education. Linda is a member of the SEISMIC Collaboration Council and co-leads the Access to Practice project with Vanessa.


Scaling Up Analysis in a SEISMIC Measurement Project

By Eben B. Witherspoon

 

The AP Project in the Measurement Working Group has been moving quickly through their project milestones. After starting strong last fall and developing shared analyses for the initial three SEISMIC institutions involved, they have started “scaling up” their project to the other seven SEISMIC institutions. While the three initial institutions are ironing out the details of their final models, the scale-up institutions are in various stages of cleaning their data and getting it ready to process. In this scale-up process, our AP Project team has learned much about what it takes to run a project across SEISMIC, and what key challenges can come up. We hope this post will provide some support for our fellow Measurement projects as they start preparing to scale up their efforts!

1. Set Common Variable Names

The first thing to consider when scaling up a Measurement project is setting common definitions and variable names for key concepts in the project. It was surprising to us how much variation there was by institution in seemingly straightforward terms such as “cohort.” Taking the time to clarify these not only facilitates discussions during meetings, but is important for ensuring that variables are being defined and generated in the same way, and that eventually, shared analysis code can be run on each institution’s dataset. 

Variable definitions and names from initial discussions in the AP and Demographic projects were merged in a working document to categorize variables across the Working Group. This could be a good jumping off point for coming up with SEISMIC-wide institutional data variable names.

An example of a method for generating project-wide variable definitions.

 

2. Attend to Dataset Formatting

Second, it is essential to decide up front on the format of the dataset for analysis (i.e. long vs. wide). Unless this is explicitly discussed, it’s easy to assume everyone is doing it the same way (but there are lots of ways people store and think about time-series data!) and this has big implications for doing shared analyses later. For example, in the AP Project, we eventually landed on a combination similar to “panel data” – our data is wide by student (i.e. each row is a single student) and stacked long by each discipline (i.e., all observations who took BIO are “stacked” on top of all observations in PHYSICS), with every student unique within each discipline, but able to be repeated across disciplines (i.e. if a student took both BIO and PHYSICS). This made the most sense for our project, as it allowed us to easily subset our analyses by discipline. It might be overkill, but creating a mock-up dataset could even be helpful to visually represent how the data looks, which variables are time-invariant or not, etc.

3. Share Model Specifications and Basic Descriptive Stats

Last but certainly not least, once you have settled on your Research Questions (RQs), it is very helpful to share clear model specifications including DV, IV and analytic sample. Even when the same model is being run, misinterpretation of patterns across institutions can easily be missed when looking at only regression tables when there are different understandings of what sample is being analyzed. Sometimes something as simple as looking at sample sizes can help catch these discrepancies early on. For example, if two schools of about the same size have vastly different Ns, something may be up. One way our project addressed this issue was by moving the part of the analysis that defines the sample (which was previously done in each individual institution’s data cleaning) to the shared script, so that each institution was literally running the same code to subset the data and generate the sample for each analysis. Of course, in order to do this, there first needed to be common variable names and similarly formatted datasets…hence parts 1 and 2 🙂 As an added “bonus,” these checks and balances worked together; if our shared code couldn’t run or gave us weird results, this led us to uncover previously undiscovered issues in our variable or sample definitions!

Additional Recommendations

We recommend each project save their analysis code in the SEISMIC-wide GitHub repository (email your GitHub username to bkoester@umich.edu to join). This is a great way to share code and track changes, without making overlapping edits. Our project also used R/RStudio and Google Co-Lab with Jupyter notebooks to share, run, and comment on each other’s code as we were developing it. Then, we saved the agreed-upon code in our AP Project (WG1-P4) GitHub repository.

We have also found it helpful to use R-Markdown to create an “Analysis Workflow” file, which acts as a guide for AP project participants in understanding the analysis process overall, including how to create a dataset that will work with our shared analysis code. It captures much of our thinking on streamlining and simplifying the scale-up process.  It also serves as a single location that links to various other relevant documents for running analyses (i.e. variable naming conventions, model specifications). In addition, the document itself is shared and editable, which allows notes to be added by institutions as more specific things pop up that might be useful to others (i.e. Pitt didn’t have a variable for the year AP was taken, so we developed and noted our work-around).

Example of the Analysis Workflow document for AP Project.

 

Calling All Demographics Project Analysts

We would love feedback from Demographics Project (WG 1 P1) analysts about our process and what you have been doing to coordinate. For example, have you found less complicated workarounds for the same issues? Are we missing key parts of scaling up that you’ve experienced? Let us know!

Interested in joining the AP Project?

Overall, the process for onboarding is:

  1. Join the Project GitHub repository (by emailing a GitHub username to bkoester@umich.edu)
  2. Read the “Analysis Workflow” file (available on GitHub)
  3. Preview the SEISMIC variable definitions doc (also linked to in the Analysis Workflow file)
  4. Create an institution-specific folder in the WG1-P4 GitHub that contains the data cleaning files for that institution’s data – these will all be slightly different, but may be useful to see how others have done it as there will be some overlap.
  5. Once the data is in the same format, run the shared analysis file (available on GitHub). 
  6. Join one of our meetings and share your findings!

 

 

 

Eben B. Witherspoon, Ph. D.

Eben Witherspoon is a post-doctoral researcher in the Learning Research and Development Center (LRDC) at the University of Pittsburgh. His main line of research examines attitudinal and environmental factors during the transition to college that influence retention in STEM career pathways for underrepresented students. Currently, he is working on a project looking at the factors influencing gendered attrition in the pre-med course sequence. Eben is an active SEISMIC member and works on the AP Project (Measurement Working Group, Project 4).