Understanding the ways in which Black folks identify as physicists can provide useful information about the facets of the physics discipline that perpetuate systems of oppression. In prior work, we developed the critical physics identity (CPI) framework to identify the structures and systems that impact the ways that people identify with the physics discipline. In this paper, we apply the CPI to deconstruct the factors that influence physics identity for a sample of undergraduate, graduate, and professional Black physicists. Using the CPI constructs, we classify important internal and external factors that influence how they identify, or not, as a physics person. We find that racialized resources were more influential than physics identity constructs, and document the subcodes of each CPI construct that are discussed most frequently among participants. We note variations between early and later career physicists and between men and women.

We have studied the impact of incoming preparation and demographic variables on student performance on the final exam in the standard introductory calculus-based mechanics course at three different institutions. Multivariable regression analysis was used to examine the extent to which exam scores can be predicted by a variety of variables that are available to most faculty and departments. The results are surprisingly consistent across the institutions, with only math SAT or ACT scores and concept inventory prescores having predictive power. They explain 20%–30% of the variation in student exam performance in all three cases. In all cases, although there appear to be gaps in exam performance if one considers only demographic variables (gender, underrepresented minority, first generation), once these two proxies of incoming preparation are controlled for, there is no longer a demographic gap. There is only a preparation gap that applies equally across the entire student population. This work shows that to properly understand differences in student performance, it is important to do statistical analyses that take multiple variables into account, covering both subject-specific and general preparation. Course designs and teaching better matched to the incoming student preparation will likely eliminate performance gaps across demographic groups, while also improving the success of all students.

We report on a study investigating the influence of context, direction of translation, and function type on undergraduate students’ ability to translate between graphical and symbolic representations of mathematical relations. Students from an algebra-based and a calculus-based physics course were asked to solve multiple-choice items in which they had to link a graph to an equation or vice versa and explain their answer. The first part of the study focuses on the accuracy of the chosen alternative. Using a generalized estimating of equations (GEE) analysis we find that mathematics items are solved better than physics or kinematics items; that items starting from a graph are solved better than those starting from an equation; and that items on inversely proportional functions are the hardest for students. Quantitatively we see big differences in the number of correct answers between the algebra-based and calculus-based cohorts, but qualitatively the effects of the item variables on the accuracy are the same for both groups. The second part of the study focuses on students’ argumentations for the chosen alternative. We observe that students use their physical knowledge 2 to even 3 times as much in kinematics items than in other physics items. When there is an effect of context on argument use, we observe that the use of an argument in a mathematics context differs significantly from the use of that argument in the physics and/or kinematics context. With regard to function type, students explain their choice of answer in items on inversely proportional functions with different arguments and fail more often to answer correctly. In general, the data also show that students from the calculus-based course use more mathematical arguments and score better on the items.

[This paper is part of the Focused Collection on Quantitative Methods in PER: A Critical Examination.] A relevant aim of research in education is to find and study the reasoning lines that students deploy when dealing with problematic situations. This can be done through an analysis of the answers students give to a questionnaire. In this paper, we discuss some methodological aspects involved in the quantitative analysis of a questionnaire by means of two different clustering methods, a hierarchical one and a nonhierarchical one. We start from the coding procedures needed to obtain analyzable data from the questionnaire and from a definition of a correlation coefficient suitable for measuring student similarity in the case of binary coding of student answers. Then, criteria to choose the optimal number of clusters are discussed, and for the same purpose a new coefficient is introduced that measures the total amount of information we can obtain from a clustering solution. We show that each cluster can be characterized by its centroid that summarizes the answers most frequently given by the cluster students to the questionnaire. Finally, an example of the application of these procedures to a student sample is given, and a comparison between the two clustering methods is discussed.

[This paper is part of the Focused Collection on Quantitative Methods in PER: A Critical Examination.] The Rasch model is a probabilistic model which describes the interaction of persons (test takers or survey respondents) with test or survey items and is governed by two parameters: item difficulty and person ability. Rasch measurement parallels physical measurement processes by constructing and using linear person and item measures that are independent of the particular characteristics of the sample and the test items along a unidimensional construct. The model’s properties make it especially suitable for test construction and evaluation as well as the development and use of surveys. The evaluation of item fit with the model can pinpoint problematic items and flag idiosyncratic respondents. The possibility of determining sample—independent item difficulties makes it possible to use the Rasch model for linking tests and tracking students’ progression. The use of the Rasch model in PER is continuously increasing. We provide an overview and examples of its use and benefits, and also outline common mistakes or misconceptions made by researchers when considering the use of the Rasch model. We focus in particular on the question of how Rasch modeling can improve some common practices in PER, such as test construction, test evaluation, and calculation of student gain on PER diagnostic instruments.

[This paper is part of the Focused Collection on Quantitative Methods in PER: A Critical Examination.] A common goal in discipline-based education research (DBER) is to determine how to improve student outcomes. Linear regression is a common technique used to test hypotheses about the effects of interventions on continuous outcomes (such as exam score) as well as control for student nonequivalence in quasirandom experimental designs. (In quasirandom designs, subjects are not randomly assigned to treatments. For example, when treatment is assigned by classroom, and observations are made on students, the design is quasirandom because treatment is assigned to classroom, not subject (students).) However, many types of outcome data cannot be appropriately analyzed with linear regression. In these instances, researchers must move beyond linear regression and implement alternative regression techniques. For example, student outcomes can be measured on binary scales (e.g., pass or fail), tightly bound scales (e.g., strongly agree to strongly disagree), or nominal scales (i.e., different discrete choices for example multiple tracks within a physics major), each necessitating alternative regression techniques. Here, we review extensions of linear modeling—generalized linear models (glms)—and specifically compare five glms that are useful for analyzing DBER data: logistic, binomial, proportional odds (also called ordinal; including censored regression), multinomial, and Poisson (including negative binomial, hurdle, and zero-inflated) regression. We introduce a diagnostic tool to facilitate a researcher’s identification of the most appropriate glm for their own data. For each model type, we explain when, why, and how to implement the regression approach. When: we provide examples of the types of research questions and outcome data that would motivate this regression approach, including citations to articles in the DBER literature. Why: we name which linear regression assumption is violated by the data type. How: we detail implementation and interpretation of this modeling approach in R, including R syntax and code, and how to discuss the regression output in research papers. Code accompanying each analysis can be found in the online github repository that is associated with this paper (https://github.com/ejtheobald/BeyondLinearRegression). This paper is not an exhaustive review of regression techniques, nor does it review nonregression-based analyses. Rather, it aims to compile and summarize regression techniques useful for the most common types of DBER data and provide examples, citations, and heavily annotated R code so that researchers can easily implement the technique in their work.

[This paper is part of the Focused Collection on Quantitative Methods in PER: A Critical Examination.] The education of physicists requires annual progress in mathematics through grades K–12, continuing on to high school physics, undergraduate instruction, and often many years of postgraduate study. The traditional domain of physics education research is the study of pedagogical techniques for a single lesson or a single course, but the physics and physics education research communities will need to study the complete educational system to increase performance and participation, especially for underrepresented groups. Analysis of long-term student outcomes and policy impacts requires predictive longitudinal techniques. We begin with concepts from fluid and statistical mechanics—trajectories and streamlines—to visualize longitudinal outcomes and make predictions. Drawing on coarse-graining procedures to model the movement of particles in a fluid, the longitudinal data are sorted into score bins to depict the flow of scores through time: trajectories depict the average scores over time for initial score bins and streamlines provide an approximate way to calculate the flow of student scores over many years based on only two consecutive years of data. However, due to the partially stochastic nature of observed scores, the coarse-graining procedure that sorts students into score bins amplifies a statistical phenomenon known as regression to the mean (RTM). As a result, streamlines do not provide an accurate prediction for the future performance of students. Here we discuss a new coarse-graining procedure, regression-corrected (RC) grouping, which reduces RTM in the streamlines. We explain the idea of RC streamlines through a toy model of freshman physics performance and then apply them to the realistic setting of the Texas State Longitudinal Data System, which contains standardized testing data for students throughout primary and secondary school since 2003. We show that the RC streamlines accurately predict trajectories, using 2 or 3 years of data. Therefore RC streamlines can be used to identify the effects of academic interventions on a time scale comparable to that of policy changes. We illustrate this assertion by examining a particular policy intervention, Texas’s Student Success Initiative.

[This paper is part of the Focused Collection on Quantitative Methods in PER: A Critical Examination.] Physics education researchers (PER) often analyze student data with single-level regression models (e.g., linear and logistic regression). However, education datasets can have hierarchical structures, such as students nested within courses, that single-level models fail to account for. The improper use of single-level models to analyze hierarchical datasets can lead to biased findings. Hierarchical models (also known as multilevel models) account for this hierarchical nested structure in the data. In this publication, we outline the theoretical differences between how single-level and multilevel models handle hierarchical datasets. We then present analysis of a dataset from 112 introductory physics courses using both multiple linear regression and hierarchical linear modeling to illustrate the potential impact of using an inappropriate analytical method on PER findings and implications. Research can leverage multi-institutional datasets to improve the field’s understanding of how to support student success in physics. There is no post hoc fix, however, if researchers use inappropriate single-level models to analyze multilevel datasets. To continue developing reliable and generalizable knowledge, PER should use hierarchical models when analyzing hierarchical datasets. The Supplemental Material includes a sample dataset, R code to model the building and analysis presented in the paper, and an HTML output from the R code.

[This paper is part of the Focused Collection on Quantitative Methods in PER: A Critical Examination.] While many studies have examined the structure, validity, and reliability of the Force Concept Inventory, far less research has been performed on other conceptual instruments in widespread use in physics education research. This study performs a confirmatory analysis of the Conceptual Survey of Electricity and Magnetism (CSEM) guided by a theoretical model of expert understanding of electricity and magnetism. Multidimensional Item Response Theory (MIRT) with the discrimination matrix constrained to the theoretical model was used to investigate two large datasets ($N_1=2014$ and $N_2=2657$) from two research universities in the United States. The optimal model identified by MIRT was similar, but not identical, for the two datasets and had very good model fit with comparative fit indices of 0.975 and 0.984, respectively. The most parsimonious optimal model required 23 independent principles of electricity and magnetism and was significantly better fitting than a more general model dividing the CSEM into 6 general topics. The optimal models for the two samples were quite similar, sharing 22 of a possible 26 conceptual principles. Most of the overall item difficulties and discriminations were significantly different between the two samples; however, the rank order of the overall difficulty and discrimination were generally similar. There was much more similarity between the discrimination by item of the individual principles. Five items had a difficulty ranking that was substantially different between the two samples, indicating that while generally similar, relative difficulty does depend on the student population and instructional environment.

[This paper is part of the Focused Collection on Quantitative Methods in PER: A Critical Examination.] Physics education researchers (PER) commonly use complete-case analysis to address missing data. For complete-case analysis, researchers discard all data from any student who is missing any data. Despite its frequent use, no PER article we reviewed that used complete-case analysis provided evidence that the data met the assumption of missing completely at random necessary to ensure accurate results. Not meeting this assumption raises the possibility that prior studies have reported biased results with inflated gains that may obscure differences across courses. To test this possibility, we compared the accuracy of complete-case analysis and multiple imputation (MI) using simulated data. We simulated the data based on prior studies such that students who earned higher grades participated at higher rates, which made the data missing at random. PER studies seldom use MI, but MI uses all available data, has less stringent assumptions, and is more accurate and more statistically powerful than complete-case analysis. Results indicated that complete-case analysis introduced more bias than MI and this bias was large enough to obscure differences between student populations or between courses. We recommend that the PER community adopt the use of MI for handling missing data to improve the accuracy in research studies.

[This paper is part of the Focused Collection on Quantitative Methods in PER: A Critical Examination.] The application of social network analysis (SNA) has recently grown prevalent in science, technology, engineering, and mathematics education research. Research on classroom networks has led to greater understandings of student persistence in physics majors, changes in their career-related beliefs (e.g., physics interest), and their academic success. In this paper, we aim to provide a practitioner’s guide to carrying out research using SNA, including how to develop data collection instruments, setup protocols for gathering data, as well as identify network methodologies relevant to a wide range of research questions beyond what one might find in a typical primer. We illustrate these techniques using student anxiety data from active-learning physics classrooms. We explore the relationship between students’ physics anxiety and the social networks they participate in throughout the course of a semester. We find that students’ with greater numbers of outgoing interactions are more likely to experience decrease in anxiety even while we control for pre-anxiety, gender, and final course grade. We also explore the evolution of student networks and find that the second half of the semester is a critical period for participating in interactions associated with decreased physics anxiety. Our study further supports the benefits of dynamic group formation strategies that give students an opportunity to interact with as many peers as possible throughout a semester. To complement our guide to SNA in education research, we also provide a set of tools for other researchers to use this approach in their work—the SNA toolbox—that can be accessed on GitHub.

[This paper is part of the Focused Collection on Quantitative Methods in PER: A Critical Examination.] A central aim of physics education research is to understand the processes of learning and use that understanding to inform instruction. To this end, researchers often conduct studies to measure the effect of classroom interventions on student outcomes. Many of these intervention studies have provided an empirical foundation of reformed teaching techniques, such as active engagement. However, many times there is not sufficient evidence to conclude that the intervention had the intended effect, and these null results often end up in the proverbial file drawer. In this paper, we argue that null results can make significant contributions to physics education research, even if the results are not statistically significant. First, we review social science and biomedical research that documents widespread publication bias against null results, exploring why it occurs and how it can hurt the field. We then present three cases from physics education research to highlight how studies that yield null results can contribute to our understanding of teaching and learning. Finally, we distill from these studies some general principles for learning from null results, proposing that we should evaluate them not on whether they reject the null hypothesis but according to their potential for generating new understanding.

[This paper is part of the Focused Collection on Quantitative Methods in PER: A Critical Examination.] How data are collected and how they are analyzed is typically described in the literature, but how the data are encoded is often not described in detail. In this paper, we discuss how data typically gathered in PER are encoded and how the choice of encoding plays a role in data analysis. We describe the kinds of data that are found when using short answer, multiple choice, Likert-scale, ranking task, and free response questions in terms of nominal, ordinal, interval, and ratio data. We discuss the mathematical operations that are available for each kind of data and how this affects ways that similarity and difference between student responses can be determined, a topic we discuss in terms of measures of distances and correlation. Finally, we use several papers from the literature to discuss ways in which data have been encoded and analyzed, with examples of normalized gain, factor analysis, model analysis, cluster analysis, and the investigation of epistemological agreement. We highlight both strengths and weaknesses of the data encoding approaches used in these studies. Our goal is not a comprehensive review, but one that is illustrative and can help researchers understand their own and each other’s work more deeply.

[This paper is part of the Focused Collection on Quantitative Methods in PER: A Critical Examination.] While other fields such as statistics and education have examined various issues with quantitative work, few studies in physics education research (PER) have done so. We conducted a two-phase study to identify and to understand the extent of these issues in quantitative PER. During phase 1, we conducted a focus group of three experts in this area, followed by six interviews. Subsequent interviews refined our plan. Both the focus group and interviews revealed issues regarding the lack of details in sample descriptions, lack of institutional or course contextual information, lack of reporting on limitation, and overgeneralization or overstatement of conclusions. During phase 2, we examined 72 manuscripts that used four conceptual or attitudinal assessments (Force Concept Inventory, Conceptual Survey of Electricity and Magnetism, Brief Electricity and Magnetism Assessment, and Colorado Learning Attitudes about Science Survey). Manuscripts were coded on whether they featured various sample descriptions, institutional or course context information, limitations, and whether they overgeneralized conclusions. We also analyzed the data to see if reporting has changed from the earlier periods to more recent times. We found that not much has changed regarding sample descriptions and institutional or course context information, but reporting and overgeneralizing conclusions have improved over time. We offer some questions for researchers, reviewers, and readers in PER to consider when conducting or using quantitative work.

[This paper is part of the Focused Collection on Quantitative Methods in PER: A Critical Examination.] The notion of quantitative physics education research (PER) so far has been mostly limited to the mere use of statistical methods or use of computational tools for analyzing numerical data. Little attention, in fact, has been given to the underpinnings of this research paradigm. To fill the gap, this theoretical paper addresses key and yet often tacit (or even misunderstood) principles of three commonly used quantitative genres in PER, which I, respectively, refer to as measurement (quantification of individual constructs), controlled exploration of relations (quantification of relationships between multiple constructs through controlled experiments), and data mining (quantification of new information from large datasets). For each genre, I elucidate the paradigmatic basis by focusing on its ontological assumptions (theories about the reality under quantitative investigation), epistemological commitments (views about the knowledge gained through quantitative investigation), and methodological implications for empirical investigations. Although framed in the context of physics education research, the discussions herein are applicable to other discipline-based or general education research employing quantitative methods.

In the present theoretical study, we introduce the entropy concept into Chi’s ontological shift theory. Chi distinguishes between two categories of process phenomena, direct and emergent, and claims that incorrectly considering emergent processes as direct ones is one of the sources of students’ robust scientific misconceptions. The present study aims to address the needs of high school and undergraduate physics, chemistry, and engineering students being already familiar with the basics of mechanics and the kinetic molecular theory. Acknowledging the contribution of the ontological shift category theory to improving the learning of science, the present paper aims at taking this theory one step further. We argue that more information about scientific phenomena could be gained if we view direct and emergent phenomena as edges of the same scale level of emergency. We show that entropy can be used to evaluate the level of emergency of physical processes. We believe that interpreting scientific phenomena in terms of level of emergency and entropy might promote students’ understanding about the underlying mechanisms explaining these phenomena, as well as about the concept of entropy itself. We provide two pedagogical examples of teaching heat and sound using the level of emergency scale and the entropy concept. We demonstrate analytically in these terms: (a) the development of the heat flow rate equation; and (b) the adiabatic nature of the sound propagation process.

Proper understanding of and learning from physics phenomena and experiments requires—among other competencies—flexible and coherent use of multiple representations (MRs). These can include everything from the “enactive” or “operational” manipulation of the experimental devices and materials to the most abstract level of a mathematical formulation of the phenomenon investigated in a given experiment. An essential prerequisite for effective work with MRs is the ability to achieve coherence between different representations. However, research indicates that the level of representational coherence ability of learners across various age groups is low. In order to improve this state of affairs, an intervention study about the use of MRs related to physics experiments was carried out (content area geometrical optics). Specific learning tasks (representational activity tasks, RATs) were designed which explicitly require various types of coherent connections, such as comparing, completing, and correcting representations. In a quasiexperimental repeated measurement study ($N=302$) using a multilevel analysis for measuring changes, a comparison of a treatment group learning with RATs vs a control group learning with conventional tasks was carried out (with identical content, lesson plans, and duration of the intervention in both groups; moreover, each of the four schools had corresponding classes of both groups. They were taught by the same teacher). Results showed a highly significant and practically relevant effect on students’ representational coherence ability ($p<0.001$; $d=0.69$). The positive effect of RATs could still be found six weeks after the end of the intervention ($p<0.001$; $d=0.43$). Several covariates (gender, pre-instructional knowledge in physics, mathematics, three facets of intelligence) were analyzed, with no or small influence on these effects. Finally, some limitations and implications of the study for classroom practice and further research are discussed.

Cultivating institutional transformation has been of recent interest in education research. This theoretical paper presents six principles for supporting sustained change efforts at the department level. Considering change efforts at the level of “principles” is valuable because principles are grounded in theoretical and empirical knowledge, but are abstract enough to be adapted to many contexts. For each principle we argue for its value, drawing on previous literature in higher education, organizational change, discipline-based education research, and design thinking. We then give illustrative examples of how each principle was embodied within the Departmental Action Team (DAT) project. The DAT project facilitates the implementation of effective changes within university science, technology, engineering, and mathematics departments. We conclude with a discussion of how these principles can be applicable across a variety of institutional transformation efforts.

Models of physical systems are used to explain and predict experimental results and observations. The Modeling Framework for Experimental Physics describes the process by which physicists revise their models to account for the newly acquired observations, or change their apparatus to better represent their models when they encounter discrepancies between actual and expected behavior of a system. While modeling is a nationally recognized learning outcome for undergraduate physics lab courses, no assessments of students’ model-based reasoning exist for upper-division labs. As part of a larger effort to create two assessments of students’ modeling abilities, we used the Modeling Framework to develop and code think-aloud problem-solving activities centered on investigating an inverting amplifier circuit. This study is the second phase of a multiphase assessment instrument development process. Here, we focus on characterizing the range of modeling pathways students employ while interpreting the output signal of a circuit functioning far outside its recommended operation range. We end by discussing four outcomes of this work: (i) Students engaged in all modeling subtasks, and they spent the most time making measurements, making comparisons, and enacting revisions; (ii) each subtask occurred in close temporal proximity to all other subtasks; (iii) sometimes, students propose causes that do not follow from observed discrepancies; (iv) similarly, students often rely on their experiential knowledge and enact revisions that do not follow from articulated proposed causes.