Educators and researchers often use research-based assessments before and after instruction to measure the efficacy of courses. Limited resources exist for interpreting assessment results, particularly for attitudinal surveys. We present analyses and representations created with data from 192 introductory physics courses that educators and researchers can use to provide a context for interpreting results from the Colorado Learning Attitudes About Science Survey. The provided data came from the online Learning About STEM Student Outcomes platform and from the scientific literature. The representations include scatter plots of pretest and post-test scores and distributions of effect sizes. Educators and researchers can use these representations to show how courses compare to the larger database before and after instruction. Results almost always associated physics courses for nonscience majors with shifts to more expertlike attitudes, whereas this was the case for approximately one-third of introductory physics courses for science majors.

High attrition rates in science, technology, engineering, and mathematics disciplines are an ongoing problem. Graduate student attrition, in particular, is understudied. Most past studies have focused on students’ attributes, undergraduate preparation, and mentoring relationships. Emerging results from the implementation of the American Physical Society Bridge Program (APSBP) suggest that departmental support structures could lead to increased retention. However, there are no validated instruments to measure students’ experience of the departmental support structures. This paper describes the development and psychometric evaluation of the aspects of student experience scale (ASES). Items were developed based on prior literature and the APSBP recommendations, revised based on APSBP feedback, and subjected to psychometric evaluation. Principal components analysis of data from 397 students at 19 physics graduate programs (2 M.S. and 17 Ph.D.) across the U.S. resulted in four distinct factors: (a) mentoring and research experience, (b) professional development, (c) social and academic integration, and (d) financial support. ASES meets the standard criteria for divergent validity, discriminant validity, and internal consistency. Results of students’ ASES response ratings are discussed, along with comparisons between students enrolled at APSBP and non-APSBP affiliated graduate programs. On average, we find that physics graduate students experience adequate mentoring and financial support; however, they report experiencing limited social and academic integration and a lack of professional development. We also find that students in APSBP affiliated programs report better experiences on all four factors than students in non-APSBP affiliated programs, with higher differences in social and academic integration and financial support. This study is a starting point for the refinement process of this instrument.

Existing research identifying common student ideas about forces focuses on students’ misunderstandings, misconceptions, and difficulties. In this paper, we characterize student thinking in terms of resources, framing student thinking as continuous with formal physics. Based on our analysis of 2048 written responses to conceptual questions, we identify six common conceptual resources for understanding forces. We document context-sensitive patterns in resource activation, and we discuss limitations of our research based on the demographics of our sample.

This study reports an analysis of the Force Concept Inventory (FCI) using item response curves (IRC)—the fraction of students selecting each response to an item as a function of their total score. Three large samples ($N=9606$, 4360, and 1439) of calculus-based physics students were analyzed. These were drawn from three land-grant institutions with very different average FCI post-test scores. A significant number of the response curves for incorrect responses have a maximum at some intermediate value of the total score on the FCI. To analyze the incorrect responses (rather than just the correct response as is typically done), we used Bock’s nominal response model (NRM), primarily because it can fit these intermediate maximum curves. The one-dimensional NRM latent ability dimension was strongly correlated ($r=0.99$) with the latent ability determined by two-parameter logistic (2PL) item response theory applied with correct or incorrect grading even though the correct responses were not identified for the NRM. To understand the relation of Newtonian and non-Newtonian thinking, higher dimensional models were fit. The two-dimensional NRM model produced one ability dimension which could be rotated to be highly correlated with the 2PL latent ability ($r\ge 0.98$) and a second dimension which was most strongly related to misconceptions involving Newton’s third law. Cluster analysis was applied to the two latent ability dimensions producing a three-cluster solution where the cluster centroids were very similar for each of the three institutions. The clusters represented three groups of students: Newtonian thinkers still retaining some misconceptions, non-Newtonian thinkers strongly applying Newton’s third law misconceptions, and non-Newtonian thinkers weakly applying those misconceptions. The differences between overall FCI scores at the three institutions could be explained by the relative populations of the three clusters.

Traditionally, scholars in physics education research have focused on students solving well-structured learning activities at the university. However, due to their constrained nature, these problems hinder collaboration and idea generation. In order to encourage student collaboration and decision making demands among undergraduate students in an introductory physics course, we utilized a real-world problem where groups were asked to compose a well-structured physics problem for younger learners. In this study we explored how they collaborated in composing physics problems. Data collection consisted of audio recording of the group discussions while they were collaborating to develop their physics problems and their respective problem solutions. Through interviews, we accessed participants’ perceptions of the task and related challenges. Results suggest that composing problems is an opportunity for participants to propose ideas and make decisions regarding the goals of the problem, concepts and procedures, contextual details, and magnitudes and units to introduce in their activities. Further, participants valued the open-ended nature of the task and recognized its benefits in utilizing physics ideas into context, which in turn enabled collaboration in a way not experienced by them with traditional well-structured problems. These findings shed light on the nature of composing physics problems in undergraduate courses, where participants assume a different role by facing a real-world activity that encourages expertise through physics-related communication and writing.

Omitted variable bias occurs in most statistical models. Whenever a confounding variable that is correlated with both dependent and independent variables is omitted from a statistical model, estimated effects of included variables are likely to be biased due to omitted variables. This issue is particularly problematic in physics education research where many research studies are quasiexperimental or observational in nature due to ethical and logistical limitations. In this paper, we illustrate the mechanisms behind omitted variable bias in explanatory modeling using authentic data and analytical solutions. We demonstrate that omitting confounding variables that are strongly correlated with included variables and have large effects on the dependent variable can significantly bias estimated effects for included variables. We also find that controlling for variables that are uncorrelated with other variables or have no effect on the dependent variable does not appreciably bias estimated effects and may or may not affect the precision of those estimates. These results suggest that removing from explanatory models variables that are not “statistically significant” can have unintended consequences on model and variable interpretations. Our results underscore the importance of carefully considering why or why not to include a variable in a model, informed by both data and theory.

The test of understanding graphs in kinematics (TUG-K) has widely been used to assess students’ understanding of this subject. The TUG-K poses different objectives to the test takers such as (1) the selection of a graph from a textual description, (2) the selection of corresponding graphs, and (3) the selection of a textual description from a graph. Whether test takers follow these task requirements is usually inferred from evaluating the test scores as correct or incorrect, yet the process of how students actually interact with the different tasks remains unknown. Recent studies have shown that eye tracking can provide rich insight into student’s interaction with multiple-choice tasks. In the current work, we analyzed the eye movement patterns of $N=115$ high school students while solving the TUG-K. Each question was divided into a question area (Q) and an option area (O), then gaze transitions between Q and O and between different options were calculated. A cluster analysis using the transition metrics revealed three item groups, containing the aforementioned objectives of the items. The clusters remain stable for different subsamples of our dataset, for instance, considering only the correct or only the incorrect responses, or considering high- or low-confidence responses. We conclude that eye movements can reflect task demands on a procedural level well beyond the classical methods of evaluating test scores, eventually making eye tracking an additional method for item analysis that can be utilized to confirm or explore test and item structures.

Previously, we had reported on the relationship between AP physics experience and exam scores, and student scores on the Force and Motion Conceptual Evaluation and in an introductory calculus-based mechanics course [E. W. Burkholder and C. E. Wieman, Phys. Rev. Phys. Educ. Res. 15, 020117 (2019)]. Two limitations to our previous work were that we aggregated data from algebra-based mechanics (AP physics 1) and calculus-based mechanics (AP physics C - Mechanics) courses, and that only 25% of the students who took AP physics reported their scores to the university; we addressed these limitations here. We found that there was no reporting bias in these scores for either AP course. We found that AP physics 1 (AP1) exam scores were predictive of students’ FMCE scores while AP physics C (APC) exam scores were not—this is consistent with our previous findings. We found that APC exam scores were more predictive of course grades in physics 1, but that students who scored a 5 on the AP1 exam also had high grades in physics 1, despite it being a calculus-based course. These results disagree with our previous findings that AP exam scores were not correlated with final exam scores after controlling for SAT or ACT math scores, likely due to the variability in course final exams from year to year. Though these results are for a single, highly selective university, they provide some key insights to educators about the value of AP courses. One important implication is that students who score a 5 on the AP1 exam can likely receive credit for the calculus-based introductory course, increasing access to higher level physics courses for students whose high schools do not have access to APC courses.

Learning about the nature of science is an important aspect of becoming part of the culture of practicing physicists. While these topics have a long-standing tradition in U.S. curricula they are currently not part of German educational standards or curricula. And while there is some research concerning primary and secondary school students’ beliefs in this area, there are hardly any data about German university students. In this study we query 100 physics students in their first to fifth semester at a German university for their views of nature of science. We then compare them to a sample of physics professors at German universities from an earlier study. The students exhibit quite adequate views in some areas but could improve on their knowledge of the variability of scientific methods and the tentativeness and limited certainty of scientific knowledge—two issues they have in common with the professors. If we interpret the sample longitudinally, we see positive development in the first but worsening in the latter area.

The COVID-19 pandemic has significantly affected the education system worldwide, which was forced to respond with a sudden shift to distance learning. While successful distance teaching requires careful thinking, planning, and the development of technological and human resources, there was no time for preparation in the current situation. Various physics courses, including lectures, tutorials, and laboratory courses, had to be transferred to online formats, resulting in a variety of simultaneous, asynchronous, and mixed activities. To investigate how physics students perceived the sudden shift to online learning, we developed a questionnaire and gathered data from $N=578$ physics students from five universities in Germany, Austria, and Croatia. In this article, we report how the problem-solving sessions (recitations) and laboratories were adapted, how students judge the different formats of the courses, and how useful and effective they perceived them. The results are correlated with the students’ self-efficacy ratings and other behavioral measures (such as self-regulated learning skills). This study is descriptive in nature, and a survey study design was implemented to examine the relationships among the variables. We found that good communication abilities ($r=0.48$, $p<0.001$) and self-organization skills ($r=0.63$, $p<0.001$) are positively correlated with perceived learning achievement. Furthermore, the previous duration of studies had a significant impact on several self-reported achievement measures, resulting in consistently lower scores of students in their first academic year compared with students who were further along academically. We draw conclusions and suggest implications for future online classes on the instructor and faculty level. Suggestions include (i) focusing on first-year courses with on-campus teaching when facing limited lecture hall capacities, (ii) offering special courses for promoting self-regulated learning skills, (iii) emphasizing the positive aspects of distance learning, and (iv) installing networking services for supporting student communication.

The American Physical Society calls on its members to improve the diversity of physics by supporting an inclusive culture that encourages women and Black, Indigenous, and people of color to become physicists. In the current educational system, it is unlikely for a student to become a physicist if they do not share the same attitudes about what it means to learn and do physics as those held by most professional physicists. Evidence shows college physics courses and degree programs do not support students in developing these attitudes. Rather physics education filters out students who do not enter college physics courses with these attitudes. To better understand the role of attitudes in the lack of diversity in physics, we investigated the intersecting relationships between racism and sexism in inequities in student attitudes about learning and doing physics using a critical quantitative framework. The analyses used hierarchical linear models to examine students’ attitudes as measured by the Colorado Learning Attitudes about Science Survey. The data came from the Learning About STEM Student Outcomes database and included 2170 students in 46 calculus-based mechanics courses and 2503 students in 49 algebra-based mechanics courses taught at 18 institutions. Like prior studies, we found that attitudes either did not change or slightly decreased for most groups. Results identified large differences across intersecting race and gender groups representing educational debts society owes these students. White students, particularly White men in calculus-based courses, tended to have more expertlike attitudes than any other group of students. Instruction that addresses society’s educational debts can help move physics toward an inclusive culture supportive of diverse students and professionals.

As a method to shorten the test time of the Force Concept Inventory (FCI), we suggest the use of computerized adaptive testing (CAT). CAT is the process of administering a test on a computer, with items (i.e., questions) selected based upon the responses of the examinee to prior items. In so doing, the test length can be significantly shortened. As a step to develop a CAT-based version of the FCI (FCI-CAT), we examined the optimal test length of the FCI-CAT such that accuracy and precision [which we measure in terms of bias, standard error, and root-mean-square error (RMSE)] of Cohen’s $d$ would be comparable to that of the full FCI for a given class size. First, we estimated the item parameters of the FCI items based on the three-parameter logistic model of item response theory, which are used in the algorithm of CAT. For this estimation, we used 2882 responses of Japanese university students. Second, we conducted a Monte Carlo simulation to analyze how the bias, standard error, and RMSE of Cohen’s $d$ depend upon the test length. Third, we conducted a post hoc simulation to examine the consistency of the Monte Carlo results with what would have been obtained using empirical responses. For this comparison, we used 86 pairs of pre- and post- test responses of Japanese university students. As a result, we found that for a class size of 40, we may reduce the test length of the FCI-CAT to 15–19 items, thereby reducing the test time of the FCI to 50%–63%, with an accompanying decrease in accuracy and precision of only 5%–10%. The results of the Monte Carlo study and the post hoc simulation were consistent, which supports the adequacy of our Monte Carlo study and its relevance in terms of administering the FCI-CAT in real classrooms.

Women continue to be underrepresented in physics in the United States. This is while many Muslim majority (MM) countries have a high representation of women in undergraduate and graduate physics programs. While there is a growing awareness of this trend, little is being done to understand why and how this trend has manifested and how it can be used to inform broadening the participation of women in physics in the U.S. To better understand how cultural experiences can influence the pursuit of physics, this study examines the lived experiences of female physics faculty members in the U.S. who came from MM countries. The study draws on seven phenomenological interviews focusing on how cultural experiences shaped participants’ gender and physics identities. The results reveal several possible hypotheses on differences and similarities in how physics and gender identities intersect in MM countries as opposed to what has been found in the West. In particular, expressions of femininity in MM countries can have a more constructive intersection with expressions of physics identity in ways that promote participation and persistence.

Although recent research has reported equity gains in the university-level physics classroom, there remains evidence of significant inequities by race. We conducted a series of multiple linear regressions to examine the magnitude of these inequities in introductory mechanics final exam scores. When we followed a common method of grouping students by underrepresented minority (URM) status in our statistical analysis, we found that when controlling for prior knowledge, based on ACT math and physics placement test scores, there was no evidence of racial inequities. However, when we performed the same analysis but replaced URM status with separate groups for each race or ethnicity reported to the university, we found that African American and Asian American students, and to a lesser extent, Hispanic students, underperformed in their physics course relative to their non-Hispanic White American peers, even when controlling for prior knowledge. We conclude that grouping students by URM status may not be sufficient to detect patterns of academic difficulties experienced by specific racial and ethnic groups of students. Furthermore, these results lead us to believe that there is still work to be done at the university level to provide a learning environment supportive of students of all backgrounds.

In two earlier studies, we developed a new method to measure students’ ability to transfer physics problem-solving skills to new contexts using a sequence of online learning modules, and implemented two interventions in the form of additional learning modules designed to improve transfer ability. The current paper introduces a new data analysis scheme that could improve the accuracy of the measurement by accounting for possible differences in students’ goal orientation and behavior, as well as revealing the possible mechanism by which one of the two interventions improves transfer ability. Based on a $2\times 2$ framework of self-regulated learning, students with a performance-avoidance oriented goal are more likely to guess on some of the assessment attempts in order to save time, resulting in an underestimation of the student populations’ transfer ability. The current analysis shows that about half of the students had frequent brief initial assessment attempts, and significantly lower correct rates on certain modules, which we think is likely to have originated at least in part from students adopting a performance-avoidance strategy. We then divided the remaining population, for which we can be certain that few students adopted a performance-avoidance strategy, based on whether they interacted with one of the intervention modules designed to develop basic problem-solving skills, or passed that module on their first attempt without interacting with the instructional material. By comparing to propensity score matched populations from a previous semester, we found that the improvement in subsequent transfer performance observed in a previous study mainly came from the latter population, suggesting that the intervention served as an effective reminder for students to activate existing skills, but fell short of developing those skills among those who have yet to master it.

Physics program for girls (PPG) is a two-week program held every summer since 2006, with the goal of engaging young women in inquiry-based physics experiments and to connect them with female science, technology, engineering, and mathematics professional role models. The program recruits young women to participate the summer after they have taken chemistry but before they enroll in their first high school physics course. Qualitative data collected over the years about PPG have shown that the program has been positively perceived by participants; 87% said they are more prepared for physics, 95% said that they think physics might be hard, but believe they can do it, and 91% said that they are positive that they will earn an A or B in their science course. However, there has not been any analysis of PPG students’ performance in their introductory physics classes. The goal of this paper is to understand this impact using chemistry and physics end of semester assessment scores for girls who participated in PPG, as well as girls who did not participate in PPG but were from the same schools and had similar demographics to the PPG participants. Girls who participated in PPG performed comparably to girls who did not participate in PPG on chemistry, but PPG girls performed better than non-PPG girls in physics. This finding persisted even after controlling for the prior year’s chemistry scores.

Recognition of interference and diffraction patterns is a difficult task for both high-school and university students. Many students fail to observe important features of particular patterns and identify the differences among similar patterns. In this study, we investigated if performing students’ investigative experiments can help high-school students in recognition of typical interference and diffraction patterns. Students in the experimental group were exposed to a teaching intervention that included five students’ investigative hands-on experiments on wave optics whereas the control group had the standard lecture-based physics teaching. We measured eye movements of students from both the experimental and control groups while they were identifying patterns produced by monochromatic light on a double slit, single slit, and diffraction grating, and by white light on a diffraction grating. Students from the experimental group had a higher percentage of correct answers than students in the control group that indicated that students’ investigative experiments had a positive effect on their recognition of interference and diffraction patterns. However, the low percentage of correct answers, even in the experimental group, confirms that distinguishing of the typical interference and diffraction patterns remains a difficult task for high-school students even if they had performed investigative hands-on experiments. Eye-tracking data showed that students from the experimental group had a shorter dwell on multiple-choice patterns, possibly because they were more familiar with interference and diffraction patterns and felt more confident in choosing the correct pattern. All students attended more to those patterns which they chose as the correct answer and that corroborates the previous findings. Overall, the results indicate that students’ recognition of interference and diffraction patterns can be improved by introducing hands-on investigative experiments in the classroom.

Despite limiting access to applicants from underrepresented racial and ethnic groups, the practice of using hard or soft Graduate Record Examination (GRE) cutoff scores in physics graduate program admissions is still a popular method for reducing the pool of applicants. The present study considers whether the undergraduate institutions of applicants have any influence on the admissions process by modeling a physics GRE cutoff score with application data from admissions offices of five universities. Two distinct approaches based on inferential and predictive modeling are conducted. While there is some disagreement regarding the relative importance between features, the two approaches largely agree that including institutional information significantly aids the analysis. Both models identify cases where the institutional effects are comparable to factors of known importance such as gender and undergraduate GPA. As the results are stable across many cutoff scores, we advocate against the practice of employing physics GRE cutoff scores in admissions.

Previous work has looked at the relationship between high school preparation and student performance in calculus-based introductory mechanics (physics 1) courses. Here, we extend that work to look at performance in introductory calculus-based electricity and magnetism (physics 2), and we look at the significance of what college math courses have been completed in addition to high school preparation. Using multiple linear regression including these measures of prior preparation, we examine the correlation between taking various math courses in college and final exam scores in introductory physics courses at a highly selective west coast university. In physics 1, we find that prior college math coursework is not a predictor of physics 1 final exam score. In physics 2, we find that having taken a course in vector calculus is a strong predictor of physics 2 exam performance (effect $\mathrm{size}=0.58$ standard deviations, $p<0.001$), even when controlling for students’ physics 1 final exam scores (effect $\mathrm{size}=0.27$ standard deviations, $p<0.01$). These effect sizes are similar in magnitude to other measures of students’ incoming physics and math preparation. Qualitative analysis of student exams from physics 2 reveal that this “vector calculus gap” is due to differences in reasoning about vectors and geometry and some differences in conceptual understanding of circuits, as vector calculus itself is not required to perform well on the final exam. That is, basic reasoning related to vector calculus appears to be important, but the formalisms of vector calculus do not.

This study examined the mediation and moderation of membership in a demographic group underrepresented in physics classes on course outcomes measured by course grades and Force and Motion Conceptual Evaluation (FMCE) post-test scores. The study used a large dataset ($N=4490$) of course grades, SAT and ACT mathematics scores (ACTM), and matched FMCE pretest and post-test scores to investigate differences by gender, underrepresented ethnic or racial minority (UERM) status, and status as a first-generation college student (FGCS). For UERM and FGCS students, ACTM and pretest scores significantly mediated the relation of membership in the demographic group and both course grade and post-test score. Differences between minority and majority members of these groups were largely removed by controlling for ACTM and pretest scores. The overwhelming majority of the effect acted through ACTM for course grade (60% and 45%, respectively), while more of the effect acted through pretest score for the post-test (36% and 48%, respectively). As such, for these groups prior preparation measures predict physics outcomes (course grades or post-test scores) differently. The mediational relations for gender were dramatically different. No mediation was detected for the relation of gender to course grade because no significant difference in course grade existed. Sixty percent of the effect of gender on post-test score was not explained by either ACTM or pretest score; pretest score accounted for 38% of the effect. As such, the majority of the difference in post-test scores between men and women was not explained by either ACTM or pretest scores. Significant moderation was also detected showing that the relation of these variables was not consistent for members of all demographic groups.