The paper introduces a set of formative assessment tasks and rubrics that were developed for use in an introductory physics instruction to help students acquire and self-assess various scientific process abilities. We will describe the rubrics, tasks, and the student outcomes in courses where the tasks and rubrics were used.

The reliability and validity of professionally written multiple-choice exams have been extensively studied for exams such as the SAT, graduate record examination, and the force concept inventory. Much of the success of these multiple-choice exams is attributed to the careful construction of each question, as well as each response. In this study, the reliability and validity of scores from multiple-choice exams written for and administered in the large introductory physics courses at the University of Illinois, Urbana-Champaign were investigated. The reliability of exam scores over the course of a semester results in approximately a 3% uncertainty in students’ total semester exam score. This semester test score uncertainty yields an uncertainty in the students’ assigned letter grade that is less than $\frac{1}{3}$ of a letter grade. To study the validity of exam scores, a subset of students were ranked independently based on their multiple-choice score, graded explanations, and student interviews. The ranking of these students based on their multiple-choice score was found to be consistent with the ranking assigned by physics instructors based on the students’ written explanations ($r>0.94$ at the 95% confidence level) and oral interviews $(r={0.94}_{-0.09}^{+0.06})$.

Previous studies have demonstrated that analogies can promote student learning in physics and can be productively taught to students to support their learning, under certain conditions. We build on these studies to explore the use of analogy by students in a large introductory college physics course. In the first large-scale study of its kind, we demonstrate that different analogies can lead to varied student reasoning. When different analogies were used to teach electromagnetic (EM) waves, we found that students explicitly mapped characteristics either of waves on strings or sound waves to EM waves, depending upon which analogy students were taught. We extend these results by investigating how students use analogies. Our findings suggest that representational format plays a key role in the use of analogy.

Recent papers document that student problem-solving competence varies (often strongly) with representational format, and that there are significant differences between the effects that traditional and reform-based instructional environments have on these competences [Kohl and Finkelstein, Phys. Rev. ST Phys. Educ. Res. 1, 010104 (2005); Kohl and Finkelstein, Phys. Rev. ST Phys. Educ. Res. 2, 010102 (2006)]. These studies focused on large-lecture introductory physics courses, and included aggregate data on student performance on quizzes and homeworks. In this paper, we complement previous papers with finer-grained in-depth problem-solving interviews. In 16 interviews of students drawn from these classes, we investigate in more detail how and when student problem-solving performance varies with problem representation (verbal, mathematical, graphical, or pictorial). We find that student strategy often varies with representation, and that in this environment students who show more strategy variation tend to perform more poorly. We also verify that student performance depends sensitively on the particular combination of representation, topic, and student prior knowledge. Finally, we confirm that students have generally robust opinions of their representational skills, and that these opinions correlate poorly with their actual performances.

The Brief Electricity and Magnetism Assessment (BEMA), developed by Chabay and Sherwood, was designed to assess student understanding of basic electricity and magnetism concepts covered in college-level calculus-based introductory physics courses. To evaluate the reliability and discriminatory power of this assessment tool, we performed statistical tests focusing both on item analyses (item difficulty index, item discrimination index, and item point biserial coefficient) and on the entire test (test reliability and Ferguson’s delta). The results indicate that BEMA is a reliable assessment tool.

This study investigates the effect of computer animation on assessment and the conditions under which animation may improve or hinder assessment of conceptual understanding in physics. An instrument was developed by replacing static pictures and descriptions of motion with computer animations on the Force Concept Inventory, a commonly used pencil and paper test. Both quantitative and qualitative data were collected. The animated and static versions of the test were given to students and the results were statistically analyzed. Think-aloud interviews were also conducted to provide additional insight into the statistical findings. We found that good verbal skills tended to increase performance on the static version but not on the animated version of the test. In general, students had a better understanding of the intent of the question when viewing an animation and gave an answer that was more indicative of their actual understanding, as reflected in separate interviews. In some situations this led students to the correct answer and in others it did not. Overall, we found that animation can improve assessment under some conditions by increasing the validity of the instrument.

Decades of education research have shown that students can simultaneously possess alternate knowledge frameworks and that the development and use of such knowledge are context dependent. As a result of extensive qualitative research, standardized multiple-choice tests such as Force Concept Inventory and Force-Motion Concept Evaluation tests provide instructors tools to probe their students’ conceptual knowledge of physics. However, many existing quantitative analysis methods often focus on a binary question of whether a student answers a question correctly or not. This greatly limits the capacity of using the standardized multiple-choice tests in assessing students’ alternative knowledge. In addition, the context dependence issue, which suggests that a student may apply the correct knowledge in some situations and revert to use alternative types of knowledge in others, is often treated as random noise in current analyses. In this paper, we present a model analysis, which applies qualitative research to establish a quantitative representation framework. With this method, students’ alternative knowledge and the probabilities for students to use such knowledge in a range of equivalent contexts can be quantitatively assessed. This provides a way to analyze research-based multiple choice questions, which can generate much richer information than what is available from score-based analysis.

In a recent study we showed that physics students’ problem-solving performance can depend strongly on problem representation, and that giving students a choice of problem representation can have a significant impact on their performance [P. B. Kohl and N. D. Finklestein, Phys. Rev. ST. Phys. Educ. Res. 1, 010104 (2005)] In this paper, we continue that study in an attempt to separate the effect of instructional technique from the effect of content area. We determine that students in a reform-style introductory physics course are learning a broader set of representational skills than those in a more traditional course. We also analyze the representations used in each course studied and find that the reformed course makes use of a richer set of representations than the traditional course and also makes more frequent use of multiple representations. We infer that this difference in instruction is the source of the broader student skills. These results provide insight into how macrolevel features of a course can influence student skills, complementary to the microlevel picture provided by the first study.

The Colorado Learning Attitudes about Science Survey (CLASS) is a new instrument designed to measure student beliefs about physics and about learning physics. This instrument extends previous work by probing additional aspects of student beliefs and by using wording suitable for students in a wide variety of physics courses. The CLASS has been validated using interviews, reliability studies, and extensive statistical analyses of responses from over 5000 students. In addition, a new methodology for determining useful and statistically robust categories of student beliefs has been developed. This paper serves as the foundation for an extensive study of how student beliefs impact and are impacted by their educational experiences. For example, this survey measures the following: that most teaching practices cause substantial drops in student scores; that a student’s likelihood of becoming a physics major correlates with their “Personal Interest” score; and that, for a majority of student populations, women’s scores in some categories, including “Personal Interest” and “Real World Connections,” are significantly different from men’s scores.

Physics education research increasingly makes use of video data for analysis of student learning and teaching practice. Collection of these data is conceptually simple but execution is often fraught with costly and time-consuming complications. This pragmatic paper discusses the development of systems to record and permanently archive audio and video data in real-time. We focus on a system based upon consumer video DVD recorders, but also give an overview of other technologies and detail issues common to all systems. We detail common yet unexpected complications, particularly with regard to sound quality and compatibility with transcription software. Information specific to fixed and transportable systems, other technology options, and generic and specific equipment recommendations are given in supplemental appendices

Student success in solving physics problems is related to the representational format of the problem. We study student representational competence in two large-lecture algebra-based introductory university physics courses with approximately 600 participants total. We examined student performance on homework problems given in four different representational formats (mathematical, pictorial, graphical, verbal), with problem statements as close to isomorphic as possible. In addition to the homeworks, we examine students’ assessment of representations by providing follow-up quizzes in which they chose between various problem formats. As a control, some parts of the classes were assigned a random-format follow-up quiz. We find that there are statistically significant performance differences between different representations of nearly isomorphic statements of quiz and homework problems. We also find that allowing students to choose which representational format they use improves student performance under some circumstances and degrades it in others. Notably, one of the two courses studied shows much greater performance differences between the groups that received a choice of format and those that did not, and we consider possible causes. Overall, we observe that student representational competence is tied to both micro- and macrolevel features of the task and environment.

This paper examines the effects of substituting a computer simulation for real laboratory equipment in the second semester of a large-scale introductory physics course. The direct current circuit laboratory was modified to compare the effects of using computer simulations with the effects of using real light bulbs, meters, and wires. Two groups of students, those who used real equipment and those who used a computer simulation that explicitly modeled electron flow, were compared in terms of their mastery of physics concepts and skills with real equipment. Students who used the simulated equipment outperformed their counterparts both on a conceptual survey of the domain and in the coordinated tasks of assembling a real circuit and describing how it worked.

We investigated the common difficulties that students have with concepts related to rotational and rolling motion covered in the introductory physics courses. We compared the performance of calculus- and algebra-based introductory physics students with physics juniors who had learned rotational and rolling motion concepts in an intermediate level mechanics course. Interviews were conducted with six physics juniors and ten introductory students using demonstration-based tasks. We also administered free-response and multiple-choice questions to a large number of students enrolled in introductory physics courses, and interviewed six additional introductory students on the test questions (during the test design phase). All students showed similar difficulties regardless of their background, and higher mathematical sophistication did not seem to help acquire a deeper understanding. We found that some difficulties were due to related difficulties with linear motion, while others were tied specifically to the more intricate nature of rotational and rolling motion.

We report a detailed study of the implementation of Tutorials in Introductory Physics at a large-scale research institution. Based on two successive semesters of evaluation, we observe students’ improved conceptual mastery (force and motion concept evaluation median normalized gain 0.77, $N=336$), albeit with some student discontent. We replicate the results of original studies of tutorial effectiveness and document how and why these results occur. Additionally, using the Colorado Learning Attitudes about Science Survey we measure the support of students’ expertlike beliefs about learning physics in our environment. We examine this implementation from a viewpoint that emphasizes varying contextual levels of this implementation, from students’ engagement in individual tasks, to the situations in which these tasks are embedded, to the broader classroom, departmental, and educational structures. We document both obvious and subtle features that help ensure the successful implementation of these reforms.