- Physical Review Journals
All JournalsPhysics Magazine

Physical Review Physics Education Research

Reuse & Permissions

It is not necessary to obtain permission to reuse this article or its components as it is available under the terms of the Creative Commons Attribution 4.0 International license. This license permits unrestricted use, distribution, and reproduction in any medium, provided attribution to the author(s) and the published article's title, journal citation, and DOI are maintained. Please note that some figures may have been included with permission from other third parties. It is your responsibility to obtain the proper permission from the rights holder directly for these figures.

Export citation

Export citation

Choose format for download:

Download Citation
  • Open Access

Multidimensional item response theory and the Brief Electricity and Magnetism Assessment

John Hansen and John Stewart*

  • Department of Physics and Astronomy, West Virginia University, Morgantown, West Virginia 26506, USA

  • *jcstewart1@mail.wvu.edu
PDF

Phys. Rev. Phys. Educ. Res. 17, 020139 – Published 30 November, 2021

DOI: https://doi.org/10.1103/PhysRevPhysEducRes.17.020139

Abstract

This work is the fourth of a series of papers applying multidimensional item response theory (MIRT) to widely used physics conceptual assessments. This study applies MIRT analysis using both exploratory and confirmatory methods to the Brief Electricity and Magnetism Assessment (BEMA) to explore the assessment’s structure and to determine a well-fitting model of student knowledge measured by the assessment. These methods were used to investigate a large dataset (N=9666) from a research university in the United States. Exploratory analysis showed that a five-factor model had the best fit statistics; the items with the highest loadings in four of the five factors were items in the same item block. Confirmatory MIRT analysis fit a theoretical model developed from expert solutions to the instrument and identified two models with superior model fit: a principle model and a topical model. The principle model consisted of 28 principles, fundamental reasoning steps needed to solve items in the instrument; this was more principles than any of the models in the previous confirmatory MIRT studies of the Force Concept Inventory, the Force and Motion Conceptual Evaluation, and the Conceptual Survey of Electricity and Magnetism. A second model, the topical model, consisted of five general subtopics of electromagnetism. Both the principle and the topical model had excellent fit statistics; however, unlike the other conceptual instruments studied, the topical model had better fit statistics. The five topical divisions were explored as possible subscales; however, none of these subscales had a Cronbach’s α of 0.7, the minimum value for required low-stakes testing.

View figure in article

Physics Subject Headings (PhySH)

Article Text

References (64)

  1. R. Chabay and B. Sherwood, Qualitative understanding and retention, AAPT Announcer 27, 96 (1997).
  2. The BEMA itself was never published in an archival journal. Early references to the instrument use Chabay and Sherwood (1997) (Ref. [1]) to cite the instrument. This issue of the AAPT Announcer is not available electronically. The citation references the program to the Summer 1997 American Association of Physics Teachers meeting then published in the Announcer. The page referenced contains Chabay and Sherwood’s contributed talk abstracts about research applying the instrument. Interestingly, Maloney, O’Kuma, Van Heuvelen, and Hieggelke discussed challenges to developing an electricity and magnetism instrument in the same session which lead to the CSEM.
  3. D. P. Maloney, T. L. O’Kuma, C. Hieggelke, and A. Van Huevelen, Surveying students’ conceptual knowledge of electricity and magnetism, Phys. Ed. Res., Am. J. Phys. 69, S12 (2001).
  4. I. A. Halloun and D. Hestenes, The initial knowledge state of college physics students, Am. J. Phys. 53, 1043 (1985).
  5. D. Hestenes, M. Wells, and G. Swackhamer, Force Concept Inventory, Phys. Teach. 30, 141 (1992).
  6. R. R. Hake, Interactive-engagement versus traditional methods: A six-thousand-student survey of mechanics test data for introductory physics courses, Am. J. Phys. 66, 64 (1998).
  7. R. K. Thornton and D. R. Sokoloff, Assessing student learning of Newton’s laws: The Force and Motion Conceptual Evaluation and the evaluation of active learning laboratory and lecture curricula, Am. J. Phys. 66, 338 (1998).
  8. J. L. Docktor and J. P. Mestre, Synthesis of discipline-based education research in physics, Phys. Rev. Phys. Educ. Res. 10, 020119 (2014).
  9. Physport, https://www.physport.org. Accessed 8/8/2017.
  10. S. DeVore, J. Stewart, and G. Stewart, Examining the effects of testwiseness in conceptual physics evaluations, Phys. Rev. Phys. Educ. Res. 12, 020138 (2016).
  11. J. Stewart, C. Zabriskie, S. DeVore, and G. Stewart, Multidimensional item response theory and the Force Concept Inventory, Phys. Rev. Phys. Educ. Res. 14, 010137 (2018).
  12. C. Zabriskie and J. Stewart, Multidimensional item response theory and the Conceptual Survey of Electricity and Magnetism, Phys. Rev. Phys. Educ. Res. 15, 020107 (2019).
  13. J. Yang, C. Zabriskie, and J. Stewart, Multidimensional item response theory and the Force and Motion Conceptual Evaluation, Phys. Rev. Phys. Educ. Res. 15, 020141 (2019).
  14. M. Planinic, L. Ivanjek, and A. Susac, Rasch model based analysis of the Force Concept Inventory, Phys. Rev. Phys. Educ. Res. 6, 010103 (2010).
  15. L. Ding, Applying Rasch theory to evaluate the construct validity of brief electricity and magnetism assessment, AIP Conf. Proc. 1413, 175 (2012).
  16. L. Ding, Seeking missing pieces in science concept assessments: Reevaluating the Brief Electricity and Magnetism Assessment through Rasch analysis, Phys. Rev. Phys. Educ. Res. 10, 010105 (2014).
  17. A. Traxler, R. Henderson, J. Stewart, G. Stewart, A. Papak, and R. Lindell, Gender fairness within the Force Concept Inventory, Phys. Rev. Phys. Educ. Res. 14, 010103 (2018).
  18. R. Henderson, P. Miller, J. Stewart, A. Traxler, and R. Lindell, Item-level gender fairness in the Force and Motion Conceptual Evaluation and the Conceptual Survey of Electricity and Magnetism, Phys. Rev. Phys. Educ. Res. 14, 020103 (2018).
  19. Y. Xiao, J. C. Fritchman, J. Y. Bao, Y. Nie, J. Han, J. Xiong, H. Xiao, and L. Bao, Linking and comparing short and full-length concept inventories of electricity and magnetism using item response theory, Phys. Rev. Phys. Educ. Res. 15, 020149 (2019).
  20. P. Eaton, K. Johnson, B. Frank, and S. Willoughby, Classical test theory and item response theory comparison of the Brief Electricity and Magnetism Assessment and the Conceptual Survey of Electricity and Magnetism, Phys. Rev. Phys. Educ. Res. 15, 010102 (2019).
  21. J. Wang and L. Bao, Analyzing Force Concept Inventory with item response theory, Am. J. Phys. 78, 1064 (2010).
  22. S. Youngsuk and B. Daniel, Nested logit models for multiple-choice item response data, Psychometrika 75, 454 (2010).
  23. J. Stewart, B. Drury, J. Wells, A. Adair, R. Henderson, Y. Ma, A. Pérez-Lemonche, and D. Pritchard, Examining the relation of correct knowledge and misconceptions using the nominal response model, Phys. Rev. Phys. Educ. Res. 17, 010122 (2021).
  24. T. F. Scott and D. Schumayer, Students’ proficiency scores within multitrait item response theory, Phys. Rev. Phys. Educ. Res. 11, 020134 (2015).
  25. J. Wells, R. Henderson, J. Stewart, G. Stewart, J. Yang, and A. Traxler, Exploring the structure of misconceptions in the Force Concept Inventory with modified module analysis, Phys. Rev. Phys. Educ. Res. 15, 020122 (2019).
  26. J. Wells, R. Henderson, A. Traxler, P. Miller, and J. Stewart, Exploring the structure of misconceptions in the Force and Motion Conceptual Evaluation with modified module analysis, Phys. Rev. Phys. Educ. Res. 16, 010121 (2020).
  27. C. Wheatley, J. Wells, R. Henderson, and J. Stewart, Applying module analysis to the Conceptual Survey of Electricity and Magnetism, Phys. Rev. Phys. Educ. Res. 17, 010102 (2021).
  28. S. J. Pollock, Longitudinal study of student conceptual understanding in electricity and magnetism, Phys. Rev. Phys. Educ. Res. 5, 020110 (2009).
  29. M. A. Kohlmyer, M. D. Caballero, R. Catrambone, R. W. Chabay, L. Ding, M. P. Haugan, M. J. Marr, B. D. Sherwood, and M. F. Schatz, Tale of two curricula: The performance of 2000 students in introductory electromagnetism, Phys. Rev. Phys. Educ. Res. 5, 020105 (2009).
  30. M. W. McColgan, R. A. Finn, D. L. Broder, and G. E. Hassel, Assessing students’ conceptual knowledge of electricity and magnetism, Phys. Rev. Phys. Educ. Res. 13, 020121 (2017).
  31. L. Ding, R. Chabay, B. Sherwood, and R. Beichner, Evaluating an electricity and magnetism assessment tool: Brief Electricity and Magnetism Assessment, Phys. Rev. Phys. Educ. Res. 2, 010105 (2006).
  32. L. Ding and R. Beichner, Approaches to data analysis of multiple-choice questions, Phys. Rev. Phys. Educ. Res. 5, 020103 (2009).
  33. R. W. Chabay and B. A. Sherwood, Matter and Interactions (John Wiley & Sons, Hoboken, NJ, 2015).
  34. S. J. Pollock, Comparing student learning with multiple research-based conceptual surveys: CSEM and BEMA, AIP Conf. Proc. 1064, 171 (2008).
  35. P. Eaton, B. Frank, K. Johnson, and S. Willoughby, Comparing exploratory factor models of the Brief Electricity and Magnetism Assessment and the Conceptual Survey of Electricity and Magnetism, Phys. Rev. Phys. Educ. Res. 15, 020133 (2019).
  36. A. Newell and H. A. Simon, Human Problem Solving (Prentice-Hall, Englewood Cliffs, NJ, 1972).
  37. S. Ohlsson, The problems with problem solving: Reflections on the rise, current status, and possible future of a cognitive research paradigm, J. Prob. Solving 5, 7 (2012).
  38. J. Larkin, J. McDermott, D. P. Simon, and H. A. Simon, Expert and novice performance in solving physics problems, Science 208, 1335 (1980).
  39. J. H. Larkin, J. McDermott, D. P. Simon, and H. A. Simon, Models of competence in solving physics problems, Cogn. Sci. 4, 317 (1980).
  40. F. Reif and J. I. Heller, Knowledge structure and problem solving in physics, Educ. Psychol. 17, 102 (1982).
  41. U.S. News & World Report: Education, https://premium.usnews.com/best-colleges. Accessed 4/30/2017.
  42. L. Chen, J. Han, J. Wang, and Y. Tu, Comparisons of Item Response Theory algorithms on Force Concept Inventory, Res. Edu. As. Learn. 2, 26 (2011).
  43. S. Osborn Popp, D. Meltzer, and M. C. Megowan-Romanowicz, Is the Force Concept Inventory biased? Investigating differential item functioning on a test of conceptual learning in physics, in 2011 American Educational Research Association Conference (American Education Research Association, Washington, DC, 2011).
  44. J. Han, L. Bao, L. Chen, T. Cai, Y. Pi, S. Zhou, Y. Tu, and K. Koenig, Dividing the Force Concept Inventory into two equivalent half-length tests, Phys. Rev. Phys. Educ. Res. 11, 010112 (2015).
  45. P. Eaton, K. Johnson, and S. Willoughby, Generating a growth-oriented partial credit grading model for the Force Concept Inventory, Phys. Rev. Phys. Educ. Res. 15, 020151 (2019).
  46. P. Eaton and S. Willoughby, Identifying a preinstruction to postinstruction factor model for the Force Concept Inventory within a multitrait item response theory framework, Phys. Rev. Phys. Educ. Res. 16, 010106 (2020).
  47. T. I. Smith, K. J. Louis, B. J. Ricci IV, and N. Bendjilali, Quantitatively ranking incorrect responses to multiple-choice questions using item response theory, Phys. Rev. Phys. Educ. Res. 16, 010107 (2020).
  48. T. F. Scott, D. Schumayer, and A. R. Gray, Exploratory factor analysis of a Force Concept Inventory data set, Phys. Rev. Phys. Educ. Res. 8, 020105 (2012).
  49. L. J. Cronbach and P. E. Meehl, Construct validity in psychological tests, Psychol. Bull. 52, 281 (1955).
  50. L. A. Clark and D. Watson, Constructing validity: Basic issues in objective scale development, Psychol. Assess. 7, 309 (1995).
  51. R. P. Chalmers, mirt: A multidimensional item response theory package for the R environment, J. Stat. Softw. 48, 1 (2012).
  52. R Core Team, R: A Language and Environment for Statistical Computing (R Foundation for Statistical Computing, Vienna, Austria 2017).
  53. L. Cai, A two-tier full-information item factor analysis model with applications, Psychometrika 75, 581 (2010).
  54. A. E. Raftery, Bayesian model selection in social research, Sociol. Methodol. 25, 111 (1995).
  55. A. Maydeu-Olivares and H. Joe, Limited information goodness-of-fit testing in multidimensional contingency tables, Psychometrika 71, 713 (2006).
  56. A. Maydeu-Olivares, Goodness-of-fit assessment of item response theory models, Measurement 11, 71 (2013).
  57. R. E. Schumacker and R. G. Lomax, A Beginner’s Guide to Structural Equation (Routledge, New York, 2016).
  58. R. B. Kline, Principles and Practices of Structural Equation Modeling, 4th edition (Guilford Publications, New York, 2016).
  59. L. Hu and P. M. Bentler, Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives, Struct. Equ. Modeling 6, 1 (1999).
  60. L. Crocker and J. Algina, Introduction to Classical and Modern Test Theory (Holt, Rinehart, and Winston, Mason, OH, 1986).
  61. D. Hestenes, M. Wells, and G. Swackhamer, Force Concept Inventory, Phys. Teach. 30, 141 (1992).
  62. J. Yang, J. Wells, R. Henderson, E. Christman, G. Stewart, and J. Stewart, Extending modified module analysis to include correct responses: Analysis of the Force Concept Inventory, Phys. Rev. Phys. Educ. Res. 16, 010124 (2020).
  63. R. C. Hudson and F. Munley, Re-score the Force Concept Inventory!, Phys. Teach. 34, 261 (1996).
  64. R. K. Thornton, D. Kuhl, K. Cummings, and J. Marx, Comparing the Force and Motion Conceptual Evaluation and the Force Concept Inventory, Phys. Rev. Phys. Educ. Res. 5, 010105 (2009).

Outline

Information

Phys. Rev. Phys. Educ. Res. 17, 020139– Published 30 November, 2021
  • Received 4 August 2021
  • Accepted 2 November 2021
Crossmark - Check for updates button
Creative Commons Logo

Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article’s title, journal citation, and DOI.

Published by the American Physical Society

Sign In to Your Journals Account

Filter

Filter

Article Lookup

Enter a citation