- Physical Review Journals
All JournalsPhysics Magazine

Physical Review Physics Education Research

Reuse & Permissions

It is not necessary to obtain permission to reuse this article or its components as it is available under the terms of the Creative Commons Attribution 4.0 International license. This license permits unrestricted use, distribution, and reproduction in any medium, provided attribution to the author(s) and the published article's title, journal citation, and DOI are maintained. Please note that some figures may have been included with permission from other third parties. It is your responsibility to obtain the proper permission from the rights holder directly for these figures.

Export citation

Export citation

Choose format for download:

Download Citation
  • Open Access

Measurement invariance across race and gender for the Force Concept Inventory

Alicen Morley1, Jayson M. Nissen2, and Ben Van Dusen1

  • 1School of Education, Iowa State University, Ames, Iowa 50011, USA
  • 2Nissen Education and Research Design, Slidell, Louisiana 70458, USA

PDF

Phys. Rev. Phys. Educ. Res. 19, 020102 – Published 6 July, 2023

DOI: https://doi.org/10.1103/PhysRevPhysEducRes.19.020102

Abstract

Instructors and researchers often use research-based assessments to identify the impact of instructional activities. These investigations often focus on issues of diversity, equity, and inclusions by comparing outcomes across social identity groups (e.g., gender, race, and class). Comparisons across groups assume the assessments measure the same factors in the same way across social identity groups. Very few research-based assessments, however, have validation evidence to support this assumption. Measurement invariance testing provides validity evidence that an assessment measures latent factors (e.g., Newtonian thinking or physics identity) equivalently across groups. We examined the measurement invariance of the Eaton and Willoughby five (EW5) factor model on the Force Concept Inventory. We found evidence for measurement invariance across the intersections of gender (men and women) and race or ethnicity (Asian, Black, Hispanic, White, and White Hispanic). These results indicate that performance differences across the five factors can further understanding of equity in physics courses. Without measurement invariance, such work could produce misleading results that undermine efforts to support equity in physics courses.

View figure in article

Physics Subject Headings (PhySH)

Article Text

References (90)

  1. S. Singer and K. A. Smith, Discipline-based education research: Understanding and improving learning in undergraduate science and engineering, J. Eng. Educ. 102, 468 (2013).
  2. J. L. Docktor and J. P. Mestre, Synthesis of discipline-based education research in physics, Phys. Rev. ST Phys. Educ. Res. 10, 020119 (2014).
  3. A. Madsen, S. B. McKagan, and E. C. Sayre, Resource letter RBAI-1: Research-based assessment instruments in physics and astronomy, Am. J. Phys. 85, 245 (2017).
  4. B. Van Dusen and J. Nissen, Equity in college physics student learning: A critical quantitative intersectionality investigation, J. Res. Sci. Teach. 57, 33 (2020).
  5. E. Brewe, V. Sawtelle, L. H. Kramer, G. E. O’Brien, I. Rodriguez, and P. Pamelá, Toward equity through participation in modeling instruction in introductory university physics, Phys. Rev. ST Phys. Educ. Res. 6, 010106 (2010).
  6. I. Rodriguez, E. Brewe, V. Sawtelle, and L. H. Kramer, Impact of equity models and statistical measures on interpretations of educational reform, Phys. Rev. ST Phys. Educ. Res. 8, 020103 (2012).
  7. P. Martinková, A. Drabinová, Y.-L. Liaw, E. A. Sanders, J. L. McFarland, and R. M. Price, Checking equity: Why differential item functioning analysis should be a routine part of developing conceptual assessments, CBE Life Sci. Educ. 16, rm2 (2017).
  8. A. Traxler, R. Henderson, J. Stewart, G. Stewart, A. Papak, and R. Lindell, Gender fairness within the force concept inventory, Phys. Rev. Phys. Educ. Res. 14, 010103 (2018).
  9. A. Traxler and E. Brewe, Equity investigation of attitudinal shifts in introductory physics, Phys. Rev. ST Phys. Educ. Res. 11, 020132 (2015).
  10. Z. Y. Kalender, E. Marshman, C. D. Schunn, T. J. Nokes-Malach, and C. Singh, Gendered patterns in the construction of physics identity from motivational factors, Phys. Rev. Phys. Educ. Res. 15, 020119 (2019).
  11. Z. Hazari, G. Sonnert, P. M. Sadler, and M.-C. Shanahan, Connecting high school physics experiences, outcome expectations, physics identity, and physics career choice: A gender study, J. Res. Sci. Teach. 47, 978 (2010).
  12. T. I. Smith and M. C. Wittmann, Comparing three methods for teaching Newton’s third law, Phys. Rev. ST Phys. Educ. Res. 3, 020105 (2007).
  13. T. I. Smith, M. C. Wittmann, and T. Carter, Applying model analysis to a resource-based analysis of the force and motion conceptual evaluation, Phys. Rev. ST Phys. Educ. Res. 10, 020102 (2014).
  14. D. Hestenes, M. Wells, and G. Swackhamer, Force concept inventory, Phys. Teach. 30, 141 (1992).
  15. B. Van Dusen and J. M. Nissen, Associations between learning assistants, passing introductory physics, and equity: A quantcrit investigation, Phys. Rev. Phys. Educ. Res. 16, 010117 (2020).
  16. B. Van Dusen and J. Nissen, Equity in college physics student learning: A critical quantitative intersectionality investigation, J. Res. Sci. Teach. 57, 33 (2019).
  17. J. M. Nissen, I. H. M. Horses, and B. Van Dusen, Investigating society’s educational debts due to racism and sexism in student attitudes about physics using quantitative critical race theory, Phys. Rev. Phys. Educ. Res. 17, 010116 (2021).
  18. R. Krakehl and A. M. Kelly, Intersectional analysis of advanced placement physics participation and performance by gender and ethnicity, Phys. Rev. Phys. Educ. Res. 17, 020105 (2021).
  19. D. Shafer, M. S. Mahmood, and T. Stelzer, Impact of broad categorization on statistical results: How underrepresented minority designation can mask the struggles of both Asian American and African American students, Phys. Rev. Phys. Educ. Res. 17, 010113 (2021).
  20. C. A. Paul and D. J. Webb, Percent grade scale amplifies racial or ethnic inequities in introductory physics, Phys. Rev. Phys. Educ. Res. 18, 020103 (2022).
  21. J. Yang, S. DeVore, D. Hewagallage, P. Miller, Q. X. Ryan, and J. Stewart, Using machine learning to identify the most at-risk students in physics classes, Phys. Rev. Phys. Educ. Res. 16, 020130 (2020).
  22. J. Stewart, G. L. Cochran, R. Henderson, C. Zabriskie, S. DeVore, P. Miller, G. Stewart, and L. Michaluk, Mediational effect of prior preparation on performance differences of students underrepresented in physics, Phys. Rev. Phys. Educ. Res. 17, 010107 (2021).
  23. A. B. Simmons and A. F. Heckler, Grades, grade component weighting, and demographic disparities in introductory physics, Phys. Rev. Phys. Educ. Res. 16, 020125 (2020).
  24. Kimberlé Crenshaw, Mapping the margins: Identity politics, intersectionality, and violence against women of color, Stanf. Law Rev. 43, 1241 (1991), https://www.jstor.org/stable/1229039.
  25. P. H. Collins and S. Bilge, Intersectionality (John Wiley & Sons, 2020).
  26. D. Shafer, M. S. Mahmood, and T. Stelzer, Impact of broad categorization on statistical results: How underrepresented minority designation can mask the struggles of both Asian American and African American students, Phys. Rev. Phys. Educ. Res. 17, 010113 (2021).
  27. Gloria Ladson-Billings, From the achievement gap to the education debt: Understanding achievement in US schools, Educ. Res. 35, 3 (2006).
  28. J. B Buncher, J. M. Nissen, B. Van Dusen, R. M. Talbot III, and H. Huvard, Bias on the force concept inventory across the intersection of gender and race, presented at PER Conf. 2021, virtual conference, 10.1119/perc.2021.pr.Buncher.
  29. M. T. Kane, Current concerns in validity theory, J. Educ. Measure. 38, 319 (2001).
  30. D. B Flora, Your coefficient alpha is probably wrong, but which coefficient omega is right? A tutorial on using r to obtain better reliability estimates, Adv. Methods Pract. Psychol. Sci. 3, 484 (2020).
  31. P. Eaton and S. D. Willoughby, Confirmatory factor analysis applied to the force concept inventory, Phys. Rev. Phys. Educ. Res. 14, 010124 (2018).
  32. L. Bowleg, When Black + lesbian + woman ≠ Black lesbian woman: The methodological challenges of qualitative and quantitative intersectionality research, Sex Roles 59, 312 (2008).
  33. G. A. Rocabado, N. A. Kilpatrick, S. R. Mooring, and J. E. Lewis, Can we compare attitude scores among diverse populations? An exploration of measurement invariance testing to support valid comparisons between Black female students and their peers in an organic chemistry course, J. Chem. Educ. 96, 2371 (2019).
  34. A. Kuhlemeier, Measurement invariance of psychological distress, substance use, and adult social support across race/ethnicity and sex among sexual minority youth, J. Sex Res. 60, 674 (2023).
  35. N. M. Else-Quest and J. S. Hyde, Intersectionality in quantitative psychological research: II. Methods and techniques, Psychol. Women Q. 40, 319 (2016).
  36. P. H. Collins, Intersectionality’s definitional dilemmas, Annu. Rev. Sociol. 41, 1 (2015).
  37. S. Cho, K. W. Crenshaw, and L. McCall, Toward a field of intersectionality studies: Theory, J. Women Cult. Soc. 38, 785 (2013).
  38. D. L. Putnick and M. H. Bornstein, Measurement invariance conventions and reporting: The state of the art and future directions for psychological research, Dev. Rev. 41, 71 (2016).
  39. Y. Dodge, D. Cox, and D. Commenges, The Oxford Dictionary of Statistical Terms (Oxford University Press on Demand, New York, 2003).
  40. J. B. Ullman and P. M. Bentler, Structural Equation Modeling, edited by J. A. Schinka and W. F. Velicer, Handbook of Psychology: Research Methods in Psychology Vol. 2 (John Wiley & Sons, Inc., Hoboken, NJ, 2003), pp. 607–634, https://doi.org/10.1002/0471264385.wei0224.
  41. T. A. Brown and M. T. Moore, Confirmatory Factor Analysis, edited by R. H. Hoyle, Handbook of Structural Equation Modeling (Guilford Publications, New York, NY, 2012), pp. 361–379.
  42. AERA, APA, and NCME, Standards for Educational and Psychological Testing (American Educational Research Association, Washington, DC, 2014).
  43. P. Eaton and S. Willoughby, Identifying a preinstruction to postinstruction factor model for the force concept inventory within a multitrait item response theory framework, Phys. Rev. Phys. Educ. Res. 16, 010106 (2020).
  44. J. Han, L. Bao, L. Chen, T. Cai, Y. Pi, S. Zhou, Y. Tu, and K. Koenig, Dividing the force concept inventory into two equivalent half-length tests, Phys. Rev. ST Phys. Educ. Res. 11, 010112 (2015).
  45. Y. Xiao, G. Xu, J. Han, H. Xiao, J. Xiong, and L. Bao, Assessing the longitudinal measurement invariance of the force concept inventory and the conceptual survey of electricity and magnetism, Phys. Rev. Phys. Educ. Res. 16, 020103 (2020).
  46. J. Bruun and E. Brewe, Talking and learning physics: Predicting future grades from network measures and force concept inventory pretest scores, Phys. Rev. ST Phys. Educ. Res. 9, 020109 (2013).
  47. M. D. Caballero, E. F. Greco, E. R. Murray, K. R. Bujak, M. Jackson Marr, R. Catrambone, M. A. Kohlmyer, and M. F. Schatz, Comparing large lecture mechanics curricula using the force concept inventory: A five thousand student study, Am. J. Phys. 80, 638 (2012).
  48. J. M. Nissen, I. H. M. Horses, B. Van Dusen, M. Jariwala, and E. Close, Providing context for identifying effective introductory mechanics courses, Phys. Teach. 60, 179 (2022).
  49. M. Good, A. Maries, and C. Singh, Impact of traditional or evidence-based active-engagement instruction on introductory female and male students’ attitudes and approaches to physics problem solving, Phys. Rev. Phys. Educ. Res. 15, 020129 (2019).
  50. E. Brewe, L. Kramer, and G. O’Brien, Modeling instruction: Positive attitudinal shifts in introductory physics measured with class, Phys. Rev. ST Phys. Educ. Res. 5, 013102 (2009).
  51. P. Eaton, Evidence of measurement invariance across gender for the force concept inventory, Phys. Rev. Phys. Educ. Res. 17, 010130 (2021).
  52. T. F. Scott, D. Schumayer, and A. R. Gray, Exploratory factor analysis of a force concept inventory data set, Phys. Rev. ST Phys. Educ. Res. 8, 020105 (2012).
  53. J. Wang and L. Bao, Analyzing force concept inventory with item response theory, Am. J. Phys. 78, 1064 (2010).
  54. M. Planinic, L. Ivanjek, and A. Susac, Rasch model based analysis of the force concept inventory, Phys. Rev. ST Phys. Educ. Res. 6, 010103 (2010).
  55. B. Van Dusen, LASSO: A new tool to support instructors and researchers. American Physics Society Forum on Education Newsletter, 12–14 (2018).
  56. Learning Assistant Alliance, https://learningassistantalliance.org/.
  57. A. Gonzalez-Barrera and M. H. Lopez, Is being Hispanic a matter of race, ethnicity or both? (2015), https://www.pewresearch.org/short-reads/2015/06/15/is-being-hispanic-a-matter-of-race-ethnicity-or-both/ (Retrieved June 2023).
  58. B. O. Muthén, Robust inference using weighted least squares and quadratic estimating equations in latent variable modeling with categorical and continuous outcomes, Psychometrika (to be published).
  59. N. K. Bowen and R. D. Masa, Conducting measurement invariance tests with ordinal data: A guide for social work researchers, J. Soc. Soc. Work Res. 6, 229 (2015).
  60. G. Hirschfeld and R. von Brachel, Improving multiple-group confirmatory factor analysis in r–a tutorial in measurement invariance with continuous and ordinal indicators, Pract. Assess. Res. Eval. 19, 7 (2014).
  61. D. Svetina, L. Rutkowski, and D. Rutkowski, Multiple-group invariance with categorical outcomes using updated guidelines: An illustration using m plus and the lavaan/semtools packages, Struct. Equ. Model. 27, 111 (2020).
  62. R. J. Vadenberg and C. E. Lance, A review and synthesis of the measurement in variance literature: Suggestions, practices, and recommendations for organizational research, Organ. Res. Meth. 3, 4 (2000).
  63. Y. Rosseel, lavaan: An R package for structural equation modeling, J. Stat. Softw. 48, 1 (2012).
  64. J. F Hair, Jr., W. C. Black, B. J. Babin, R. E. Anderson, and R. L. Tatham, Data Analysis Multivariate (New Jersey. Upper Saddle River. Pearson Education, 2006).
  65. T. D. Jorgensen, S. Pornprasertmanit, A. M. Schoemann, and Y. Rosseel, semTools: Useful tools for structural equation modeling (2022), r package version 0.5–6.
  66. A. Madsen, S. B. McKagan, and E. C. Sayre, Gender gap on concept inventories in physics: What is consistent, what is inconsistent, and what factors influence the gap?, Phys. Rev. ST Phys. Educ. Res. 9, 020121 (2013).
  67. V. Sawtelle, E. Brewe, and L. H. Kramer, Exploring the relationship between self-efficacy and retention in introductory physics, J. Res. Sci. Teach. 49, 1096 (2012).
  68. L. E. Kost, S. J. Pollock, and N. D. Finkelstein, Characterizing the gender gap in introductory physics, Phys. Rev. ST Phys. Educ. Res. 5, 010101 (2009).
  69. L. E. Kost-Smith, S. J. Pollock, and N. D. Finkelstein, Gender disparities in second-semester college physics: The incremental effects of a “smog of bias”, Phys. Rev. ST Phys. Educ. Res. 6, 020112 (2010).
  70. S. Kanim and X. C. Cid, Demographics of physics education research, Phys. Rev. Phys. Educ. Res. 16, 020106 (2020).
  71. J. M. Nissen, I. H. M. Horses, B. V. Dusen, M. Jariwala, and E. W. Close, Tools for identifying courses that support development of expertlike physics attitudes, Phys. Rev. Phys. Educ. Res. 17, 013103 (2021).
  72. P. Barrett, Structural equation modelling: Adjudging model fit, Pers. Individ. Differ. 42, 815 (2007).
  73. R. B. Kline, Principles and Practice of Structural Equation Modeling (Guilford Publications, New York, 2015).
  74. D. Hooper, J. Coughlan, and M. R. Mullen, Structural equation modelling: Guidelines for determining model fit, Electron. J. Bus. Res. Methods 6, 53 (2008).
  75. L.-t. Hu and P. M. Bentler, Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives, Struct. Equ. Model. 6, 1 (1999).
  76. C. N. McIntosh, Rethinking fit assessment in structural equation modelling: A commentary and elaboration on Barrett (2007), Pers. Individ. Differ. 42, 859 (2007).
  77. D. A. Kenny and D. B. McCoach, Effect of the number of variables on measures of fit in structural equation modeling, Struct. Equ. Model. 10, 333 (2003).
  78. B. M. Byrne, Structural Equation Modeling with LISREL, PRELIS, and SIMPLIS: Basic Concepts, Applications, and Programming (Psychology Press, New York, 2013).
  79. A. Diamantopoulos, J. A. Siguaw, and J. A. Siguaw, Introducing LISREL: A Guide for the Uninitiated (Sage, London, 2000).
  80. Y. Xia and Y. Yang, RMSEA, CFI, and TLI in structural equation modeling with ordered categorical data: The story they tell depends on the estimation methods, Behav. Res. Meth. Instrum. Comput. 51, 409 (2019).
  81. R. C MacCallum, M. W. Browne, and H. M. Sugawara, Power analysis and determination of sample size for covariance structure modeling, Psychol. Methods 1, 130 (1996).
  82. B. G. Tabachnick, L. S. Fidell, and J. B. Ullman, Using Multivariate Statistics (Pearson, Boston, MA, 2007), Vol. 5.
  83. P. M. Bentler, Comparative fit indexes in structural models, Psychol. Bull. 107, 238 (1990).
  84. S. Cangur and I. Ercan, Comparison of model fit indices used in structural equation modeling under multivariate normality, J. Mod. Appl. Stat. Methods 14, 14 (2015).
  85. J. T Newsom, Structural models for binary repeated measures: Linking modern longitudinal structural equation models to conventional categorical data analysis for matched pairs, Struct. Equ. Model. 24, 626 (2017).
  86. M. Rhemtulla, P. É. Brosseau-Liard, and V. Savalei, When can categorical variables be treated as continuous? A comparison of robust continuous and categorical SEM estimation methods under suboptimal conditions, Psychol. Methods 17, 354 (2012).
  87. C.-H. Li, Confirmatory factor analysis with ordinal data: Comparing robust maximum likelihood and diagonally weighted least squares, Behav. Res. Meth. Instrum. Comput. 48, 936 (2016).
  88. G. W. Cheung and R. B. Rensvold, Evaluating goodness-of-fit indexes for testing measurement invariance, Struct. Equ. Model. 9, 233 (2002).
  89. L. Rutkowski and D. Svetina, Assessing the hypothesis of measurement invariance in the context of large-scale international surveys, Educ. Psychol. Meas. 74, 31 (2014).
  90. F. F. Chen, Sensitivity of goodness of fit indexes to lack of measurement invariance, Struct. Equ. Model. 14, 464 (2007).

Outline

Information

Phys. Rev. Phys. Educ. Res. 19, 020102– Published 6 July, 2023
  • Received 5 November 2022
  • Accepted 8 June 2023
Crossmark - Check for updates button
Creative Commons Logo

Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article’s title, journal citation, and DOI.

Published by the American Physical Society

Sign In to Your Journals Account

Filter

Filter

Article Lookup

Enter a citation