- Physical Review Journals
All JournalsPhysics Magazine

Physical Review Physics Education Research

Reuse & Permissions

It is not necessary to obtain permission to reuse this article or its components as it is available under the terms of the Creative Commons Attribution 4.0 International license. This license permits unrestricted use, distribution, and reproduction in any medium, provided attribution to the author(s) and the published article's title, journal citation, and DOI are maintained. Please note that some figures may have been included with permission from other third parties. It is your responsibility to obtain the proper permission from the rights holder directly for these figures.

Export citation

Export citation

Choose format for download:

Download Citation
  • Editors' Suggestion
  • Open Access

Critical issues in statistical causal inference for observational physics education research

Vidushi Adlakha and Eric Kuo

  • University of Illinois Urbana-Champaign, Urbana, Illinois 61801, USA

PDF

Phys. Rev. Phys. Educ. Res. 19, 020160 – Published 20 November, 2023

DOI: https://doi.org/10.1103/PhysRevPhysEducRes.19.020160

Abstract

Recent critiques of physics education research (PER) studies have revoiced the critical issues when drawing causal inferences from observational data where no intervention is present. In response to a call for a “causal reasoning primer” in PER, this paper discusses some of the fundamental issues in statistical causal inference. In reviewing these issues, we discuss well-established causal inference methods commonly applied in other fields and discuss their application to PER. Using simulated data sets, we illustrate (i) why analysis for causal inference should control for confounders but not control for mediators and colliders and (ii) that multiple proposed causal models can fit a highly correlated dataset. Finally, we discuss how these causal inference methods can be used to represent and explain existing issues in quantitative PER. Throughout, we discuss a central issue in observational studies: A good quantitative model fit for a proposed causal model is not sufficient to support that proposed model over alternative models. To address this issue, we propose an explicit role for observational studies in PER that draw statistical causal inferences: Proposing future intervention studies and predicting their outcomes. Mirroring the way that theory can motivate experiments in physics, observational studies in PER can predict the causal effects of interventions, and future intervention studies can test those predictions directly.

View figure in article

Physics Subject Headings (PhySH)

Article Text

References (82)

  1. M. B. Weissman, Policy recommendations from causal inference in physics education research, Phys. Rev. Phys. Educ. Res. 17, 020118 (2021).
  2. M. B. Weissman, Invalid methods and false answers: Physics education research and the use of GREs, Econ. J. Watch 19, 4 (2022).
  3. H. R. Varian, Causal inference in economics and marketing, Proc. Natl. Acad. Sci. U.S.A. 113, 7310 (2016).
  4. J. M. Rohrer, Thinking clearly about correlations and causation: Graphical causal models for observational data, Adv. Methods Pract. Psychol. Sci. 1, 27 (2018).
  5. E. M. Foster, Causal inference and developmental psychology, Dev. Psychol. 46, 1454 (2010).
  6. T. A. Glass, S. N. Goodman, M. A. Hernán, and J. M. Samet, Causal inference in public health, Annu. Rev. Public health 34, 61 (2013).
  7. M. Gangl, Causal inference in sociological research, Annu. Rev. Sociol. 36, 21 (2010).
  8. L. Keele, The statistics of causal inference: A view from political methodology, Political Anal. 23, 313 (2015).
  9. R. J. Murnane and J. B. Willett, Methods Matter: Improving Causal Inference in Educational and Social Science Research (Oxford University Press, New York, 2010).
  10. D. Freedman, R. Pisani, and R. Purves, Statistics (W.W. Norton & Company, New York, 2007).
  11. G. W. Imbens, Potential outcome and directed acyclic graph approaches to causality: Relevance for empirical practice in economics, J. Econ. Lit. 58, 1129 (2020).
  12. T. C. Williams, C. C. Bach, N. B. Matthiesen, T. B. Henriksen, and L. Gagliardi, Directed acyclic graphs: A tool for causal studies in paediatrics, Pediatr. Res. 84, 487 (2018).
  13. G. W. Imbens and D. B. Rubin, Causal Inference in Statistics, Social, and Biomedical Sciences (Cambridge University Press, Cambridge, England, 2015).
  14. J. Pearl and D. Mackenzie, The Book of Why: The New Science of Cause and Effect (Basic Books, New York, NY, 2018).
  15. S. Greenland, J. Pearl, and J. M. Robins, Causal diagrams for epidemiologic research, Epidemiology, 10, 37 (1999).
  16. M. Hernán and J. Robins, Causal Inference: What If (Chapman & Hall/CRC, Boca Raton, FL, 2020).
  17. M. Glymour, J. Pearl, and N. P. Jewell, Causal Inference in Statistics: A Primer (John Wiley & Sons, New York, 2016).
  18. C. Glymour, K. Zhang, and P. Spirtes, Review of causal discovery methods based on graphical models, Front. Genet. 10, 524 (2019).
  19. S. L. Morgan, Handbook of Causal Analysis for Social Research (Springer, New York, 2013).
  20. J. Peters, D. Janzing, and B. Schölkopf, Elements of Causal Inference: Foundations and Learning Algorithms (The MIT Press, Cambridge, MA, 2017).
  21. S. L. Morgan and C. Winship, Counterfactuals and Causal Inference (Cambridge University Press, Cambridge, England, 2015).
  22. J. Pearl, Linear models: A useful “microscope” for causal analysis, J. Causal Infer. 1, 155 (2013).
  23. G. Shmueli, To explain or to predict?, Stat. Sci. 25, 289 (2010).
  24. M. Kuhn, K. Johnson et al., Applied Predictive Modeling (Springer, New York, 2013), Vol. 26.
  25. E. W. Burkholder, G. Murillo-Gonzalez, and C. Wieman, Importance of math prerequisites for performance in introductory physics, Phys. Rev. Phys. Educ. Res. 17, 010108 (2021).
  26. S. McCammon, J. Golden, and K. L. Wuensch, Predicting course performance in freshman and sophomore physics courses: Women are more predictable than men, J. Res. Sci. Teach. 25, 501 (1988).
  27. M. Verostek, C. W. Miller, and B. Zwickl, Analyzing admissions metrics as predictors of graduate GPS and whether graduate GPA mediates Ph. D. completion, Phys. Rev. Phys. Educ. Res. 17, 020115 (2021).
  28. J. Pearl, Causal diagrams for empirical research, Biometrika 82, 669 (1995).
  29. P. Spirtes, C. N. Glymour, and R. Scheines, Causation, Prediction, and Search (MIT Press, Cambridge, MA, 2000).
  30. R Core Team, R: A Language and Environment for Statistical Computing (R Foundation for Statistical Computing, Vienna, Austria 2022).
  31. A. Gelman and J. Hill, Data Analysis Using Regression and Multilevel/Hierarchical Models (Cambridge University Press, Cambridge, England, 2006).
  32. K. A. Clarke, The phantom menace: Omitted variable bias in econometric research, Confl. Manag. Peace Sci. 22, 341 (2005).
  33. K. A. Clarke, Return of the phantom menace: Omitted variable bias in political research, Confl. Manag. Peace Sci. 26, 46 (2009).
  34. S. K. Riegg, Causal inference and omitted variable bias in financial aid research: Assessing solutions, Rev. High. Educ. 31, 329 (2008).
  35. C. Walsh, M. M. Stein, R. Tapping, E. M. Smith, and N. G. Holmes, Exploring the effects of omitted variable bias in physics education research, Phys. Rev. Phys. Educ. Res. 17, 010119 (2021).
  36. D. Hutchison and B. Styles, A Guide to Running Randomised Controlled Trials for Educational Researchers (NFER, Slough, 2010).
  37. C. Torgerson, A. Wiggins, D. Torgerson, H. Ainsworth, and C. Hewitt, Every child counts: Testing policy effectiveness using a randomised controlled trial, designed, conducted and reported to consort standards, Res. Math. Educ. 15, 141 (2013).
  38. L. V. Hedges and J. Schauer, Randomised trials in education in the USA, Educ. Res. 60, 265 (2018).
  39. B. Ripley, B. Venables, D. M. Bates, K. Hornik, A. Gebhardt, D. Firth, and M. B. Ripley, Package ‘mass’, Cran r 538, 113 (2013).
  40. T. Richardson, Markov properties for acyclic directed mixed graphs, Scand. J. Stat. Theory Appl. 30, 145 (2003).
  41. S. Greenland, Quantifying biases in causal models: Classical confounding vs collider-stratification bias, Epidemiology 14, 300 (2003).
  42. B. W. Whitcomb, E. F. Schisterman, N. J. Perkins, and R. W. Platt, Quantification of collider-stratification bias and the birthweight paradox, Paediatr. Perinat. Epidemiol. 23, 394 (2009).
  43. F. Elwert and C. Winship, Endogenous selection bias: The problem of conditioning on a collider variable, Annu. Rev. Sociol. 40, 31 (2014).
  44. H. R. Banack and J. S. Kaufman, From bad to worse: Collider stratification amplifies confounding bias in the “obesity paradox”, Eur. J. Epidemiol. 30, 1111 (2015).
  45. M. Sperrin, J. Candlish, E. Badrick, A. Renehan, and I. Buchan, Collider bias is only a partial explanation for the obesity paradox, Epidemiology 27, 525 (2016).
  46. C. Coscia, D. Gill, R. Benítez, T. Pérez, N. Malats, and S. Burgess, Avoiding collider bias in Mendelian randomization when performing stratified analyses, Eur. J. Epidemiol. 37, 671 (2022).
  47. D. J. Del Junco, E. M. Bulger, E. E. Fox, J. B. Holcomb, K. J. Brasel, D. B. Hoyt, J. J. Grady, S. Duran, P. Klotz, M. A. Dubick et al., Collider bias in trauma comparative effectiveness research: The stratification blues for systematic reviews, Injury 46, 775 (2015).
  48. F. R. Leite, G. G. Nascimento, K. G. Peres, F. F. Demarco, B. L. Horta, and M. A. Peres, Collider bias in the association of periodontitis and carotid intima-media thickness, Community Dent. Oral Epidemiol. 48, 264 (2020).
  49. M. Sanni Ali, R. H. Groenwold, W. R. Pestman, S. V. Belitser, A. W. Hoes, A. De Boer, and O. H. Klungel, Time-dependent propensity score and collider-stratification bias: An example of beta 2-agonist use and the risk of coronary heart disease, Eur. J. Epidemiol. 28, 291 (2013).
  50. T. Tönnies, S. Kahl, and O. Kuss, Collider bias in observational studies, Dtsch. Aerztebl. Int. 119, 107 (2022).
  51. M. J. Holmberg and L. W. Andersen, Collider bias, JAMA, J. Am. Med. Assoc. 327, 1282 (2022).
  52. G. J. Griffith, T. T. Morris, M. J. Tudball, A. Herbert, G. Mancano, L. Pike, G. C. Sharp, J. Sterne, T. M. Palmer, G. Davey Smith et al., Collider bias undermines our understanding of COVID-19 disease risk and severity, Nat. Commun. 11, 5749 (2020).
  53. M. A. Hernán, S. Hernández-Díaz, and J. M. Robins, A structural approach to selection bias, Epidemiology 15, 615 (2004).
  54. T. J. VanderWeele and J. M. Robins, Directed acyclic graphs, sufficient causes, and the properties of conditioning on a common effect, Am. J. Epidemiol. 166, 1096 (2007).
  55. S. R. Cole, R. W. Platt, E. F. Schisterman, H. Chu, D. Westreich, D. Richardson, and C. Poole, Illustrating bias due to conditioning on a collider, Int. J. Epidemiol. 39, 417 (2010).
  56. M. B. Weissman, Do GRE scores help predict getting a physics Ph.D.? A comment on a paper by Miller et al., Sci. Adv. 6, eaax3787 (2020).
  57. J. Yerushalmy, The relationship of parents’ cigarette smoking to outcome of pregnancy—implications as to the problem of inferring causation from observed associations, Am. J. Epidemiol. 93, 443 (1971).
  58. J. M. Nissen, M. Jariwala, E. W. Close, and B. V. Dusen, Participation and performance on paper-and computer-based low-stakes assessments, Int. J. STEM Educ. 5, 21 (2018).
  59. J. Nissen, R. Donatello, and B. Van Dusen, Missing data and bias in physics education research: A case for using multiple imputation, Phys. Rev. Phys. Educ. Res. 15, 020106 (2019).
  60. A. Bandura, Social Foundations of Thought and Action (Englewood Cliffs, NJ, 1986), p. 23.
  61. E. A. Locke, Self-efficacy: The exercise of control, Pers. Psychol. 50, 801 (1997).
  62. A. D. Liem, S. Lau, and Y. Nie, The role of self-efficacy, task value, and achievement goals in predicting learning strategies, task disengagement, peer relationship, and achievement outcome, Contemp. Educ. Psychol. 33, 486 (2008).
  63. C. M. Vogt, Faculty as a critical juncture in student retention and performance in engineering programs, J. Eng. Educ. 97, 27 (2008).
  64. B. D. Jones, M. C. Paretti, S. F. Hein, and T. W. Knott, An analysis of motivation constructs with first-year engineering students: Relationships among expectancies, values, achievement, and career plans, J. Eng. Educ. 99, 319 (2010).
  65. T. Honicke and J. Broadbent, The influence of academic self-efficacy on academic performance: A systematic review, Educ. Res. Rev. 17, 63 (2016).
  66. J. B. Vancouver, C. M. Thompson, and A. A. Williams, The changing signs in the relationships among self-efficacy, personal goals, and performance, J. Appl. Psychol. 86, 605 (2001).
  67. J. Hattie and E. M. Anderman, International Guide to Student Achievement (Routledge, London, 2013).
  68. F. Pajares, Current directions in self-efficacy research, Adv. Motiv. Achiev. 10, 1 (1997).
  69. B. J. Zimmerman, Self-efficacy: An essential motive to learn, Contemp. Educ. Psychol. 25, 82 (2000).
  70. S. L. Britner and F. Pajares, Sources of science self-efficacy beliefs of middle school students, J. Res. Sci. Teach. 43, 485 (2006).
  71. E. L. Usher and F. Pajares, Sources of self-efficacy in school: Critical review of the literature and future directions, Rev. Educ. Res. 78, 751 (2008).
  72. R. W. Lent, F. G. Lopez, and K. J. Bieschke, Mathematics self-efficacy: Sources and relation to science-based career choice, J. Counsel. Psychol. 38, 424 (1991).
  73. R. M. Klassen, A cross-cultural investigation of the efficacy beliefs of South Asian Immigrant and Anglo Canadian nonimmigrant early adolescents, J. Educ. Psychol. 96, 731 (2004).
  74. T. Matsui, K. Matsui, and R. Ohnishi, Mechanisms underlying math self-efficacy learning of college students, J. Vocat. Behav. 37, 225 (1990).
  75. K. D. Multon, S. D. Brown, and R. W. Lent, Relation of self-efficacy beliefs to academic outcomes: A meta-analytic investigation, J. Counsel. Psychol. 38, 30 (1991).
  76. H. P. Phan, Informational sources, self-efficacy and achievement: A temporally displaced approach, Educ. Psychol. 32, 699 (2012).
  77. K. Boden, E. Kuo, T. Nokes-Malach, T. Wallace, and M. Menekse, What is the role of motivation in procedural and conceptual physics learning? An examination of self-efficacy and achievement goals, presented at PER Conf. 2017, Cincinnati, OH, 10.1119/perc.2017.pr.010.
  78. K. Talsma, B. Schüz, R. Schwarzer, and K. Norris, I believe, therefore i achieve (and vice versa): A meta-analytic cross-lagged panel analysis of self-efficacy and academic performance, Learn. Individ. Diff. 61, 136 (2018).
  79. M. Mund and S. Nestler, Beyond the cross-lagged panel model: Next-generation statistical tools for analyzing interdependencies across the life course, Adv. Life Course Res. 41, 100249 (2019).
  80. E. L. Hamaker, R. M. Kuiper, and R. P. Grasman, A critique of the cross-lagged panel model, Psychol. Methods 20, 102 (2015).
  81. S. Usami, N. Todo, and K. Murayama, Modeling reciprocal effects in medical research: Critical discussion on the current practices and potential alternative models, PLoS One 14, e0209133 (2019).
  82. Y. Li and C. Singh, How to select suitable models from many statistically equivalent models: An example from physics identity, arXiv:2303.13786.

Outline

Information

Phys. Rev. Phys. Educ. Res. 19, 020160– Published 20 November, 2023
  • Received 23 May 2023
  • Accepted 9 October 2023
Crossmark - Check for updates button
Creative Commons Logo

Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article’s title, journal citation, and DOI.

Published by the American Physical Society

Sign In to Your Journals Account

Filter

Filter

Article Lookup

Enter a citation